A neural network-based animation data generation method, device and related product

Abstract

The application discloses a neural network-based animation data generation method and device and related products. It can be applied to cloud technology, artificial intelligence, intelligent transportation, auxiliary driving, digital people, virtual people, games, virtual reality, extended reality and other scenes. The trajectory features and skeleton features of virtual objects in a virtual scene are used as query features, and a trained neural network is used to generate animation data of the virtual objects based on the query features. The neural network increases the feature dimension of the virtual objects based on the query features and generates the animation data of the virtual objects based on the high-dimensional features, meeting the generation requirements of the animation data. The use of the neural network eliminates the need to store massive data in the memory and query matching animations from the data when generating animation data. Only the weight data related to the neural network needs to be stored in advance, the memory occupation is low, and the problems of high memory occupation and poor query performance when generating animation data are avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a neural network-based animation data generation method and device and related products. BACKGROUND

[0002] In game development and animation production scenarios, in order to enhance the experience of the viewer, flexible and realistic animation effects often need to be designed for the motion of virtual objects. In a game scenario, for example, if the motion of running, jumping, crouching, slight breathing or swaying when idle, and panic raising of arms when falling can be displayed with flexible and realistic animation effects, the visual experience of the player can be enriched and the interaction of the player in the game can be enhanced.

[0003] In order to produce animations of virtual objects with different motions, the traditional way is to control the various complex playback and conversion control logic of the animation by designing a state machine. Figure 1 A schematic diagram of an animation state machine is shown in FIG. 1. Figure 1 In the state machine shown in FIG. 1, Defend, Upset, Victory, and Idle represent four different animations, respectively, and the bidirectional arrows between the four animations indicate that the animations are switched. If the traditional state machine method is used to generate animations in a game development or animation production scenario, when the motion of the virtual object is complex, the amount of design of the state machine will be very large, and subsequent updates and maintenance will be very difficult, requiring a large amount of time and being prone to failure.

[0004] In the field of animation production, there is another technology called Motion Matching. This technology can select the most matching animation frame from a large amount of animation for playback. However, when the Motion Matching technology drives animation data, a large amount of data needs to be stored in the memory during runtime, and motion matching also needs to be performed on these large amounts of data, resulting in high memory usage and poor query performance. This problem limits the development of the Motion Matching technology in game animation engines. SUMMARY

[0005] Embodiments of the present application provide a neural network-based animation data generation method, device, and related products, which aims to generate animation data with low memory usage.

[0006] In a first aspect, a neural network-based animation data generation method is provided. The neural network-based animation data generation method includes:

[0007] query features of the virtual object are generated according to running data of the virtual scene, and the query features are taken as inputs of the pre-trained neural network; the query features include trajectory features and skeleton features of the virtual object;

[0008] The feature dimension of the virtual object is increased through a feature generation network in the neural network, to obtain combined features matched with the query features;

[0009] Animation data of the virtual object are generated through an animation generation network in the neural network based on the combined features of the virtual object.

[0010] In a second aspect, the application provides a neural network-based animation data generation apparatus. The neural network-based animation data generation apparatus comprises:

[0011] A query feature generation unit is configured to generate query features of a virtual object according to running data of a virtual scene, and take the query features as inputs of a pre-trained neural network; the query features include trajectory features and skeleton features of the virtual object;

[0012] A combined feature generation unit is configured to increase the feature dimension of the virtual object through a feature generation network in the neural network, to obtain combined features matched with the query features;

[0013] An animation data generation unit is configured to generate animation data of the virtual object through an animation generation network in the neural network based on the combined features of the virtual object.

[0014] In a third aspect, the application provides a neural network-based animation data generation device. The neural network-based animation data generation device comprises a processor and a memory:

[0015] The memory is configured to store program codes and transmit the program codes to the processor;

[0016] The processor is configured to execute steps of the neural network-based animation data generation method according to instructions in the program codes.

[0017] In a fourth aspect, the application provides a computer readable storage medium. The computer readable storage medium is configured to store program codes, and the program codes are configured to execute steps of the neural network-based animation data generation method.

[0018] In a fifth aspect, the application provides a computer program product. The computer program product comprises computer programs or instructions, and the computer programs or instructions are configured to execute steps of the neural network-based animation data generation method when executed by the neural network-based animation data generation device.

[0019] From the above technical solutions, the embodiments of the present application have the following advantages:

[0020] The present application provides a neural network-based animation data generation method, device and related products. In the technical solution of the present application, the trajectory features and skeletal features of virtual objects in a virtual scene are used as query features, and on the basis of the query features, a pre-trained neural network is used to generate animation data of the virtual objects. Since the pre-trained neural network has the function of increasing the feature dimension of the virtual objects on the basis of the query features and generating the animation data of the virtual objects based on high-dimensional features, it can meet the generation demand of animation data. In addition, due to the use of neural networks, it is no longer necessary to store massive data in memory and query matching animations from them in the manner of previous action matching technology when generating animation data. The use of neural networks only needs to store weight data related to neural networks in advance, so the implementation of the whole scheme has a low memory occupancy, and real-time queries from massive data are not required. Thus, the problems of high memory occupancy and poor query performance when generating animation data are avoided. In addition, due to the reduction of memory occupancy and the reduction of query demand, in the case of a game scene, the game runs more smoothly, more storage space can be used for other purposes, thereby facilitating the improvement of other performance of the game, such as game quality, etc. Thus, the game experience of players is improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A schematic diagram of an animation state machine;

[0022] Figure 2 A scene architecture diagram for implementing a neural network-based animation data generation method provided by an embodiment of the present application;

[0023] Figure 3 A flowchart of a neural network-based animation data generation method provided by an embodiment of the present application;

[0024] Figure 4 A structure schematic diagram of a neural network provided by an embodiment of the present application;

[0025] Figure 5A Another structure schematic diagram of a neural network provided by an embodiment of the present application;

[0026] Figure 5B Another flowchart of a neural network-based animation data generation method provided by an embodiment of the present application;

[0027] Figure 6 A structure schematic diagram of a feature generation network provided by an embodiment of the present application;

[0028] Figure 7 A structural diagram of a feature updating network provided by an embodiment of the present application is shown in FIG. 7;

[0029] Figure 8 A structural diagram of an animation generation network provided by an embodiment of the present application is shown in FIG. 8;

[0030] Figure 9A A flowchart of a training process of a neural network provided by an embodiment of the present application is shown in FIG. 9;

[0031] Figure 9B A schematic diagram of a root skeleton trajectory before noise reduction is shown in FIG. 10;

[0032] Figure 9C A schematic diagram of a root skeleton trajectory after noise reduction is shown in FIG. 11;

[0033] Figure 10A A structural diagram of a deep learning network capable of extracting auxiliary query features provided by an embodiment of the present application is shown in FIG. 12;

[0034] Figure 10B A schematic diagram of an animation effect obtained by a traditional action matching method and an animation data generation method provided by an embodiment of the present application is shown in FIG. 13;

[0035] Figure 11 A structural diagram of an animation data generation device based on a neural network provided by an embodiment of the present application is shown in FIG. 14;

[0036] Figure 12 A structural diagram of another animation data generation device based on a neural network provided by an embodiment of the present application is shown in FIG. 15;

[0037] Figure 13 A structural diagram of a server in an embodiment of the present application is shown in FIG. 16;

[0038] Figure 14 A structural diagram of a terminal device in an embodiment of the present application is shown in FIG. 17. DETAILED DESCRIPTION

[0039] In an animation production scenario or a game scenario, the action matching technology solves the problems of a large amount of design, complex logic, and inconvenient maintenance of a traditional animation state machine. However, the action matching technology needs to store a large amount of animation data in advance for query matching, and therefore has a high memory occupancy, which leads to poor storage and query performance.

[0040] In view of the above problems, a neural network-based animation data generation method, device and related product are provided in the present application. When it is necessary to generate animation of a virtual object, only the pre-trained neural network needs to be used to generate animation data of the virtual object on the basis of the obtained query features of the virtual object. Compared with the implementation mode of obtaining animation effects of a virtual object by pre-storing massive animation data for query matching, the storage and query performance can be improved due to the small memory occupation of the weight data of the neural network. This advantage enables the embodiments of the present application to obtain better application and development in game animation engines.

[0041] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0042] With the research and progress of artificial intelligence technology, artificial intelligence technology is researched and applied in many fields, such as common smart home, smart wearable device, virtual assistant, smart speaker, smart marketing, unmanned vehicle, autonomous vehicle, unmanned aerial vehicle, robot, smart medical treatment, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0043] The scheme provided by the embodiments of the present application relates to machine learning of artificial intelligence, which will be specifically explained by the embodiments hereinafter. The neural network trained by machine learning solves the storage and query performance problems of existing action matching technology in animation production and film production.

[0044] First, some terms that may be involved in the embodiments hereinafter are explained.

[0045] 1) Machine learning (Machine Learning):

[0046] Machine learning is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and a branch of artificial intelligence. It is a fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning. The history of artificial intelligence research has a natural and clear context from focusing on "reasoning" to focusing on "knowledge" and then focusing on "learning". Obviously, machine learning is a way to achieve artificial intelligence, i.e. using machine learning to solve problems in artificial intelligence.

[0047] 2) Neural Network:

[0048] Artificial neural network, simply neural network or neural network, is a mathematical model or computational model that simulates the structure and function of biological neural networks in the field of machine learning and cognitive science, used to estimate or approximate functions. Neural networks are calculated by a large number of artificial neurons. Most artificial neural networks can change their internal structure on the basis of external information, and are self-adaptive systems, which are commonly known as learning functions.

[0049] 3) Motion capture:

[0050] Motion capture, also known as dynamic capture, is a technology that records and processes the motion of people or other objects. It is widely used in entertainment, sports, medical applications, computer vision, and robotics. In the fields of animation production, film production and electronic game development, it usually records the motion of human actors and converts it into digital model motion, and generates two-dimensional or three-dimensional computer animation. When it captures the subtle movements of the face or fingers, it is usually called performance capture.

[0051] 4) Virtual scene:

[0052] Virtual scene is a virtual scene displayed (or provided) by an application when running on a terminal. The virtual scene can be a simulation scene of the real world, a semi-simulation and semi-fictional three-dimensional scene, or a purely fictional three-dimensional scene. The virtual scene can be any one of a two-dimensional virtual scene, a 2.5-dimensional virtual scene and a three-dimensional virtual scene. The following embodiments take the virtual scene as a three-dimensional virtual scene as an example, but are not limited thereto. Optionally, the virtual scene is also used for virtual scene battles between at least two virtual objects.

[0053] 4) Virtual object:

[0054] Refers to a movable object in a virtual scene. The movable object can be at least one of a virtual person, a virtual animal, and an animation character. Alternatively, when the virtual scene is a three-dimensional virtual scene, the virtual object can be a three-dimensional model created based on an animation skeleton technology. Each virtual object has its own shape and volume in the three-dimensional virtual scene and occupies a part of the space in the three-dimensional virtual scene.

[0055] The execution subject of the neural network-based animation data generation method provided in the embodiments of the present application can be a terminal device. That is, the query feature is generated on the terminal device, and the animation data is generated according to the pre-trained neural network. As an example, the terminal device can include but is not limited to a mobile phone, a computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, and the like. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, digital human, virtual human, game, virtual reality, extended reality (XR, Extended Reality), and the like. In addition, the execution subject of the neural network-based animation data generation method provided in the embodiments of the present application can also be a server, that is, the query feature can be generated on the server, and the animation data can be generated according to the pre-trained neural network.

[0056] In some other implementations, the neural network-based animation data generation method provided in the embodiments of the present application can also be implemented by a terminal device and a server together. Figure 2 A scene architecture diagram for implementing the neural network-based animation data generation method is provided in the embodiments of the present application. In order to facilitate understanding of the technical solutions provided in the embodiments of the present application, the implementation scenario of the scheme is introduced as follows. Figure 2 In the implementation scenario, a terminal device and a server are involved. For example, the running data of a virtual scene can be extracted on the terminal device to generate the query feature of a virtual object, the weight data of a neural network can be called from the server, and the animation data of the virtual object can be generated on the terminal device based on the neural network. In addition, the query feature of the virtual object can also be generated in the server according to the running data of the virtual scene, the query feature can be sent to the terminal device, and the generation of the animation data can be implemented on the terminal device by using the neural network. Therefore, the implementation subject for implementing the technical solutions of the present application is not limited in the embodiments of the present application.

[0057] Figure 2 The server shown can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers. In addition, the server can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms, and the like basic cloud computing services.

[0058] Figure 3 A flowchart of a neural network-based animation data generation method is provided for an embodiment of the present application. The specific implementation of the method is introduced below with a terminal device as the execution subject. As shown in the neural network-based animation data generation method, the method comprises the following steps. Figure 3

[0059] S301: generating a query feature of a virtual object according to running data of a virtual scene, and taking the query feature as an input of a pre-trained neural network.

[0060] Taking a game scene as an example, when a player controls a virtual object, the virtual object needs to show corresponding animation effects according to the player's control. For example, the virtual object performs a walking action, and when the player controls the virtual object to perform a crouching action, the virtual object needs to show the animation of performing the crouching action in the virtual scene (i.e., the game scene). The animation data of the crouching action needs to be generated and implemented through the technical solution of the present application. In order to generate animation data that matches the purpose of the player's operation of the virtual object, the technical solution of the present application first needs to generate the query feature of the virtual object, which is used as the input of the neural network in the subsequent steps, and finally generates the animation data.

[0061] The query feature can include a trajectory feature and a skeleton feature of the virtual object. The trajectory feature refers to a feature related to the trajectory of the virtual object in the virtual scene. The trajectory feature is a feature from the perspective of the virtual object as a whole. In contrast, the skeleton feature is a feature from the perspective of individual bones of the virtual object. For example, the trajectory feature in the query feature can include a trajectory speed and a trajectory direction. In addition, the trajectory feature can also include a trajectory point position. The skeleton feature in the query feature can include left foot bone position information, left foot bone rotation information, right foot bone position information, and right foot bone rotation information. In addition, the skeleton feature can also include left foot bone speed and right foot bone speed.

[0062] ​It should be noted that the trajectory referred to in the trajectory feature refers to the trajectory of the root joint of the virtual object. It can be a path formed according to the projection of the hip bone of the virtual object on the ground. If the virtual object is a humanoid character, the generation method is to project the hip bone information of the humanoid skeleton to the ground, so that the trajectory point information of the virtual object is formed by connecting a plurality of animation frames. The ground referred to here can be the ground in the virtual scene coordinate system. The biped skeleton feature is included in the query feature, and since the foot is an important part of the human body that represents the posture, the information of its skeleton in position, rotation, etc. is conducive to generating matching animation through the neural network. In this application, the trajectory feature and the skeleton feature are used as query features, which represent the features of the virtual object from the whole and individual bones respectively, so that the combination of these two types of features is conducive to accurate generation of animation data and ensures that the generated animation data realistically depicts the display effect of the virtual object action.

[0063] As an optional implementation of the step, the query feature of the virtual object is generated according to the running data of the virtual scene, and specifically can include:

[0064] First, the action control signal for the virtual object is extracted from the running data of the virtual scene. Then, according to the control parameters in the action control signal and the historical control parameters in the historical action control signal for the virtual object, the trajectory feature and the skeleton feature of the virtual object are generated.

[0065] During game running, the walking and running of the character mainly depends on the input of the player. If the player wants to run, the corresponding action control signal will be input through the keyboard and handle, and then a reasonable running speed will be calculated as the trajectory feature according to the action control signal inside the game engine. In the calculation, the historical control parameters in the historical action control signal can be combined. For example, the control parameters can include the action type (running, jumping, walking, etc.). In addition, the trajectory feature and the skeleton feature of the virtual object can also be generated in combination with the character attributes of the virtual object. For example, different character attributes have different maximum and minimum speeds. Here, the historical action control signal can be the action control signal received before the latest received action control signal, or the action control signal received within a predetermined time before. With the historical control parameters in the historical action control signal, it is beneficial to generate more accurate and real-time trajectory features and skeleton features.

[0066] S302: Increase the feature dimension of the virtual object through the feature generation network in the neural network, and obtain the combined feature matched with the query feature.

[0067] Figure 4A structural diagram of a neural network is provided in an embodiment of the present application. In the network structure shown in the diagram, a feature generation network and an animation generation network are included. The feature generation network can be used to increase the feature dimension of a virtual object, i.e., the feature generation network is used to enrich the query feature dimension of the virtual object. For example, the query features input into the feature generation network include trajectory speed, trajectory direction, left foot skeleton position information, left foot skeleton rotation information, right foot skeleton position information, and right foot skeleton rotation information. Through processing of the feature generation network, the output features include not only the feature information in the input query features, but also other auxiliary features that are helpful to accurately generate animation data. The manner of obtaining the auxiliary features will be described in more detail later. In order to distinguish between the input features and the output features in the text, the features with increased dimensions obtained through the feature generation network in the embodiment of the present application are referred to as combined features. Since the combined features are obtained based on the input query features, it can be understood that the combined features output by the feature generation network match the query features input into the feature generation network.

[0068] S303: Based on the combined features of the virtual object, animation data of the virtual object is generated through an animation generation network in the neural network.

[0069] As shown in the network structure, Figure 4 The network structure also includes an animation generation network. The function of the network is to generate animation data of a virtual object based on the combined features of the virtual object input into the network. In actual applications, the combined features of the virtual object of each frame of the animation engine can be input into the animation generation network to generate the animation data of the frame. According to the time sequence, the animation data of each frame of the animation engine forms a coherent animation. Therefore, based on the functional requirements of the neural network, when the neural network needs to meet the above functional requirements, the output of the feature generation network can be directly used as the input of the animation generation network for training and use.

[0070] In the animation data generation method based on the neural network introduced in the embodiment of the present application, since the pre-trained neural network has the function of increasing the feature dimension of the virtual object based on the query features and generating the animation data of the virtual object based on the high-dimensional features, the generation requirement of the animation data can be met. In addition, due to the use of the neural network, when generating the animation data, it is no longer necessary to store a large amount of data in the memory and query the matched animation from the data in the manner of the previous action matching technology; the neural network only needs to store the weight data related to the neural network in advance, so that the implementation of the whole scheme occupies a relatively low amount of memory, thereby avoiding the problems of high memory occupation and poor query performance when generating the animation data. Therefore, the scheme in the embodiment of the present application can be better applied and developed in a game animation engine.

[0071] In some possible implementation manners, the feature generation network does not run every frame, so as to improve the performance of the scheme in running and reduce animation jitter. For example, when the change of the query feature meets a first preset condition and / or the time interval of the feature generation network from the previous output of the combined feature meets a second preset condition, the feature generation network outputs the combined feature according to the latest input query feature. That is, in this possible implementation manner, the running of the feature generation network needs to meet a precondition, which can be a condition related to the feature change (for example, the first preset condition), or a condition related to the running time interval (for example, the second preset condition), or a combination of the two conditions.

[0072] In view of the fact that the feature generation network does not run every frame in some possible implementation manners, in order to ensure the animation effect and ensure that a smooth animation can be generated, the present application can use another neural network structure to generate animation data. Figure 5A Another neural network structure provided by an embodiment of the present application is shown in the figure. Compared with the neural network structure shown in Figure 4 The neural network shown in Figure 5A Further includes a feature update network. In the structure shown in Figure 5A The output of the feature generation network is used as the input of the feature update network; the output of the feature update network is used as the input of the animation generation network. When the feature generation network does not run, the feature update network is used to drive the generation of the next frame of animation, so as to ensure the smoothness and continuity of the animation. Figure 5B The flowchart of another neural network-based animation data generation method provided by an embodiment of the present application is shown in the figure. The neural network structure used in the method shown in the figure is consistent with the neural network structure shown in Figure 5A That is, the neural network includes a feature generation network, a feature update network and an animation generation network.

[0073] The neural network-based animation data generation method shown in Figure 5B includes the following steps.

[0074] S501: generating a query feature of a virtual object according to running data of a virtual scene, and using the query feature as the input of a pre-trained neural network.

[0075] S502: increasing the feature dimension of the virtual object by using a feature generation network in the neural network, to obtain a combined feature matched with the query feature.

[0076] In the embodiment of the present application, the implementation manners of steps S501-S502 are basically the same as the implementation manners of steps S301-S302 of the foregoing embodiment, so the related description can refer to the foregoing embodiment, and will not be described herein again.

[0077] S503: Based on the combined features of the virtual object in the current frame output by the feature generation network, the combined features of the virtual object in the next frame of the current frame are output by the feature update network in the neural network.

[0078] This step S503 embodies the function of the feature update network in the neural network. Figure 5A As an optional implementation manner, the feature update network can output the combined features of the virtual object in the next frame of the current frame based on the combined features of the virtual object in the current frame output by the feature generation network and the interframe difference of the engine of the virtual scene. The interframe difference (deltaTime) refers to the time difference between two updates of the animation logic thread of the engine. Generally, it is close to the game update time, for example, the update rate of the game is 60 frames per second, and then deltaTime is 1 / 60 second. That is, in the embodiment of the application, the feature update network can obtain the combined features of the same dimension of the virtual object in the next frame based on the combined features of the current frame. That is, the feature update network realizes the update of the combined features of the virtual object in the adjacent frames, and updates the combined features of the next frame based on the combined features of the previous frame. In this way, the continuity and smoothness of the animation data output by the subsequent animation generation network can be realized by the function of the feature update network when the feature generation network does not work in real time.

[0079] S504: Based on the combined features of the virtual object in the next frame of the current frame, the animation data of the virtual object is generated by the animation generation network in the neural network.

[0080] Unlike the method shown in Figure 3 In step S504 of the embodiment of the application, the output of the feature update network is taken as the input of the animation generation network, so the animation generation network directly generates the animation data according to the combined features of the next frame in the input and outputs the animation data.

[0081] In the embodiment of the application, the feature generation network does not run every frame, thereby improving the performance of the scheme during running and reducing animation jitter. At the same time, through the feature update network, even if the feature generation network does not run every frame, the continuity and smoothness of the animation can be ensured.

[0082] Figure 6 A structural schematic diagram of a feature generation network provided in the embodiment of the application. Figure 7 A structural schematic diagram of a feature update network provided in the embodiment of the application. Figure 8 A structural schematic diagram of an animation generation network provided in the embodiment of the application. In Figure 6 to Figure 8In the example, the structure of the feature generation network is a six-layer fully connected network with four hidden layers, each of which has 512 units. The feature update network is a four-layer fully connected network with two hidden layers, each of which has 512 units. The animation generation network is a three-layer fully connected network with one hidden layer. Each of the hidden layers has 512 units. In other implementations, the three networks described above can include other numbers of hidden layers or other numbers of units in the hidden layers. Thus, the network structure of 6+4+2 layers and 512 units in the neural network is only an implementation and is not limited herein.

[0083] The neural network used to generate animation data of a virtual object based on a query feature is introduced in the method embodiment, and the structure thereof is exemplarily described. The training method of the neural network with the network structure shown in FIG. 1 is introduced below. Figure 9A The neural network training method is introduced for the neural network with the network structure shown in FIG. 1. Figure 5A The neural network training method is introduced for the neural network with the network structure shown in FIG. 1. Figure 9A A training flowchart of a neural network provided in an embodiment of the present application is shown in FIG. 9. As shown in FIG. 9, training the neural network includes the following steps: Figure 9A The training flowchart of the neural network provided in the embodiment of the present application is shown in FIG. 9. As shown in FIG. 9, training the neural network includes the following steps:

[0084] S901: Obtain motion capture data of a real scene.

[0085] The purpose of obtaining the motion capture data of the real scene is to train the neural network. The motion capture technology has been introduced above and belongs to a technology that is currently maturely applied in the fields of film production, animation production, and game development. In the embodiment of the present application, the motion capture data of a human body in a real scene is obtained by means of this technology. As an example, the step can be implemented in the following manner:

[0086] A motion capture route and a plurality of preset actions to be captured are designed. When a motion subject (generally a human being, such as an actor, or an animal) moves according to the preset motion capture route and performs the preset actions in a real scene, the motion subject is captured to obtain initial motion capture data. The initial motion capture data can be directly applied to the subsequent step S902.

[0087] In some scenarios, due to the signal noise existing in the acquisition device for collecting motion capture data, noise also exists in the captured data. If the data with serious noise is used to train the neural network, it is easy to affect the training effect. To solve this problem, for the initial motion capture data, noise reduction measures can be taken. For example, the scheme of using an SG filter for filtering can be used to process the initial motion capture data. For the position of the skeleton root node in each frame of motion capture data, the data of the previous and next N frames, that is, a total of 2N+1 frames of data, is used to perform a least squares fitting. The least squares method requires that the square difference of the data be as small as possible. Then the value of the current frame is selected on the fitting curve as the result after fitting. It should be noted that the value of N is related to the number of animation frames and the data change between animation frames. If the number of animation frames is large and the change between frames is small, N needs to be larger to successfully smooth and reduce noise. Generally, the larger N is, the stronger the noise reduction effect is. As an example, N=50 is selected. In actual application, other ways of filtering can also be used, and the SG filter is only an implementation example. Through filtering, the trajectory curve of the motion capture data is smoother, and the disturbance is reduced. Figure 9B and Figure 9C are schematic diagrams of the root skeleton trajectory before and after noise reduction. In combination with Figure 9B and Figure 9C It is not difficult to find that after noise reduction, the motion capture data taking the root skeleton trajectory as an example becomes less noisy and the trajectory is smoother.

[0088] In some scenarios, the initial motion capture data has a small amount of data, and in order to improve the performance of the subsequent trained neural network, the initial motion capture data can be expanded. Among them, the expansion method can include expanding the initial motion capture data by mirroring method and / or expanding the initial motion capture data by scaling the time axis. As an example, the mirroring method can mirror the left walk in the motion capture as the right walk, and the right walk as the left walk, thereby increasing the amount of data for each mode. In the animation data, only the data of the subject walking once may be captured, such as this data is that the left foot advances first and then the right foot advances. In order to expand the data set, such as the right foot advances first and then the left foot follows, the mirroring method needs to be used for expansion. The way of expanding data by scaling the time axis is to expand or reduce the speed of the track to expand the data, and this method is mainly to adjust the speed in the animation data to simulate the generation of motion capture data under different action speeds. For example, the initial motion capture data is a walking action of 100 meters in a path completed in 30 seconds. By enlarging the time axis, for example, to a time axis of 2 times the length, the original motion capture data is transformed into a walking action of 100 meters in a path completed in 60 seconds. As can be seen, enlarging the time axis reduces the action speed of the data corresponding to the execution subject. Similarly, shortening the time axis corresponds to increasing the action speed of the data corresponding to the execution subject. For example, by shortening the time axis, for example, to a time axis of half the length, the original motion capture data is transformed into a walking action of 100 meters in a path completed in 15 seconds. For the implementation of enlarging the time axis, the extra time is linearly interpolated. For the implementation of shortening the time axis, the data can be regularly filtered according to the time sequence. Through the above way of expanding the motion capture data, a large amount of data is provided for training the neural network, which is conducive to improving the performance of the neural network.

[0089] In addition to noise reduction and data expansion, due to the difference between the coordinate system of the real scene and the virtual scene, and the final obtained animation data needs to be corresponded to the coordinate system of the virtual scene, therefore, the data in the coordinate system of the virtual scene engine can also be generated based on the initial motion capture data in this step. In this way, a basic database for training the motion matching technology of the neural network can be constructed. For example, the initial motion capture data is the data of the right-hand coordinate system, and the coordinate system of the virtual scene engine is the left-hand coordinate system with the Z-axis upward, which can be converted according to the coordinate system relationship to generate the motion capture data in the coordinate system of the virtual scene engine.

[0090] That is, in practical applications, in order to improve the training effect, the initial motion capture data can be processed by at least one of the following preprocessing methods to obtain processed motion capture data: noise reduction, data augmentation, or generating data in a coordinate system adapted to the virtual scene engine.

[0091] S902: The root motion data, the skeleton pose information and the basic query feature of the motion subject are obtained according to the motion capture data.

[0092] The basic query feature includes trajectory features and skeleton features of the motion subject. Here, the basic query feature is consistent with the data type of the query feature that needs to be input to the feature generation network after the neural network is trained. The trajectory feature in the basic query feature can be generated according to the moving direction and position of the motion subject; the skeleton feature in the basic query feature can be obtained according to the motion information of the current motion subject's feet.

[0093] The root motion data and the skeleton pose information of the motion subject obtained from the motion capture data are information that is helpful for training the feature generation network and increasing the dimension of the query feature in addition to the basic query feature.

[0094] S903: The feature value of the motion subject is extracted from the root motion data and the skeleton pose information of the motion subject, and the feature value is used as an auxiliary query feature.

[0095] In the embodiments of the present application, S903 can be completed by another trained deep learning network. The function of the neural network is to extract the feature value as an auxiliary query feature. It should be noted that the features referred to in the embodiments of the present application, such as query features, basic query features, auxiliary query features, combined features, etc., can be represented by feature vectors. The vector representation of the auxiliary query feature can also be referred to as an auxiliary vector. The auxiliary vector is a number generated by the deep learning network executing S903. The dimension of the vector is consistent with the dimension of the feature. Figure 10A A structure diagram of a deep learning network capable of extracting an auxiliary query feature is provided in the embodiments of the present application. In Figure 10A The deep learning network shown can be a five-layer fully connected network with three hidden layers. After passing through each hidden layer, a low-dimensional feature vector representing the input data is gradually obtained. The final output is the auxiliary vector that needs to be used together with the vector representation of the basic query feature to train the feature generation network.

[0096] S904: Obtain the combined feature of the motion subject according to the basic query feature and the auxiliary query feature.

[0097] The base query feature and the auxiliary query feature can obtain the combined feature of the motion subject, and the auxiliary query feature realizes adding dimensions to the query feature on the basis of the base query feature. As mentioned above, the function of the feature generation network is to add feature dimensions to the query feature, and therefore, in the embodiments of the present application, the feature generation network in the neural network can be trained by taking the base query feature and the combined feature as a set of training data. The base query feature is taken as the input of the feature generation network in the training stage, and the combined feature of the motion subject is taken as the target output result for the foregoing input. See S905 below.

[0098] S905: training the feature generation network in the neural network by using the base query feature and the combined feature of the motion subject.

[0099] In actual application, a training termination condition for the feature generation network can be set. For example, whether the training needs to be terminated can be determined by the number of training iterations and / or the loss function. Similarly, a training termination condition can also be set for the training of the feature update network and the animation generation network. In the embodiments of the present application, the process of training the neural network is performed in sequence, that is, the feature generation network is trained first, then the feature update network is trained, and finally the animation generation network is trained. Training the above networks in this way can guarantee the performance of each network after training as much as possible. The process of training the feature generation network and the animation generation network in this way is shown in S906 and S907 below.

[0100] S906: after the training of the feature generation network is completed, the feature update network in the neural network is trained by using the combined feature of the current frame output by the feature generation network and the combined feature of the motion subject in the next frame obtained according to the motion capture data of the motion subject.

[0101] Among them, the combined feature of the motion subject in the next frame obtained according to the motion capture data of the motion subject is taken as the target output result of the feature update network being trained, and the combined feature of the current frame output by the feature generation network is taken as the actual input of the feature update network being trained.

[0102] S907: after the training of the feature update network is completed, the animation generation network is trained by using the root motion data and the skeletal pose information of the motion subject and the combined feature of the motion subject in the next frame output by the feature generation network.

[0103] Among them, the root motion data and the skeletal pose information of the motion subject are taken as the target output result of the animation generation network being trained, and the combined feature of the motion subject in the next frame output by the feature generation network is taken as the actual input of the animation generation network being trained.

[0104] The whole neural network is trained through the above steps, i.e., can be used in the animation data generation method provided in the embodiments of the present application. Table 1 compares the storage space occupied by the traditional action matching method and the storage space occupied by the animation generation method provided in the embodiments of the present application.

[0105] Table 1

[0106] Conventional action matching techniques The present application Animation database 43M 0M Feature database 10M 0M Network weight data 0M 5M Total storage 53M 5M

[0107] As can be seen from Table 1, compared with the traditional action matching technical solution, the present application can greatly save the storage space occupied when generating animation data, and improve the storage performance. Figure 10B The animation effect diagrams obtained by the traditional action matching method and the animation data generation method provided in the embodiments of the present application are shown in FIG. 2. The humanoid animation on the left is obtained by the traditional action matching method, and the humanoid animation on the right is obtained by the technical solution of the present application. It can be seen from FIG. 2 that the animation effect obtained by the technical solution of the present application is very close to the animation effect obtained by the traditional action matching method. Figure 10B As can be seen from the animation effect diagrams on the left and right, the animation effect obtained by the technical solution of the present application is very close to the animation effect obtained by the traditional action matching method. That is, a good effect is achieved, and the demand for animation data generation is met. On the basis of ensuring the animation effect, the improvement of the storage performance makes the game run more smoothly, and the animation viewing more smoothly. The improvement in storage performance makes more room for storage to support improvements in other aspects, such as further improving the game quality, storing more game data of users, increasing more rich virtual role related data or scene data, etc. Thus, the game experience of the player is further improved.

[0108] The application will be described below in combination with a game scenario. A certain game runs on a terminal device, and a player controls a virtual object in the game scenario to make running, jumping, dodging and other actions in real time through a mouse and a keyboard. When the player presses the F key of the keyboard, the virtual object controlled by the player needs to make a jumping action in the virtual scenario according to the game settings. When the player presses the T key of the keyboard, the virtual object controlled by the player needs to run in the virtual scenario according to the game settings. By using the method provided in the application, the terminal device can determine the action control intention of the player through the control parameters and historical control parameters in the action control signal input by the player through the mouse and / or the keyboard, and calculate the query feature of the virtual object. After the query feature of the virtual object is calculated, the terminal device communicates with a remote server to call a neural network. The weight data of the neural network is stored locally after being called. The terminal device takes the query feature as the input of the neural network. The neural network is pre-trained based on some real scene motion capture data in the server, so in fact, the terminal device can store the weight data of the neural network locally or call the weight data of the neural network from the server and store it locally to operate based on the input content and finally output the animation data of the virtual object. The terminal device renders the animation data of the virtual object into an animation effect that can be seen in the game scenario displayed on the terminal device for the player by using some rendering methods of the game engine. When the player presses the F key of the keyboard, the virtual object controlled by the player jumps in the virtual scenario displayed on the screen of the terminal device by using the above method, and the animation shows the body posture of the virtual object when jumping and the large separation distance between the two feet that is different from other postures. When the player presses the T key of the keyboard, the virtual object controlled by the player makes a running posture in the virtual scenario displayed on the screen of the terminal device by using the above method, and the two arms swing back and forth regularly to show the alternating leg movement that is beyond the walking posture. From the time when the player starts to control on the terminal device to the time when the corresponding animation effect is displayed in the virtual scenario picture, the whole time-consuming is very short, and other picture displays of the game will not be affected by the control instruction to cause lag and regional mosaic effect.

[0109] Figure 11 A structure diagram of a neural network-based animation data generation device provided in the application is shown.

[0110] As shown in Figure 11 , the neural network-based animation data generation device comprises:

[0111] The query feature generation unit 111 is configured to generate a query feature of a virtual object according to running data of a virtual scenario, and take the query feature as the input of a pre-trained neural network; the query feature comprises a trajectory feature and a skeleton feature of the virtual object.

[0112] The combination feature generation unit 112 is configured to increase the feature dimension of the virtual object by a feature generation network in the neural network, to obtain a combination feature matching the query feature;

[0113] The animation data generation unit 113 is configured to generate animation data of the virtual object by an animation generation network in the neural network based on the combination feature of the virtual object.

[0114] Since the pre-trained neural network has the function of increasing the feature dimension of the virtual object based on the query feature and generating the animation data of the virtual object based on the high-dimensional feature, the generation requirement of the animation data can be met. In addition, due to the use of the neural network, it is no longer necessary to store a large amount of data in the memory and query the matching animation from the data in the manner of the previous action matching technology when generating the animation data. The use of the neural network only needs to store the weight data related to the neural network in advance, so that the implementation of the whole scheme has a low memory occupation amount, thereby avoiding the problems of high memory occupation amount and poor query performance when generating the animation data.

[0115] Figure 12 Another structure schematic diagram of the animation data generation device based on the neural network provided by the embodiment of the present application is provided. Figure 12 In the schematic device structure, the animation data generation unit 113 specifically comprises:

[0116] The combination feature update sub-unit is configured to output the combination feature of the virtual object in the next frame of the current frame by a feature update network in the neural network based on the combination feature of the virtual object in the current frame output by the feature generation network.

[0117] The animation data generation sub-unit is configured to generate the animation data of the virtual object by an animation generation network in the neural network based on the combination feature of the virtual object in the next frame of the current frame.

[0118] Optionally, the combination feature generation unit 112 is specifically configured to:

[0119] When the change of the query feature meets the first preset condition and / or the time interval of the feature generation network from the previous output of the combination feature meets the second preset condition, the feature generation network outputs the combination feature according to the latest input query feature.

[0120] Optionally, the combination feature update sub-unit is specifically configured to:

[0121] The combination feature update sub-unit is configured to output the combination feature of the virtual object in the next frame of the current frame based on the combination feature of the virtual object in the current frame output by the feature generation network and the interframe difference of the engine of the virtual scene.

[0122] Optionally, the neural network-based animation data generation apparatus can further include a network training unit configured to obtain the neural network through training.

[0123] an action capture data acquisition subunit configured to acquire action capture data of a real scene;

[0124] a data analysis subunit configured to acquire root motion data, skeleton pose information and basic query features of a motion subject according to the action capture data;

[0125] a feature value extraction subunit configured to extract feature values of the motion subject from the root motion data and the skeleton pose information of the motion subject, and take the feature values as auxiliary query features;

[0126] a feature combination subunit configured to obtain combined features of the motion subject according to the basic query features and the auxiliary query features;

[0127] a first training subunit configured to train a feature generation network in the neural network by using the basic query features and the combined features of the motion subject;

[0128] a second training subunit configured to train a feature update network in the neural network by using combined features of a current frame output by the feature generation network and combined features of a next frame of the motion subject obtained according to the action capture data of the motion subject after the training of the feature generation network is completed;

[0129] a third training subunit configured to train an animation generation network by using root motion data and skeleton pose information of the motion subject and the combined features of the next frame of the motion subject output by the feature generation network after the training of the feature update network is completed.

[0130] Optionally, the action capture data acquisition subunit is specifically configured to:

[0131] when the motion subject moves according to a preset action capture route and performs a preset action in the real scene, capture the motion subject to obtain initial action capture data;

[0132] process the initial action capture data through at least one of the following preprocessing methods to obtain processed action capture data:

[0133] de-noising, data expansion or generating data in a coordinate system adapted to an engine of the virtual scene.

[0134] Optionally, the data augmentation method can include but is not limited to:

[0135] augmenting the initial motion capture data by a mirroring method; and / or,

[0136] augmenting the initial motion capture data by a time axis scaling method.

[0137] Optionally, the query feature generation unit 111 includes:

[0138] a signal extraction subunit configured to extract a motion control signal for the virtual object from running data of the virtual scene;

[0139] a feature generation subunit configured to generate a trajectory feature and a skeleton feature of the virtual object according to a control parameter in the motion control signal and a historical control parameter in a historical motion control signal for the virtual object.

[0140] Optionally, the trajectory feature includes a trajectory speed and a trajectory direction, and the skeleton feature includes left foot skeleton position information, left foot skeleton rotation information, right foot skeleton position information, and right foot skeleton rotation information; wherein the trajectory is formed according to a projection of a hip skeleton.

[0141] The structure of the animation data generation device based on the neural network will be introduced in the following in the form of a server and a terminal device respectively.

[0142] Figure 13 is a schematic diagram of a server structure provided by an embodiment of the present application. The server 900 can have great differences due to different configurations or performances, and can include one or more central processing units (CPUs) 922 (for example, one or more processors) and a memory 932, one or more storage media 930 (for example, one or more mass storage devices) storing application programs 942 or data 944. The memory 932 and the storage media 930 can be temporary storage or persistent storage. The programs stored in the storage media 930 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the server. Further, the central processing unit 922 can be configured to communicate with the storage media 930 and execute a series of instruction operations in the storage media 930 on the server 900.

[0143] The server 900 can also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input and output interfaces 958, and / or one or more operating systems 941, such as Windows ServerTM Mac OS X TM Unix TM Linux TM FreeBSD TM and so on.

[0144] The CPU 922 is configured to perform the following steps:

[0145] generate query features of the virtual object according to running data of the virtual scene, and take the query features as inputs of a pre-trained neural network; the query features include trajectory features and skeleton features of the virtual object;

[0146] increase a feature dimension of the virtual object through a feature generation network in the neural network, to obtain combined features matching the query features;

[0147] generate animation data of the virtual object through an animation generation network in the neural network based on the combined features of the virtual object.

[0148] Another neural network-based animation data generation device is also provided in the embodiments of the present application, as shown in Figure 14 For ease of illustration, only parts related to the embodiments of the present application are shown, and specific technical details not disclosed are referred to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English: Personal Digital Assistant, English abbreviation: PDA), a point of sales (English: Point of Sales, English abbreviation: POS), a vehicle-mounted computer, etc. Taking the terminal as a mobile phone for example:

[0149] Figure 14 A block diagram of part of the structure of the mobile phone related to the terminal provided by the embodiments of the present application is shown. Referring to Figure 14 , the mobile phone includes a radio frequency (English: Radio Frequency, English abbreviation: RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (English: wireless fidelity, English abbreviation: WiFi) module 1070, a processor 1080, and a power supply 1090, etc. Those skilled in the art can understand that Figure 14 the structure of the mobile phone shown in the embodiments of the present application does not constitute a limitation on the mobile phone, and can include more or fewer components than those shown in the figure, or combine certain components, or different component arrangements.

[0150] The various constituent components of the mobile phone will be specifically introduced below: Figure 14 ​

[0151] The RF circuit 1010 can be configured to receive and send signals during a process of receiving or transmitting information or a call. In particular, the RF circuit 1010 can receive downlink information from a base station and send uplink data to the base station. Generally, the RF circuit 1010 can include, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 1010 can communicate with a network and other devices through wireless communication. The wireless communication can use any communication standards or protocols, including but not limited to Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0152] The memory 1020 can be configured to store software programs and modules. The processor 1080 can execute various functions of the mobile phone and process data by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required for a function (such as a sound play function, an image play function, etc.), etc. The data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one disk storage device, a flash memory device, or other volatile solid state memory device.

[0153] The input unit 1030 can be used to receive inputted digital or character information, and to generate key signal input with respect to user setting of the mobile phone and function control. Specifically, the input unit 1030 can include a touch panel 1031 and other input devices 1032. The touch panel 1031, also called a touch screen, can collect a touch operation (such as an operation of a user using a finger, a stylus, or any suitable object or accessory on or near the touch panel 1031) of the user on or near the touch panel 1031, and drive a corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 can include two parts of a touch detection device and a touch controller. The touch detection device detects a touch position of the user and detects a signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch coordinates, and sends it to the processor 1080, and can receive a command from the processor 1080 and execute it. In addition, the touch panel 1031 can be implemented in various types such as a resistive type, a capacitive type, an infrared type, and a surface acoustic wave type. In addition to the touch panel 1031, the input unit 1030 can also include other input devices 1032. Specifically, the other input devices 1032 can include one or more of a physical keyboard, a function key (such as a volume control key, an on-off key, etc.), a trackball, a mouse, a joystick, etc.

[0154] The display unit 1040 can be used to display information input by the user or information provided to the user, and various menus of the mobile phone. The display unit 1040 can include a display panel 1041, which can be configured in the form of a liquid crystal display (English full name: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (English full name: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 can cover the display panel 1041, and when the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event, and then the processor 1080 provides corresponding visual output on the display panel 1041 according to the type of touch event. Although in the above embodiment, the touch panel 1031 and the display panel 1041 are realized as two independent components to realize the input and output functions of the mobile phone, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone. Figure 14

[0155] ​The mobile phone can further include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor can include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 1041 according to the brightness of ambient light, and the proximity sensor can turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and when at rest, it can detect the magnitude and direction of gravity, which can be used for identifying the posture of the mobile phone (such as switching between landscape and portrait, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), and the like. As for other sensors that can also be configured on the mobile phone, such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, they will not be described here.

[0156] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can convert the received audio data into an electrical signal, transmit it to the speaker 1061, and convert it into a sound signal output by the speaker 1061; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then output to the processor 1080 for processing, and then transmitted to another mobile phone through the RF circuit 1010, or output to the memory 1020 for further processing.

[0157] WiFi belongs to a short-range wireless transmission technology, and the mobile phone can help the user to send and receive emails, browse web pages, and access streaming media through the WiFi module 1070, which provides the user with wireless broadband Internet access. Although Figure 14 The WiFi module 1070 is shown, but it can be understood that it does not belong to the essential components of the mobile phone, and can be omitted as needed without changing the essence of the application.

[0158] The processor 1080 is the control center of the mobile phone, which connects all parts of the mobile phone through various interfaces and lines, executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, thereby collecting overall data and information of the mobile phone. Optionally, the processor 1080 can include one or more processing units; preferably, the processor 1080 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1080.

[0159] The mobile phone further includes a power supply 1090 (such as a battery) for supplying power to various components, and preferably, the power supply is logically connected to the processor 1080 through a power management system, so that the power management system can realize functions such as charge management, discharge management, and power consumption management.

[0160] Although not shown, the mobile phone can further include a camera, a Bluetooth module, and the like, which will not be described herein.

[0161] In the embodiments of the present application, the processor 1080 included in the terminal further has the following functions:

[0162] generating a query feature of a virtual object according to running data of a virtual scene, and taking the query feature as an input of a pre-trained neural network; the query feature includes a trajectory feature and a skeleton feature of the virtual object;

[0163] increasing a feature dimension of the virtual object through a feature generation network in the neural network, to obtain a combined feature matched with the query feature;

[0164] generating animation data of the virtual object through an animation generation network in the neural network based on the combined feature of the virtual object.

[0165] The embodiments of the present application further provide a computer readable storage medium for storing program codes, the program codes being used for executing any one of the embodiments of the method for generating animation data based on a neural network.

[0166] The embodiments of the present application further provide a computer program product including instructions, when the instructions are executed on a computer, causing the computer to execute any one of the embodiments of the method for generating animation data based on a neural network.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the foregoing method embodiments, which will not be described herein.

[0168] In several embodiments provided in the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the system is only a logical function division, and actual implementation can have another division manner, for example, a plurality of systems can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0169] The system described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0170] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0171] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), a random access memory (English full name: Random Access Memory, English abbreviation: RAM), a magnetic disk or an optical disk, and various program code storage media.

[0172] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A neural network-based animation data generation method, characterized by, The method comprises the following steps: generating query features of a virtual object according to running data of a virtual scene, and taking the query features as inputs of a pre-trained neural network; the query features comprise trajectory features and skeleton features of the virtual object; increasing feature dimensions of the virtual object by a feature generation network in the neural network, to obtain combined features matching the query features; generating animation data of the virtual object by an animation generation network in the neural network based on the combined features of the virtual object; the neural network is trained in the following way: obtaining motion capture data of a real scene; obtaining root motion data, skeleton posture information and basic query features of a motion subject respectively according to the motion capture data; the basic query features comprise trajectory features and skeleton features of the motion subject; extracting feature values of the motion subject from the root motion data and the skeleton posture information, and taking the feature values as auxiliary query features; obtaining combined features of the motion subject according to the basic query features and the auxiliary query features; training a feature generation network in the neural network by using the basic query features and the combined features of the motion subject; after the training of the feature generation network is completed, training a feature update network in the neural network by using combined features of a current frame output by the feature generation network and combined features of a next frame of the motion subject obtained according to the motion capture data of the motion subject; after the training of the feature update network is completed, training an animation generation network by using the root motion data and the skeleton posture information of the motion subject and the combined features of the next frame of the motion subject output by the feature generation network.

2. The method of claim 1, wherein, the method of generating animation data of the virtual object based on the combined features of the virtual object by the animation generation network in the neural network comprises: outputting combined features of a next frame of the virtual object in a current frame by a feature update network in the neural network based on the combined features of the virtual object in the current frame output by the feature generation network; generating animation data of the virtual object by the animation generation network in the neural network based on the combined features of the next frame of the virtual object in the current frame.

3. The method of claim 2, wherein, the method of increasing feature dimensions of the virtual object by the feature generation network in the neural network to obtain combined features matching the query features comprises: when changes of the query features satisfy a first preset condition and / or a time interval of the feature generation network from a previous output of combined features satisfies a second preset condition, the feature generation network outputs combined features according to the latest input query features.

4. The method of claim 2, wherein, the method of outputting combined features of a next frame of the virtual object in a current frame by a feature update network in the neural network based on the combined features of the virtual object in the current frame output by the feature generation network comprises: The feature update network outputs the combined feature of the virtual object in the next frame of the current frame based on the combined feature of the virtual object in the current frame output by the feature generation network and the inter-frame difference of the engine of the virtual scene.

5. The method of claim 1, wherein, The action capture data of the real scene comprises: When the action subject moves along a preset action capture route and performs a preset action in the real scene, the action subject is action captured to obtain initial action capture data; The initial action capture data is processed by at least one of the following preprocessing methods to obtain processed action capture data: Noise reduction, data augmentation, or generating data in a coordinate system adapted to the engine of the virtual scene.

6. The method of claim 5, wherein, The data augmentation of the initial action capture data comprises: The initial action capture data is augmented by a mirroring method; and / or The initial action capture data is augmented by scaling the time axis.

7. The method according to any one of claims 1 to 6, characterized in that, The query feature of the virtual object is generated according to the running data of the virtual scene, comprising: An action control signal for the virtual object is extracted from the running data of the virtual scene; A trajectory feature and a skeleton feature of the virtual object are generated according to a control parameter in the action control signal and a historical control parameter in a historical action control signal for the virtual object.

8. The method according to any one of claims 1 to 6, characterized in that, The trajectory feature comprises a trajectory speed and a trajectory direction, and the skeleton feature comprises left foot skeleton position information, left foot skeleton rotation information, right foot skeleton position information, and right foot skeleton rotation information; wherein the trajectory is formed according to a projection of a hip skeleton.

9. A neural network-based animation data generation apparatus characterized by comprising: Comprise: A query feature generation unit is configured to generate a query feature of a virtual object according to running data of a virtual scene, and to use the query feature as an input of a pre-trained neural network; the query feature comprises a trajectory feature and a skeleton feature of the virtual object; A combined feature generation unit is configured to increase a feature dimension of the virtual object by a feature generation network in the neural network to obtain a combined feature matching the query feature; An animation data generation unit is configured to generate animation data of the virtual object by an animation generation network in the neural network based on the combined feature of the virtual object. The neural network-based animation data generation apparatus further comprises a network training unit configured to obtain the neural network by training; wherein the network training unit specifically comprises: An action capture data acquisition subunit is configured to acquire action capture data of a real scene; A data analysis subunit is configured to acquire root motion data, skeleton posture information, and basic query features of an action subject respectively according to the action capture data; the basic query features comprise a trajectory feature and a skeleton feature of the action subject; A feature value extraction subunit is configured to extract feature values of the action subject from the root motion data and the skeleton posture information of the action subject, and use the feature values as auxiliary query features; A feature combination subunit is configured to obtain a combined feature of the action subject according to the basic query features and the auxiliary query features. The first training subunit is configured to train a feature generation network in the neural network by using the basic query feature and the combined feature of the action subject. The second training subunit is configured to train a feature update network in the neural network by using the combined feature of the current frame output by the feature generation network and the combined feature of the action subject in the next frame obtained according to the action capture data of the action subject after the feature generation network is trained. The third training subunit is configured to train an animation generation network by using the root motion data and the skeletal pose information of the action subject and the combined feature of the action subject in the next frame output by the feature generation network after the feature update network is trained.

10. The apparatus of claim 9, wherein, The animation data generation unit specifically comprises: The combined feature update subunit is configured to output the combined feature of the virtual object in the next frame of the current frame by the feature update network in the neural network based on the combined feature of the virtual object in the current frame output by the feature generation network. The animation data generation subunit is configured to generate the animation data of the virtual object by the animation generation network in the neural network based on the combined feature of the virtual object in the next frame of the current frame.

11. The apparatus of claim 10, wherein, The combined feature generation unit is specifically configured to: When the change of the query feature satisfies the first preset condition and / or the time interval of the feature generation network from the previous output of the combined feature satisfies the second preset condition, the feature generation network outputs the combined feature according to the latest input query feature.

12. The apparatus of claim 10, wherein, The combined feature update subunit is specifically configured to: Output the combined feature of the virtual object in the next frame of the current frame based on the combined feature of the virtual object in the current frame output by the feature generation network and the interframe difference of the engine of the virtual scene.

13. The apparatus of claim 9, wherein, The action capture data acquisition subunit is specifically configured to: Capture the action of the action subject to obtain initial action capture data when the action subject moves according to a preset action capture route and performs a preset action in the real scene. Process the initial action capture data by at least one of the following preprocessing methods to obtain processed action capture data: Noise reduction, data expansion, or generation of data adapted to the coordinate system of the engine of the virtual scene.

14. The apparatus of claim 13, wherein, Expand the initial action capture data by mirroring; and / or Expand the initial action capture data by scaling the time axis.

15. The apparatus of any of claims 9-14, wherein, The query feature generation unit comprises: The signal extraction subunit is configured to extract an action control signal for the virtual object from the running data of the virtual scene. The feature generation subunit is configured to generate a trajectory feature and a skeletal feature of the virtual object according to a control parameter in the action control signal and a historical control parameter in a historical action control signal for the virtual object.

16. The apparatus of any one of claims 9-14, wherein, The trajectory features include a trajectory speed and a trajectory direction, and the skeleton features include left foot skeleton position information, left foot skeleton rotation information, right foot skeleton position information, and right foot skeleton rotation information; wherein the trajectory is formed according to a projection of a hip skeleton.

17. A neural network-based animation data generation device, characterized by comprising: The device comprises a processor and a memory: The memory is configured to store program code and transmit the program code to the processor; The processor is configured to execute the steps of the neural network-based animation data generation method according to instructions in the program code.

18. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program code, and the program code is configured to execute the steps of the neural network-based animation data generation method.

19. A computer program product, characterised in that, The computer program or instructions are executed by the neural network-based animation data generation device to implement the steps of the neural network-based animation data generation method.

Images