Method for training action generation model, action generation method, device and equipment

Abstract

The application discloses a method for training an action generation model, an action generation method, an action generation device and an action generation equipment, and relates to machine learning of artificial intelligence. The method comprises the following steps: calling an action encoder to perform action feature extraction on an action state of a virtual character to obtain an action code; calling a style encoder to perform style feature extraction on the action state of the virtual character to obtain a style code; calling a decoder to decode the action code and the style code to obtain a predicted action of the virtual character; calculating a style loss according to the style codes of different action states; calculating an action loss according to the action codes of different action states; calculating a reconstruction loss according to a real action corresponding to an action state and the predicted action; and training the action generation model according to the action loss, the style loss and the reconstruction loss. The method can distinguish the action styles of different virtual characters and improve the action quality of the generated virtual characters.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to machine learning of artificial intelligence, and in particular, to a method for training a motion generation model, a method for generating motion, and devices thereof. BACKGROUND

[0002] Virtual characters in games usually exhibit diverse and flexible motions, such as running, jumping, etc. Traditionally, these motions are collected by professional motion actors to form a material library, and different motion materials are played to achieve the corresponding motion effect in game use. In recent years, the development of deep learning and neural network technology provides a new solution to this task, i.e., data-driven motion generation technology.

[0003] In related technologies, given the current virtual character state, including virtual character joint position and speed, target trajectory position and direction, surrounding terrain data, etc., and the control instruction input by the user, the deep learning model will predict the next frame of character state according to the current state and player instruction, and update the game character state in real time. The existing data-driven motion generation technology is mostly based on phase function neural network, which can dynamically change the network weight, thereby improving the expressiveness of the network while maintaining the compactness of the structure. This enables it to learn how to move from a large amount of high-dimensional data that is highly coupled with environmental geometry and character motion, and then generate character motion in real time and stably online.

[0004] However, the method in related technologies does not consider that different virtual characters have different motion styles. In actual game scenarios, different virtual characters have a lot of overlapping motion types, such as walking, running, jumping, etc., but the walking styles of different virtual characters are different, for example, an old man should walk in an old and infirm manner, and a child should walk in an energetic manner. The method in related technologies does not distinguish the motion styles of virtual characters, causing the motion styles of different virtual characters to be confused and reducing the motion quality of virtual characters. SUMMARY

[0005] Embodiments of the present application provide a method for training a motion generation model, a method for generating motion, and devices thereof, which can distinguish the motion styles of different virtual characters and improve the motion quality of generated virtual characters. The technical solutions are as follows:

[0006] In one aspect, a method for training a motion generation model is provided, the motion generation model including a motion encoder, a style encoder, and a decoder; the method comprising:

[0007] The style encoder is invoked to perform style feature extraction on the action state of the virtual character, to obtain a style code, the style code including features of an action style of the virtual character; the action encoder is invoked to perform action feature extraction on the action state of the virtual character, to obtain an action code, the action code including features of an action type of the virtual character;

[0008] The decoder is invoked to decode the action code and the style code to obtain a predicted action of the virtual character;

[0009] The style loss is calculated according to the style codes of different action states; the action loss is calculated according to the action codes of different action states; the reconstruction loss is calculated according to the real action corresponding to the action state and the predicted action;

[0010] The action generation model is trained according to the action loss, the style loss, and the reconstruction loss; the action loss is used to train the action encoder to output similar action codes for the action states of the same action type, the style loss is used to train the style encoder to output similar style codes for the action states of the same action style, and the reconstruction loss is used to make the predicted action close to the real action.

[0011] In another aspect, an action generation method is provided, the method comprising:

[0012] The action encoder is invoked to perform action feature extraction on an input action state of a virtual character, to obtain an input action code, the action code including features of an action type of the virtual character, the action encoder being configured to output similar action codes for the action states of the same action type;

[0013] The style encoder is invoked to perform style feature extraction on the input action state of the virtual character, to obtain an input style code, the style code including features of an action style of the virtual character, the style encoder being configured to output similar style codes for the action states of the same action style;

[0014] The decoder is invoked to decode the input action code and the input style code to obtain an output action of the virtual character.

[0015] In another aspect, a training device of an action generation model is provided, the action generation model including an action encoder, a style encoder, and a decoder; the device comprising:

[0016] a style encoding module, configured to invoke the style encoder to perform style feature extraction on the action state of the virtual character to obtain a style code, the style code comprising features of an action style of the virtual character;

[0017] an action encoding module, configured to invoke the action encoder to perform action feature extraction on the action state of the virtual character to obtain an action code, the action code comprising features of an action type of the virtual character;

[0018] a decoding module, configured to invoke the decoder to decode the action code and the style code to obtain a predicted action of the virtual character;

[0019] a loss module, configured to calculate a style loss according to the style codes of different action states, calculate an action loss according to the action codes of different action states, and calculate a reconstruction loss according to a real action corresponding to the action state and the predicted action;

[0020] a training module, configured to train the action generation model according to the action loss, the style loss, and the reconstruction loss; the action loss is used to train the action encoder to output similar action codes for the action states of the same action type, the style loss is used to train the style encoder to output similar style codes for the action states of the same action style, and the reconstruction loss is used to make the predicted action close to the real action.

[0021] In another aspect, an action generation apparatus is provided, and the apparatus comprises:

[0022] an action feature extraction module, configured to invoke an action encoder to perform action feature extraction on an input action state of a virtual character to obtain an input action code, the action code comprising features of an action type of the virtual character, the action encoder being configured to output similar action codes for the action states of the same action type;

[0023] a style feature extraction module, configured to invoke a style encoder to perform style feature extraction on the input action state of the virtual character to obtain an input style code, the style code comprising features of an action style of the virtual character, the style encoder being configured to output similar style codes for the action states of the same action style;

[0024] a generation module, configured to invoke a decoder to decode the input action code and the input style code to obtain an output action of the virtual character.

[0025] In another aspect, a computer device is provided, which includes a processor and a memory having stored therein at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to implement the action generation model training method or the action generation method as described in the above aspects.

[0026] In another aspect, a computer readable storage medium is provided, which has stored therein at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by a processor to implement the action generation model training method or the action generation method as described in the above aspects.

[0027] In another aspect, the embodiments of the present application provide a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the action generation model training method or the action generation method provided in the above optional implementation manners.

[0028] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0029] By using the action encoder and the style encoder on the action state of the virtual role, the action code and the feature code are extracted respectively, and the action encoder and the style encoder are trained respectively by using the action loss and the style loss, so that the action generation model can accurately distinguish the action features and the style features in the action state. The action and the action style are decoupled and encoded into the hidden space respectively, so that the style code difference of the action states of the same style and different actions is as small as possible, and the action code difference is as large as possible; the action code difference of the action states of the same action type and different styles is as small as possible, and the style code difference is as large as possible, so as to achieve the purpose of decoupling the style and the action. The style code and the action code can extract the corresponding style features and action features from the input action state, thereby reducing the mutual influence when different styles of data are trained together, and improving the quality of the predicted action generated by the action generation model. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical scheme in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0031] Figure 1 is a structural block diagram of a computer system according to an example embodiment of the present application;

[0032] Figure 2 is a method flowchart of a training method of a motion generation model according to another example embodiment of the present application;

[0033] Figure 3 is a graph of left and right foot movement speed of a training method of a motion generation model according to another example embodiment of the present application;

[0034] Figure 4 is a phase diagram of a training method of a motion generation model according to another example embodiment of the present application;

[0035] Figure 5 is a method flowchart of a training method of a motion generation model according to another example embodiment of the present application;

[0036] Figure 6 is a method flowchart of a motion generation method according to another example embodiment of the present application;

[0037] Figure 7 is a method flowchart of a training method of a motion generation model according to another example embodiment of the present application;

[0038] Figure 8 is a method flowchart of a motion generation method according to another example embodiment of the present application;

[0039] Figure 9 is a device block diagram of a training device of a motion generation model according to another example embodiment of the present application;

[0040] Figure 10 is a device block diagram of a motion generation device according to another example embodiment of the present application;

[0041] Figure 11 is a block diagram of a terminal according to another example embodiment of the present application;

[0042] Figure 12 is a block diagram of a server according to another example embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0044] Artificial Intelligence (AI) is the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making.

[0045] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other several directions.

[0046] Machine Learning (ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a branch of computer science that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0047] Figure 1 The structural block diagram of the computer system provided by an exemplary embodiment of the present application is given. The computer system includes a terminal 101 and a server 102.

[0048] For example, the terminal 101 includes at least one of a smart phone, a tablet computer, an electronic book reader, an MP3 player, an MP4 player, a laptop computer and a desktop computer; the server 102 includes at least one of a server, multiple servers, a cloud computing platform and a virtualization center.

[0049] For example, the terminal 101 runs a client, and the terminal 101 interacts with the server through the client. For example, the client is a client of an application program supporting a three-dimensional virtual environment, for example, the client can be a client of a game program, and the game program can be any one of a Strategy Game, a Multiplayer Online Battle Arena Games (MOBA) program, a Battle Royale shooting game program, a virtual reality game, an augmented reality game program, a First-Person Shooting Game (FPS) program, a Third-Person Shooting Game (TPS) program, and a Simulation Game (SLG) program. For another example, the client can be a client of at least one of a simulation program, a Virtual Reality (VR) application program, an Augmented Reality (AR) program, and a three-dimensional map program.

[0050] In this embodiment, the client is taken as an example of an FPS game. The terminal 101 is a terminal used by a first user, and in a game session, the first user uses the terminal 101 to control a virtual character in a virtual environment to perform activities, and the virtual character can be referred to as a master virtual character of the first user in the game session. The activities of the virtual character include, but are not limited to, at least one of adjusting a body posture, crawling, walking, running, riding, flying, jumping, driving, picking up, shooting, attacking, and throwing.

[0051] The terminal 101 is connected to the server 102 through a wireless network or a wired network.

[0052] The server 102 is configured to provide background services for the client running on the terminal. Optionally, the server 102 undertakes main computing work, and the terminal undertakes secondary computing work; or the server 102 undertakes secondary computing work, and the terminal undertakes main computing work; or the server 102 and the terminal adopt a distributed computing architecture to perform collaborative computing.

[0053] For example, the training method of the action generation model provided in the present application can be executed by the terminal 101 or the server 102; and the action generation method provided in the present application can be executed by the terminal 101 or the server 102.

[0054] For example, the action generation model 103 is stored on the server or the terminal.

[0055] In the action generation method, the action generation model 103 comprises a first encoder 104, a style encoder 105, an action encoder 106, a multi-layer perceptron (MLP), and a decoder 107.

[0056] For example, the action state m of the virtual role is input into the action generation model, and the action generation model outputs a predicted action s After the action generation model is input, a first encoding is obtained through the first encoder; the first encoding is input into the style encoder 105 and the action encoder 106 respectively to obtain a style encoding and an action encoding respectively; the style encoding is input into the multi-layer perceptron to obtain a normalization coefficient; the action encoding is subjected to instance normalization (IN) to obtain a normalized action encoding; the normalized action encoding is subjected to adaptive instance normalization (AdaIN) using the normalization coefficient to obtain a normalization result; and the decoder performs action prediction according to the normalization result to output a predicted action

[0057] In the training method of the action generation model, the action generation model 103 further comprises a discriminator 108.

[0058] For example, based on the above action generation method, the action state m of the virtual role is input into the action generation model, and the action generation model outputs a predicted action s The corresponding real action, i.e., the predicted action is the real value. The discriminator is used to determine whether the input action is a real action, so as to train the generation model to output a more real action by using the idea of generative adversarial network (GAN). The predicted action is input into the discriminator 108 to obtain a first prediction result, and the real action is input into the discriminator 108 to obtain a second prediction result. An adversarial loss is calculated based on the first prediction result and the second prediction result. A style loss is calculated based on two style encodings output by the style encoder for two action states of the same style and two style encodings output by the style encoder for two action states of different styles. An action loss is calculated based on two action encodings output by the action encoder for two action states of the same action and two action encodings output by the action encoder for two action states of different actions. A reconstruction loss is calculated based on the real action and the predicted action. A weighted sum of the reconstruction loss, the adversarial loss, the style loss, and the action loss is determined as a final loss value, and the action generation model is trained according to the weighted sum.

[0059] In combination with the above introduction of the virtual environment and the implementation environment, the training method of the action generation model and the action generation method provided by the embodiments of the present application are described. The execution subject of the method isFigure 1 The server or the terminal shown is used for illustration.

[0060] Figure 2 A flowchart of a method for training an action generation model is shown. The method can be performed by a computer device, for example, the computer device can be a server or a terminal in the above Figure 1 The method includes:

[0061] In step 201, a style encoder is called to extract style features of the action state of the virtual character, to obtain a style code, the style code including features of the action style of the virtual character; an action encoder is called to extract action features of the action state of the virtual character, to obtain an action code, the action code including features of the action type of the virtual character.

[0062] The action generation model includes an action encoder, a style encoder and a decoder. The action encoder is used to extract action type features, the style encoder is used to extract action style features, and the decoder is used to output a predicted action according to the extracted features.

[0063] The action generation model includes at least two encoders: an action encoder and a style encoder, which are respectively used to extract features of the action state in the action type and the action style. The style code is the feature of the action state in the action style, and the action code is the feature of the action state in the action type.

[0064] The virtual character corresponds to a three-dimensional virtual model in the three-dimensional virtual environment. The virtual character can move in the three-dimensional virtual environment. The virtual character moving in the three-dimensional virtual environment includes moving at least one of the limbs, head, neck, torso and fingers.

[0065] The action refers to the activity of the virtual character by moving at least one part. The action refers to the periodic activity of the virtual character in the three-dimensional virtual environment according to a certain movement rule. For example, the virtual character walks, runs, climbs, swims in the three-dimensional virtual environment. The action can also refer to the activity of the virtual character in the three-dimensional virtual environment to achieve a certain purpose. For example, jumping, opening the door, holding a gun, shooting, picking up, squatting, lying down, etc.

[0066] The action of the virtual character can be divided into multiple action types. For example, the action type includes but is not limited to at least one of walking, running, jumping, climbing, swimming, crawling, squatting, lying down, sitting down, holding a gun, shooting, attacking, picking up, squatting, dropping, driving, swinging a knife, throwing.

[0067] Exemplarily, due to different people, the amplitude, speed, position, direction of the body part movement when performing the same action type will be different, and different action styles will be formed; correspondingly, different virtual roles also have different action styles when performing the same action type.

[0068] Exemplarily, the action style refers to a feature of a moving way of a virtual role when performing an action. For example, an old man usually moves slowly and with small amplitude. For another example, a zombie usually moves stiffly and with large force but small amplitude. For another example, a guard usually moves standardly, quickly and with power.

[0069] Exemplarily, the action generation model can extract the action style features in the action state according to the input action state, and output the predicted action of the same action style based on the action style features learned from the action state. That is, the action style in the embodiment is autonomously learned and judged by the action generation model, rather than artificially defined.

[0070] Of course, using some action data of different action styles artificially defined as training samples in the training stage can enable the action generation model to better recognize the action styles in the training samples. For example, using a series of action data of an old man, a series of action data of a zombie, a series of action data of a young man, and a series of action data of a child to train the model can enable the trained model to better extract the action style features of the old man, the zombie, the young man, and the child, thereby predicting more accurate predicted actions for these action styles.

[0071] Exemplarily, the action state refers to the action data of the current state or the historical state of the virtual role. Exemplarily, the action state input into the model can be a set of action states of the virtual role at one time (one frame), or a plurality of sets of action states at a plurality of times (a plurality of frames).

[0072] Exemplarily, taking a set of action states at one time as an example, the data types contained in the action state are described.

[0073] Exemplarily, the action state includes at least one of the information contained in the input X in Table 1. That is, the action state includes at least one of the bone joint information, the motion trajectory information, the target point information, and the phase. Exemplarily, the predicted state output by the model includes at least one of the information contained in the output Y in Table 1. That is, the predicted action includes at least one of the predicted bone information, the predicted motion trajectory information, the predicted target point information, and the phase change amount.

[0074] Table 1

[0075]

[0076] For example, when the input action state is an action state at a moment, for example, the action state at time t is input, the output predicted action is a predicted action at the next moment, for example, the predicted action at time t+1 is output, and the real action corresponding to the action state is the real action at time t+1, that is, the moment corresponding to the predicted action and the real action is the same.

[0077] For example, when the input action state is an action state in a period of time, for example, six action states from time t to time t+5 are input, the output predicted action is a predicted action at the next moment, for example, the predicted action at time t+6 is output, and the real action corresponding to the action state is the real action at time t+6.

[0078] The following describes each data in the action state.

[0079] 1) Bone joint information.

[0080] The bone joint information in the action state is used to describe the position and motion state of at least one point on the three-dimensional virtual model of the virtual character. That is, the bone joint information is used to describe the relative position, relative speed and relative direction of each feature point on the three-dimensional virtual model of the virtual character at the current moment.

[0081] For example, the relative position, relative speed and relative direction refer to the position, speed and direction described by a three-dimensional coordinate system established with the root node of the three-dimensional virtual model of the virtual character as the origin. For example, the relative position corresponds to the absolute position, and the absolute position refers to the position of a point in the world coordinate system of the three-dimensional virtual environment, and the relative position refers to the position of a point in the root node coordinate system of the three-dimensional virtual model of the virtual character. Taking a point on the head of the virtual character as an example, when the virtual character moves in the three-dimensional virtual environment, but the head does not move relative to the root node of the three-dimensional virtual model of the virtual character, the absolute position of the head changes, but the relative position of the head does not change. That is, the relative position is used to describe the relative position relationship and relative position change of each point on the three-dimensional virtual model.

[0082] For example, at least one point (feature point) on the three-dimensional virtual model of the virtual character refers to any point on the three-dimensional virtual model of the virtual character. For example, the feature point can be used to represent the position of each body part of the virtual character, for example, a point at each joint of the virtual character is taken as a feature point, five points on each limb of the virtual character are taken as feature points, ten points on the torso of the virtual character are taken as feature points, and the like. The use of feature points can describe the relative position relationship of each part of the virtual character's body.

[0083] For example, when the virtual character is performing an action, each part of the body moves continuously, and the relative speed is the moving speed of the feature point at a certain moment, and the relative direction is the moving direction of the feature point at a certain moment.

[0084] 2) Motion trajectory information.

[0085] The motion trajectory information is used to describe the motion trajectory of the virtual character from a certain moment in the past to the current moment. The motion trajectory information includes at least one of the position information, the moving direction information, the action type information, and the terrain information of the motion trajectory point of the virtual character. For example, if the action state refers to the action state of the virtual character at the t-th frame, the motion trajectory information can refer to the motion trajectory information of at least one trajectory point of the virtual character from the (t-1)-th frame to the t-th frame. Alternatively, if the motion state refers to the motion state of the virtual character at the t-th frame, the motion trajectory can refer to the motion trajectory information of one trajectory point corresponding to each frame from the (t-n)-th frame to the t-th frame of the virtual character. That is, the motion trajectory is the trajectory information corresponding to at least one trajectory point of the virtual character from a certain moment in the past to the current moment corresponding to the action state.

[0086] For example, the trajectory point (motion trajectory point) refers to each position point of the root node of the virtual character when the virtual character moves as a whole.

[0087] For example, the root node is used to describe the overall position of the three-dimensional virtual model of the virtual character. For example, the root node can be a point on the three-dimensional virtual model, or a point at which the center of gravity of the three-dimensional virtual model is located, or a point outside the three-dimensional virtual model (for example, taking the intersection of the central axis of the three-dimensional virtual model and the foot surface as the root node). For example, when determining the absolute position of each point on the three-dimensional virtual model in the three-dimensional virtual environment, first, the absolute position of the root node of the three-dimensional virtual model is obtained, and then the absolute position of each point is obtained by adding the absolute position of the root node and the relative position of each point on the three-dimensional virtual model.

[0088] For example, the motion trajectory information includes the relative position, the relative direction, the action type, and the terrain information of at least one trajectory point.

[0089] The action type is, for example, a type of action performed by the virtual character at the current track point. The action type is determined, for example, according to a control instruction for controlling the virtual character. For example, when the virtual character is a virtual character controlled by a user, the user controls the virtual character to walk forward, and the action type of each track point passed through by the virtual character during walking is walking. When the user controls the virtual character to jump forward, the action type of each track point passed through by the virtual character during jumping is jumping. The action type is, for example, a type of action corresponding to an instruction received by the virtual character at a previous track point. For example, the virtual character walks according to a walking instruction from a first track point to a second track point, and the action type of the second track point is walking. Then, the virtual character receives a jumping instruction from the second track point to a third track point, and the action type of the third track point is jumping.

[0090] The terrain information is, for example, terrain information corresponding to the track point in a three-dimensional virtual environment. The terrain information includes, for example, at least one of a height of the track point (a height from a horizontal plane), a terrain type of a terrain where the track point is located (a flat surface, a swamp, a staircase, a grassland, etc.), and a tilt of a plane where the track point is located.

[0091] 3) Target point information.

[0092] The target point information includes information of a target point determined according to an activity instruction for controlling an activity of the virtual character. The target point information in the action state is, for example, information of a target point calculated according to a control instruction for controlling the virtual character. The target point information includes, for example, at least one of a relative position of the target point, a relative direction of the target point, and an action type of the target point.

[0093] For example, the user controls the virtual character to move forward, and according to the control instruction, it can be determined that the virtual character moves 1 m forward. The position point at 1 m is the target point obtained according to the control instruction.

[0094] The relative position of the target point is, for example, a position of the target point in a coordinate system of a root node. The relative direction of the target point is, for example, a relative direction of the target point from a current position of the virtual character.

[0095] The action type of the target point in the target point information is, for example, determined according to a control instruction for controlling the virtual character. That is, the action type of the target point is a type of action corresponding to an instruction received by the virtual character at a current time (a time corresponding to the action state).

[0096] 4) Phase.

[0097] The phase information is used to describe a standing or walking state of the virtual character.

[0098] For example, the phase is determined according to the contact of the virtual role's foot with the ground. For example, in the training data used when training the model, the phase of each action state is artificially labeled according to the contact of the virtual role's foot with the ground. Since part of the action is a periodic action, that is, the cycle of the left foot landing-the right foot landing-the left foot landing, the phase is used to describe the stage of the virtual role in the action cycle. For example, the phase 0 represents the standing state.

[0099] For example, taking the left foot landing-the right foot landing-the left foot landing as a cycle, the phase is from 0 to 1, that is, the phase is 0 when the left foot lands for the first time, the phase is 1 when the left foot lands for the second time, and then the second left foot landing is taken as the phase 0 of the next cycle, and the third left foot landing is taken as the phase 1. In this way, the phase diagram of the virtual role can be obtained according to the time when the virtual role's left and right feet land.

[0100] For example, as shown in Figure 3 , it is a curve diagram of the moving speed of the virtual role's left and right feet with time, the vertical axis is the speed, the horizontal axis is the frame number (time), the upper curve diagram 501 is the right foot, and the lower curve diagram 502 is the left foot. When the foot lands, its speed is zero, and the time when the left and right feet just contact the ground is marked with a vertical line in the curve diagram, and then the phase diagram 503 as shown in Figure 4 can be obtained according to the landing time of the left and right feet. For example, according to Figure 3 , the phase diagram 503 as shown in Figure 4 can be obtained. The way can be arbitrary, for example, the phase of the first left foot landing time can be taken as 0, the phase of the second left foot landing time can be taken as 1, a straight line is drawn connecting the two points, and then the phase of the second left foot landing time is changed to 0, and the phase of the third left foot landing time is changed to 1, and the phase diagram 503 as shown in Figure 4 is obtained in turn. For another example, the phase of the first left foot landing time can be taken as 0, the phase of the first right foot landing time can be taken as 0.5, a straight line is drawn connecting the two points, the phase of the second left foot landing time is taken as 1, a straight line is drawn connecting the two points, and the phase diagram 503 as shown in Figure 4 is obtained in turn.

[0101] For example, after obtaining the phase diagram, the phase information of the action state at each time in a smooth action can be determined according to the phase diagram.

[0102] For example, training the action generation model based on the training data artificially labeled with phase information can make the action generation model learn the phase information and predict the change of the phase information.

[0103] The following describes the data in the predicted action.

[0104] 1) Predict the bone information.

[0105] The predicted skeleton information is used to describe the position and motion state of at least one point on the three-dimensional model of the virtual character in the next frame. The predicted skeleton information includes the relative position, relative speed and relative direction of each feature point on the three-dimensional virtual model of the virtual character in the next frame.

[0106] For example, the predicted skeleton information corresponds to the skeleton information in the action state, and is the position and motion state information of each feature point on the three-dimensional virtual model of the virtual character in the next frame.

[0107] 2) Predicted motion trajectory information.

[0108] The predicted motion trajectory information includes at least one of the position information, the moving direction information and the action type information of the predicted motion trajectory point of the virtual character. That is, the predicted motion trajectory information includes the information of at least one trajectory point of the virtual character in the future.

[0109] The predicted motion trajectory information includes at least one of the relative position, the relative direction and the action type of the predicted trajectory point.

[0110] 3) Predicted target point information.

[0111] The predicted target point information includes the information of the target point of the virtual character in the future.

[0112] For example, the model predicts a target point according to the input action state. The actual target point of the virtual character is determined according to the predicted target point in the predicted action and the target point in the action state, so that the final target point combines the prediction result of the model and the calculation result of the control instruction.

[0113] 4) Phase change amount.

[0114] For example, the action generation model predicts the phase change amount of the next frame of action state (predicted action) and the previous frame of action state according to the phase information of the input action state.

[0115] For example, when the action generation model is actually applied to generate an action, the initial state of the virtual character is usually a standing state. Therefore, when the action generation model is actually applied, the phase of the first frame of input action state is 0. Then, the model outputs the phase change amount of the next frame of action state. The computer device can obtain the phase of the next frame of action state according to the phase change amount, and then input the next frame of action state into the model to continue to predict the next frame of action state.

[0116] In step 202, a decoder is called to decode the action encoding and the style encoding to obtain the predicted action of the virtual character.

[0117] Exemplarily, based on the two features of the action state on the action type and the action style, a decoder is called to decode to obtain a predicted action (action state) of the next frame of the virtual character.

[0118] In step 203, a style loss is calculated according to the style encoding of different action states; an action loss is calculated according to the action encoding of different action states; and a reconstruction loss is calculated according to the real action and the predicted action corresponding to the action state.

[0119] Exemplarily, in the action encoding of the action state of the same action style and different action types, the feature of the action type should be as large as possible, and the feature of the action style should be as small as possible; in the style encoding of the action state of the same action type and different action styles, the feature of the action type should be as small as possible, and the feature of the action style should be as large as possible. The above idea is used to train the action encoder and the style encoder to distinguish the action type feature and the action style feature, and the action encoder and the style encoder are trained by using the action loss and the style loss respectively.

[0120] Exemplarily, based on the fact that the predicted action should be close to the real action, the action generation model is trained by using the reconstruction loss.

[0121] In step 204, the action generation model is trained according to the action loss, the style loss and the reconstruction loss; the action loss is used to train the action encoder to output similar action encodings for the action states of the same action type, the style loss is used to train the style encoder to output similar style encodings for the action states of the same action style, and the reconstruction loss is used to make the predicted action close to the real action.

[0122] In summary, the method provided in the embodiment can accurately distinguish the action feature and the style feature in the action state of the virtual character by using the action encoder and the style encoder to respectively extract the action encoding and the feature encoding, and training the action encoder and the style encoder by using the action loss and the style loss respectively. The action and the action style are decoupled and encoded into the hidden space respectively, so that the style encoding difference of the action states of the same style and different actions is as small as possible, and the action encoding difference is as large as possible; the action encoding difference of the action states of the same action type and different styles is as small as possible, and the style encoding difference is as large as possible, so as to achieve the purpose of decoupling the style and the action. The style encoding and the action encoding can extract the corresponding style feature and action feature from the input action state, thereby reducing the mutual influence of different styles when training together, and improving the quality of the predicted action generated by the action generation model.

[0123] Exemplarily, an exemplary embodiment for calculating the style loss and the action loss is given, and an exemplary embodiment for decoding after fusing the style code and the action code is also given, and an exemplary embodiment for training the model using the idea of the generative adversarial network is also given.

[0124] Figure 5 A flowchart of a method for training an action generation model provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device, for example, the computer device can be a server or a terminal in the above-mentioned Figure 1 Based on the embodiment shown in Figure 2 , step 201 further includes step 2011 and step 2012. Step 201 further includes step 301 and step 302, step 202 further includes step 2021, step 203 further includes step 2031 to step 2034, and step 204 further includes step 2041.

[0125] In step 2011, a style encoder is called to perform style feature extraction on the first action state to obtain a first style code; the style encoder is called to perform style feature extraction on the second action state to obtain a second style code; and the style encoder is called to perform style feature extraction on the third action state to obtain a third style code.

[0126] Exemplarily, the action states include the first action state, the second action state, and the third action state, the action style of the first action state is the same as that of the second action state, and the action style of the first action state is different from that of the third action state.

[0127] Exemplarily, the action types of the first action state, the second action state, and the third action state can be the same or different.

[0128] In an optional implementation, the action generation model further includes a first encoder, which is located before the style encoder and the action encoder and connected to the style encoder and the action encoder respectively.

[0129] Exemplarily, a first encoder is called to perform feature extraction on the action state of the virtual character to obtain a first code; a style encoder is called to perform style feature extraction on the first code to obtain a style code; and an action encoder is called to perform action feature extraction on the first code to obtain an action code.

[0130] Taking the first action state as an example, a first encoder is called to perform feature extraction on the first action state of the virtual character to obtain a first code of the first action state; a style encoder is called to perform style feature extraction on the first code of the first action state to obtain a first style code; and an action encoder is called to perform action feature extraction on the first code of the first action state to obtain a first action code.

[0131] The manner in which the other action states obtain the style encoding and the action encoding through the first encoder can be analogous to the first action state, that is, replacing the first action state with the second action state, the third action state, the fourth action state, the fifth action state, or other action states.

[0132] At step 2012, the action encoder is called to perform action feature extraction on the first action state to obtain a first action encoding; the action encoder is called to perform action feature extraction on the fourth action state to obtain a fourth action encoding; and the action encoder is called to perform action feature extraction on the fifth action state to obtain a fifth action encoding.

[0133] Illustratively, the action states include the first action state, the fourth action state, and the fifth action state, the first action state and the fourth action state have the same action type, and the first action state and the fifth action state have different action types.

[0134] Illustratively, the action styles of the first action state, the fourth action state, and the fifth action state can be the same or different.

[0135] At step 301, the fully connected layer is called to predict the coefficients of the style encoding to obtain a normalization coefficient, and the normalization coefficient is used to represent the action style of the action state.

[0136] Illustratively, the action generation model further includes a fully connected layer.

[0137] Illustratively, the normalization coefficient includes a mean and a variance corresponding to the action style of the action state.

[0138] At step 302, the action encoding is subjected to instance normalization to obtain a normalized action encoding.

[0139] Illustratively, the action encoding is subjected to instance normalization (IN) processing.

[0140] At step 2021, the normalized action encoding is subjected to adaptive instance normalization according to the normalization coefficient to obtain a normalization result, and the action is predicted according to the normalization result to obtain a predicted action.

[0141] In the field of computer vision, there is a task called picture style transfer, for example, given an artist's painting and a real photo, the real photo can be displayed as an artistic style of an artistic painting, and one of the techniques is called adaptive instance normalization (AdaIN). Inspired by the style transfer in the field of computer vision, the AdaIN technique is applied in the action generation field of virtual characters. The style of the action of the virtual character can be represented by the statistical information of the feature map, that is, the means and variance β s The action generation model learns the mean and variance of a certain action style, so as to realize the learning of the action style.

[0142] For example, the formula of adaptive instance normalization is as follows:

[0143]

[0144] wherein, AdaIN(x, s) is the normalization result, x is the normalized action code obtained by instance normalizing the action code, s is the style, and a s and β s are the normalization coefficients (the mean and variance corresponding to the style) output by the full connection layer, μ(x) is the mean of the normalized action code, and δ(x) is the variance of the normalized action code.

[0145] For example, another full connection layer is included in the decoder. The computer device calls the full connection layer in the decoder to perform action prediction on the normalization result to obtain a predicted action.

[0146] In step 2031, the sum of the first difference value and the style boundary distance is determined as the style loss, the first difference value is the difference between the first loss and the second loss, the first loss includes the distance between the first style code and the second style code, the second loss includes the distance between the first style code and the third style code, and the style boundary distance is a constant.

[0147] For example, the formula for calculating the style loss is as follows:

[0148]

[0149] wherein, L Style is the style loss, m s is the first action state, w s is the second action state, x t is the third action state, E S (m s ) is the first style code, E S (w s ) is the second style code, E S (x t ) is the third style code, ‖E S (m s )-E S (w s )‖2 is the first loss, ‖E S (m s )-E S (x t )‖2 is the second loss, δ is the style boundary distance, and ‖·‖2 represents the 2-norm.

[0150] For example, s≠t, and δ can be 5.

[0151] The style loss encourages the distance between the style encodings of action states of the same action style to be smaller than the distance between the style encodings of action states of different action styles by δ.

[0152] In step 2032, a second difference value and a sum of action boundary distances are determined as the action loss, the second difference value being a difference between a third loss and a fourth loss, the third loss including a distance between the first action encoding and the fourth action encoding, the fourth loss including a distance between the first action encoding and the fifth action encoding, and the action boundary distance being a constant.

[0153] For example, the calculation formula of the action loss is as follows:

[0154]

[0155] wherein, L Action is the action loss, m s is the first action state, w s is the fourth action state, x S is the fifth action state, E A (m s ) is the first action encoding, E A (w s ) is the fourth action encoding, E A (x S ) is the fifth action encoding, ‖E A (m s )-E A (w s )‖2 is the third loss, and ‖E A (m s )-E A (x S )‖2 is the fourth loss, and δ is the action boundary distance.

[0156] The action loss encourages the distance between the action encodings of action states of the same action type to be smaller than the distance between the action encodings of action states of different action types by δ.

[0157] In step 2033, a reconstruction loss is calculated according to the real action and the predicted action corresponding to the action state.

[0158] For example, the calculation formula of the reconstruction loss is as follows:

[0159]

[0160] wherein, L Rec is the reconstruction loss, m s′ is the real action, For predicting the action, ‖·‖1 represents the L1 norm.

[0161] In step 2034, the discriminator is called to determine whether the predicted action is a real action to obtain a first prediction result, and the discriminator is called to determine whether the real action is a real action to obtain a second prediction result. An adversarial loss is calculated according to the first prediction result and the second prediction result, and the adversarial loss is used to train the discriminator to distinguish between the predicted action and the real action.

[0162] For example, the action generation model further includes a discriminator for determining whether the input action is a real action.

[0163] For example, in order to improve the overall effect of the action generation model, the action generation model is designed into a structure of a generator and a discriminator according to the idea of GAN, and the realism of the action generated by the model is further improved. The generator includes an encoder (a first encoder, an action encoder, and a style encoder), a full connection layer, and a decoder.

[0164] For example, the discriminator is used to distinguish whether the predicted action is a real action, thereby encouraging the generator (the action generation model) to generate a predicted action that is closer to the real action.

[0165] For example, the formula of the adversarial loss is as follows:

[0166]

[0167] wherein, L Adv is the adversarial loss, m s′ is the real action, D s (m s′ ) is the second prediction result, is the predicted action, is the first prediction result.

[0168] For example, the embodiment also provides a feature matching loss, which can minimize the distance between the predicted action input into the discriminator and the average value of the real action input into the discriminator, so as to make the training process more stable.

[0169]

[0170] wherein, is the predicted action is the i th real action; M s is a set of all action states of the style s; D f represents all layers before the last layer of the discriminator; is the output of the second-to-last layer of the discriminator after the predicted action is input into the discriminator, is the output of the second-to-last layer of the discriminator after the i-th real action is input into the discriminator.

[0171] In step 2041, the action generation model is trained according to the action loss, the style loss, the reconstruction loss and the adversarial loss.

[0172] In an optional embodiment, the computer device trains the action generation model according to the action loss, the style loss and the reconstruction loss.

[0173] That is, the computer device calculates a weighted sum of the action loss, the style loss and the reconstruction loss; and trains the action generation model according to the weighted sum.

[0174] That is, the total loss function is:

[0175] L = L Rec + a Style L Style + a Action L Action

[0176] wherein L is the total loss, L Rec is the reconstruction loss, a Style is the weight of the style loss, L Style is the style loss, a Action is the weight of the action loss, L Action is the action loss.

[0177] In an optional embodiment, the computer device calculates a weighted sum of the action loss, the style loss, the reconstruction loss and the adversarial loss; and trains the action generation model according to the weighted sum.

[0178] That is, the total loss function is:

[0179] L = L Rec + a Adv L Adv + a Style L Style + a Action L Action

[0180] wherein L is the total loss, L Rec is the reconstruction loss, a Adv is the weight of the adversarial loss, L Adv is the adversarial loss, a Style is the weight of the style loss, L Style is the style loss, a Action is the weight of the action loss, L Action is the action loss.

[0181] In an optional embodiment, the computer device can further train the action generation model according to the action loss, the style loss, the reconstruction loss, the adversarial loss and the feature matching loss.

[0182] That is, the computer device calculates a weighted sum of the action loss, the style loss, the reconstruction loss, the adversarial loss and the feature matching loss; and trains the action generation model according to the weighted sum.

[0183] That is, the total loss function is:

[0184] L = L Rec + a Adv L Adv + a Reg L Reg + a Style L Style + a Action L Action

[0185] wherein L is the total loss, L Rec is the reconstruction loss, a Adv is the weight of the adversarial loss, L Adv is the adversarial loss, a Reg is the weight of the feature matching loss, L Reg is the feature matching loss, a Style is the weight of the style loss, L Style is the style loss, a Action is the weight of the action loss, L Action is the action loss.

[0186] To sum up, the method provided in the embodiment can use one action generation model to learn virtual character actions of multiple different action styles, reduces the game memory usage under the premise of ensuring that each action style has obvious distinguishability, thereby reducing the computational requirements of the data-driven action generation algorithm, and providing great convenience for deploying these algorithms, especially to mobile phones.

[0187] The method provided in the embodiment calculates a loss function according to the first distance and the second distance by using the first distance of the style encodings of two action states of the same action style and the second distance of the style encodings of two action states of different action styles, so that the difference between the first distance and the second distance is as large as possible, thereby enabling the style encoder to distinguish different action styles. The method calculates a loss function according to the third distance and the fourth distance by using the third distance of the action encodings of two action states of the same action type and the fourth distance of the action encodings of two action states of different action types, so that the difference between the third distance and the fourth distance is as large as possible, thereby enabling the action encoder to distinguish different action types.

[0188] The method provided in the embodiment uses a discriminator to determine the authenticity of the predicted action predicted by the generator (action generation network) based on the idea of a generative adversarial network.

[0189] The method provided in the embodiment combines the action type features and the action style features by using AdaIN, so that the action generation model learns the style features of the action state. For a specific action style, AdaIN can maintain the quality of the action while distinguishing different action styles, achieving the purpose of learning all action styles by one action generation model.

[0190] An exemplary embodiment of applying the trained action generation model to action generation is given as an example.

[0191] Figure 6 A flowchart of an action generation method provided in an exemplary embodiment of the application is shown. The method can be executed by a computer device, for example, the computer device can be a server or a terminal in the above Figure 1 The computer device executing the training method of the action generation model can be the same as or different from the computer device executing the action generation method. The method includes the following steps.

[0192] Step 401, calling a style encoder to extract style features from the input action state of the virtual character to obtain input style encoding. The style encoding includes the features of the action style of the virtual character. The style encoder is used to output similar style encoding for action states with the same action style.

[0193] The action generation model trained by the training method of any of the above action generation models is used to implement the embodiment as an example. The action generation model includes a style encoder, an action encoder, and a decoder as an example.

[0194] As an example, in an optional implementation, the action generation model further includes a first encoder.

[0195] The computer device calls the first encoder to extract features from the input action of the virtual character to obtain a second encoding. The style encoder is called to extract style features from the second encoding to obtain input style encoding. The action encoder is called to extract action features from the second encoding to obtain input action encoding.

[0196] As an example, the data type of the input action state can refer to the explanation and description of the action state in the above embodiments.

[0197] At step 402, an action encoder is invoked to perform action feature extraction on the input action state of the virtual role, to obtain an input action code. The action code includes features of the action type of the virtual role. The action encoder is configured to output similar action codes for action states of the same action type.

[0198] At step 403, a decoder is invoked to decode the input action code and the input style code to obtain an output action of the virtual role.

[0199] For example, the action generation model further includes a fully connected layer.

[0200] The computer device invokes the fully connected layer to perform coefficient prediction on the input style code, to obtain an input normalization coefficient. The input normalization coefficient is used to represent the action style of the input action state. The input action code is normalized by instance, to obtain an input normalized action code. The input normalized action code is adaptively normalized by instance according to the input normalization coefficient, to obtain an input normalization result. The output action is predicted according to the input normalization result.

[0201] For example, the method of performing instance normalization calculation and adaptive instance normalization calculation can refer to the above explanations.

[0202] In summary, the method provided in this embodiment can perform action generation of a virtual role by invoking an action generation model, can output the next frame of action of the virtual role on the basis of distinguishing the action styles of different virtual roles, and improves the action accuracy.

[0203] For example, the training method of the action generation model and the action generation method provided in the embodiments of this application can be used to train all action style data at the same time by using one action generation model, instead of training and deploying an action generation model for each virtual role.

[0204] For example, as shown in Figure 7 When training the action generation model 601, action data of different action styles is used as a training sample, the action generation model 601 is trained, the action generation model 601 is caused to learn the action style features of different actions and the action type features of different action types, and then accurately predict the action of the virtual role according to different action styles and action types.

[0205] When the game is running, as shown in Figure 8As shown, only one action generation model needs to be deployed. Given the current action state of a virtual character (which can be of any style), the action generation model 601 automatically learns the action style in the action state and automatically generates the next frame of action of this action style. Since all virtual characters in the game share the same action generation model 601, the memory and computing resources occupied are greatly reduced.

[0206] The following is an apparatus embodiment of the present application. For details not described in the apparatus embodiment, reference can be made to the above method embodiments.

[0207] Figure 9 is a block diagram of an apparatus for training an action generation model provided by an exemplary embodiment of the present application. The action generation model includes an action encoder, a style encoder, and a decoder. The apparatus includes:

[0208] An action encoding module 901 is configured to invoke the action encoder to perform action feature extraction on the action state of a virtual character to obtain an action code, the action code including features of the action type of the virtual character.

[0209] A style encoding module 902 is configured to invoke the style encoder to perform style feature extraction on the action state of the virtual character to obtain a style code, the style code including features of the action style of the virtual character.

[0210] A decoding module 904 is configured to invoke the decoder to decode the action code and the style code to obtain a predicted action of the virtual character.

[0211] A loss module 905 is configured to calculate a style loss according to the style codes of different action states, calculate an action loss according to the action codes of different action states, and calculate a reconstruction loss according to a real action corresponding to the action state and the predicted action.

[0212] A training module 906 is configured to train the action generation model according to the action loss, the style loss, and the reconstruction loss. The action loss is used to train the action encoder to output similar action codes for the action states of the same action type, the style loss is used to train the style encoder to output similar style codes for the action states of the same action style, and the reconstruction loss is used to make the predicted action close to the real action.

[0213] In an optional embodiment, the action state includes a first action state, a second action state, and a third action state, the action style of the first action state is the same as that of the second action state, and the action style of the first action state is different from that of the third action state.

[0214] The style encoding module 902 is configured to: call the style encoder to perform style feature extraction on the first action state to obtain a first style code; call the style encoder to perform style feature extraction on the second action state to obtain a second style code; and call the style encoder to perform style feature extraction on the third action state to obtain a third style code.

[0215] In an optional embodiment, the loss module 905 is configured to determine a sum of a first difference value and a style boundary distance as the style loss, the first difference value being a difference between a first loss and a second loss, the first loss including a distance between the first style code and the second style code, the second loss including a distance between the first style code and the third style code, and the style boundary distance being a constant.

[0216] In an optional embodiment, the action states include a first action state, a fourth action state, and a fifth action state, the first action state and the fourth action state being of the same action type, and the first action state and the fifth action state being of different action types.

[0217] The action encoding module 901 is configured to: call the action encoder to perform action feature extraction on the first action state to obtain a first action code; call the action encoder to perform action feature extraction on the fourth action state to obtain a fourth action code; and call the action encoder to perform action feature extraction on the fifth action state to obtain a fifth action code.

[0218] In an optional embodiment, the loss module 905 is configured to determine a sum of a second difference value and an action boundary distance as the action loss, the second difference value being a difference between a third loss and a fourth loss, the third loss including a distance between the first action code and the fourth action code, the fourth loss including a distance between the first action code and the fifth action code, and the action boundary distance being a constant.

[0219] In an optional embodiment, the action generation model further includes a fully connected layer, the decoding module 904 includes an adaptive instance normalization submodule 909 and a prediction submodule 910, and the apparatus further includes:

[0220] The fully connected module 907 is configured to call the fully connected layer to perform coefficient prediction on the style code to obtain a normalization coefficient, the normalization coefficient being used to represent the action style of the action state.

[0221] The instance normalization module 908 is configured to perform instance normalization on the action code to obtain a normalized action code.

[0222] An adaptive instance normalization submodule 909 is configured to perform adaptive instance normalization on the normalized action encoding according to the normalization coefficient, to obtain a normalization result.

[0223] A prediction submodule 910 is configured to perform action prediction according to the normalization result to obtain the predicted action.

[0224] In an optional embodiment, the action generation model further includes a discriminator in the training stage; and the device further includes:

[0225] A discrimination module 911 is configured to call the discriminator to determine whether the predicted action is a real action, to obtain a first prediction result.

[0226] The discrimination module 911 is configured to call the discriminator to determine whether the real action is a real action, to obtain a second prediction result.

[0227] The loss module 905 is configured to calculate an adversarial loss according to the first prediction result and the second prediction result, where the adversarial loss is used to train the discriminator to distinguish between a predicted action and a real action.

[0228] The training module 906 is configured to train the action generation model according to the action loss, the style loss, the reconstruction loss, and the adversarial loss.

[0229] In an optional embodiment, the loss module 905 is configured to calculate a weighted sum of the action loss, the style loss, the reconstruction loss, and the adversarial loss.

[0230] The training module 906 is configured to train the action generation model according to the weighted sum.

[0231] In an optional embodiment, the action generation model further includes a first encoder; and the device further includes:

[0232] A first encoding module 903 is configured to call the first encoder to perform feature extraction on the action state of the virtual character, to obtain a first encoding.

[0233] The action encoding module 901 is configured to call the action encoder to perform action feature extraction on the first encoding, to obtain the action encoding.

[0234] The style encoding module 902 is configured to call the style encoder to perform style feature extraction on the first encoding, to obtain the style encoding.

[0235] In an optional embodiment, the action state includes at least one of bone joint information, motion trajectory information, target point information, and phase.

[0236] The bone joint information is used to describe the position and motion state of at least one point on a three-dimensional model of the virtual character; the motion trajectory information includes at least one of position information, moving direction information, action type information and terrain information of a motion trajectory point of the virtual character; the target point information includes information of a target point determined according to an activity instruction for controlling the virtual character to perform an activity; and the phase information is used to describe a standing or walking state of the virtual character.

[0237] In an optional embodiment, the predicted action includes at least one of predicted bone joint information, predicted motion trajectory information, predicted target point information and a phase change amount.

[0238] The predicted bone joint information is used to describe the position and motion state of at least one point on a three-dimensional model of the virtual character in a next frame; the predicted motion trajectory information includes at least one of predicted position information, moving direction information and action type information of a motion trajectory point of the virtual character; the predicted target point information includes information of a target point predicted to be moved by the virtual character; and the phase change amount is used to describe a change degree of a standing or walking state of the virtual character.

[0239] Figure 10 is a block diagram of an action generation device provided by an example embodiment of the present application. The device includes:

[0240] An action feature extraction module 1002 is configured to call an action encoder to perform action feature extraction on an input action state of a virtual character, to obtain an input action code, wherein the action code includes features of an action type of the virtual character, and the action encoder is configured to output similar action codes for the action states with the same action type.

[0241] A style feature extraction module 1001 is configured to call a style encoder to perform style feature extraction on the input action state of the virtual character, to obtain an input style code, wherein the style code includes features of an action style of the virtual character, and the style encoder is configured to output similar style codes for the action states with the same action style.

[0242] A generation module 1003 is configured to call a decoder to decode the input action code and the input style code to obtain an output action of the virtual character.

[0243] In an optional embodiment, the action generation model further includes a fully connected layer, and the device further includes:

[0244] The full connection layer module 1004 is configured to call the full connection layer to perform coefficient prediction on the input style code to obtain an input normalization coefficient, where the input normalization coefficient is used to represent the action style of the input action state.

[0245] The normalization module 1005 is configured to perform instance normalization on the input action code to obtain an input normalized action code.

[0246] The generation module 1003 is configured to perform adaptive instance normalization on the input normalized action code according to the input normalization coefficient to obtain an input normalization result.

[0247] The generation module 1003 is configured to perform action prediction according to the input normalization result to obtain the output action.

[0248] It should be noted that the training apparatus of the action generation model provided in the above embodiments is only exemplified by the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the training apparatus of the action generation model and the training method of the action generation model provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0249] The present application also provides a terminal, which includes a processor and a memory, and the memory stores at least one instruction, which is loaded and executed by the processor to implement the action generation model training method or the action generation method provided in each of the above method embodiments. It should be noted that the terminal can be Figure 11 The provided terminal.

[0250] Figure 11 The structure block diagram of the terminal 1100 provided in an example embodiment of the present application is shown in FIG. 11. The terminal 1100 can be a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a notebook computer or a desktop computer. The terminal 1100 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0251] Generally, the terminal 1100 includes a processor 1101 and a memory 1102.

[0252] The processor 1101 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1101 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1101 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also known as a CPU, and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1101 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content required to be displayed by the display screen. In some embodiments, the processor 1101 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0253] The memory 1102 can include one or more computer-readable storage media that can be non-transitory. The memory 1102 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one instruction for being executed by the processor 1101 to implement the training method of the action generation model provided by the method embodiments in the present application.

[0254] In some embodiments, the terminal 1100 can also optionally include a peripheral device interface 1103 and at least one peripheral device. The processor 1101, the memory 1102, and the peripheral device interface 1103 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1103 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1104, a display screen 1105, a camera 1106, an audio circuit 1107, and a power supply 1109.

[0255] The peripheral interface 1103 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 1101 and the memory 1102. In some embodiments, the processor 1101, the memory 1102 and the peripheral interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1101, the memory 1102 and the peripheral interface 1103 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this regard.

[0256] The radio frequency circuit 1104 is used to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1104 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1104 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1104 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1104 can also include NFC (Near Field Communication) related circuitry, and the present application is not limited in this regard.

[0257] The display screen 1105 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1105 is a touch display screen, the display screen 1105 is further configured to capture touch signals on or above the surface of the display screen 1105. The touch signals can be input to the processor 1101 as control signals for processing. In this case, the display screen 1105 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 1105 can be one, arranged on the front panel of the terminal 1100; in other embodiments, the display screen 1105 can be at least two, arranged on different surfaces of the terminal 1100 or in a folding design; in still other embodiments, the display screen 1105 can be a flexible display screen, arranged on a curved surface or a folding surface of the terminal 1100. Even, the display screen 1105 can also be arranged in an irregular shape, i.e., a special-shaped screen. The display screen 1105 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0258] The camera assembly 1106 is configured to capture images or videos. Optionally, the camera assembly 1106 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is arranged on the front panel of the terminal, and the rear-facing camera is arranged on the back of the terminal. In some embodiments, the rear-facing camera is at least two, which is any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 1106 can further include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0259] The audio circuit 1107 can include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into an electrical signal input to the processor 1101 for processing, or input to the radio frequency circuit 1104 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, respectively arranged at different parts of the terminal 1100. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert the electrical signal from the processor 1101 or the radio frequency circuit 1104 into sound waves. The speaker can be a traditional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can it convert electrical signals into sound waves that humans can hear, but it can also convert electrical signals into sound waves that humans cannot hear for ranging purposes. In some embodiments, the audio circuit 1107 can also include a headphone jack.

[0260] The power supply 1109 is used to supply power to each component in the terminal 1100. The power supply 1109 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 1109 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0261] In some embodiments, the terminal 1100 also includes one or more sensors 1110. The one or more sensors 1110 include, but are not limited to, an acceleration sensor 1111, a gyroscope sensor 1112, a pressure sensor 1113, an optical sensor 1115, and a proximity sensor 1116.

[0262] The acceleration sensor 1111 can detect the acceleration in three coordinate axes of the coordinate system established by the terminal 1100. For example, the acceleration sensor 1111 can be used to detect the components of the gravitational acceleration in three coordinate axes. The processor 1101 can control the display screen 1105 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1111. The acceleration sensor 1111 can also be used for game or user motion data collection.

[0263] The gyroscope sensor 1112 can detect the body direction and rotation angle of the terminal 1100, and the gyroscope sensor 1112 can collect 3D actions of the user on the terminal 1100 in cooperation with the acceleration sensor 1111. The processor 1101 can realize the following functions according to the data collected by the gyroscope sensor 1112: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization when shooting, game control, and inertial navigation.

[0264] The pressure sensor 1113 can be disposed on the side bezel of the terminal 1100 and / or on the lower layer of the display screen 1105. When the pressure sensor 1113 is disposed on the side bezel of the terminal 1100, it can detect the user's grip signal on the terminal 1100, and the processor 1101 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1113. When the pressure sensor 1113 is disposed on the lower layer of the display screen 1105, the processor 1101 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1105. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0265] An optical sensor 1115 is used to collect ambient light intensity. In one embodiment, the processor 1101 can control the display brightness of the display screen 1105 based on the ambient light intensity collected by the optical sensor 1115. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1105 is increased; when the ambient light intensity is low, the display brightness of the display screen 1105 is decreased. In another embodiment, the processor 1101 can also dynamically adjust the shooting parameters of the camera assembly 1106 based on the ambient light intensity collected by the optical sensor 1115.

[0266] The proximity sensor 1116, also known as a distance sensor, is typically mounted on the front panel of the terminal 1100. The proximity sensor 1116 is used to detect the distance between the user and the front of the terminal 1100. In one embodiment, when the proximity sensor 1116 detects that the distance between the user and the front of the terminal 1100 is gradually decreasing, the processor 1101 controls the display screen 1105 to switch from a screen-on state to a screen-off state; when the proximity sensor 1116 detects that the distance between the user and the front of the terminal 1100 is gradually increasing, the processor 1101 controls the display screen 1105 to switch from a screen-off state to a screen-on state.

[0267] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on terminal 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0268] The memory further includes one or more programs stored in the memory, and the one or more programs include a training method or an action generation method for performing the action generation model provided in the embodiments of this application.

[0269] Figure 12is a structural schematic diagram of a server provided by an embodiment of the present application. Specifically, the server 1200 includes a central processing unit (English: Central Processing Unit, for short: CPU) 1201, a system memory 1204 including a random access memory (English: Random Access Memory, for short: RAM) 1202 and a read-only memory (English: Read-Only Memory, for short: ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the central processing unit 1201. The server 1200 also includes a basic input / output system (I / O system) 1206 to help transfer information between various devices in the computer, and a mass storage device 1207 for storing an operating system 1213, application programs 1214 and other program modules 1215.

[0270] The basic input / output system 1206 includes a display 1208 for displaying information and an input device 1209 such as a mouse, keyboard, etc. for user input information. Among them, the display 1208 and the input device 1209 are connected to the central processing unit 1201 through the input / output controller 1210 connected to the system bus 1205. The basic input / output system 1206 can also include an input / output controller 1210 for receiving and processing input from a keyboard, mouse, or electronic stylus, and other devices. Similarly, the input / output controller 1210 also provides output to the display screen, printer or other types of output devices.

[0271] The mass storage device 1207 is connected to the central processing unit 1201 through a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1207 and its associated computer readable medium provide non-volatile storage for the server 1200. That is, the mass storage device 1207 can include a computer readable medium (not shown) such as a hard disk or a read-only optical disc (English: Compact Disc Read-Only Memory, for short: CD-ROM) drive.

[0272] Without loss of generality, the computer readable medium can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid state memory technology, CD-ROM, digital versatile discs (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices. Of course, computer storage media does not limit to the above-mentioned several kinds. The system memory 1204 and the mass storage device 1207 mentioned above can be collectively referred to as memory.

[0273] According to various embodiments of the present application, the server 1200 can also run on a remote computer connected to the network through a network connection such as the Internet. That is, the server 1200 can be connected to the network 1212 through the network interface unit 1211 connected to the system bus 1205, or can be connected to other types of networks or remote computer systems (not shown) using the network interface unit 1211.

[0274] The present application provides a computer readable storage medium, the storage medium stores at least one instruction, the at least one instruction is loaded and executed by the processor to realize the action generation model training method or the action generation method provided by each method embodiment.

[0275] The present application also provides a computer program product or computer program, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer readable storage medium. The processor of the computer equipment reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer equipment executes the action generation model training method or the action generation method provided in the above optional implementation manner.

[0276] The above-mentioned embodiment number of the present application is only for description, not representing the advantages and disadvantages of the embodiments.

[0277] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed to relevant hardware by program. The program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0278] The above only describes optional embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training an action generation model, characterized by, The action generation model comprises a first encoder, an action encoder, a style encoder, and a decoder; the method comprises: calling the first encoder to perform feature extraction on the action state of the virtual character to obtain a first code; the action state comprises at least one of bone joint information, motion trajectory information, target point information, and phase; calling the style encoder to perform style feature extraction on the first code to obtain a style code, the style code comprising features of the action style of the virtual character; The action state comprises a first action state, a fourth action state of the same action type as the first action state, and a fifth action state of a different action type from the first action state; calling the action encoder to perform action feature extraction on the first, fourth, and fifth action states respectively corresponding to the first codes to obtain a first, fourth, and fifth action code; the action code comprises features of the action type of the virtual character; normalizing the action code to obtain a normalized action code; and adaptively normalizing the normalized action code according to a normalization coefficient to obtain a normalization result; calling the decoder to decode the normalization result to obtain the predicted action of the virtual character; calculating a style loss according to the style codes of different action states; determining the sum of a second difference value and an action boundary distance as an action loss, the second difference value being the difference between a third loss comprising the distance between the first action code and the fourth action code, and a fourth loss comprising the distance between the first action code and the fifth action code, the action boundary distance being a constant, the action loss encouraging the distance between the action codes of action states of the same action type to be smaller than the distance between the action codes of action states of different action types by the action boundary distance; calculating a reconstruction loss according to the distance between the real action corresponding to the action state and the predicted action, the reconstruction loss being used to make the predicted action close to the real action; calling the discriminator to determine whether the predicted action is a real action, taking the determination result as a first prediction result; calling the discriminator to determine whether the real action is a real action, taking the determination result as a second prediction result, and taking the difference between the second prediction result and a preset constant as a third prediction result; calculating an adversarial loss according to the first prediction result and the third prediction result, the adversarial loss being used to train the discriminator to distinguish between predicted actions and real actions; calculating a feature matching loss based on the distance between a first value and a second value, the first value being the output of the second-to-last layer of the discriminator after inputting the predicted action into the discriminator, and the second value being the output of the second-to-last layer of the discriminator after inputting the real action into the discriminator, and the ratio between the action states of the same style, the feature matching loss being used to minimize the distance between the average of the predicted action input into the discriminator and the real action input into the discriminator. training the action generation model according to the action loss, the style loss, the adversarial loss, the feature matching loss and the reconstruction loss; the action loss is used for training the action encoder to output similar action encodings for action states of the same action type, and the style loss is used for training the style encoder to output similar style encodings for action states of the same action style; for any virtual role, calling the trained action generation model to predict the action of the virtual role at the next moment according to the input action state of the virtual role at a moment or within a period of time.

2. The method of claim 1, wherein, The action state further includes a second action state and a third action state, the action style of the first action state is the same as that of the second action state, and the action style of the first action state is different from that of the third action state. The style feature extraction of the first encoding by the style encoder includes: The style feature extraction of the first encoding by the style encoder includes:

3. The method of claim 2, wherein, The style feature extraction of the first encoding by the style encoder includes: The style loss is calculated according to the style encodings of different action states, including:

4. The method according to any one of claims 1 to 3, characterized in that, The sum of the first difference value and the style boundary distance is determined as the style loss, the first difference value is the difference between the first loss and the second loss, the first loss includes the distance between the first style encoding and the second style encoding, the second loss includes the distance between the first style encoding and the third style encoding, and the style boundary distance is a constant. The action generation model further includes a fully connected layer; the method further includes:

5. The method of claim 1, wherein, The fully connected layer is called to predict the coefficients of the style encodings to obtain normalized coefficients, and the normalized coefficients are used to represent the action style of the action state. The action generation model is trained according to the action loss, the style loss, the adversarial loss, the feature matching loss and the reconstruction loss, including: The weighted sum of the action loss, the style loss, the reconstruction loss, the feature matching loss and the adversarial loss is calculated; 6. A motion generation method characterized by comprising: The action generation model is trained according to the weighted sum. The method includes: For any virtual role, the action encoder in the trained action generation model is called to perform action feature extraction on the input action state of the virtual role to obtain input action encoding, the action encoding includes the features of the action type of the virtual role, and the action encoder is used to output similar action encodings for the action states of the same action type; the input action state includes the action state of the virtual role at a moment or within a period of time; The style encoder in the action generation model is invoked to perform style feature extraction on the input action state of the virtual character, to obtain an input style code, the style code including features of an action style of the virtual character, the style encoder being configured to output similar style codes for the action states with the same action style; The decoder is invoked to decode the input action code and the input style code to obtain an output action of the virtual character, the output action being a predicted action of the virtual character at a next time point; The training process of the action generation model includes: A first encoder in the action generation model is invoked to perform feature extraction on an action state of a virtual character, to obtain a first code; the action state including at least one of bone joint information, motion trajectory information, target point information, and phase; The style encoder is invoked to perform style feature extraction on the first code, to obtain a style code; The action state includes a first action state, a fourth action state with the same action type as the first action state, and a fifth action state with a different action type from the first action state; the action encoder is invoked to perform action feature extraction on the first code corresponding to the first action state, the fourth action state, and the fifth action state respectively, to obtain a first action code, a fourth action code, and a fifth action code; Instance normalization is performed on the action code to obtain a normalized action code; adaptive instance normalization is performed on the normalized action code according to a normalization coefficient to obtain a normalization result; The decoder is invoked to decode the normalization result to obtain a predicted action of the virtual character; A style loss is calculated according to the style codes of different action states; a second difference and a sum of action boundary distances are determined as an action loss, the second difference being a difference between a third loss including a distance between the first action code and the fourth action code, and a fourth loss including a distance between the first action code and the fifth action code, the action boundary distance being a constant, the action loss encouraging the distance between the action codes of the action states with the same action type to be smaller than the distance between the action codes of the action states with different action types by the action boundary distance; a reconstruction loss is calculated according to a distance between a real action corresponding to the action state and the predicted action, the reconstruction loss being used to make the predicted action close to the real action; A discriminator is invoked to determine whether the predicted action is a real action, and a determination result is taken as a first prediction result; the discriminator is invoked to determine whether a real action is a real action, and a determination result is taken as a second prediction result; a difference between the second prediction result and a preset constant is taken as a third prediction result; an adversarial loss is calculated according to the first prediction result and the third prediction result, the adversarial loss being used to train the discriminator to distinguish between the predicted action and the real action; calculate a feature matching loss based on a distance between a first value and a second value, the first value being an output of a penultimate layer of the discriminator after the predicted action is input into the discriminator, the second value being an output of the penultimate layer of the discriminator after the real action is input into the discriminator, a ratio between the set of action states of the same style, the feature matching loss being used to minimize a distance between the predicted action input into the discriminator and an average of the real actions input into the discriminator; train the action generation model according to the action loss, the style loss, the adversarial loss, the feature matching loss, and the reconstruction loss, the action loss being used to train the action encoder to output the action encodings similar to each other for the action states of the same action type, the style loss being used to train the style encoder to output the style encodings similar to each other for the action states of the same action style.

7. The method of claim 6, wherein, the action generation model further comprises a fully connected layer; the method further comprises: invoke the fully connected layer to perform coefficient prediction on the input style encoding, to obtain an input normalization coefficient, the input normalization coefficient being used to represent the action style of the input action state; perform instance normalization on the input action encoding, to obtain an input normalized action encoding; the invoking the decoder to decode the input action encoding and the input style encoding to obtain the output action of the virtual character comprises: perform adaptive instance normalization on the input normalized action encoding according to the input normalization coefficient, to obtain an input normalization result; perform action prediction according to the input normalization result to obtain the output action.

8. A training device for an action generation model, characterized in that, the action generation model comprises a first encoder, an action encoder, a style encoder, and a decoder; the apparatus comprises: a first encoding module configured to invoke the first encoder to perform feature extraction on an action state of a virtual character, to obtain a first encoding, the action state comprising at least one of bone joint information, motion trajectory information, target point information, and phase; a style encoding module configured to invoke the style encoder to perform style feature extraction on the first encoding, to obtain a style encoding, the style encoding comprising a feature of an action style of the virtual character; an action state comprises a first action state, a fourth action state of the same action type as the first action state, and a fifth action state of a different action type from the first action state; an action encoding module is configured to invoke the action encoder to perform action feature extraction on respective first encodings of the first action state, the fourth action state, and the fifth action state, to obtain a first action encoding, a fourth action encoding, and a fifth action encoding; the action encoding comprises a feature of an action type of the virtual character; an instance normalization module configured to perform instance normalization on the action encoding, to obtain a normalized action encoding; a decoding module, the decoding module comprising an adaptive instance normalization submodule and a prediction submodule; the adaptive instance normalization submodule is configured to perform adaptive instance normalization on the normalized action encoding according to a normalization coefficient, to obtain a normalization result; The prediction submodule is configured to call the decoder to decode the normalization result to obtain a predicted action of the virtual character. The loss module is configured to calculate a style loss according to the style encodings of different action states, and determine a sum of a second difference value and an action boundary distance as an action loss, the second difference value being a difference between a third loss including a distance between the first action encoding and the fourth action encoding, and a fourth loss including a distance between the first action encoding and the fifth action encoding, the action boundary distance being a constant, the action loss encouraging the distance between the action encodings of action states of the same action type to be smaller than the distance between the action encodings of action states of different action types by the action boundary distance. The loss module is further configured to calculate a reconstruction loss according to a distance between a real action corresponding to the action state and the predicted action, the reconstruction loss being used to make the predicted action close to the real action. The discrimination module is configured to call the discriminator to determine whether the predicted action is a real action, and take a result of the determination as a first prediction result, call the discriminator to determine whether the real action is a real action, and take a result of the determination as a second prediction result, and take a difference between the second prediction result and a preset constant as a third prediction result. The loss module is further configured to calculate an adversarial loss according to the first prediction result and the third prediction result, the adversarial loss being used to train the discriminator to distinguish between the predicted action and the real action. The loss module is further configured to calculate a feature matching loss based on a distance between a first value and a second value, the first value being an output of a penultimate layer of the discriminator after the predicted action is input into the discriminator, and the second value being a ratio between a set of action states of the same style and an average value of the real action input into the discriminator and the predicted action input into the discriminator, the feature matching loss being used to minimize the distance between the average value of the real action input into the discriminator and the predicted action input into the discriminator. The training module is configured to train the action generation model according to the action loss, the style loss, the adversarial loss, the feature matching loss, and the reconstruction loss, the action loss being used to train the action encoder to output the action encodings close to each other for the action states of the same action type, and the style loss being used to train the style encoder to output the style encodings close to each other for the action states of the same action style. The modules are configured to perform the following steps: for any virtual character, calling the trained action generation model to predict an action of the virtual character at a next time according to action states of the virtual character at a time or a period of time.

9. The apparatus of claim 8, wherein, The action states further include a second action state and a third action state, the action style of the first action state being the same as that of the second action state, and the action style of the first action state being different from that of the third action state. The style encoding module is configured to: call the style encoder to perform style feature extraction on the first encoding of the first action state to obtain a first style encoding; call the style encoder to perform style feature extraction on the first encoding of the second action state to obtain a second style encoding; and call the style encoder to perform style feature extraction on the first encoding of the third action state to obtain a third style encoding.

10. The apparatus of claim 9, wherein, The loss module is configured to: determine a sum of a first difference value and a style boundary distance as the style loss, the first difference value being a difference between a first loss and a second loss, the first loss including a distance between the first style encoding and the second style encoding, the second loss including a distance between the first style encoding and the third style encoding, and the style boundary distance being a constant.

11. The apparatus of any one of claims 8 to 10, wherein, The action generation model further includes a fully connected layer, and the decoding module includes an adaptive instance normalization submodule and a prediction submodule. The fully connected module is configured to call the fully connected layer to perform coefficient prediction on the style encoding to obtain a normalization coefficient, the normalization coefficient being used to represent the action style of the action state.

12. The apparatus of claim 8, wherein, The loss module is configured to calculate a weighted sum of the action loss, the style loss, the reconstruction loss, the feature matching loss, and the adversarial loss. The training module is configured to train the action generation model according to the weighted sum.

13. An action generation apparatus characterized by comprising: The device includes: An action feature extraction module is configured to, for any virtual character, call an action encoder in a trained action generation model to perform action feature extraction on an input action state of the virtual character to obtain an input action encoding, the action encoding including features of an action type of the virtual character, and the action encoder being configured to output similar action encodings for the action states of the same action type; and the input action state including an action state of the virtual character at a time or over a period of time. A style feature extraction module is configured to call a style encoder in the action generation model to perform style feature extraction on the input action state of the virtual character to obtain an input style encoding, the style encoding including features of an action style of the virtual character, and the style encoder being configured to output similar style encodings for the action states of the same action style. A generation module is configured to call a decoder to decode the input action encoding and the input style encoding to obtain an output action of the virtual character, the output action being a predicted action of the virtual character at a next time. The training process of the action generation model includes: calling a first encoder in the action generation model to perform feature extraction on an action state of a virtual character to obtain a first encoding, the action state including at least one of bone joint information, motion trajectory information, target point information, and phase; calling a style encoder to perform style feature extraction on the first encoding to obtain a style encoding; and The action states include a first action state, a fourth action state of the same action type as the first action state, and a fifth action state of a different action type from the first action state; the action encoder is called to perform action feature extraction on respective first encodings of the first action state, the fourth action state, and the fifth action state, to obtain a first action encoding, a fourth action encoding, and a fifth action encoding; The action encodings are instance-normalized to obtain normalized action encodings; the normalized action encodings are adaptively instance-normalized according to a normalization coefficient to obtain a normalization result; The decoder is called to decode the normalization result to obtain a predicted action of the virtual character; A style loss is calculated according to the style encodings of different action states; a second difference and a sum of action boundary distances are determined as an action loss, the second difference being a third loss including a distance between the first action encoding and the fourth action encoding, and a fourth loss being a difference between a distance between the first action encoding and the fifth action encoding, the action boundary distance being a constant, the action loss encouraging a distance between action encodings of action states of the same action type to be smaller than a distance between action encodings of action states of different action types by the action boundary distance; a reconstruction loss is calculated according to a distance between a real action corresponding to the action state and the predicted action, the reconstruction loss being used to make the predicted action close to the real action; The discriminator is called to determine whether the predicted action is a real action, and a determination result is taken as a first prediction result; the discriminator is called to determine whether the real action is a real action, and a determination result is taken as a second prediction result; a difference between the second prediction result and a preset constant is taken as a third prediction result; an adversarial loss is calculated according to the first prediction result and the third prediction result, the adversarial loss being used to train the discriminator to distinguish between the predicted action and the real action; A feature matching loss is calculated based on a distance between a first value and a second value, the first value being an output of a penultimate layer of the discriminator after the predicted action is input into the discriminator, the second value being a ratio between a set of action states of the same style, the feature matching loss being used to minimize a distance between an average of the predicted action input into the discriminator and the real action input into the discriminator; The action generation model is trained according to the action loss, the style loss, the adversarial loss, the feature matching loss, and the reconstruction loss; the action loss is used to train the action encoder to output similar action encodings for action states of the same action type, and the style loss is used to train the style encoder to output similar style encodings for action states of the same action style.

14. The apparatus of claim 13, wherein, The action generation model further includes a fully connected layer; The device further includes: The fully connected layer module is configured to call the fully connected layer to perform coefficient prediction on the input style encoding to obtain an input normalization coefficient, the input normalization coefficient being used to represent the action style of the input action state. The normalization module is configured to perform instance normalization on the input action code to obtain an input normalized action code. The generation module is configured to perform adaptive instance normalization on the input normalized action code according to the input normalized coefficient to obtain an input normalized result, and perform action prediction according to the input normalized result to obtain the output action.

15. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the action generation model training method or the action generation method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the action generation model training method or the action generation method according to any one of claims 1 to 7.

17. A computer program product, comprising computer instructions stored in a computer readable storage medium, wherein a processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to implement the action generation model training method or the action generation method according to any one of claims 1 to 7.

Images