Generation method, model, device and electronic equipment

Abstract

The invention discloses a generation method, model and device and electronic equipment. The method comprises the steps that action description information input by a user is obtained; determining text features based on the action description information; processing the text features, and gradually obtaining an action feature set with multiple time sequence lengths according to a pre-trained codebook; the action feature set of the time sequence length comprises a plurality of action features with different time sequences; the codebook is used for storing a mapping relation between a plurality of discrete code values and a plurality of action characteristics; and processing the action feature set with the multiple time sequence lengths to obtain a target action sequence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a generation method, a model, an apparatus and an electronic device. BACKGROUND

[0002] In the field of human action generation, with the continuous improvement of requirements for action quality and generation efficiency in various application scenarios, how to realize the rapid generation of human action sequences while ensuring that the generated actions have high realism has become a key problem to be solved. SUMMARY

[0003] The technical scheme provided by the present application is as follows:

[0004] The first aspect of the present application provides a generation method, comprising:

[0005] obtaining action description information input by a user;

[0006] determining a text feature based on the action description information;

[0007] processing the text feature to gradually obtain a plurality of action feature sets of different time lengths according to a pre-trained codebook; the action feature set of the time length comprises a plurality of action features of different time sequences; the codebook is used to store a mapping relationship between a plurality of discrete code values and a plurality of action features;

[0008] processing the plurality of action feature sets of different time lengths to obtain a target action sequence.

[0009] In a possible implementation, the processing the text feature to gradually obtain a plurality of action feature sets of different time lengths according to a pre-trained codebook comprises:

[0010] taking the text feature as a candidate action feature set of a first time length, processing the candidate action feature set of the first time length, and selecting, from the pre-trained codebook, action features corresponding in number to the first time length as an action feature set of the first time length;

[0011] processing the action feature set of the first time length to obtain a candidate action feature set of a second time length; the second time length is greater than the first time length;

[0012] taking the candidate action feature set of the second time length as a new candidate action feature set of the first time length, and continuing to perform the processing of the candidate action feature set of the first time length until a plurality of action feature sets of different time lengths are obtained.

[0013] In a possible implementation, the processing of the candidate action feature set of the first time length includes: selecting, from a pre-trained codebook, a number of action features corresponding to the first time length as the action feature set of the first time length.

[0014] The candidate action feature set of the first time length is input into a prediction module, and the prediction module determines a conditional probability corresponding to each candidate action feature in the candidate action feature set of the first time length; the conditional probability represents a matching degree between an action feature corresponding to each discrete code value in the pre-trained codebook and the candidate action feature.

[0015] A maximum probability is selected from the conditional probability corresponding to each candidate action feature.

[0016] An action feature corresponding to each maximum probability is selected from the codebook to determine the action feature set of the first time length.

[0017] In another aspect of the present application, a generation model is provided, comprising:

[0018] A codebook is configured to store a mapping relationship between a plurality of discrete code values and a plurality of action features.

[0019] A prediction module is configured to process a text feature to gradually obtain a plurality of action feature sets of time lengths according to the codebook; the action feature set of the time length includes a plurality of action features of different time lengths; and the text feature is from action description information input by a user.

[0020] A generation module is configured to process the plurality of action feature sets of time lengths to obtain a target action sequence.

[0021] In a possible implementation, the generation model further comprises an encoding module.

[0022] The encoding module, the codebook, and the generation module are trained in the following manner:

[0023] A first sample action sequence is obtained.

[0024] The first sample action sequence is processed by the encoding module to obtain a first candidate sample action feature set of a first sample time length.

[0025] Based on the codebook and the first candidate sample action feature set of the first sample time length, a plurality of first candidate sample action feature sets of sample time lengths and a first sample action feature set are obtained.

[0026] The first sample action feature set of various sample time series lengths is input into the generation module, and the generation module processes it to generate a predicted action sequence.

[0027] If the first loss and the second loss do not meet the first set condition, the parameters of the encoding module and the generation module and the codebook are adjusted, and the step of obtaining the first sample action sequence continues to be executed; the first loss represents the difference between the predicted action sequence and the first sample action sequence; the second loss represents the semantic consistency between the first candidate sample action feature set and the first sample action feature set.

[0028] If the first loss and the second loss satisfy the first set condition, the training step ends.

[0029] In one possible implementation, the process of obtaining a first candidate sample action feature set and a first sample action feature set with multiple sample time series lengths based on the codebook and the first candidate sample action feature set with the first sample time series length includes:

[0030] Select action features that match the first candidate sample action feature set with the first sample time sequence length from the codebook to obtain the first sample action feature set with the first sample time sequence length;

[0031] Determine the residual between the first candidate sample action feature set and the first sample action feature set to obtain a new first candidate sample action feature set with the first sample temporal length;

[0032] The action feature set of the new first candidate sample with the first sample time series length is processed to obtain the action feature set of the first candidate sample with the second sample time series length; the second sample time series length is less than the first sample time series length.

[0033] The first candidate sample action feature set of the second sample time sequence length is used as the new first candidate sample action feature set of the first sample time sequence length. The step of selecting action features that match the first candidate sample action feature set of the first sample time sequence length from the codebook is continued until the first sample action feature set of multiple sample time sequence lengths is obtained.

[0034] In one possible implementation, the processing of the first sample action sequence based on the encoding module to obtain a first candidate sample action feature set with a first sample time sequence length includes:

[0035] Extract multiple sample sub-action sequences from the first sample action sequence;

[0036] The encoding module processes each of the sample sub-action sequences to obtain the first candidate sample sub-action feature set with the first sample time sequence length.

[0037] In one possible implementation, the prediction module is trained in the following manner:

[0038] Obtain the second sample action sequence and the corresponding text description of the second sample action sequence;

[0039] The second sample action sequence is processed by the encoding module to obtain a second candidate sample action feature set with a first sample time sequence length;

[0040] Based on the codebook and the second candidate sample action feature set with the first sample time sequence length, a second sample action feature set with multiple sample time sequence lengths is obtained;

[0041] The third sample action feature set is obtained by concatenating the text descriptions corresponding to each second sample action feature set and the second sample action sequence.

[0042] The third sample action feature set is input into the prediction module to obtain the sample conditional probability determined by the prediction module; the sample conditional probability represents the degree of matching between the action feature corresponding to each discrete code value in the codebook and the third sample action feature set;

[0043] If the third loss does not meet the second set condition, keep the parameters of the encoding module and the codebook unchanged, adjust the parameters of the prediction module, and continue to execute the step of obtaining the second sample action sequence; the third loss represents the difference between the maximum probability in the sample conditional probability and the discrete code value in the codebook corresponding to the third sample action feature set;

[0044] If the third loss satisfies the second set condition, the training step ends.

[0045] A third aspect of this application provides a generating apparatus, comprising:

[0046] The acquisition unit is used to acquire action description information input by the user.

[0047] The determining unit is used to determine text features based on the action description information;

[0048] The processing unit is used to process the text features and gradually obtain action feature sets of various temporal lengths based on a pre-trained codebook; the action feature sets of various temporal lengths contain multiple action features with different temporal lengths; the codebook is used to store multiple discrete code values ​​and the mapping relationship between multiple action features;

[0049] The generation unit is used to process the action feature sets of the various time lengths to obtain the target action sequence.

[0050] In a fourth aspect, this application provides an electronic device comprising:

[0051] Memory is used to store computer programs;

[0052] The processor is used to execute the computer program to enable the electronic device to perform the following method steps:

[0053] Obtain description information of the user's input action;

[0054] Based on the action description information, text features are determined;

[0055] The text features are processed to gradually obtain action feature sets of various temporal lengths based on a pre-trained codebook; each temporal length action feature set contains multiple action features with different temporal lengths; the codebook is used to store multiple discrete code values ​​and the mapping relationship between multiple action features.

[0056] The action feature sets of the various time lengths are processed to obtain the target action sequence. Attached Figure Description

[0057] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0058] Figure 1 This is a flowchart illustrating a generation method provided in Embodiment 1 of this application;

[0059] Figure 2 This is a flowchart illustrating a generation method provided in Embodiment 2 of this application;

[0060] Figure 3 A schematic diagram illustrating an implementation scenario of the generation method provided in this application;

[0061] Figure 4 This application provides an interactive diagram between an encoding module and a generation module;

[0062] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0063] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0064] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0065] The terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0066] Reference Figure 1 This is a flowchart illustrating a generation method provided in Embodiment 1 of this application, as shown below. Figure 1 As shown, the method may include, but is not limited to, the following steps:

[0067] Step 101: Obtain the action description information input by the user.

[0068] In this embodiment, users can input descriptive information about the human body movements they wish to generate in natural language, according to their own needs.

[0069] For example, if a user wants to generate an action of "a character running quickly and then suddenly stopping and turning around," they can enter a text description like this. This description can cover multiple aspects such as the type of action (e.g., walking, jumping, fighting), the speed of the action (fast, slow), the amplitude of the action (large, small), and the scene in which the action takes place (indoor, outdoor).

[0070] Step 102: Determine text features based on the action description information.

[0071] In this embodiment, the word embedding method can be used to convert each word in the action description information into a word vector, which can capture the semantic information of the words.

[0072] Then, the word vectors can be further processed by a text encoder (such as a recurrent neural network (RNN), a long short-term memory network (LSTM), or a Transformer) to obtain fixed-dimensional text feature vectors. Text feature vectors are abstract representations of user input actions, containing key information about the action, such as its type, direction, and intensity, which can provide semantic guidance for subsequent generation of action sequences.

[0073] Step 103: Process the text features and gradually obtain action feature sets of various time lengths based on the pre-trained codebook; the action feature sets of various time lengths contain multiple action features with different time sequences; the codebook is used to store multiple discrete code values ​​and the mapping relationship between multiple action features.

[0074] In practical applications, a single-time-length set of motion features often fails to comprehensively and accurately describe a motion. Motion is a continuously changing process over time, with different stages exhibiting distinct characteristics. For example, walking involves multiple stages such as starting, accelerating, maintaining a constant speed, decelerating, and stopping, each with unique features in posture, trajectory, and joint angles. A single-time-length set of motion features can only capture the characteristics of a motion within a specific timeframe, failing to fully represent the entirety of the action.

[0075] To fully represent the entirety of an action, a set of action features of various time lengths can be gradually obtained through a codebook. These sets of action features of different time lengths can characterize the action from different temporal dimensions. For example, a set of action features with a current time length of two consecutive time points (e.g., two frames) primarily focuses on local features of the action, accurately describing the posture and motion details of the action within a very short timeframe (e.g., a moment or a minute movement phase). For instance, in the initial stage of a character walking, a set of action features from two consecutive time points can clearly capture the posture and joint angle changes at the moment the foot leaves the ground. However, it cannot reflect the overall trend and patterns of change of the action over a longer time period.

[0076] A motion feature set with a temporal length of eight consecutive time points (e.g., eight frames) can cover a wider time range and capture the overall characteristics and development trend of the motion. In the example of a person walking, the motion feature set of eight consecutive time points can present the complete process from starting to accelerating and then walking at a constant speed, including the gradual changes in body posture, the extension of the movement trajectory, and the continuous adjustment of joint angles. However, due to the longer time range, its description of the details of the motion at each instant may be relatively less accurate than that of a motion feature set with a shorter temporal length.

[0077] The discrete code values ​​in the codebook are a discretized representation of actions. Each discrete code value uniquely corresponds to a specific action feature, and the action features corresponding to each discrete code value are different.

[0078] The motion features corresponding to discrete code values ​​can contain multi-dimensional numerical information, and different dimensions of numerical information can reflect the motion from different perspectives. For example, the motion features corresponding to discrete code values ​​can contain 263 dimensions of numerical information. The first 50 dimensions of numerical information may be used to describe the posture information of the motion, including the spatial position and relative angle of various parts of the body; the 51st to 150th dimensions of numerical information can be used to characterize the motion trajectory, such as the movement path of the body's center of gravity in space; and the 151st to 263rd dimensions of numerical information can be used to record changes in joint angles, such as the degree of bending of the arm and leg joints at different times.

[0079] In this embodiment, the text features obtained in step 102 can be further processed to map them to an initial action feature. Then, the discrete code value that best matches the initial action feature is selected from the codebook. Based on the mapping relationship in the codebook, the action feature corresponding to the discrete code value is found to obtain the action feature set for the current time sequence length (e.g., 2 consecutive time points).

[0080] Then, based on the action feature set of the current time sequence length (e.g., two consecutive time points), discrete code values ​​matching the new time sequence length (e.g., four consecutive time points) are searched from the codebook. The new time sequence length can be an extension of the current time sequence length according to predetermined rules (e.g., upsampling). After finding matching discrete code values, action features corresponding to each discrete code value are obtained from the codebook according to the mapping relationships in the codebook, and these action features are combined to form the action feature set of the new time sequence length.

[0081] By following the above method and repeatedly searching for new time-length matching discrete code values ​​based on the current time-length action feature set, and then obtaining the corresponding action features to form a new set, action feature sets of various time-lengths can be gradually obtained. For example, for action description information describing "a person walking slowly," a shorter time-length action feature set might be obtained first (e.g., a set containing action features at two consecutive time points), representing the initial stage of the person starting to walk; then, based on the set containing action features at two consecutive time points, a set containing action features at four consecutive time points is obtained; and then, based on the set containing action features at four consecutive time points, a set containing action features at eight consecutive time points is obtained. In this way, the entire process of a person walking slowly can be completely represented. These action feature sets of different time-lengths can capture the feature changes of the action at different time scales, providing rich and accurate information for generating coherent and natural action sequences.

[0082] In this embodiment, the codebook can be trained based on a large amount of real motion data, so that the motion features in the codebook can accurately reflect the physical laws and motion characteristics of human motion.

[0083] Step 104: Process the action feature sets of the various time lengths to obtain the target action sequence.

[0084] In this embodiment, since the time ranges corresponding to action feature sets with different time lengths are different, time alignment operations can be performed to ensure the continuity of actions in the time dimension.

[0085] Specifically, time interpolation or time scaling methods can be used to map action feature sets of different time series lengths onto the same time axis. For example, for a short-time action feature set, interpolation methods can be used to extend it on the time axis so that it corresponds to the long-time action feature set in time.

[0086] The time-aligned sets of motion features are input into the decoder. The decoder can process and integrate the motion features based on the input sets of motion features, combined with its internal learning mechanism and parameter settings, to generate motion data at multiple consecutive time points, and finally output a complete sequence of the target motion.

[0087] For example, when a user inputs a description of a martial arts movement, including a starting stance, kicking, punching, and finishing stance, the preceding steps generate a set of movement features of varying temporal lengths. After time alignment, these feature sets are input into a decoder, which generates a complete target movement sequence. This sequence clearly demonstrates the character's continuous movement changes, from the initial standing posture to the leg extension and weight shift during the kick, the arm swing and power generation during the punch, and finally the stable standing posture at the finishing stance.

[0088] In this embodiment, based on the pre-stored mapping relationship between a large number of discrete code values ​​and action features in the codebook, action description information input by the user is obtained, and text features are determined based on the action description information. The text features are then processed, and action feature sets of various time lengths are gradually obtained according to the pre-trained codebook. This can transform the complex action generation problem into a relatively simple operation of matching and searching in the codebook, eliminating the need to calculate and generate action features from scratch each time, avoiding a large amount of repetitive calculation, and greatly reducing the amount of computation.

[0089] The reduced computational load allows for faster action sequence generation, enabling quicker responses to generation requests. For example, in real-time interactive applications such as game development or virtual reality experiences, users can input action descriptions, leading to rapid generation of corresponding action sequences, thus improving user experience and interaction efficiency.

[0090] Furthermore, the motion features in the codebook are carefully trained and selected, learned from a large amount of real motion data, and can cover various types of actions, speeds, amplitudes, and scenarios. Regardless of the action description information input by the user, suitable motion features can be found in the codebook for combination and expansion, thereby ensuring that the generated actions are highly realistic and meet the user's personalized needs.

[0091] As another optional embodiment of this application, refer to Figure 2 This is a flowchart illustrating a generation method provided in Embodiment 2 of this application. This embodiment is mainly an implementation of step S103 in Embodiment 1 above, such as... Figure 2 As shown, the following steps may be included, but are not limited to:

[0092] Step S1031: Use the text features as a set of candidate action features with a first temporal length.

[0093] Step S1032: Process the candidate action feature set of the first time sequence length, and select action features from the pre-trained codebook with a number corresponding to the first time sequence length as the action feature set of the first time sequence length.

[0094] In this embodiment, the similarity between each candidate action feature in the candidate action feature set of the first time-series length and each action feature in the pre-trained codebook can be calculated. For example, a cosine similarity algorithm can be used, which measures the similarity between two vectors by calculating the cosine of the angle between them; the closer the value is to 1, the higher the similarity. The action features with the highest similarity in the codebook are selected to form the action feature set of the first time-series length. This is done because action features with high similarity are semantically and behaviorally closer to the action features expected in the text description, and can better reflect the action description information input by the user.

[0095] For example, assuming the user inputs the action description "a person slowly raises their arm", step 102 obtains text features as a candidate action feature set for the first time sequence length. The codebook contains a large number of different action features. By calculating similarity, it is found that one or more action features in the codebook are most similar to the description "slowly raises their arm" in terms of posture, movement trend, etc., and these one or more action features are then selected as the action feature set for the first time sequence length.

[0096] Step S1033: Process the action feature set of the first time sequence length to obtain the candidate action feature set of the second time sequence length; the second time sequence length is greater than the first time sequence length.

[0097] Based on the first time-length action feature set, a larger time-length action feature set can be generated to gradually capture the feature changes of actions in different time dimensions.

[0098] Specifically, we can first upsample the action feature set for the first time series length. Taking linear interpolation as an example, assuming the first time series length is 2 consecutive time points and the action features are A and B, when upsampling to the second time series length of 4 consecutive time points, we insert two new action features between A and B, thus obtaining the candidate action feature set for the second time series length.

[0099] For example, if the first time sequence length is two consecutive time points, representing the instant when a person's arm just begins to lift, and after upsampling to obtain four consecutive time points (i.e., one implementation of the second time sequence length) of candidate action features, the process of the arm lifting from the beginning, to lifting it slightly, and then lifting it more fully can be described in more detail, and the local feature changes of the action can be presented more comprehensively.

[0100] Step S1034: Determine whether a set of action features with multiple time series lengths has been obtained.

[0101] If not, proceed to step S1035; if yes, proceed to step S104.

[0102] Step S1035: Use the candidate action feature set of the second time series length as the new candidate action feature set of the first time series length, and continue to execute step S1032.

[0103] By iteratively executing the above steps, the temporal length is continuously expanded, gradually obtaining action feature sets of various temporal lengths. For example, after the first execution, a set of candidate action features containing four consecutive time points is obtained (i.e., one implementation of the candidate action feature set for the second temporal length). This set is used as a new candidate action feature set for the first temporal length. The similarity with action features in the codebook is calculated again, and action features with high similarity are selected to form a new action feature set. At this point, a set of action features containing four consecutive time points is obtained (i.e., one implementation of the action feature set for the second temporal length). This process is repeated until action feature sets of various temporal lengths that meet the requirements are obtained.

[0104] For example, taking a person's walking action as an example, the first loop obtains a set of action features containing two consecutive time points, describing the moment when walking begins; the second loop obtains a set of action features containing four consecutive time points, describing the process from starting to taking a small step; the third loop obtains a set of action features containing eight consecutive time points, which can more completely present the stage of walking from starting to accelerating.

[0105] In this embodiment, text features are first used as a candidate action feature set for a first time sequence length. This set is then processed to select action features from a pre-trained codebook, with the number corresponding to the first time sequence length, thus forming an action feature set for the first time sequence length. Next, the action feature set for the first time sequence length is further processed to obtain a candidate action feature set for a second time sequence length. This set is then used as a new candidate action feature set for the first time sequence length, and the processing operation on the candidate action feature set for the first time sequence length is repeated. This process is repeated iteratively until action feature sets of multiple time sequence lengths are generated. This method can generate action feature sets of different time sequence lengths as needed, effectively avoiding a large amount of repetitive computation and reducing the computational load. This allows for rapid generation of action sequences and quick response to relevant needs, significantly improving user experience and interaction efficiency in real-time interactive scenarios (such as games and virtual reality).

[0106] As another optional embodiment of this application, a generation method provided in Embodiment 3 of this application, this embodiment mainly processes the candidate action feature set of the first time series length in Embodiment 2 above, and selects a number of action features corresponding to the first time series length from a pre-trained codebook as an implementation of the action feature set of the first time series length. Specifically, it may include, but is not limited to, the following steps:

[0107] Step S11: Input the candidate action feature set of the first time series length into the prediction module, and the prediction module determines the conditional probability corresponding to each candidate action feature in the candidate action feature set of the first time series length.

[0108] The prediction module can be a trained neural network model, such as a model based on the Transformer architecture. This model can receive a set of candidate action features of a first temporal length as input. Through internal self-attention mechanisms and other structures, it can capture the temporal correlation between various candidate features in the set of candidate action features of the first temporal length. When calculating the conditional probability, it not only considers the matching degree between a single candidate action feature and the action features in the codebook, but also constrains the action features in the codebook corresponding to adjacent candidate action features at different time points to conform to the same action logic, thus avoiding the break in action logic.

[0109] The conditional probability can represent the degree of matching between the action feature corresponding to each discrete code value in the pre-trained codebook and the candidate action feature. The higher the probability value in the conditional probability, the higher the degree of matching between the action feature corresponding to the discrete code value and the candidate action feature, that is, the two are more similar in terms of action patterns, dynamic characteristics, etc.

[0110] For example, suppose there are two candidate action features in the candidate action feature set of the first time sequence length, denoted as candidate action features A11 and A12 respectively. The codebook includes 512 discrete code values. These two candidate action features are input to the prediction module. The prediction module can calculate 512 probability values ​​for candidate action features A11 and A13 respectively, and obtain two conditional probabilities, which can be represented as P11 = [p11_1, p11_2, …, p11_512] and P12 = [p12_1, p12_2, …, p12_512] respectively.

[0111] Where p11_i (i = 1, 2, …, 512) represents the probability value between the action feature corresponding to the i-th discrete code value in the codebook and the candidate action feature A11. Similarly, p12_i (i = 1, 2, …, 512) represents the probability value between the action feature corresponding to the i-th discrete code value in the codebook and the candidate action feature A12.

[0112] Step S12: Select the highest probability from the conditional probabilities corresponding to each candidate action feature.

[0113] In this embodiment, the probability values ​​of the conditional probabilities corresponding to the candidate action features can be compared, and the maximum probability can be selected from the various probability values.

[0114] For example, for the conditional probability P11 = [p11_1, p11_2, …, p11_512] corresponding to the candidate action feature A11, we can iterate through these 512 probability values. During the iteration, we compare the size of each probability value one by one, and by continuously updating the current maximum value, we finally find the maximum value, which is denoted as max_p11.

[0115] Similarly, for the conditional probability P12 = [p12_1, p12_2, …, p12_512] corresponding to the candidate action feature A12, the same traversal comparison algorithm can be used to find the maximum value, denoted as max_p12.

[0116] Step S13: Select the action feature corresponding to each of the maximum probabilities from the codebook and determine it as the action feature set of the first temporal length.

[0117] Since each discrete code value in the codebook corresponds to a specific action feature, and the correlation between the conditional probability and the discrete code values ​​in the codebook has already been established when the conditional probability is obtained based on the prediction module in step S11, once the maximum probability is found, its corresponding discrete code value can be determined, and then the action feature corresponding to that discrete code value can be selected from the codebook. For example, once the maximum probability max_p11 is found, its corresponding discrete code value in the codebook can be determined, and then the action feature corresponding to that discrete code value can be selected from the codebook, denoted as Feature_A11. Similarly, for the maximum probability max_p12, the action feature corresponding to its corresponding discrete code value in the codebook is selected, denoted as Feature_A12.

[0118] These selected action features can be combined to form an action feature set of the first time length, for example, {Feature_A11, Feature_A12}.

[0119] In this embodiment, a candidate action feature set of the first time-series length is input into the prediction module. This prediction module (such as a model based on the Transformer architecture) uses internal self-attention mechanisms and other structures to not only consider the matching degree between individual candidate action features and action features in the codebook, but also to constrain action features in the codebook corresponding to adjacent candidate action features to conform to the same action logic, avoiding action logic breaks and ensuring the rationality and reliability of conditional probabilities. Based on this, the maximum probability is selected from the conditional probabilities corresponding to each candidate action feature. This maximum probability represents the optimal probability of combining the candidate action feature with a certain action feature in the codebook. Finally, based on these selected maximum probabilities, corresponding action features are selected from the codebook, and these action features are aggregated to ultimately determine the action feature set of the first time-series length. Through such a rigorous and comprehensive calculation and selection process, the determined action feature set can be effectively guaranteed to achieve a high level of accuracy and rationality, providing a reliable basis for subsequently generating high-quality action sequences.

[0120] In this embodiment, the generation method is illustrated using the user input "The person walks forward, then turns to walk diagonally to the left." (i.e., one implementation of the action description information). This action description information indicates that the person first walks forward, then turns to walk diagonally to the left. The subsequent process of generating the action feature set will revolve around this action change logic.

[0121] For example, such as Figure 3 As shown, action description information can be input into the CLIP B / 32 model to obtain text features determined by the CLIP B / 32 model. These text features are then used as a candidate action feature set with a temporal length of 1 (this set contains one candidate action feature, denoted as candidate action feature A1). Candidate action feature A1 can be initially understood as a feature representation related to the character's initial forward movement; however, it is only a preliminary candidate feature that requires further processing to match suitable action features in the codebook. The candidate action feature set with a temporal length of 1 is input into the prediction module, which determines the conditional probability corresponding to the candidate action feature, denoted as conditional probability P1.

[0122] Select the highest probability from the conditional probabilities P1, denoted as max_p1. Select the action feature corresponding to max_p1 from the codebook, denoted as action feature B1, as the action feature set with a time length of 1. Unlike the candidate action feature A1, B1 can accurately represent the initial forward movement, and is the feature directly found from the codebook that best reflects the initial forward movement.

[0123] Upsample the action feature set with a time sequence length of 1 to obtain a candidate action feature set with a time sequence length of 2 (this set contains candidate action features A2 and A3). A2 and A3 correspond to the continuation of the forward walking action and the feature that begins to show a turning trend, respectively. They are also preliminary candidate features used to find suitable action features in the codebook later.

[0124] The candidate action feature set with a time series length of 2 is input into the prediction module, and the prediction module determines the conditional probabilities corresponding to these two candidate action features, namely conditional probabilities P2 and P3.

[0125] From conditional probabilities P2, select the highest probability, denoted as max_p2. From the codebook, select the action feature corresponding to max_p2, denoted as action feature B2. B2 further emphasizes the forward walking action and is a feature directly obtained from the codebook that more accurately reflects the continuation of the forward walking action. Also, from conditional probabilities P3, select the highest probability, denoted as max_p3. From the codebook, select the action feature corresponding to max_p3, denoted as action feature B3. B3 begins to reflect a left-slanting forward turning action, echoing the character's turning action in the original description. It is the most precise feature selected from the codebook that best reflects the start of the turning trend.

[0126] Combine action features B2 and B3 into a set of action features with a temporal length of 2.

[0127] Upsampling is performed on the action feature set with a time sequence length of 2 to obtain a candidate action feature set with a time sequence length of 3 (this set contains candidate action features A4, A5, and A6). A4, A5, and A6 correspond to features at different stages of the turning action, respectively, which more meticulously depict the action process of the character from the start of the turning to the gradual completion of the turning. However, they are still only candidate action features and are used to match precise action features in the codebook later.

[0128] The candidate action feature set with a time series length of 3 is input into the prediction module, and the prediction module determines the conditional probabilities corresponding to these 3 candidate action features, namely conditional probabilities P4, P5 and P6.

[0129] Select the highest probability from conditional probabilities P4, denoted as max_p4. Select the action feature corresponding to max_p4 from the codebook, denoted as action feature B4. B4 enhances the initial turning action and is a feature directly found from the codebook that most accurately reflects the initial turning action. Also, select the highest probability from conditional probabilities P5, denoted as max_p5. Select the action feature corresponding to max_p5 from the codebook, denoted as action feature B5. B5 reflects the action during the turning process and is the most precise action feature found from the codebook that best reflects the turning process. Finally, select the highest probability from conditional probabilities P6, denoted as max_p6. Select the action feature corresponding to max_p6 from the codebook, denoted as action feature B16. B6 represents the action of the character walking diagonally to the left after the turning is basically complete. It is closely related to the original description "turns to walk diagonally to the left" and is the feature found from the codebook that most accurately reflects the action of completing the turning and walking diagonally to the left.

[0130] Combining action features B4, B5, and B6 into a set of action features with a temporal length of 3.

[0131] Upsampling is performed on the action feature set with a time sequence length of 3 to obtain a candidate action feature set with a time sequence length of 4 (this set contains candidate action features A7, A8, A9, and A10). These upsampled features further refine the action details of the character walking diagonally forward to the left, but they are still only candidate features and are used to match precise action features in the codebook later.

[0132] The candidate action feature set with a time series length of 4 is input into the prediction module, and the prediction module determines the conditional probabilities corresponding to these 4 candidate action features, namely conditional probabilities P7, P8, P9 and P10.

[0133] Select the highest probability from conditional probabilities P7, denoted as max_p7. Select the action feature corresponding to max_p7 from the codebook, denoted as action feature B7. B7 emphasizes the initial movement of walking diagonally forward to the left, and is a feature directly found from the codebook that more accurately reflects the initial movement of walking diagonally forward to the left. Also, select the highest probability from conditional probabilities P8, denoted as max_p8. Select the action feature corresponding to max_p8 from the codebook, denoted as action feature B8. B8 reflects the movement during this walking process and is the most precise action feature found from the codebook that best reflects the movement during the walking process. Also, select the highest probability from conditional probabilities P9, denoted as max_p9. Select the action feature corresponding to max_p9 from the codebook, denoted as action feature B9. B9 further refines the walking movement and is a feature found from the codebook that more accurately reflects the details of the walking movement. Finally, select the highest probability from conditional probabilities P10, denoted as max_p10. Select the action feature corresponding to max_p10 from the codebook and represent it as action feature B10. B10 can represent the action of the character walking steadily to the left and diagonally forward. It fully presents the action process of the character in the original description. It is the most accurate action feature found in the codebook that can accurately reflect the entire action process.

[0134] The action features B7, B8, B9, and B10 are combined into a set of action features with a temporal length of 4.

[0135] In another embodiment of this application, a generative model is provided for embodiment 4 of this application. The generative model may include, but is not limited to:

[0136] A codebook is used to store the mapping relationship between multiple discrete code values ​​and multiple action features.

[0137] The prediction module is used to process text features and gradually obtain action feature sets of various time lengths based on the codebook; the action feature sets of various time lengths contain multiple action features with different time sequences; the text features are derived from action description information input by the user;

[0138] The generation module is used to process the action feature sets of the various time lengths to obtain the target action sequence.

[0139] In this embodiment, for details regarding the specific functional implementation, working principles, and collaborative mechanisms of the codebook, prediction module, and generation module, please refer to the detailed descriptions in the above generation method. To avoid repetition, these details will not be repeated here.

[0140] In another embodiment of this application, a generative model is provided for embodiment 5 of this application. In addition to including a codebook, a prediction module and a generation module, the generative model may also include an encoding module.

[0141] In this embodiment, refer to Figure 4 The encoding module, the codebook, and the generation module can be, but are not limited to, being trained in the following ways:

[0142] Step S21: Obtain the first sample action sequence.

[0143] In this embodiment, the first sample action sequence can be obtained from captured real human motion data, publicly available motion databases, or custom motion datasets. This data should cover a variety of motion types (such as walking, running, jumping, dancing, etc.), speeds, amplitudes, and scenarios.

[0144] Step S22: Process the first sample action sequence based on the encoding module to obtain the first candidate sample action feature set with the first sample time sequence length.

[0145] In this embodiment, structures such as convolutional neural networks (CNN), recurrent neural networks (RNN), or Transformers can be used to extract the spatiotemporal features of the first sample action sequence. During the extraction process, the extracted spatiotemporal features are closely related to the temporal length of the first sample action sequence, that is, the extracted spatiotemporal features can completely cover the entire temporal range of the first sample action sequence, and the final feature set is the first candidate sample action feature set corresponding to the temporal length of the first sample.

[0146] For example, if the first sample action sequence contains motion data at 32 consecutive time points, then the first sample time sequence length can be 32.

[0147] Step S23: Based on the codebook and the first candidate sample action feature set with the first sample time sequence length, obtain the first candidate sample action feature set and the first sample action feature set with multiple sample time sequence lengths.

[0148] In this embodiment, the similarity between each first candidate sample action feature in the first candidate sample action feature set with the first sample time sequence length and each action feature in the codebook can be calculated. The first sample action feature with the highest similarity to each first candidate sample action feature is then selected from the codebook. For example, if the first sample time sequence length is 32, and the first candidate sample action feature set contains first candidate sample action features at 32 consecutive time points, the 32 most similar first sample action features are found from the codebook to form the first sample action feature set with a sample time sequence length of 32.

[0149] Next, based on the first sample action feature set with a sample time sequence length of 32, a first candidate sample action feature set with a sample time sequence length of 16 (containing the first candidate sample action features at 16 consecutive time points) is obtained through a specific method (e.g., downsampling). Then, the 16 first sample action features most similar to the first candidate sample action features at 16 consecutive time points are found from the codebook to form the first sample action feature set with a sample time sequence length of 16.

[0150] Then, based on the first sample action feature set with a sample time sequence length of 16, a first candidate sample action feature set with a sample time sequence length of 8 is obtained (containing the first candidate sample action features at 8 consecutive time points). The 8 first sample action features most similar to the first candidate sample action features at 8 consecutive time points are found from the codebook to form the first sample action feature set with a sample time sequence length of 8.

[0151] Then, based on the first sample action feature set with a sample time sequence length of 8, a first candidate sample action feature set with a sample time sequence length of 4 is obtained (containing the first candidate sample action features at 4 consecutive time points). From the codebook, the 4 first sample action features most similar to the first candidate sample action features at 4 consecutive time points are found to form the first sample action feature set with a sample time sequence length of 4.

[0152] Then, based on the first sample action feature set with a sample time sequence length of 4, a first candidate sample action feature set with a sample time sequence length of 2 is obtained (containing the first candidate sample action features at 2 consecutive time points). From the codebook, the two first sample action features most similar to the first candidate sample action features at 2 consecutive time points are found to form the first sample action feature set with a sample time sequence length of 2.

[0153] Finally, based on the first sample action feature set with a sample time series length of 2, a first candidate sample action feature set with a sample time series length of 1 is obtained (containing the first candidate sample action feature at one consecutive time point). The first sample action feature most similar to the first candidate sample action feature at one consecutive time point is found from the codebook, and the first sample action feature set with a sample time series length of 1 is formed.

[0154] Step S24: Input the first sample action feature set of various sample time series lengths into the generation module, and generate the predicted action sequence after processing by the generation module.

[0155] In this embodiment, the generation module may include, but is not limited to, a deconvolutional network, an RNN decoder, or a Transformer decoder.

[0156] For a detailed description of step S24, please refer to the relevant description of step S104 in Example 1, which will not be repeated here.

[0157] Step S25: If the first loss and the second loss do not meet the first set condition, adjust the parameters of the encoding module and the generation module and the codebook, and continue to execute the step of obtaining the first sample action sequence; the first loss represents the difference between the predicted action sequence and the first sample action sequence; the second loss represents the semantic consistency between the first candidate sample action feature set and the first sample action feature set.

[0158] When calculating the second loss, for each sample's time series length, the similarity between each first sample action feature in its first sample action feature set and the corresponding first candidate sample action feature is compared to a similarity threshold. If the similarity is greater than the similarity threshold, it is considered a hit. The hit rate is calculated by counting the number of hits, and the hit rate is used as the second loss.

[0159] Specifically, for a first candidate sample action feature set with a sample time sequence length of 32, 32 similar first sample action features are found in the codebook. The similarity between these 32 first sample action features in the codebook and the first candidate sample action features is compared with the similarity threshold. If the similarity of 30 first sample action features is greater than the threshold, the number of hits is 30.

[0160] For a first candidate sample action feature set with a sample time sequence length of 16, find 16 similar first sample action features in the codebook. Similarly, compare the similarity with the similarity threshold. If the similarity of 14 first sample action features is greater than the threshold, then the number of hits is 14.

[0161] For a first candidate sample action feature set with a sample time sequence length of 8, find 8 similar first sample action features in the codebook. If the similarity of 7 first sample action features is greater than the threshold after comparison, the number of hits is 7.

[0162] For a first candidate sample action feature set with a sample time sequence length of 4, find 4 similar first sample action features in the codebook. If the similarity of 3 first sample action features is greater than the threshold, the number of hits is 3.

[0163] For a first candidate sample action feature set with a sample time sequence length of 2, find two similar first sample action features in the codebook. If the similarity of two first sample action features is greater than the threshold, the number of hits is 2.

[0164] For a first candidate sample action feature set with a sample time sequence length of 1, find 1 similar first sample action feature in the codebook. If its similarity is greater than the threshold, the number of hits is 1.

[0165] The total number of hits is 1 + 2 + 3 + 7 + 14 + 30 = 57. Since the total number of features to be compared under all sample time lengths is 63, the hit rate is 57 / 63.

[0166] In this embodiment, the first setting condition can be set as needed and is not limited in this application. For example, the first setting condition may include, but is not limited to, any of the following: both the first loss and the second loss converge; the number of training iterations is greater than a set training iteration threshold.

[0167] Step S26: If the first loss and the second loss satisfy the first set condition, the training step ends.

[0168] In this embodiment, if the first loss and the second loss do not meet the first set condition, the parameters and codebook of the encoding module and the generation module are adjusted, and the step of obtaining the first sample action sequence continues. This adaptive parameter adjustment mechanism enables the model to continuously optimize its parameters and codebook based on feedback information during training, thereby improving the performance of the generative model.

[0169] During training, by processing action feature sets with different sample time lengths, multi-level information about the action sequence is captured. For example, feature sets with different time lengths (16, 8, 4, 2, 1, etc.) are obtained by progressively downsampling from a feature set with a sample time length of 32. This process is similar to multi-scale analysis, where different lengths correspond to action features of different granularities. Coarse-grained features can grasp the overall trend and structure of the action, while fine-grained features focus on the detailed changes in the action. Through this hierarchical representation, the model can gain a more comprehensive and in-depth understanding of human action sequences.

[0170] Simultaneously, feature decoupling is achieved through vector quantization to enhance generalization. The process of calculating the similarity between the action feature set of the first candidate samples and the action features of samples in the codebook, and selecting the most similar features from the codebook to form the action feature sets of the first samples with different temporal lengths, essentially involves vector quantization of the action features. This quantization method maps continuous action features to a discrete codebook space, achieving feature decoupling. After feature decoupling, the generative model can more clearly identify and learn the relationships between different action features, thereby enhancing the generalization ability of the generative model. This allows the generative model to learn action primitives from limited data and generate unseen action combinations through cross-scale recombination. For example, if basic action primitives such as walking and running are learned during training, when encountering new and unseen action combination requirements, the generative model can use these learned primitives to recombine and generate action sequences that meet the requirements.

[0171] Minimizing the first loss ensures that the predicted action sequences generated by the generative model closely approximate the real sample action sequences, guaranteeing that the generated actions are feasible and physically plausible. Minimizing the second loss ensures that the codebook accurately and comprehensively represents various candidate actions, improving the accuracy and detail of action generation. The dual loss function guides the generative model to optimize from multiple dimensions during training, enhancing its training performance.

[0172] Furthermore, because the first sample action sequence encompasses diverse action types, speeds, amplitudes, and scenarios, the trained generative model can adapt to different application scenarios and generate action sequences that meet the requirements of each scenario. For example, in the field of game development, this generative model can accurately generate natural, smooth, and expressive character actions based on the game character's design and plot requirements. Whether it's the unique actions of a magician casting spells in a fantasy game or the intense actions of soldiers in battle in a war game, it can be easily achieved, adding a realistic experience and rich fun to the game.

[0173] In terms of virtual reality experience, when users interact with the virtual environment, the generated model can generate interactive actions in real time that match the user's operation and the virtual scene, making the user feel as if they are truly in the virtual world and have a highly immersive experience. For example, in a virtual tourism scenario, the model can generate corresponding body postures and action feedback based on the user's walking, turning and other actions.

[0174] In the field of animation production, it can endow characters with vivid and logical animated movements, greatly improving the efficiency and quality of animation production. Whether creating exaggerated movements for characters in cartoon animation or delicate movements for characters in realistic animation, this model can provide precise motion sequence support, helping creators to create more attractive animated works.

[0175] Furthermore, this generative model has wide and important applications in many fields such as medical rehabilitation and human-computer interaction. In medical rehabilitation scenarios, for patients with movement disorders, the model can combine the patient's physical condition and rehabilitation goals to generate personalized rehabilitation movement simulations, helping patients to conduct scientific and effective rehabilitation training and improve rehabilitation outcomes. In smart home environments, it can achieve natural and smooth human-computer interaction, such as controlling the switching and adjustment of home appliances based on the user's limb movement commands, promoting natural collaboration and deep integration between humans and machines in a smart society.

[0176] In another embodiment of this application, a generative model is provided for embodiment 6 of this application. In this embodiment, it is mainly an implementation of step S23 in embodiment 5 above, and may include, but is not limited to, the following steps:

[0177] Step S31: Select action features from the codebook that match the first candidate sample action feature set with the first sample time sequence length to obtain the first sample action feature set with the first sample time sequence length.

[0178] In this embodiment, the similarity between each first candidate sample action feature in the first candidate sample action feature set with the first sample time sequence length and each action feature in the codebook can be calculated. There are various methods for calculating similarity, such as the cosine similarity algorithm and the Euclidean distance algorithm.

[0179] Taking the cosine similarity algorithm as an example, for each action feature of the first candidate sample and an action feature in the codebook, the cosine value of the angle between the two features is calculated to measure their similarity. The value of cosine similarity ranges from -1 to 1. The closer the value is to 1, the more similar the two features are in the direction, that is, the closer the corresponding action features are in semantics and action performance.

[0180] Taking the Euclidean distance algorithm as an example, the Euclidean distance between the action features of the first candidate sample and the action features in the codebook is calculated. The smaller the distance, the more similar the two action features are.

[0181] After calculating the similarity, for each first candidate sample action feature, select the action feature with the highest similarity (under the cosine similarity algorithm) or the shortest distance (under the Euclidean distance algorithm) from the codebook, and combine these action features to obtain the first sample action feature set with the first sample time sequence length.

[0182] Step S32: Determine the residual between the first candidate sample action feature set and the first sample action feature set to obtain a new first candidate sample action feature set with the first sample time sequence length.

[0183] In this embodiment, the residual can be represented as the difference between the action features of the first candidate sample and the matching action features in the codebook.

[0184] The action features in the codebook are learned from a large amount of real action data and have a certain degree of representativeness and universality. However, the action features of actual first candidate samples may contain some details or unique information that cannot be fully covered by the codebook action features. Residuals can capture these differences and use them as a new set of first candidate sample action features. This helps to further mine the detailed features of the action sequence in subsequent processing, improving the accuracy and detail representation of the generated actions.

[0185] For example, suppose the time series length of the first sample is 32, and the action feature set of the first candidate sample contains 32 action features, namely A1, A2, ..., A32. From the codebook, select the action features most similar to each Ai, namely B1, B2, ..., B32. For A1 and B1, calculate the residual R1 = A1 - B1; for A2 and B2, calculate the residual R2 = A2 - B2; and so on, calculating R1, R2, ..., R32. Combining these residuals R1, R2, ..., R32 yields a new action feature set of the first candidate sample with a time series length of 32.

[0186] Step S33: Process the new first candidate sample action feature set with the first sample time series length to obtain the first candidate sample action feature set with the second sample time series length; the second sample time series length is less than the first sample time series length.

[0187] In this embodiment, the action features of two consecutive adjacent time points in the new first candidate sample action feature set can be proportionally merged into a new candidate action feature. This processing method can ensure a smooth transition of candidate sample action features in the time dimension, avoid feature abrupt changes, and thus improve the stability and reliability of the feature set.

[0188] A specific merging calculation method can include: New candidate action feature = 0.5 × the action feature of the first candidate sample at the i-th consecutive time point + 0.5 × the action feature of the first candidate sample at the (i+1)-th consecutive time point. This proportionally weighted average calculation method can effectively fuse feature information from adjacent time points, generating more representative new candidate action features.

[0189] For example, if the initial time series length of the first sample is 32, after the downsampling process described above, the time series length of the second sample can be 16. In this case, the action feature set of the first candidate sample with the second sample time series length will contain 16 action features of the first candidate sample after merging. Each feature incorporates feature information from adjacent time points, thereby reducing the time series length while retaining key action feature information.

[0190] The specific merging calculation method is not limited to proportional weighted average. For example, a merging method based on time weight decay can be used. It is assumed that the closer a time point is to the current moment, the greater its influence on the new candidate action features; therefore, the weight is set to be inversely proportional to time. This calculation method based on time weight decay can better highlight the contribution of recent time point features to the new candidate action features, effectively integrate feature information from adjacent time points, and generate more representative new candidate action features.

[0191] Step S34: Take the first candidate sample action feature set of the second sample time sequence length as the new first candidate sample action feature set of the first sample time sequence length, and continue to execute the step of selecting action features that match the first candidate sample action feature set of the first sample time sequence length from the codebook, until the first sample action feature set of multiple sample time sequence lengths is obtained.

[0192] For example, if the first sample time sequence length is 32, and after downsampling the second sample time sequence length becomes 16, then the first candidate sample feature set with a time sequence length of 16 can be used as the new first candidate sample action feature set with the first sample time sequence length. Continue to execute the step of selecting action features that match the first candidate sample action feature set with the first sample time sequence length from the codebook, and gradually obtain the first sample action feature set with different sample time sequence lengths.

[0193] In this embodiment, by selecting action features from the codebook that match the first candidate sample action feature set with the first sample time sequence length, the first sample action feature set with the first sample time sequence length is obtained. The residual between the first candidate sample action feature set and the first sample action feature set is determined and used as the new first candidate sample action feature set. This can accurately capture the difference and help to further mine the detailed features of the action sequence in subsequent processing, making the generated action richer and more realistic in detail.

[0194] Furthermore, the new first candidate sample action feature set is downsampled, converting the first sample time series length to the second sample time series length (the second sample time series length is shorter than the first sample time series length). By proportionally merging the first candidate sample action features at adjacent time points, using either a proportionally weighted average or a merging method based on time weight decay, a smooth transition of candidate sample action features in the time dimension is ensured, avoiding feature abrupt changes. This processing method guarantees the coherence and consistency of action features, whether in the generation of long-term action sequences or in real-time interactive scenarios.

[0195] After obtaining sample action feature sets with multiple sample time series lengths, the actions can be described comprehensively and in detail based on these action feature sets, thereby generating higher quality and more accurate action sequences. This makes the generated actions more realistic and natural, conforming to the performance rules of actual actions.

[0196] In another embodiment of this application, a generative model is provided for embodiment 7 of this application. In this embodiment, it is mainly an implementation of step S22 in embodiment 5 above, and may include, but is not limited to, the following steps:

[0197] Step S41: Extract multiple sample sub-action sequences from the first sample action sequence.

[0198] In this embodiment, the size and step size of the sliding window can be set according to the temporal length and action complexity of the first sample action sequence. For example, setting the window size to 64 frames and the step size to 32 frames can increase the diversity of sub-sequences while ensuring the integrity of the sub-sequences.

[0199] Starting from the beginning of the first sample action sequence, the sliding window gradually slides backward according to the set step size. After each slide, the action sequence within the window is extracted as a sample sub-action sequence. For example, for the first sample action sequence with a total length of 128 frames, using a window of size 64 frames and a step size of 32 frames, three sample sub-action sequences can be extracted (starting frames 1, 33, and 65 respectively), representing different stages of the walking action.

[0200] By adjusting the window size and step size, it is ensured that the captured sample sub-action sequences have both overlapping parts in time to maintain action continuity and non-overlapping parts to increase diversity. For example, a smaller step size can increase the correlation between sub-sequences, which is suitable for capturing gradual changes in actions; a larger step size can increase the differences between sub-sequences, which is suitable for capturing abrupt changes in actions.

[0201] Step S42: Process each of the sample sub-action sequences based on the encoding module to obtain the first candidate sample sub-action feature set with the first sample time sequence length.

[0202] In this embodiment, step S42 can be understood by referring to the method of processing the first sample action sequence based on the encoding module, and will not be repeated here.

[0203] In this embodiment, the process of processing the first candidate sample sub-action feature set based on the first sample time sequence length can be understood by referring to the relevant description of steps S23-S25 in embodiment 5, and will not be repeated here.

[0204] In this embodiment, by decomposing the first sample action sequence into multiple sample sub-action sequences and extracting features from each sub-sequence, the local features and details of the action can be captured more accurately. Compared to extracting features from the entire long sequence, this method avoids information loss and feature blurring caused by excessive sequence length, thereby improving the accuracy of feature extraction.

[0205] Furthermore, the extraction of sample sub-action sequences and the feature extraction process enable the model to learn more diverse action features. Different sample sub-action sequences may correspond to different stages or different manifestations of an action. By learning these diverse features, the model can better adapt to various action generation needs and enhance its generalization ability.

[0206] Furthermore, decomposing the first sample action sequence into sample sub-action sequences makes subsequent processing and analysis more convenient and efficient. For example, when matching and selecting action features, the feature sets of each sample sub-action sequence can be processed independently, and then the results can be integrated. This reduces the complexity of the problem, improves processing efficiency, and also facilitates the debugging and optimization of the generated action sequences.

[0207] In another embodiment of this application, a generative model is provided for embodiment 8 of this application. In this embodiment, it is mainly an implementation of the prediction module in embodiment 5 above. The prediction module can be trained, but is not limited to, in the following ways:

[0208] Step S51: Obtain the second sample action sequence and the corresponding text description.

[0209] In this embodiment, the second sample motion sequence can be obtained from captured real human motion data, publicly available motion databases, or custom motion datasets. This data should cover a variety of motion types (such as walking, running, jumping, dancing, etc.), speeds, amplitudes, and scenarios.

[0210] At the same time, each second sample action sequence is labeled with a corresponding text description, which can accurately and in detail describe the second sample action sequence.

[0211] Step S52: Process the second sample action sequence based on the encoding module to obtain the second candidate sample action feature set with the first sample time sequence length.

[0212] In this embodiment, step S52 can be understood by referring to the processing method based on the encoding module in embodiment 5, and will not be repeated here.

[0213] Step S53: Based on the codebook and the second candidate sample action feature set with the first sample time sequence length, obtain a second sample action feature set with multiple sample time sequence lengths.

[0214] In this embodiment, step S53 can be understood by referring to the relevant description of step S23 in the previous embodiments, and will not be repeated here.

[0215] Step S54: Concatenate the text descriptions corresponding to each second sample action feature set and the second sample action sequence to obtain the third sample action feature set.

[0216] In this embodiment, action features and text descriptions in each of the second sample action feature sets can be sequentially connected to form a longer feature set, namely the third sample action feature set.

[0217] This concatenation operation allows the third sample action feature set to simultaneously contain the spatiotemporal features of the action and the corresponding semantic information, providing the prediction module with more comprehensive input data and helping the prediction module learn the complex relationship between action features and text descriptions.

[0218] Step S55: Input the third sample action feature set into the prediction module to obtain the sample conditional probability determined by the prediction module; the sample conditional probability represents the degree of matching between the action feature corresponding to each discrete code value in the codebook and the third sample action feature set.

[0219] In this embodiment, the conditional probability of the sample can be immediately calculated by referring to the relevant introduction of conditional probability in Embodiment 3, which will not be repeated here.

[0220] Step S56: If the third loss does not meet the second set condition, keep the parameters of the encoding module and the codebook unchanged, adjust the parameters of the prediction module, and continue to execute the step of obtaining the second sample action sequence; the third loss represents the difference between the maximum probability in the sample conditional probability and the discrete code value in the codebook corresponding to the third sample action feature set.

[0221] The third loss can be calculated using the following formula: Cross-entropy loss

[0222]

[0223] in, The third loss is represented by C, which represents the total number of discrete code values ​​in the codebook (e.g., if the codebook contains 512 discrete code values, then C=512). For the One-Hot encoding of the real label, if the i-th discrete code value corresponds to the third sample action feature set (i.e., the action feature corresponding to this code value best matches a certain sample action feature in the third sample action feature set), then =1, the rest =0; This represents the highest probability among the sample conditional probabilities output by the prediction module.

[0224] During the optimization of the prediction module based on this loss, the AdaLN Transformer layer of the prediction module can dynamically adjust the normalization parameters of the layer according to the text description contained in the action feature set of the third sample. For example, when the text description is "a person is walking slowly", the AdaLN Transformer layer pays more attention to the action features corresponding to the discrete code values ​​related to "walking" and "low speed" in the codebook. When calculating the conditional probability of the samples, the probability allocation is tilted towards these code values ​​to ensure that the prediction result is highly matched with the text semantics and avoid the bias of "the text describes walking but predicts running code values".

[0225] Meanwhile, to further improve the quality of actions generated after the prediction module is trained, Classifier-free Guidance (CFG) technology can be incorporated into the training phase. Specifically, by randomly discarding the text description conditions in 10% of the third sample action feature set (i.e., replacing the text features of these 10% of samples with empty unconditional embedding vectors), the prediction module can learn the probabilistic prediction logic in both "text-guided" and "text-free" scenarios: when the input sample contains text description, the model learns "accurate prediction under semantic constraints"; when the input sample does not contain text description, the model learns "prediction based on the inherent patterns of the action features themselves".

[0226] Simultaneous training of these two capabilities allows the prediction module to output both conditional and unconditional log probabilities when applying CFG technology for inference.

[0227] Subsequently, the generative model can fuse the conditional log probability and the unconditional log probability according to the preset CFG guidance strength (e.g., scale parameter 1.5) to obtain the conditional probability.

[0228] The parameters can be adjusted using the backpropagation algorithm. The gradient of each parameter in the prediction module is calculated based on the third loss, and then the parameters are updated according to a certain learning rate. This allows the prediction module to gradually optimize its prediction ability and improve the accuracy of the sample conditional probability.

[0229] Afterwards, a second sample action sequence can be obtained, and the above training process can be repeated to continuously iterate and optimize the prediction module.

[0230] The second setting condition can be set as needed and is not limited in this application. For example, the second setting condition may include, but is not limited to, any of the following: the third loss converges; the number of training iterations exceeds a set training iteration threshold.

[0231] Step S57: If the third loss satisfies the second set condition, the training step ends.

[0232] In this embodiment, by concatenating action features with text descriptions, the prediction module can simultaneously capture the physical characteristics of the action (such as joint movement trajectory) and semantic information (such as kicking force).

[0233] By dynamically adjusting the parameters of the prediction module using the third loss, it is ensured that the prediction module can correctly find the code value with the highest semantic matching degree from the codebook based on the input action features.

[0234] Furthermore, during training, the parameters of the encoding module and the codebook remain unchanged, and only the parameters of the prediction module are optimized. This decoupled training method can effectively avoid convergence difficulties that may occur when multiple modules are trained together, such as parameter conflicts, gradient vanishing or exploding, and ensure the stable improvement of the prediction module's performance.

[0235] The generation apparatus provided in this application will be described below. The generation apparatus described below can be referred to in correspondence with the generation method described above.

[0236] The generating apparatus may include:

[0237] The acquisition unit is used to obtain action description information input by the user.

[0238] The determining unit is used to determine text features based on the action description information.

[0239] The processing unit is used to process the text features and gradually obtain action feature sets of various temporal lengths based on a pre-trained codebook; the action feature sets of various temporal lengths contain multiple action features with different temporal lengths; the codebook is used to store multiple discrete code values ​​and the mapping relationship between multiple action features.

[0240] The generation unit is used to process the action feature sets of the various time lengths to obtain the target action sequence.

[0241] The processing unit can specifically be used for:

[0242] The text features are used as a candidate action feature set for a first time length. The candidate action feature set for the first time length is processed, and action features with a number corresponding to the first time length are selected from the pre-trained codebook as the action feature set for the first time length.

[0243] The action feature set with the first time sequence length is processed to obtain a candidate action feature set with a second time sequence length; the second time sequence length is greater than the first time sequence length.

[0244] The candidate action feature set of the second time series length is used as the new candidate action feature set of the first time series length, and the step of processing the candidate action feature set of the first time series length is continued until action feature sets of multiple time series lengths are obtained.

[0245] In this embodiment, the processing unit processes the candidate action feature set of the first time series length, selecting a number of action features corresponding to the first time series length from a pre-trained codebook as the action feature set of the first time series length. Specifically, this may include:

[0246] The candidate action feature set of the first time series length is input into the prediction module, and the prediction module determines the conditional probability corresponding to each candidate action feature in the candidate action feature set of the first time series length; the conditional probability represents the degree of matching between the action feature corresponding to each discrete code value in the pre-trained codebook and the candidate action feature;

[0247] Select the highest probability from the conditional probabilities corresponding to each of the candidate action features;

[0248] Select the action feature corresponding to each of the maximum probabilities from the codebook and determine it as the action feature set of the first temporal length.

[0249] In another embodiment of this application, an electronic device is provided.

[0250] Reference Figure 5The electronic device includes a memory 100 and a processor 200. The memory 100 and the processor 200 communicate via a bus.

[0251] The memory 100 is used to store computer programs.

[0252] Processor 200 is configured to execute the computer program to enable the electronic device to perform the following method steps:

[0253] Obtain description information of the user's input action;

[0254] Based on the action description information, text features are determined;

[0255] The text features are processed to gradually obtain action feature sets of various temporal lengths based on a pre-trained codebook; each temporal length action feature set contains multiple action features with different temporal lengths; the codebook is used to store multiple discrete code values ​​and the mapping relationship between multiple action features.

[0256] The action feature sets of the various time lengths are processed to obtain the target action sequence.

[0257] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0258] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0259] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0260] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A generation method comprising: ‌obtaining action description information input by a user; determining text features based on the action description information; processing the text features to obtain a plurality of action feature sets of different time lengths step by step according to a pre-trained codebook; the action feature sets of different time lengths comprise a plurality of action features of different time sequences; the codebook is used to store mapping relationships between a plurality of discrete code values and a plurality of action features; processing the plurality of action feature sets of different time lengths to obtain a target action sequence.

2. The generation method of claim 1, wherein the processing the text features to obtain a plurality of action feature sets of different time lengths step by step according to a pre-trained codebook comprises: taking the text features as a candidate action feature set of a first time length, processing the candidate action feature set of the first time length, and selecting, from the pre-trained codebook, action features corresponding in number to the first time length as an action feature set of the first time length; processing the action feature set of the first time length to obtain a candidate action feature set of a second time length; the second time length is greater than the first time length; taking the candidate action feature set of the second time length as a new candidate action feature set of the first time length, and continuing to perform the processing of the candidate action feature set of the first time length until a plurality of action feature sets of different time lengths are obtained.

3. The generation method of claim 2, wherein the processing the candidate action feature set of the first time length to select, from the pre-trained codebook, action features corresponding in number to the first time length as an action feature set of the first time length comprises: inputting the candidate action feature set of the first time length into a prediction module to determine, by the prediction module, a conditional probability corresponding to each candidate action feature in the candidate action feature set of the first time length; the conditional probability represents a matching degree between an action feature corresponding to each discrete code value in the pre-trained codebook and the candidate action feature; selecting a maximum probability from the conditional probability corresponding to each candidate action feature; selecting, from the codebook, an action feature corresponding to each maximum probability to determine as an action feature set of the first time length.

4. A generation model comprising: a codebook used to store mapping relationships between a plurality of discrete code values and a plurality of action features; a prediction module used to process text features to obtain a plurality of action feature sets of different time lengths step by step according to the codebook; the action feature sets of different time lengths comprise a plurality of action features of different time sequences; the text features are from action description information input by a user; a generation module used to process the plurality of action feature sets of different time lengths to obtain a target action sequence.

5. The generative model of claim 4, further comprising: an encoding module; the encoding module, the codebook, and the generation module are trained in the following manner: obtaining a first sample action sequence; obtaining a first candidate sample action feature set of a first sample time length based on the encoding module processing the first sample action sequence; obtaining a first candidate sample action feature set of a plurality of sample time lengths and a first sample action feature set based on the codebook and the first candidate sample action feature set of the first sample time length; inputting the first sample action feature set of each sample time length into the generation module, and generating a predicted action sequence after processing by the generation module; if the first loss and the second loss do not satisfy the first set condition, adjusting the parameters of the encoding module and the generation module and the codebook, and continuing to perform the step of obtaining the first sample action sequence; the first loss represents the difference between the predicted action sequence and the first sample action sequence; the second loss represents the semantic consistency between the first candidate sample action feature set and the first sample action feature set; if the first loss and the second loss satisfy the first set condition, ending the training step.

6. The generation model of claim 5, wherein the obtaining a first candidate sample action feature set of a plurality of sample time lengths and a first sample action feature set based on the codebook and the first candidate sample action feature set of the first sample time length comprises: selecting an action feature matching the first candidate sample action feature set of the first sample time length from the codebook to obtain the first sample action feature set of the first sample time length; determining a residual error between the first candidate sample action feature set and the first sample action feature set to obtain a new first candidate sample action feature set of the first sample time length; processing the new first candidate sample action feature set of the first sample time length to obtain a first candidate sample action feature set of a second sample time length; the second sample time length is less than the first sample time length; taking the first candidate sample action feature set of the second sample time length as a new first candidate sample action feature set of the first sample time length, and continuing to perform the step of selecting an action feature matching the first candidate sample action feature set of the first sample time length from the codebook until a first sample action feature set of a plurality of sample time lengths is obtained.

7. The generation model of claim 5, wherein the obtaining a first candidate sample action feature set of a first sample time length based on the encoding module processing the first sample action sequence comprises: extracting a plurality of sample sub-action sequences from the first sample action sequence; processing each sample sub-action sequence based on the encoding module to obtain a first candidate sample sub-action feature set of a first sample time length.

8. The generation model of claim 5, wherein the prediction module is trained by: obtaining a second sample action sequence and a text description corresponding to the second sample action sequence; processing the second sample action sequence based on the encoding module to obtain a second candidate sample action feature set of a first sample time sequence length; obtaining a second sample action feature set of multiple sample time sequence lengths based on the codebook and the second candidate sample action feature set of the first sample time sequence length; splicing the text description corresponding to each of the second sample action feature sets and the second sample action sequence to obtain a third sample action feature set; inputting the third sample action feature set into a prediction module to obtain a sample conditional probability determined by the prediction module; the sample conditional probability represents a matching degree between action features corresponding to each discrete code value in the codebook and the third sample action feature set; if the third loss does not satisfy the second set condition, keeping the parameters of the encoding module and the codebook unchanged, adjusting the parameters of the prediction module, and continuing to perform the step of obtaining the second sample action sequence; the third loss represents a difference between the maximum probability in the sample conditional probability and a discrete code value in the codebook corresponding to the third sample action feature set; if the third loss satisfies the second set condition, ending the training step.

9. A generation apparatus, comprising: an obtaining unit configured to obtain action description information input by a user; a determining unit configured to determine a text feature based on the action description information; a processing unit configured to process the text feature to gradually obtain action feature sets of multiple time sequence lengths according to a pre-trained codebook; the action feature sets of the time sequence lengths include multiple action features that are different in time sequence; the codebook is used to store a mapping relationship between multiple discrete code values and multiple action features; a generation unit configured to process the action feature sets of the multiple time sequence lengths to obtain a target action sequence.

10. An electronic device, comprising: a memory configured to store a computer program; a processor configured to execute the computer program to enable the electronic device to implement the following method steps: obtaining action description information input by a user; determining a text feature based on the action description information; processing the text feature to gradually obtain action feature sets of multiple time sequence lengths according to a pre-trained codebook; the action feature sets of the time sequence lengths include multiple action features that are different in time sequence; the codebook is used to store a mapping relationship between multiple discrete code values and multiple action features; processing the action feature sets of the multiple time sequence lengths to obtain a target action sequence.

Images