Method and device for generating action sequence, method and device for training related model

Abstract

The present disclosure relates to a method and apparatus for generating an action sequence, a method and apparatus for training a model for generating an action sequence, an electronic device, and a computer storable medium, and relates to the field of computer technology. The method for generating an action sequence includes: obtaining a reference sound sequence; based on the reference sound sequence, using a model for generating an action sequence, generating a predicted action sequence corresponding to the reference sound sequence, the model for generating an action sequence includes an associated first sub-model and a second sub-model, the first sub-model's attention to local short-time features is greater than its attention to global long-time features, and the second sub-model's attention to global long-time features is greater than its attention to local short-time features. According to the present disclosure, the accuracy of the generated action sequence can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method and device for generating an action sequence, a method and device for training a model for generating an action sequence, an electronic device, and a computer storable medium. Background Art

[0002] The multimodal human motion driving and generation method refers to a method of generating a human motion sequence corresponding to a sound sequence given a sound sequence.

[0003] In the related art, a sound sequence is input into a trained Transformer model to obtain an action sequence corresponding to the sound sequence. Summary of the invention

[0004] In the related art, the Transformer model will continuously accumulate local errors during feature processing, resulting in poor accuracy of the generated action sequence.

[0005] In view of the above technical problems, the present disclosure proposes a solution to improve the accuracy of the generated action sequence.

[0006] According to a first aspect of the present disclosure, a method for generating an action sequence is provided, comprising: obtaining a reference sound sequence; based on the reference sound sequence, generating a predicted action sequence corresponding to the reference sound sequence using a model for generating an action sequence, the model for generating an action sequence comprising an associated first sub-model and a second sub-model, the first sub-model paying more attention to local short-time features than to global long-time features, and the second sub-model paying more attention to global long-time features than to local short-time features.

[0007] According to a second aspect of the present disclosure, a method for training a model for generating an action sequence is provided, comprising: obtaining a reference sound sequence and its corresponding actual action sequence, the actual action sequence being a training label of the reference sound sequence; generating a predicted action sequence corresponding to the reference sound sequence based on the reference sound sequence using a model for generating an action sequence, the model for generating an action sequence comprising an associated first sub-model and a second sub-model, the first sub-model paying more attention to local short-time features than to global long-time features, and the second sub-model paying more attention to global long-time features than to local short-time features; and training the model for generating an action sequence using the predicted action sequence and the actual action sequence.

[0008] In some embodiments, based on the reference sound sequence, using a model for generating action sequences, generating a predicted action sequence corresponding to the reference sound sequence includes: based on the reference sound sequence, using a model for generating action sequences, generating an initial action sequence corresponding to the reference sound sequence; performing a first adjustment and a second adjustment on the initial action sequence to obtain the predicted action sequence, the first adjustment being used to adjust the action executor itself, and the second adjustment being used to adjust the relationship between the action executor and its surrounding environment.

[0009] In some embodiments, performing a second adjustment on the initial action sequence includes: adjusting at least one of a relative position and a relative speed between one or more key points of the action execution body and an object in its surrounding environment.

[0010] In some embodiments, performing a first adjustment on the initial action sequence includes: adjusting a relative rotation speed between a plurality of key points of the action execution body.

[0011] In some embodiments, performing a first adjustment on the initial action sequence further includes: adjusting at least one of a relative position and a relative rotation angle between a plurality of key points of the action execution body.

[0012] In some embodiments, in the process of generating the predicted action sequence, the first sub-model maps the features related to the reference sound sequence to a local area of ​​the input space of the first sub-model, and the second sub-model maps the features related to the reference sound sequence to the entire area of ​​the input space of the second sub-model.

[0013] In some embodiments, based on the reference sound sequence, using a model for generating action sequences, generating a predicted action sequence corresponding to the reference sound sequence includes: obtaining a reference action sequence related to a partial sound sequence of the reference sound sequence; based on the reference sound sequence and the reference action sequence, using the model for generating action sequences, generating the predicted action sequence.

[0014] In some embodiments, based on the reference sound sequence and the reference action sequence, using the model for generating action sequences, generating the predicted action sequence includes: using the first sub-model and the second sub-model to encode the reference sound sequence and the reference action sequence to obtain target semantic coding features; and generating the predicted action sequence based on the target semantic coding features.

[0015] In some embodiments, the first sub-model and the second sub-model are connected in series or in parallel.

[0016] In some embodiments, when the first sub-model and the second sub-model are connected in series, the reference sound sequence and the reference action sequence are encoded using the first sub-model and the second sub-model to obtain target semantic coding features, including: using the first sub-model to encode the reference sound sequence and the reference action sequence to obtain initial semantic coding features; using the second sub-model to encode the initial semantic coding features to obtain the target semantic coding features.

[0017] In some embodiments, when the first sub-model and the second sub-model are connected in series, the reference sound sequence and the reference action sequence are encoded using the first sub-model and the second sub-model to obtain target semantic coding features, including: using the second sub-model to encode the reference sound sequence and the reference action sequence to obtain initial semantic coding features; using the first sub-model to encode the initial semantic coding features to obtain the target semantic coding features.

[0018] In some embodiments, when the first sub-model and the second sub-model are connected in parallel, the reference sound sequence and the reference action sequence are encoded using the first sub-model and the second sub-model to obtain target semantic coding features, including: using the first sub-model to encode the reference sound sequence and the reference action sequence to obtain a first initial semantic coding feature; using the second sub-model to encode the reference sound sequence and the reference action sequence to obtain a second initial semantic coding feature; and fusing the first initial semantic coding feature and the second initial semantic coding feature to obtain the target semantic coding feature.

[0019] In some embodiments, the reference sound sequence and the reference action sequence are encoded using the first sub-model and the second sub-model to obtain target semantic coding features, including: determining composite features related to sound and action based on the reference sound sequence and the reference action sequence; and encoding the composite features using the first sub-model and the second sub-model to obtain the target semantic coding features.

[0020] In some embodiments, determining the composite features related to sound and action based on the reference sound sequence and the reference action sequence includes: performing feature extraction on the reference sound sequence to obtain audio features; performing feature extraction on the reference action sequence to obtain action features; and determining the composite features based on the audio features and the action features.

[0021] In some embodiments, the audio feature includes at least one of a Mel-frequency cepstral coefficient feature and an onset intensity feature, and the action feature includes a multi-dimensional rotation matrix associated with multiple key points of an action executor.

[0022] In some embodiments, the model for generating the action sequence also includes a third sub-model, and generating the predicted action sequence according to the target semantic coding features includes: using the third sub-model to decode the target semantic coding features to obtain the predicted action sequence.

[0023] In some embodiments, the third sub-model is a long short-term memory (LSTM) model, and during the training process, the LSTM model uses the output of the previous time step of each time step as the input of the time step.

[0024] In some embodiments, the first sub-model includes one or more time-domain convolutional neural network models.

[0025] In some embodiments, the second sub-model includes an encoding structure of one or more Transformer models.

[0026] According to a third aspect of the present disclosure, there is provided a device for generating an action sequence, comprising: an acquisition module configured to acquire a reference sound sequence; a generation module configured to generate a predicted action sequence corresponding to the reference sound sequence based on the reference sound sequence and using a model for generating action sequences, wherein the model for generating action sequences comprises an associated first sub-model and a second sub-model, wherein the first sub-model pays more attention to local short-time features than to global long-time features, and the second sub-model pays more attention to global long-time features than to local short-time features.

[0027] According to a fourth aspect of the present disclosure, there is provided a device for training a model for generating an action sequence, comprising: an acquisition module, configured to acquire a reference sound sequence and its corresponding actual action sequence, wherein the actual action sequence is a training label of the reference sound sequence; a generation module, configured to generate a predicted action sequence corresponding to the reference sound sequence based on the reference sound sequence and using a model for generating an action sequence, wherein the model for generating an action sequence comprises an associated first sub-model and a second sub-model, wherein the first sub-model pays more attention to local short-time features than to global long-time features, and the second sub-model pays more attention to global long-time features than to local short-time features; and a training module, configured to train the model for generating an action sequence by using the predicted action sequence and the actual action sequence.

[0028] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the method described in any one of the above embodiments based on instructions stored in the memory.

[0029] According to a sixth aspect of the present disclosure, a computer storable medium is provided, on which computer program instructions are stored, and when the instructions are executed by a processor, the method described in any of the above embodiments is implemented.

[0030] In the above embodiment, the accuracy of the generated action sequence can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0032] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0033] Figure 1 is a flowchart illustrating a method of generating an action sequence according to some embodiments of the present disclosure;

[0034] Figure 2 is a flowchart illustrating a method for training a model for generating action sequences according to some embodiments of the present disclosure;

[0035] Figure 3 is a flowchart illustrating generating a predicted action sequence corresponding to a reference sound sequence according to some embodiments of the present disclosure;

[0036] Figure 4 is a flowchart illustrating generating a predicted action sequence corresponding to a reference sound sequence according to some other embodiments of the present disclosure;

[0037] Figure 5 is a flowchart illustrating generation of a predicted action sequence according to some embodiments of the present disclosure;

[0038] Figure 6 is a block diagram illustrating an apparatus for generating an action sequence according to some embodiments of the present disclosure;

[0039] Figure 7 is a block diagram illustrating an apparatus for training a model for generating action sequences according to some embodiments of the present disclosure;

[0040] Figure 8 is a block diagram illustrating an electronic device according to some embodiments of the present disclosure;

[0041] Fig. 9 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0042] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0043] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0044] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0045] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0046] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0047] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0048] The method for generating an action sequence and the method for training a model for generating an action sequence in some embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0049] Figure 1 is a flowchart illustrating a method of generating an action sequence according to some embodiments of the present disclosure.

[0050] like Figure 1 As shown, the method for generating an action sequence includes: step S110, obtaining a reference sound sequence; and step S120, generating a predicted action sequence corresponding to the reference sound sequence.

[0051] In step S110, a reference sound sequence is obtained. In this case, the reference sound sequence is data to be predicted. In some embodiments, the sound sequence includes music.

[0052] In step S120, a predicted action sequence corresponding to the reference sound sequence is generated using a model for generating an action sequence according to a reference sound sequence. The model for generating an action sequence includes an associated first sub-model and a second sub-model. The first sub-model pays more attention to local short-time features than to global long-time features. The second sub-model pays more attention to global long-time features than to local short-time features. For example, in the process of generating the predicted action sequence, the first sub-model maps the features related to the reference sound sequence to a local area of ​​the input space of the first sub-model, so the first sub-model pays more attention to local short-time features than to global long-time features. In the process of generating the predicted action sequence, the second sub-model maps the features related to the reference sound sequence to the entire area of ​​the input space of the second sub-model, so the second sub-model pays more attention to global long-time features than to short-time features. In some embodiments, taking the sound sequence as music as an example, the action sequence includes dance.

[0053] In the above embodiment, by fusing the first sub-model and the second sub-model that are good at processing different types of features into the model for generating action sequences, the model for generating action sequences can pay more attention to and process local short-time features and global long-time features in a more balanced manner, taking into account local dependencies and long-time dependencies, thereby reducing the accumulation of local errors of the model in long-time actions and improving the accuracy of the generated action sequences.

[0054] Figure 2 is a flowchart illustrating a method for training a model for generating action sequences according to some embodiments of the present disclosure.

[0055] like Figure 2 As shown, the method for training a model for generating an action sequence includes: step S210, obtaining a reference sound sequence and its corresponding actual action sequence; step S220, generating a predicted action sequence corresponding to the reference sound sequence; and step S230, using the predicted action sequence and the actual action sequence to train a model for generating an action sequence.

[0056] In step S210, a reference sound sequence and its corresponding actual action sequence are obtained. The actual action sequence is a training label of the reference sound sequence. In this case, the reference sound sequence is training data.

[0057] In step S220, a predicted action sequence corresponding to the reference sound sequence is generated based on the reference sound sequence using a model for generating an action sequence. The model for generating an action sequence includes an associated first sub-model and a second sub-model. The first sub-model pays more attention to local short-time features than to global long-time features. The second sub-model pays more attention to global long-time features than to local short-time features. For example, in the process of generating the predicted action sequence, the first sub-model maps the features related to the reference sound sequence to a local area of ​​the input space of the first sub-model, so the first sub-model pays more attention to local short-time features than to global long-time features. In the process of generating the predicted action sequence, the second sub-model maps the features related to the reference sound sequence to the entire area of ​​the input space of the second sub-model, so the second sub-model pays more attention to global long-time features than to short-time features.

[0058] In step S230, the model for generating the action sequence is trained using the predicted action sequence and the actual action sequence. In some embodiments, the loss value of the loss function is determined based on the predicted action sequence and the actual action sequence, and the model for generating the action sequence is trained using the loss value. For example, the model for generating the action sequence is trained with the goal of minimizing the loss value. The loss function can use some common loss functions in the art such as the cross entropy function.

[0059] In the above embodiment, by fusing the first sub-model and the second sub-model that are good at processing different types of features into the model for generating action sequences, during the model training process, the model for generating action sequences can be trained to pay more attention to and process local short-time features and global long-time features in a more balanced manner, taking into account local dependencies and long-time dependencies, thereby reducing the accumulation of local errors in the model on long-time actions and improving the accuracy of the generated action sequences.

[0060] In some embodiments, the first sub-model may include one or more temporal convolutional neural network models (Temporal Convolutional Network, TCN). For example, when the first sub-model includes multiple temporal convolutional neural network models, the multiple temporal convolutional neural network models may be connected in series. In some embodiments, each temporal convolutional neural network model includes causal convolution, dilated convolution, and residual block structure. For example, more convolutional layers can be stacked in each temporal convolutional neural network model, and a larger dilation rate can be used to flexibly change the receptive field of the network model, so that the model has better scalability and is more conducive to the optimization of local short-term action sequences. The number of convolutional layers and the size of the dilation rate can be used as training parameters and determined through the training process.

[0061] In some embodiments, the second sub-model may include an encoding structure of one or more Transformer models. In the case where the first sub-model includes multiple Transformer models, the multiple Transformer models may be connected in series.

[0062] In some embodiments, the Figure 3 The method shown implements the step of "generating a predicted action sequence corresponding to the reference sound sequence using a model for generating action sequences based on the reference sound sequence".

[0063] Figure 3 is a flowchart illustrating generating a predicted action sequence corresponding to a reference sound sequence according to some embodiments of the present disclosure.

[0064] like Figure 3 As shown, generating a predicted action sequence corresponding to a reference sound sequence includes steps S321 and S322.

[0065] In step S321, based on the reference sound sequence, an initial action sequence corresponding to the reference sound sequence is generated using a model for generating an action sequence.

[0066] In some embodiments, feature extraction can be performed on a reference sound sequence to obtain audio features. The audio features are input into a model for generating an action sequence to obtain an initial action sequence corresponding to the reference sound sequence. For example, the target semantic coding features can be obtained by sequentially performing encoding operations on the first sub-model and the second sub-model, and the initial action sequence can be determined based on the target semantic coding features. For another example, the first semantic coding features and the second semantic coding features can be obtained after respectively performing encoding operations on the first sub-model and the second sub-model, and the target semantic coding features can be obtained by fusing the first semantic coding features and the second semantic coding features. Then, the initial action sequence is determined based on the target semantic coding features.

[0067] In step S322, the initial action sequence is adjusted first and second to obtain a predicted action sequence. The first adjustment is used to adjust the action execution body itself. The second adjustment is used to adjust the relationship between the action execution body and its surrounding environment. By adjusting the action execution body itself and the relationship between the action execution body and its surrounding environment, the action sequence can be adjusted from a global and local perspective, respectively, so as to further improve the accuracy, authenticity and rationality of the generated action sequence.

[0068] In some embodiments, the initial action sequence may be adjusted first and second in sequence to obtain a predicted action sequence. In other embodiments, the initial action sequence may be adjusted second and first in sequence to obtain a predicted action sequence. In still other embodiments, the initial action sequence may be adjusted first and second at the same time to obtain a first initial action sequence and a second initial action sequence, respectively, and then the first initial action sequence and the second initial action sequence may be fused to obtain a predicted action sequence. For example, the first initial action sequence and the second initial action sequence are merged to obtain a predicted action sequence.

[0069] In some embodiments, the initial action sequence can be first adjusted by adjusting the relative rotation speed between multiple key points of the action execution body. Taking the action execution body as a human body as an example, the multiple key points of the action execution body are multiple joints of the human body. For example, the multiple joints include the shoulder joint, knee joint, wrist joint, elbow joint and ankle joint of the human body. By adjusting the relative rotation speed between the multiple key points, the accuracy and authenticity of the generated action sequence can be further improved.

[0070] In some embodiments, the initial action sequence may be first adjusted by adjusting at least one of the relative position and the relative rotation angle between the plurality of key points of the action execution body.

[0071] In some embodiments, the first adjustment of the initial action sequence can be achieved by presetting action constraints. For example, the action constraints include the value ranges of the relative rotation speed, relative position, and relative rotation angle of each key point. The first adjustment is achieved by detecting and adjusting the relative rotation speed, relative position, and relative rotation angle of each key point so that the action constraints are met.

[0072] In some embodiments, at least one of the relative position and relative speed between one or more key points of the action execution body and one or more objects in its surrounding environment can be adjusted. For example, the one or more objects in the surrounding environment can be static objects such as tables, chairs, floors, walls, etc., or can be moving vehicles or other objects in motion.

[0073] In some embodiments, the second adjustment of the initial action sequence can be achieved by presetting environmental constraints. For example, the environmental constraints include the value ranges of the relative speed and relative position of each key point of the action executor. The second adjustment is achieved by detecting and adjusting the relative speed and relative position of each key point so that the environmental constraints are satisfied. Taking the environmental constraints including the contact constraints between the foot and the floor as an example, the relative position relationship between the foot position and the floor is obtained by detecting the foot joints and the floor. When the foot position is below the floor, the foot position is adjusted so that the foot position is above the floor.

[0074] In some embodiments, it is also possible to Figure 4 The method shown implements the step of "generating a predicted action sequence corresponding to the reference sound sequence using a model for generating action sequences based on the reference sound sequence".

[0075] Figure 4 is a flowchart illustrating generating a predicted action sequence corresponding to a reference sound sequence according to some other embodiments of the present disclosure.

[0076] like Figure 4 As shown, generating a predicted action sequence corresponding to a reference sound sequence includes: step S421, obtaining a reference action sequence related to a partial sound sequence of the reference sound sequence; and step S422, generating a predicted action sequence based on the reference sound sequence and the reference action sequence using a model for generating an action sequence.

[0077] In the above embodiment, the reference action sequence related to the partial sound sequence of the reference sound sequence can assist the model to more accurately learn the mapping relationship between the sound features and the action features, thereby further improving the accuracy and authenticity of the generated action sequence.

[0078] In step S421, a reference action sequence related to a partial sound sequence of a reference sound sequence is obtained. In some embodiments, the reference action sequence is an actual action sequence corresponding to the partial sound sequence of the reference sound sequence. For example, taking the sound sequence as music and the action sequence as dance, a dance action choreographed by humans according to a period of time at the beginning of the music can be used as the reference action sequence.

[0079] In step S422, a predicted action sequence is generated based on the reference sound sequence and the reference action sequence using a model for generating action sequences.

[0080] For example, one can Figure 5 The steps shown implement the above-mentioned step S422.

[0081] Figure 5 is a flow chart illustrating the generation of a predicted action sequence according to some embodiments of the present disclosure.

[0082] like Figure 5 As shown, generating a predicted action sequence includes: step S4221, using the first sub-model and the second sub-model to encode the reference sound sequence and the reference action sequence to obtain target semantic coding features; and step S4222, generating a predicted action sequence based on the target semantic coding features.

[0083] In step S4221, the reference sound sequence and the reference action sequence are encoded using the first sub-model and the second sub-model to obtain target semantic coding features. For example, the semantic coding features represent mapping features between sound features and action features.

[0084] In some embodiments, the first sub-model and the second sub-model are connected in series or in parallel.

[0085] When the first sub-model and the second sub-model are connected in series, the step of "encoding the reference sound sequence and the reference action sequence to obtain the target semantic encoding features" can be implemented in the following way.

[0086] Firstly, the reference sound sequence and the reference action sequence are encoded using the first sub-model to obtain initial semantic encoding features.

[0087] Then, the second sub-model is used to encode the initial semantic coding features to obtain the target semantic coding.

[0088] When the first sub-model and the second sub-model are connected in series, the step of "encoding the reference sound sequence and the reference action sequence to obtain the target semantic coding features" can also be implemented in the following way.

[0089] Firstly, the second sub-model is used to encode the reference sound sequence and the reference action sequence to obtain the initial semantic encoding features.

[0090] Then, the first sub-model is used to encode the initial semantic coding features to obtain the target semantic coding.

[0091] When the first sub-model and the second sub-model are connected in parallel, the step of "encoding the reference sound sequence and the reference action sequence to obtain the target semantic encoding features" can be implemented in the following manner.

[0092] Firstly, the reference sound sequence and the reference action sequence are encoded using the first sub-model to obtain the first initial semantic encoding feature.

[0093] Then, the reference sound sequence and the reference action sequence are encoded using the second sub-model to obtain a second initial semantic encoding feature.

[0094] Finally, the first initial semantic coding feature and the second initial semantic coding feature are fused to obtain the target semantic coding. In some embodiments, the first initial semantic coding feature and the second initial semantic coding feature can be concatenated to obtain the target semantic coding.

[0095] In some embodiments, in the process of encoding the reference sound sequence and the reference action sequence, composite features related to the sound and the action may be determined based on the reference sound sequence and the reference action sequence, and then the composite features may be encoded.

[0096] In some embodiments, the composite features related to sound and motion may be determined through the following steps 1)-3).

[0097] 1) Extracting features from a reference sound sequence to obtain audio features. In some embodiments, the audio features include at least one of Mel frequency cepstral coefficient features and onset intensity features. For example, the audio features are represented by a one-dimensional vector.

[0098] 2) Feature extraction is performed on the reference action sequence to obtain action features. In some embodiments, the action features include a multidimensional rotation matrix associated with multiple key points of the action executor. For example, the multidimensional rotation matrix includes at least one of a three-dimensional rotation angle matrix and a nine-dimensional rotation matrix. The three-dimensional rotation angle matrix is ​​a matrix composed of the rotation angles of multiple key points of the action executor relative to the origin. The nine-dimensional rotation matrix is ​​a matrix composed of the rotation angles, rotation directions, relative positions, etc. of multiple key points of the action executor relative to the origin. For example, the origin is other parts other than one or more key points among the multiple key points. For example, the action feature is represented by a one-dimensional vector.

[0099] 3) Determine composite features based on audio features and motion features.

[0100] In step S4222, a predicted action sequence is generated based on the target semantic coding features.

[0101] In some embodiments, the model for generating the action sequence further includes a third sub-model. The target semantic coding features are decoded using the third sub-model to obtain the predicted action sequence. The decoding process is the process of mapping the sound features to the action features.

[0102] In some other embodiments, the third sub-model can also be used to decode the target semantic coding features to obtain an initial action sequence, and the initial action sequence is adjusted first and second to obtain a predicted action sequence. The process of the first adjustment and the second adjustment has been described in Figure 3 Description, no further elaboration is given here.

[0103] In some embodiments, the third sub-model is an LSTM (Long Short-Term Memory) model. During the training process, the LSTM model uses the output of the previous time step of each time step as the input of the time step. For example, during the training process, after obtaining the output of the i-th time step, the output of the i-th time step is used as the input of the i+1-th time step to obtain the output of the i+1-th time step, where i is an integer. By using the output of the previous time step of each time step as the input of the time step during the training of the LSTM model, the problem of action freezing (i.e., the problem of a single action that remains unchanged for a period of time) that may be caused by long-term action prediction can be solved, thereby further improving the accuracy of the generated action sequence.

[0104] Figure 6 is a block diagram illustrating an apparatus for generating an action sequence according to some embodiments of the present disclosure.

[0105] like Figure 6 As shown, the device 6 for generating an action sequence includes an acquisition module 61 and a generation module 62 .

[0106] The acquisition module 61 is configured to acquire a reference sound sequence, for example, by performing the following steps: Figure 1 Step S110 is shown. In some embodiments, the acquisition module 61 is further configured to acquire a reference action sequence related to a partial sound sequence of the reference sound sequence.

[0107] The generation module 62 is configured to generate a predicted action sequence corresponding to the reference sound sequence using a model for generating an action sequence according to the reference sound sequence, for example, Figure 1 Step S120 is shown. The model for generating an action sequence includes an associated first sub-model and a second sub-model. The first sub-model pays more attention to local short-time features than to global long-time features. The second sub-model pays more attention to global long-time features than to local short-time features. The first sub-model is configured to map features associated with a reference sound sequence to a local area of ​​an input space of the first sub-model. The second sub-model is configured to map features associated with a reference sound sequence to the entire area of ​​an input space of the second sub-model.

[0108] In some embodiments, the generation module 62 includes an audio feature extraction module 621. The audio feature extraction module 621 is configured to extract features from the reference sound sequence to obtain audio features.

[0109] In some embodiments, the generating module 62 further includes an action feature extraction module 622. The action feature extraction module 622 is configured to extract features from the reference action sequence to obtain action features.

[0110] In some embodiments, the generation module 62 further includes a first fusion module 623. The first fusion module 623 is configured to receive the audio features of the audio feature extraction module 621 and the motion feature extraction module 622, and determine a composite feature according to the audio features and the motion features.

[0111] In some embodiments, the generation module 62 further includes a model 624 for generating action sequences. The model 624 for generating action sequences includes an associated first sub-model 6241 and a second sub-model 6242. For example, the first sub-model 6241 and the second sub-model 6242 constitute an encoding module.

[0112] Here, taking the second sub-model 6242 connected in series after the first sub-model 6241 as an example, the first sub-model 6241 is configured to encode the composite features from the first fusion module 623, obtain initial semantic coding features, and send the initial semantic coding features to the second sub-model 6242.

[0113] The second sub-model 6242 is configured to receive the initial semantic coding features from the first sub-model 6241 and encode the initial semantic coding features to obtain target semantic coding features.

[0114] In some embodiments, the model 624 for generating an action sequence further includes a third sub-model 6243. The third sub-model 6243 is configured to decode the target semantic encoding features from the second sub-model 6242 to obtain an initial action sequence. For example, the third sub-model 6243 constitutes a decoding module.

[0115] In some embodiments, the model 624 for generating action sequences further includes a first adjustment module 6244 and a second adjustment module 6245 .

[0116] In some embodiments, Figure 6 As shown, the first adjustment module 6244 and the second adjustment module 6245 are connected in parallel. In this case, the model 624 for generating action sequences further includes a second fusion module 6246.

[0117] The first adjustment module 6244 is configured to receive the initial action sequence from the third sub-model 6243, and perform a first adjustment on the initial action sequence to obtain a first initial action sequence. The first adjustment is used to adjust the action execution body itself.

[0118] The second adjustment module 6245 is configured to receive the initial action sequence from the third sub-model 6243, and perform a second adjustment on the initial action sequence to obtain a second initial action sequence. The second adjustment is used to adjust the relationship between the action execution body and its surrounding environment.

[0119] The second fusion module 6246 is configured to fuse the first initial action sequence from the first adjustment module 6244 and the second initial action sequence from the second adjustment module 6245 to obtain a predicted action sequence.

[0120] In some other embodiments, the first adjustment module and the second adjustment module are connected in series. For example, the first adjustment module is connected in series after the second adjustment module. In this case, the first adjustment module directly outputs the predicted action sequence. For another example, the second adjustment module is connected in series after the first adjustment module. In this case, the second adjustment module directly outputs the predicted action sequence.

[0121] Figure 7 is a block diagram illustrating an apparatus for training a model for generating action sequences according to some embodiments of the present disclosure.

[0122] like Figure 7 As shown, the device 7 for training a model for generating an action sequence includes an acquisition module 71 , a generation module 72 and a training module 73 .

[0123] The acquisition module 71 is configured to acquire a reference sound sequence and its corresponding actual action sequence, for example, Figure 2 In step S210 shown, the actual action sequence is a training label of the reference sound sequence.

[0124] The generation module 72 is configured to generate a predicted action sequence corresponding to the reference sound sequence using a model for generating an action sequence according to the reference sound sequence, for example, Figure 2 Step S220 shown. The model for generating the action sequence includes a first sub-model and a second sub-model associated with each other. The first sub-model pays more attention to the local short-term features than the global long-term features. The second sub-model pays more attention to the global long-term features than the local short-term features.

[0125] The training module 73 is configured to train a model for generating action sequences using the predicted action sequences and the actual action sequences, for example, Figure 2 Step S230 is shown.

[0126] Figure 8 is a block diagram illustrating an electronic device according to some embodiments of the present disclosure.

[0127] like Figure 8 As shown, the electronic device 8 includes a memory 81; and a processor 82 coupled to the memory 81. The memory 81 is used to store instructions for executing the method for generating an action sequence or training the method for generating an action sequence corresponding to an embodiment. The processor 82 is configured to execute the method for generating an action sequence or training the method for generating an action sequence in any of the embodiments of the present disclosure based on the instructions stored in the memory 81.

[0128] Fig. 9 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0129] like Fig. 9 As shown, the computer system 90 may be embodied in the form of a general-purpose computing device. The computer system 90 includes a memory 910, a processor 920, and a bus 900 that connects the various system components.

[0130] The memory 910 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium may store, for example, instructions for executing at least one of the corresponding embodiments of the method for generating an action sequence or the method for training the generation of an action sequence. The non-volatile storage medium includes, but is not limited to, a disk storage, an optical storage, a flash memory, etc.

[0131] The processor 920 can be implemented by a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors, etc. Discrete hardware components. Accordingly, each module such as the judgment module and the determination module can be implemented by a central processing unit (CPU) running instructions in a memory that execute corresponding steps, or can be implemented by a dedicated circuit that executes corresponding steps.

[0132] The bus 900 may use any of a variety of bus architectures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0133] The computer system 90 may also include an input / output interface 930, a network interface 940, a storage interface 950, etc. These interfaces 930, 940, 950, the memory 910, and the processor 920 may be connected via a bus 900. The input / output interface 930 may provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 940 may provide a connection interface for various networked devices. The storage interface 950 may provide a connection interface for external storage devices such as a floppy disk, a USB flash drive, and an SD card.

[0134] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames can be implemented by computer-readable program instructions.

[0135] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0136] These computer-readable program instructions may also be stored in a computer-readable memory, which cause the computer to work in a specific manner to produce an article of manufacture, including instructions for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0137] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0138] The accuracy of the generated action sequence can be improved by using the method and device for generating an action sequence, the method and device for training a model for generating an action sequence, the electronic device, and the computer storable medium in the above embodiments.

[0139] So far, the method and device for generating an action sequence, the method and device for training a model for generating an action sequence, the electronic device, and the computer storable medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

Claims

1. A method for generating an action sequence, comprising: Obtain a reference sound sequence; According to the reference sound sequence, a predicted action sequence corresponding to the reference sound sequence is generated using a model for generating an action sequence, wherein the model for generating an action sequence includes an associated first sub-model and a second sub-model, wherein the first sub-model pays more attention to local short-time features than to global long-time features, and the second sub-model pays more attention to global long-time features than to local short-time features, wherein the first sub-model and the second sub-model constitute an encoding module, and the first sub-model and the second sub-model are connected in series or in parallel.

2. A method for training a model for generating an action sequence, comprising: Acquire a reference sound sequence and its corresponding actual action sequence, wherein the actual action sequence is a training label of the reference sound sequence; According to the reference sound sequence, a predicted action sequence corresponding to the reference sound sequence is generated using a model for generating an action sequence, wherein the model for generating an action sequence includes an associated first sub-model and a second sub-model, wherein the first sub-model pays more attention to local short-time features than to global long-time features, and the second sub-model pays more attention to global long-time features than to local short-time features, wherein the first sub-model and the second sub-model constitute an encoding module, and the first sub-model and the second sub-model are connected in series or in parallel; The model for generating the action sequence is trained by using the predicted action sequence and the actual action sequence.

3. The method according to claim 1 or 2, wherein: According to the reference sound sequence, using a model for generating an action sequence, generating a predicted action sequence corresponding to the reference sound sequence comprises: According to the reference sound sequence, using a model for generating action sequences, generating an initial action sequence corresponding to the reference sound sequence; The initial action sequence is subjected to a first adjustment and a second adjustment to obtain the predicted action sequence, wherein the first adjustment is used to adjust the action execution body itself, and the second adjustment is used to adjust the relationship between the action execution body and its surrounding environment.

4. The method according to claim 3, wherein: The second adjustment of the initial action sequence includes: At least one of a relative position and a relative speed between one or more key points of the action execution body and objects in its surrounding environment is adjusted.

5. The method according to claim 3, wherein: The first adjustment of the initial action sequence includes: The relative rotation speed between the multiple key points of the action execution body is adjusted.

6. The method according to claim 5, wherein: The first adjustment of the initial action sequence further includes: At least one of a relative position and a relative rotation angle between a plurality of key points of the action execution body is adjusted.

7. The method according to claim 1 or 2, wherein: In the process of generating the predicted action sequence, the first sub-model maps the features related to the reference sound sequence to the local area of ​​the input space of the first sub-model, and the second sub-model maps the features related to the reference sound sequence to the entire area of ​​the input space of the second sub-model.

8. The method according to claim 1 or 2, wherein: According to the reference sound sequence, using a model for generating an action sequence, generating a predicted action sequence corresponding to the reference sound sequence comprises: Acquire a reference action sequence related to a partial sound sequence of the reference sound sequence; The predicted action sequence is generated according to the reference sound sequence and the reference action sequence using the model for generating the action sequence.

9. The method according to claim 8, wherein: Generating the predicted action sequence according to the reference sound sequence and the reference action sequence using the model for generating the action sequence comprises: Encoding the reference sound sequence and the reference action sequence using the first sub-model and the second sub-model to obtain target semantic encoding features; The predicted action sequence is generated according to the target semantic encoding feature.

10. The method according to claim 9, wherein: In the case where the first sub-model and the second sub-model are connected in series, the reference sound sequence and the reference action sequence are encoded using the first sub-model and the second sub-model to obtain target semantic encoding features including: Using the first sub-model, encoding the reference sound sequence and the reference action sequence to obtain initial semantic coding features; The initial semantic coding features are encoded using the second sub-model to obtain the target semantic coding features.

11. The method according to claim 9, wherein: In the case where the first sub-model and the second sub-model are connected in series, the reference sound sequence and the reference action sequence are encoded using the first sub-model and the second sub-model to obtain target semantic encoding features including: Using the second sub-model, encoding the reference sound sequence and the reference action sequence to obtain initial semantic coding features; The initial semantic coding features are encoded using the first sub-model to obtain the target semantic coding features.

12. The method according to claim 9, wherein: In the case where the first sub-model and the second sub-model are connected in parallel, the reference sound sequence and the reference action sequence are encoded using the first sub-model and the second sub-model to obtain target semantic encoding features including: Using the first sub-model, encoding the reference sound sequence and the reference action sequence to obtain a first initial semantic coding feature; Using the second sub-model, encoding the reference sound sequence and the reference action sequence to obtain a second initial semantic coding feature; The first initial semantic coding feature and the second initial semantic coding feature are fused to obtain the target semantic coding feature.

13. The method according to claim 9, wherein: The reference sound sequence and the reference action sequence are encoded using the first sub-model and the second sub-model to obtain target semantic encoding features including: Determining a composite feature related to sound and action according to the reference sound sequence and the reference action sequence; The composite feature is encoded using the first sub-model and the second sub-model to obtain the target semantic encoding feature.

14. The method according to claim 13, wherein: Determining, according to the reference sound sequence and the reference action sequence, a composite feature related to the sound and the action comprises: Extracting features from the reference sound sequence to obtain audio features; Extracting features from the reference action sequence to obtain action features; The composite feature is determined according to the audio feature and the action feature.

15. The method according to claim 14, wherein: The audio feature includes at least one of a Mel frequency cepstral coefficient feature and a starting intensity feature, and the action feature includes a multi-dimensional rotation matrix associated with a plurality of key points of an action execution body.

16. The method according to claim 9, wherein: The model for generating an action sequence further includes a third sub-model, and generating the predicted action sequence according to the target semantic encoding feature includes: The target semantic coding features are decoded using the third sub-model to obtain the predicted action sequence.

17. The method according to claim 16, wherein: The third sub-model is a long short-term memory (LSTM) model, which uses the output of the previous time step of each time step as the input of the time step during the training process.

18. The method according to claim 1 or 2, wherein: The first sub-model includes one or more time-domain convolutional neural network models.

19. The method according to claim 1 or 2, wherein: The second sub-model includes an encoding structure of one or more Transformer models.

20. A device for generating an action sequence, comprising: An acquisition module, configured to acquire a reference sound sequence; A generation module is configured to generate a predicted action sequence corresponding to the reference sound sequence based on the reference sound sequence and using a model for generating action sequences, wherein the model for generating action sequences includes an associated first sub-model and a second sub-model, wherein the first sub-model pays more attention to local short-time features than to global long-time features, and the second sub-model pays more attention to global long-time features than to local short-time features, wherein the first sub-model and the second sub-model constitute an encoding module, and the first sub-model and the second sub-model are connected in series or in parallel.

21. A device for training a model for generating an action sequence, comprising: An acquisition module is configured to acquire a reference sound sequence and its corresponding actual action sequence, wherein the actual action sequence is a training label of the reference sound sequence; A generating module is configured to generate a predicted action sequence corresponding to the reference sound sequence using a model for generating an action sequence according to the reference sound sequence, wherein the model for generating an action sequence includes an associated first sub-model and a second sub-model, wherein the first sub-model pays more attention to local short-time features than to global long-time features, and the second sub-model pays more attention to global long-time features than to local short-time features, wherein the first sub-model and the second sub-model constitute an encoding module, and the first sub-model and the second sub-model are connected in series or in parallel; The training module is configured to train the model for generating the action sequence by using the predicted action sequence and the actual action sequence.

22. An electronic device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the method according to any one of claims 1 to 19 based on instructions stored in the memory.

23. A computer storable medium having computer program instructions stored thereon, which, when executed by a processor, implement the method according to any one of claims 1 to 19.

Images