Action generation method, device, electronic device and storage medium

Abstract

The present invention provides an action generation method, device, electronic device and storage medium, wherein the method comprises: obtaining audio data and a preceding action sequence in a current scene; performing feature extraction based on the audio data and the preceding action sequence respectively, and fusing the audio features and action features obtained by the feature extraction to obtain fused features; performing action generation based on the fused features to obtain a current action sequence in the current scene, thereby achieving matching of the action sequence with the scene audio, allowing the generated action sequence to change with changes in input information, achieving body movement with sound, ensuring diversity in action generation, and consistency and coordination between the current action sequence and the preceding action sequence, overcoming the defects of lack of authenticity and diversity in actions generated in traditional solutions, and improving the authenticity, timeliness and naturalness of action generation.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an action generation method, device, electronic device and storage medium. Background Art

[0002] With the development of virtual reality, augmented reality and other technologies, 3D (3-dimension) motion generation technology has been widely used in games, film and television and other fields.

[0003] However, most of the existing 3D motion generation methods are based on preset motion libraries, and new 3D motions are generated by combining and modifying the motions in the library. Although this method is simple and easy, it has the following problems: First, the motions in the library are limited, and there are obvious limitations in motion generation, and the types and number of generated animations are limited; second, the generated motions are relatively abrupt and cannot match the actual usage scenarios. They will not change with different scenarios, input information, etc., and their authenticity and diversity are poor. Summary of the invention

[0004] The present invention provides an action generation method, device, electronic device and storage medium to solve the defects of lack of authenticity and diversity of actions generated in the prior art. It can generate adaptive action sequences for different input information, ensure the vividness, realism and diversity of action generation, and realize real and natural action generation.

[0005] The present invention provides an action generation method, comprising:

[0006] Get the audio data of the current scene and the preceding action sequence;

[0007] Extracting features based on the audio data and the preceding action sequence respectively, and fusing the audio features and action features obtained by the feature extraction to obtain fused features;

[0008] Action generation is performed based on the fusion features to obtain a current action sequence in the current scene.

[0009] According to an action generation method provided by the present invention, the action generation based on the fusion feature to obtain the current action sequence in the current scene includes:

[0010] Get the style type corresponding to the current scene;

[0011] Determining an action probability distribution based on the audio feature and the style type;

[0012] Action generation is performed based on the fusion feature and the action probability distribution to obtain a current action sequence in the current scene.

[0013] According to an action generation method provided by the present invention, determining the action probability distribution based on the audio feature and the style type includes:

[0014] Based on the audio features and the style type, applying an action prediction model to perform action prediction to obtain an action probability distribution corresponding to the audio data and the style type;

[0015] The action prediction model is obtained by training on the basis of the initial model using sample audio data in sample scenes and corresponding sample style types.

[0016] According to an action generation method provided by the present invention, the action prediction model is trained based on the following steps:

[0017] Based on the initial model, determining a sample action probability distribution corresponding to the sample audio data and the sample style type;

[0018] Selecting positive samples and negative samples from the sample audio data and their corresponding sample style types, wherein the positive samples are at least two sample audio data and their corresponding sample style types in adjacent time domains under the same sample scene, and the negative samples are at least two sample audio data and their corresponding sample style types in different sample scenes and / or non-adjacent time domains;

[0019] Based on the similarity between the sample action probability distribution corresponding to each sample audio data and its corresponding sample style type in the positive sample, and the similarity between the sample action probability distribution corresponding to each sample audio data and its corresponding sample style type in the negative sample, the initial model is iterated on parameters to obtain an action prediction model.

[0020] According to an action generation method provided by the present invention, the initial model includes a downsampling module, a residual module and an upsampling module;

[0021] The downsampling module is used to downsample the sample fusion feature, where the sample fusion feature is obtained by fusion of the sample audio feature of the sample audio data and the corresponding sample style type;

[0022] The residual module is used to determine the initial action probability distribution based on the features obtained by downsampling; the upsampling module is used to upsample the initial action probability distribution to obtain the sample action probability distribution corresponding to the sample audio data and the sample style type.

[0023] According to an action generation method provided by the present invention, the action generation is performed based on the fusion feature and the action probability distribution to obtain the current action sequence in the current scene, including:

[0024] Obtaining a preset lower body action sequence in the current scene;

[0025] Perform feature extraction based on the lower body action sequence to obtain lower body action features;

[0026] Action generation is performed based on the fusion feature, the action probability distribution, and the lower body action feature to obtain a current action sequence in the current scene.

[0027] According to an action generation method provided by the present invention, the action generation is performed based on the fusion feature, the action probability distribution, and the lower body action feature to obtain the current action sequence in the current scene, including:

[0028] Based on the body movement constraints in the current scene, applying the lower body movement sequence to generate an upper body movement sequence in the current scene;

[0029] Extract features based on the upper body action sequence to obtain upper body action features;

[0030] Action generation is performed based on the fusion feature, the action probability distribution, and the lower body action feature and the upper body action feature to obtain a current action sequence in the current scene.

[0031] According to an action generation method provided by the present invention, the action generation is performed based on the fusion feature, the action probability distribution, and the lower body action feature to obtain the current action sequence in the current scene, including:

[0032] Acquire an action tag, where the action tag includes a key action in the current scene and an action attribute of the key action;

[0033] Action generation is performed based on the fusion feature, the action probability distribution, the lower body action feature, the style type and the action label to obtain a current action sequence.

[0034] According to an action generation method provided by the present invention, the action generation is performed based on the fusion feature, the action probability distribution, the lower body action feature, the style type and the action label to obtain the current action sequence, including:

[0035] Determine a fused action feature based on the action probability distribution, the lower body action feature, the style type, and the action label;

[0036] Action generation is performed based on the fused features and the fused action features to obtain a current action sequence.

[0037] According to an action generation method provided by the present invention, the step of obtaining the style type corresponding to the current scene includes:

[0038] Classify the audio data and / or the transcribed text corresponding to the audio data to obtain an emotion category and / or a scene type;

[0039] Based on the emotion category and / or the scene type, the style type corresponding to the current scene is determined from a preset style type library.

[0040] According to an action generation method provided by the present invention, the step of fusing the audio features and action features obtained by feature extraction to obtain fused features includes:

[0041] Performing time domain alignment processing on the audio feature and the action feature to obtain aligned audio feature and action feature;

[0042] Based on the correlation between the aligned audio features and the action features, the aligned audio features and the action features are fused to obtain fused features.

[0043] The present invention also provides an action generation method, comprising:

[0044] Acquire first modal data and second modal data in a current scene, where the first modal data is used to reflect scene information of the current scene;

[0045] Performing feature extraction on the first modal data to obtain a first modal feature, and performing feature extraction on the second modal data to obtain a second modal feature;

[0046] Action generation is performed based on the first modal feature and the second modal feature to obtain a current action sequence in the current scene.

[0047] According to an action generation method provided by the present invention, the first modal data is audio data in the current scene, and the first modal feature is an audio feature of the audio data;

[0048] The second modal data is a preceding action sequence in the current scene, and the second modal feature is an action feature of the preceding action sequence; or, the second modal data is a style type corresponding to the current scene, and the second modal feature is a style feature.

[0049] The present invention also provides an action generating device, comprising:

[0050] An acquisition unit, used to acquire audio data in the current scene and a preceding action sequence;

[0051] A fusion unit, configured to extract features based on the audio data and the preceding action sequence, respectively, and fuse the audio features and action features obtained by the feature extraction to obtain fused features;

[0052] A generating unit is used to generate actions based on the fusion features to obtain a current action sequence in the current scene.

[0053] The present invention also provides an action generating device, comprising:

[0054] A data acquisition unit, used to acquire first modal data and second modal data in a current scene, wherein the first modal data is used to reflect scene information of the current scene;

[0055] a feature extraction unit, configured to perform feature extraction on the first modal data to obtain a first modal feature, and perform feature extraction on the second modal data to obtain a second modal feature;

[0056] An action generation unit is used to generate actions based on the first modal feature and the second modal feature to obtain a current action sequence in the current scene.

[0057] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the action generation method described in any one of the above is implemented.

[0058] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the action generation method as described in any one of the above is implemented.

[0059] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the action generation method as described above is implemented.

[0060] The action generation method, device, electronic device and storage medium provided by the present invention extract audio features and action features through audio data and previous action sequences in the current scene, fuse the audio features and action features to obtain fused features, generate actions based on the fused features, obtain the current action sequence, achieve matching of the action sequence with the scene audio, and enable the generated action sequence to change with changes in input information, thereby achieving movement with sound, ensuring the diversity of action generation, and the continuity and coordination between the current action sequence and the previous action sequence, overcoming the defects of lack of authenticity and diversity of actions generated in traditional solutions, and effectively improving the authenticity, timeliness and naturalness of action generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 It is one of the flowcharts of the action generation method provided by the present invention;

[0063] Figure 2 It is a structural schematic diagram of the initial model provided by the present invention;

[0064] Figure 3 It is the overall framework diagram of the action generation method provided by the present invention;

[0065] Figure 4 This is the second flow chart of the action generation method provided by the present invention;

[0066] Figure 5 It is one of the structural schematic diagrams of the action generation device provided by the present invention;

[0067] Figure 6 This is the second structural schematic diagram of the action generation device provided by the present invention;

[0068] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] At present, the action generation method based on the preset action library has obvious limitations on the types and number of 3D actions that can be generated due to the limited capacity of the action library. In addition, when generating actions based on the action library, it is impossible to take into account the information in the real usage scenario. The information in the real scene is crucial for understanding the user's intention and generating corresponding actions. Therefore, when the information in the real scene cannot be processed and utilized in real time, the generated actions often cannot be well integrated with the scene, and the generated actions will not change with the changes in the scene, input information, etc., and their authenticity and diversity are poor.

[0071] In this regard, the present invention provides an action generation method, which aims to fuse action features of a preceding action sequence and audio features of audio data in a current scene to obtain fused features, and generate actions based on the fused features to obtain the current action sequence, thereby achieving matching of the action sequence with the scene audio, being well adapted to specific scenes, and generating different action sequences for different inputs, which can change accordingly according to changes in the input information, thereby achieving movement with the sound, and greatly improving the vividness, realism and naturalness of the generated action sequence while ensuring action diversity. Figure 1 is one of the flow charts of the action generation method provided by the present invention, such as Figure 1 As shown, the method includes:

[0072] Step 110, obtaining audio data in the current scene and a preceding action sequence;

[0073] In order to avoid the defects of traditional solutions based on preset action libraries in action generation, realize real, natural and diversified action generation, and ensure the adaptability of the generated action sequence to the scene, in the embodiments of the present invention, when generating actions, the information under the specific scene can be considered. By incorporating the scene information, it can be used as a reference in the subsequent action generation link to generate corresponding actions to ensure that they are consistent with the scene. At the same time, they can be connected and coordinated with previous actions to ensure the adaptability and coordination of the generated action sequence.

[0074] In view of this, in the embodiment of the present invention, before generating an action, the information of the current scene can be obtained first, and the information here can be voice, text, background, etc., and considering that in actual scenes, the information represented by voice is usually more sufficient, comprehensive and accurate, and is more helpful to understand the user's intention and generate a corresponding action sequence. For example, voice can more directly reflect the atmosphere of the current scene, the user's emotions, attitude, etc.;

[0075] Therefore, in the embodiment of the present invention, in order to simplify the difficulty of information acquisition, ensure the simplicity and convenience of collection, and the timeliness of action generation, the information used for action generation in the current scene can be determined to include voice, that is, the audio data in the current scene can be obtained. The audio data here can be understood as real-time voice in the current scene, which can be obtained by real-time audio pickup through a sound pickup device, and the sound pickup device here can be a microphone, a recorder, a pickup, etc.; it can be set / installed in the current scene to pick up sound in a specific area, range, etc. in the current scene, or it can be carried by the person in the current scene to better and more accurately pick up the human voice, and the embodiment of the present invention does not make specific limitations on this.

[0076] Among them, the scenes that need to generate actions can be various fields, for example, scenes corresponding to education, games, transportation, new media, etc., and specifically can be course teaching, simulation games, rail transportation, film and television interpretation, shopping live broadcast, etc. However, due to the differences in styles and atmospheres of different scenes, in order to better adapt to specific scenes, so that the generated actions can be accurately integrated with the scenes without being awkward, etc., in the embodiment of the present invention, before generating actions, it is necessary to determine the current scene, that is, determine the current scene, and obtain the audio data in the current scene, so as to generate actions.

[0077] It is worth noting that after obtaining the audio data, if action generation is performed only based on this, it can only ensure the adaptability of the generated action sequence to the scene, but cannot guarantee its coherence and coordination with the previous action. This will undoubtedly cause the newly generated action sequence to have a gap in connection, unable to be coherent with the previous action, and poor vividness and naturalness.

[0078] Therefore, in order to ensure the continuity and coordination of the action, in the embodiment of the present invention, before performing action generation, it is necessary to obtain the previous action sequence in the current scene, that is, the action sequence before the current moment, which is called the previous action sequence of the current action sequence to be generated. In essence, it is the historical action sequence relative to the current moment, and the process of action generation is based on the historical action sequence and the audio data in the current scene. The process can obtain an action sequence that is adapted to the current scene and has good continuity and coordination with the previous action sequence, realizing efficient and accurate action generation and ensuring the authenticity, naturalness and timeliness of action generation.

[0079] Step 120, extracting features based on the audio data and the preceding action sequence respectively, and fusing the audio features and action features obtained by the feature extraction to obtain fused features;

[0080] Specifically, after determining the audio data and the preceding action sequence in the current scene, feature extraction can be performed on the audio data and the preceding action sequence to obtain audio features and action features, and the audio features and action features can be fused to obtain fused features.

[0081] It is understandable that after obtaining the audio data and the previous action sequence, before generating the action based on them, feature extraction is required first, so as to extract the rhythm, tempo and other features in the audio data that can help the action generation, as well as the limbs, key actions, etc. in the previous action sequence, so as to ensure that the action sequence generated during the subsequent action generation can be consistent with the rhythm and tempo of the audio data, and coordinated and coherent with the actions of the previous action sequence, so as to achieve real and natural action generation.

[0082] Specifically, the audio data and the preceding action sequence may be encoded respectively to extract the audio features of the audio data and the action features of the preceding action sequence. Here, specifically, the audio data and the preceding action sequence may be taken as input, and an encoder may be used to encode the input audio data and the preceding action sequence respectively to extract the action features of the preceding action sequence, as well as the rhythm and tempo related features in the audio data. That is, the encoder may be used to encode the information related to rhythm and tempo in the audio data, as well as the action related information in the preceding action sequence, so as to obtain the audio features and action features output by the encoder.

[0083] Here, it is worth noting that the encoder that audio data is encoded and the encoder that the preceding action sequence is encoded can be the same or different. And as preferably, in the embodiment of the present invention, for being able to extract features respectively from audio data and preceding action sequence more efficiently and accurately, select to adopt different encoders to carry out audio encoding and motion encoding, specifically can be to select the encoder that is adapted to audio characteristics (rhythm rhythm, volume, etc.) as audio encoder (AudioEncoder), select the encoder that is adapted to motion characteristics (coherence, coordination, etc.) as motion encoder (MotionEncoder). By audio encoder, audio data is encoded, to obtain the audio features of audio encoder output, by motion encoder, preceding action sequence is encoded, to obtain the motion features of motion encoder output.

[0084] Afterwards, the extracted audio features and action features can be fused to obtain fused features. Since audio and action correspond to different modalities, the extracted audio features and action features also belong to features of different modalities. Therefore, the process of fusing the two is essentially the fusion of information from different modalities. During the fusion, attention can be paid to information from multiple modalities, thereby obtaining fused features including the rhythmic beat of the audio and key actions.

[0085] Specifically, a multimodal encoder may be used here to fuse audio features and motion features, that is, audio features and motion features may be used as input, and the input audio features and motion features may be processed by a multimodal encoder so that audio features and motion features that are not completely corresponding in the time domain can be aligned in the time domain, and both may be enhanced to obtain fused multimodal features, i.e., fused features. The multimodal encoder here is essentially an encoder (Multi-modal Encoder) that can jointly model the input audio features and motion features and output the multimodal features after the joint modeling.

[0086] Step 130, generating actions based on the fused features to obtain a current action sequence in the current scene.

[0087] Specifically, after obtaining the fusion features through the above process, action generation can be performed based on the fusion features to obtain the required action sequence, that is, the current action sequence in the current scene, and based on the generated actions at each moment, the complete action sequence in the current scene can be obtained. In this way, the action generation in the current scene is realized, which has good vividness, realism and real-time performance, and realizes real and natural action generation.

[0088] It can be understood that after obtaining the fused multimodal features through feature fusion, action generation can be performed based on the fused features containing multimodal information to obtain an action sequence that matches the rhythmic beat of the audio in the current scene and is connected to the previous action sequence, that is, the action sequence at the current moment in the current scene, which is called the current action sequence.

[0089] Specifically, after the fused features are obtained through the encoder fusion, when generating actions, the fused features can be decoded to decode the actions, thereby obtaining the current action sequence, that is, the fused features can be input into the decoder so that the decoder decodes the fused features and outputs the action sequence, thereby obtaining the current action sequence in the current scene.

[0090] Here, the decoder that decodes the fused features may be a motion decoder. Based on the fused features, actions are generated through the motion decoder, and the current action sequence in the current scene output by the motion decoder can be obtained.

[0091] In the embodiment of the present invention, the generated action sequence can change accordingly with the change of input information, thereby ensuring the diversity of action generation. At the same time, it can also make the generated action sequence fit the current scene and be coherent and coordinated with the previous action sequence, avoiding the problems of limited action generation and lack of authenticity and diversity in traditional solutions based on preset action libraries, and improving the vividness and realism of the generated action sequence.

[0092] It should be noted that the method provided by the present invention can be applied to an artificial intelligence-based adolescent mental health diagnosis and treatment system. For example, if the present invention is used to generate a virtual digital person, the generated virtual person can provide companionship and psychological health treatment to adolescents, which greatly improves the current problem that the companionship and psychological health treatment of adolescents are completely dependent on human intervention, ensures the timeliness and frequency of psychological monitoring of adolescents, and can effectively improve the current situation of frequent psychological problems among adolescents and improve the efficiency of mental health diagnosis and treatment.

[0093] The action generation method provided by the present invention extracts audio features and action features through audio data and previous action sequences in the current scene, fuses the audio features and action features to obtain fused features, generates actions based on the fused features, obtains the current action sequence, achieves matching of the action sequence with the scene audio, enables the generated action sequence to change with changes in input information, achieves movement with sound, ensures diversity of action generation, and consistency and coordination between the current action sequence and the previous action sequence, overcomes the defects of lack of authenticity and diversity of actions generated in traditional solutions, and effectively improves the authenticity, timeliness and naturalness of action generation.

[0094] Based on the above embodiment, step 130 includes:

[0095] Get the style type corresponding to the current scene;

[0096] Determine the action probability distribution based on audio features and style type;

[0097] Action generation is performed based on fusion features and action probability distribution to obtain the current action sequence in the current scene.

[0098] Specifically, in step 130, the process of generating actions according to the fusion features to obtain the current action sequence in the current scene may specifically include:

[0099] Taking into account that different scenes may require different styles, for example, the style type for course teaching may tend to be quiet, rigorous, etc., while the style type for shopping live broadcasts is more noisy, enthusiastic, and exciting. Therefore, in order to adapt to specific scenes more accurately, the style of the scene can be considered when generating actions, so that the final generated action sequence can adapt to the style type of the current scene and be more in line with it.

[0100] Based on this, in an embodiment of the present invention, when generating actions, the style type required by the current scene can be obtained. The style type here corresponds to the specific scene. For example, it can be a comic style, science fiction style, magic style, etc. corresponding to a simulation game scene, or an experimental style, historical style, etc. corresponding to course teaching, or a humanistic style, natural style, war style, etc. corresponding to cultural popularization, or a comedy style, tragedy style, modern style, etc. corresponding to film and television interpretation, or a rigorous style corresponding to rail transit, or a radical style corresponding to shopping live broadcasts.

[0101] Here, the style type can be pre-set, and then when generating actions for a specific scene, the style type that matches the current scene can be directly selected from the set various style types, and the selected style type can be one or more, and the embodiment of the present invention does not specifically limit this. For example, when the current scene is a film and television interpretation, the selected style type can be a comedy style or a tragedy style, or a comedy style and a modern style, or a tragedy style and a modern style. As a preference, when the set style type supports, multiple style types are preferred for the current scene, so that the generated action sequence is more realistic and closer to the real scene through the fusion of multiple styles. In addition, the style type can also be determined based on the information in the specific scene. For example, the field corresponding to the current scene can be determined based on the audio data, text, background, etc. in the current scene, thereby determining the corresponding style type.

[0102] Then, this style type and the audio features of the audio data in the current scene can be used to predict actions in order to predict the action at the current moment, thereby obtaining the action probability distribution at the current moment. That is, based on the audio features and style types, the actions that may be performed at the current moment are predicted, thereby obtaining the predicted action probability distribution at the current moment. This action probability distribution can be well matched to the style of the current scene and the rhythmic rhythm of the audio.

[0103] Here, the action probability distribution obtained by action prediction not only includes the predicted actions at the current moment, but also includes the confidence corresponding to each action, that is, the probability. The greater the probability, the greater the possibility of making the action at the current moment. Conversely, the lower the probability, the lower the possibility of making the action at the current moment. Action generation based on action probability distribution can consider the scene adaptability (rhythm consistency and style matching) of the generated action sequence, as well as the possibility of the predicted action, to ensure the rationality of the generated action sequence.

[0104] Then, actions can be generated based on the action probability distribution to obtain the action sequence in the current scene. That is, actions can be generated based on the action probability distribution and the fusion features obtained by encoder fusion. Specifically, based on the action probability distribution and the fusion features, an action decoder can be used to generate actions so that the generated actions can fit the style, audio rhythm, and cadence of the current scene, and connect with the previous action, thereby obtaining the current action sequence in the current scene decoded and output by the action decoder.

[0105] In an embodiment of the present invention, style is introduced in the process of action generation, and action generation is performed based on the style type corresponding to the current scene, so that the generated action sequence can better match the current scene. While being consistent with its audio, its style is integrated, so that the action sequence generated for the current scene can be more realistic and natural, avoiding awkwardness or abruptness.

[0106] Based on the above embodiment, determining the action probability distribution based on the audio features and the style type includes:

[0107] Based on the audio features and style types, the action prediction model is applied to predict the action and obtain the action probability distribution corresponding to the audio data and style types;

[0108] The action prediction model is trained based on the initial model using sample audio data in sample scenarios and corresponding sample style types.

[0109] Specifically, the process of determining the action probability distribution based on the audio features and style type can be implemented through an action prediction model, that is, the action prediction model can be used to perform action prediction to obtain the action probability distribution corresponding to the audio and style in the current scene.

[0110] The action probability distribution here can be regarded as an implicit label in the current scene, which implicitly indicates the action that matches the current style and the current audio. Therefore, the action prediction process based on the action prediction model can be understood as the process of predicting the implicit label at the current moment through the implicit label prediction model, that is, the current implicit label in the current scene can be predicted by the implicit label prediction model based on the audio features and style type.

[0111] Specifically, the audio features and style types can be used as inputs into the action prediction model. The action prediction model can perform action prediction based on the input audio features and style types to predict the actions that may be taken at the current moment, as well as the probabilities corresponding to each action. Finally, the action probability distribution output by the action prediction model can be obtained.

[0112] Before inputting the audio features and style types into the action prediction model, that is, before applying the action prediction model to perform action prediction (predicting the implicit label at the current moment through the implicit label prediction model), it is necessary to pre-train the action prediction model (implicit label prediction model). Different from the traditional supervised or multi-task training method, the embodiment of the present invention takes into account the difficulty of obtaining labels in the supervised training method, and the training method of multi-task learning is prone to deviation when multi-modal information is not fully shared, resulting in poor model performance. Based on this, in the embodiment of the present invention, the model training is performed by contrastive learning to obtain a trained action prediction model.

[0113] Specifically, when conducting model training, first, a large amount of sample data, namely, sample audio data and sample style types, needs to be collected in each scene, so as to obtain the sample audio data in the sample scene, and the sample style types corresponding to the sample scene; then, based on the sample audio features of the sample audio data and the sample style types, the sample action probability distribution of the corresponding sample scene can be determined through the initial model; thereafter, the initial model can be trained according to the similarity / difference between the sample action probability distributions corresponding to the sample audio data in adjacent time domains under the same sample scene, and the similarity / difference between the sample action probability distributions corresponding to the sample audio data in different sample scenes and / or non-adjacent time domains, so as to obtain a trained action prediction model.

[0114] Here, the sample data can be obtained by querying or downloading from the Internet, or can be obtained by screening from an existing data set, or can be obtained by shooting or recording an actual scene, etc., and the embodiments of the present invention do not specifically limit this. For example, in the actual application process, a studio can be set up and shooting or recording can be performed to obtain a large amount of sample data for model training, thereby ensuring the adequacy of the training process and the model training effect.

[0115] It is worth noting that in order to speed up the training of the model, reduce the constraints between the probability distributions of sample actions corresponding to sample audio data in adjacent time domains during training, reduce resource consumption, and reduce computing power requirements, in an embodiment of the present invention, when constructing the initial model, a network with a downsampling function can be selected as the basis for constructing the initial model.

[0116] In addition, it should be noted that the learning and training process of the initial model here can be carried out separately, for example, the initial model performs contrastive learning alone to train the motion prediction model, or it can be learned together with various encoders (such as audio encoders, motion encoders, multimodal encoders, etc.) and motion decoders. For example, the initial model can be end-to-end learned together with each encoder and decoder. At the same time, the initial model also performs contrastive learning, and the contrastive learning task is used as the main learning task, and the motion prediction model is trained accordingly.

[0117] In an embodiment of the present invention, during model training, a network with a downsampling function is selected as the basis for constructing an initial model. The initial model constructed based on this determines the sample action probability distribution, and the initial model is trained based on the similarity between the sample action probability distributions. This can greatly reduce the constraints on the sample action probability corresponding to the sample audio data in the adjacent time domain during the comparative training process while ensuring the training effect, thereby speeding up the model training process and promoting the overall process of action generation.

[0118] Based on the above embodiment, the action prediction model is trained based on the following steps:

[0119] Based on the initial model, determine the sample action probability distribution corresponding to the sample audio data and the sample style type;

[0120] Select positive samples and negative samples from the sample audio data and their corresponding sample style types, where the positive samples are at least two sample audio data and their corresponding sample style types in adjacent time domains under the same sample scene, and the negative samples are at least two sample audio data and their corresponding sample style types in different sample scenes and / or non-adjacent time domains;

[0121] Based on the similarity between the sample action probability distribution corresponding to each sample audio data and its corresponding sample style type in the positive sample, and the similarity between the sample action probability distribution corresponding to each sample audio data and its corresponding sample style type in the negative sample, the parameters of the initial model are iterated to obtain the action prediction model.

[0122] Specifically, the training process of the action prediction model includes the following steps:

[0123] Before model training, you first need to determine the sample data required for the training process and the initial model to be trained. The sample data here refers to the sample audio data in the sample scene and the corresponding sample style type. There can be multiple sample scenes, and the initial model can be a model built on the basis of a network with downsampling function.

[0124] Then, based on the sample audio data and its corresponding sample style type, the initial model can be used to perform action prediction to obtain the sample action probability distribution corresponding to the sample audio data and the sample style type. Specifically, the sample audio features of the sample audio data are first extracted, and then, based on the sample audio features and the sample style type, the initial model is applied to perform action prediction to obtain the action probability distribution corresponding to the sample audio data and the sample style type output by the initial model, that is, the sample action probability distribution. It is worth noting that the sample audio features and the sample style type input to the model correspond to the same sample scene.

[0125] Subsequently, the similarity between the sample action probability distribution corresponding to each sample audio data and its corresponding sample style type can be determined, and the loss in the comparative training process, i.e., the comparative loss, can be calculated based on this similarity; here, specifically, the similarity between the sample action probability distribution corresponding to the sample audio data and its corresponding sample style type in the adjacent time domain under the same sample scene and the similarity between the sample action probability distribution corresponding to the sample audio data and its corresponding sample style type in different sample scenes and / or non-adjacent time domains can be determined first, and then the comparative loss can be calculated based on the two.

[0126] Since the training goal of the initial model is to make the sample action probability distributions corresponding to the multiple sample data (including sample audio data and their corresponding sample style types) as close as possible (the similarity is as high as possible) when different sample data (including sample audio data and their corresponding sample style types) can constitute positive samples, that is, when multiple sample data correspond to the same sample scene and are adjacent in time domain; correspondingly, when different sample data constitute negative samples, that is, when multiple sample data correspond to different sample scenes and / or are not adjacent in time domain, the sample action probability distributions corresponding to the multiple sample data are as distant as possible (the similarity is as low as possible).

[0127] Therefore, after obtaining the sample action probability distribution corresponding to each sample data, when performing loss measurement based on it, the positive samples and negative samples can be first determined from each sample data. Specifically, at least two sample data in the same sample scenario and adjacent in time domain are selected from each sample data to form a positive sample. Correspondingly, sample data in different sample scenarios and / or in non-adjacent time domains can be selected to form a negative sample. In this way, the selected positive and negative samples can be obtained.

[0128] Here, time domain proximity can be understood as being adjacent / adjacent / close in time. For example, if the corresponding times of sample data A and B in the same scene are t and t+1 respectively, then it can be considered that the two are adjacent in time domain and can constitute positive samples; and for sample data C and D in the same scene, the corresponding times of the two are t and t+5 respectively. At this time, it can be judged whether the two are adjacent in time domain based on the proximity range, so as to determine whether they constitute positive samples or negative samples; for sample data E and F in different scenes, it can be determined that the two constitute negative samples. Here, t can be time, such as seconds, minutes, etc., or it can be an audio frame, such as the 1st frame, the 6th frame, etc., and the embodiments of the present invention do not make specific limitations on this.

[0129] Next, the similarity between the sample action probability distributions corresponding to each sample data in the positive sample can be calculated based on the sample action probability distributions corresponding to each sample data in the positive sample, and the similarity between the sample action probability distributions corresponding to each sample data in the negative sample can be calculated based on the sample action probability distributions corresponding to each sample data in the negative sample, that is, the similarity between the sample action probability distributions corresponding to each sample audio data in the positive sample and its corresponding sample style type, as well as the similarity between the sample action probability distributions corresponding to each sample audio data in the negative sample and its corresponding sample style type are determined, and the loss of the initial model in the comparative training process, that is, the comparative loss, is determined based on the two.

[0130] Here, when the similarity between the sample action probability distributions corresponding to each sample data in the positive sample is high, and the similarity between the sample action probability distributions corresponding to each sample data in the negative sample is low, it can be determined that the contrast loss is small; accordingly, when the similarity between the sample action probability distributions corresponding to each sample data in the positive sample is low, and / or the similarity between the sample action probability distributions corresponding to each sample data in the negative sample is high, it can be determined that the contrast loss is large.

[0131] Among them, the contrast loss can be measured by KLD (Kullback-Leibler divergence, relative entropy), that is, KLD can be introduced, and the relative entropy of the predicted posterior probability of the sample action corresponding to each sample data in the positive sample and the negative sample is used as the difference loss between the probability distribution of the sample action corresponding to each sample data in the positive sample and the negative sample, that is, the contrast loss.

[0132] Thereafter, the parameters of the initial model are iterated according to the contrast loss to obtain the action prediction model. Specifically, the parameters of the initial model are adjusted according to the contrast loss so that the adjusted initial model can make the similarity between the sample action probability distributions output for each sample data as high as possible when each sample data can constitute a positive sample, and make the similarity between the sample action probability distributions output for each sample data as low as possible when each sample data can constitute a negative sample. Finally, the trained action prediction model can be obtained.

[0133] Based on the above embodiment, the initial model includes a downsampling module, a residual module and an upsampling module; the downsampling module is used to downsample the sample fusion feature, and the sample fusion feature is obtained by fusion of the sample audio feature of the sample audio data and the corresponding sample style type;

[0134] The residual module is used to determine the initial action probability distribution based on the features obtained by downsampling; the upsampling module is used to upsample the initial action probability distribution to obtain the sample action probability distribution corresponding to the sample audio data and the sample style type.

[0135] Specifically, Figure 2 It is a structural schematic diagram of the initial model provided by the present invention, such as Figure 2 As shown, the initial model includes a downsampling module, a residual module and an upsampling module. Among them, the downsampling module can be constructed on the basis of a convolutional network, so it can also be called a convolutional downsampling module. This module (Downsample Block) is used to downsample the features of the input model to downsample it in the time dimension, thereby obtaining the downsampled features. The features of the input model here are the features after the fusion of the sample audio data and the sample style type, that is, the sample fusion features. The sample fusion features here can be obtained by splicing the sample audio features and the sample style type of the sample audio data.

[0136] After obtaining the downsampled features through the downsampling module, the features can be processed by the residual module to obtain the initial action probability distribution. Specifically, the downsampled features are further extracted and processed by the multi-layer convolution residual layer in the residual module to output a set of probability distributions in the time dimension, namely, the initial action probability distribution.

[0137] Furthermore, a linear weighted combination can be performed based on the initial action probability distribution and the learnable dictionary (Codebook) to obtain the target action probability distribution, that is, based on the initial action probability distribution and the learnable dictionary, the downsampled action implicit label, that is, the target action probability distribution, is obtained by linear weighted combination. Among them, the learnable dictionary is a technology used in machine learning and deep learning, especially in unsupervised learning, which is used to represent or encode specific features of input data; specifically in the embodiment of the present invention, the initial action probability distribution can be converted into a specific feature representation through processing based on the learnable dictionary, which is convenient for processing.

[0138] After that, upsampling can be performed through the upsampling module to upsample the downsampled action implicit label obtained in the previous step to the time scale of the original input, so as to obtain the final sample action probability distribution, that is, the sample action probability distribution corresponding to the sample audio data and the sample style type.

[0139] Based on the above embodiment, action generation is performed based on the fusion features and the action probability distribution to obtain the current action sequence in the current scene, including:

[0140] Get the preset lower body action sequence in the current scene;

[0141] Extract features based on the lower body action sequence to obtain lower body action features;

[0142] Action generation is performed based on fusion features, action probability distribution, and lower body action features to obtain the current action sequence in the current scene.

[0143] Specifically, the above process of generating actions according to the fusion features and the action probability distribution to obtain the current action sequence in the current scene may specifically include:

[0144] In order to make the generated action sequence more in line with expectations and meet personalized requirements, in an embodiment of the present invention, before action generation, a lower body action sequence can also be pre-set to fix the lower body actions in the current action sequence to be generated, so that the final action sequence generated at the current moment can meet expectations and be consistent with the preset lower body action sequence.

[0145] In view of this, in an embodiment of the present invention, the lower body action sequence in the current scene can be first obtained. The lower body action sequence here can be pre-set and stored according to actual needs, actual conditions, etc., and can be directly called during the action generation process to wait for action generation, thereby obtaining the current action sequence.

[0146] Subsequently, feature extraction can be performed on the lower body action sequence to extract key information about the lower limb movements in the lower body action sequence, thereby encoding the lower body action features for action generation, thereby ensuring that the lower limb movements in the subsequently generated action sequence are consistent with the expected lower body movements, and the lower limb movements in the generated action sequence can be coordinated and adapted with the upper limb movements, thereby achieving real and natural action generation.

[0147] Specifically, the lower body motion sequence (Lower Body Motion) may be encoded here to extract the lower body motion features, and specifically, the lower body motion sequence may be used as input, and an encoder may be used to encode the input lower body motion sequence, and the motion-related information in the lower body motion sequence may be encoded, thereby obtaining the lower body motion features output by the encoder. Here, it is worth noting that the encoder for encoding the lower body motion sequence may be the same as or different from the aforementioned motion encoder for encoding the preceding motion sequence, and preferably, in an embodiment of the present invention, in order to more accurately extract the corresponding motion features from different motion sequences, a corresponding encoder may be set for each motion sequence, that is, a lower body motion encoder (Lower Body Motion Encoder) may be used here to encode the lower body motion sequence, thereby obtaining the lower body motion features output by the lower body motion encoder.

[0148] After that, action generation can be performed based on the lower body action features, as well as the fused features and the action probability distribution obtained by action prediction, so as to obtain the current action sequence in the current scene. That is, on the basis of the fused features and the lower body action features, combined with the action probability distribution, the action at the current moment can be predicted, so as to obtain an action sequence that matches the style type and the rhythmic rhythm of the audio in the current scene, is connected to the previous action sequence, and is consistent with the expected lower body action, that is, the current action sequence in the current scene.

[0149] Here, the motion decoder is used to generate motion based on the lower body motion features, fusion features and motion probability distribution, that is, the motion at the current moment is decoded by the motion decoder to obtain the motion sequence at the current moment. When generating motion here, not only the scene adaptability (rhythmic rhythm consistency and style matching) of the generated motion sequence is considered, but also the consistency with the expected motion, so that the generated motion sequence can fit the style, audio rhythm and rhythm of the current scene, and is consistent with the expected lower limb motion, and can be connected to the previous motion, ensuring the coherence and coordination of the generated motion sequence, with good authenticity and naturalness.

[0150] In the embodiment of the present invention, the generated action sequence can change accordingly with the change of input information, thereby ensuring the diversity of action generation. At the same time, it can also make the generated action sequence fit the current scene, be consistent with the expected action, and be coherent and coordinated with the previous action, avoiding the problems of limited action generation and lack of authenticity and diversity in traditional solutions based on preset action libraries, and improving the vividness and realism of the generated action sequence.

[0151] Based on the above embodiment, action generation is performed based on the fusion features, the action probability distribution, and the lower body action features to obtain the current action sequence in the current scene, including:

[0152] Based on the body motion constraints in the current scene, the lower body motion sequence is applied to generate the upper body motion sequence in the current scene;

[0153] Feature extraction is performed based on the upper body action sequence to obtain upper body action features;

[0154] Action generation is performed based on fusion features, action probability distribution, and lower body action features and upper body action features to obtain the current action sequence in the current scene.

[0155] Specifically, the process of generating actions according to the fusion features, the action probability distribution, and the lower body action features to obtain the current action sequence in the current scene may specifically include:

[0156] Considering that in the process of normal body movements, the upper body generally changes accordingly with the changes in the lower limb movements. For example, when a person walks, his legs are moving, and it can be determined that his upper body will also move. It can be seen that there is an implicit constraint relationship between the upper and lower body movements. Therefore, after obtaining the lower body movement sequence, when generating movements based on it, the corresponding upper body movement sequence can be generated through this lower body movement sequence, and then the movement can be generated based on this upper and lower body movement sequence to ensure the consistency and coordination between the generated movement sequence and the expected movement.

[0157] Based on this, in an embodiment of the present invention, first, based on the preset lower body action sequence in the current scene, the upper body action sequence in the current scene can be generated, that is, based on this lower body action sequence, the corresponding upper body action sequence can be generated; and specifically when generating, it can be based on the constraint relationship between the upper and lower body actions in the usual body actions, that is, the body action constraints in the current scene can be referred to, and based on the lower body action sequence, a corresponding upper body action sequence can be generated, that is, the upper body action sequence in the current scene.

[0158] Here, the body motion constraints may be pre-set for the current scene, or obtained through learning / observation / analysis of a large number of whole-body motion sequences, or directly captured from existing motion constraint relationships and applicable to the current scene. The embodiments of the present invention do not make specific limitations on this.

[0159] After obtaining the upper body action sequence, it is also necessary to perform feature extraction on it to extract the key information about the upper limb movements in the upper body action sequence, so as to encode the upper body action features. That is, the upper body action sequence can be encoded to extract the upper body action features. Specifically, the lower body action sequence is taken as input, and the input upper body action sequence is encoded through the encoder, and the action-related information in the upper body action sequence is encoded to obtain the upper body action features output by the encoder.

[0160] After that, actions can be generated based on the upper body action features, lower body action features, fusion features and action probability distribution, so as to obtain the current action sequence in the current scene. That is, based on the fusion features, upper body action features and lower body action features, combined with the action probability distribution, the action at the current moment can be predicted, so as to obtain an action sequence that matches the style type and rhythmic rhythm of the audio in the current scene, is connected to the previous action sequence, and is coordinated with the expected action, that is, the current action sequence in the current scene.

[0161] Here, specifically, the upper body action features, lower body action features, fusion features and action probability distribution can be used as a benchmark, and the action at the current moment can be decoded through the action decoder to obtain the current action sequence. The current action sequence thus decoded not only has good scene adaptability (rhythmic consistency and style matching) and action coordination (coherence and coordination with the previous action sequence), but can also be consistent with the expected lower limb actions, and ensure the coordination of upper and lower limb actions, thereby achieving real and natural action generation.

[0162] Based on the above embodiment, action generation is performed based on the fusion features, the action probability distribution, and the lower body action features to obtain the current action sequence in the current scene, including:

[0163] Get action tags, which include key actions in the current scene and action attributes of key actions;

[0164] Action generation is performed based on fusion features, action probability distribution, lower body action features, style type and action labels to obtain the current action sequence.

[0165] Specifically, the process of generating actions according to the fusion features, the action probability distribution, and the lower body action features to obtain the current action sequence in the current scene may specifically include:

[0166] Taking into account that there will be some key actions, special actions, etc. at certain specific times in real scenes, such as raising hands, making a heart shape, cheering, etc., so as to better convey emotions and express feelings, therefore, in order to further fit with the actual usage scenarios and further meet personalized settings and needs, in the embodiment of the present invention, during the action generation process, under specific circumstances, some key actions can also be triggered to make the generated actions more flexible, real, and in line with actual conditions and user needs.

[0167] In view of this, in an embodiment of the present invention, the action label of the current scene can also be obtained, which includes the key actions that need to be added / triggered in the current scene, and the action attributes of each key action. Here, the key action can be one or more, and each key action has corresponding action attributes. The action attributes can include the action type of the key action, the body part corresponding to the action, the time of occurrence, the timing of triggering, etc.

[0168] Among them, the action label can be pre-set according to actual needs, and can be set for a specific scene. For example, based on the emotions and atmosphere reflected by the audio data of the current scene, and the theme and background displayed by the style type, the key actions that need to be triggered in the current scene can be determined, and corresponding action attributes can be generated for each key action.

[0169] Then, based on this action label, the above-mentioned fusion features, lower body action features, action probability distribution and style type can be combined to generate actions together to obtain the action sequence at the current moment, that is, the action decoder can be used to decode the action to predict the action at the current moment based on the fusion features, lower body action features, action probability distribution, action labels and style types, so as to obtain an action sequence that matches the current scene, connects to the previous action sequence, is consistent with the expected action, and can trigger the key action at the set time or trigger timing, that is, the current action sequence in the current scene.

[0170] Based on the above embodiment, action generation is performed based on fusion features, action probability distribution, lower body action features, style type and action label to obtain the current action sequence, including:

[0171] Determine the fused action features based on the action probability distribution, lower body action features, style type, and action label;

[0172] Based on the fused features and fused action features, action generation is performed to obtain the current action sequence.

[0173] Specifically, when generating actions based on fused features, action probability distribution, lower body action features, style types and action labels, the multimodal information can be fused first, that is, the action probability distribution, lower body action features, style types and action labels can be fused. By fusing this multimodal information, the fused features can be obtained. The features are used to guide subsequent action generation to generate an action sequence that matches it. Therefore, they can be regarded as guiding features of the action generation process and can be called fused action features.

[0174] After that, action generation can be performed based on this fused action feature and the above-mentioned fused features, and the action at the current moment can be decoded through the action decoder to obtain the current action sequence. Specifically here, the fused feature and the fused action feature are used as the input of the action decoder, so that the action decoder can decode the corresponding action sequence according to the input features, and use this action sequence as the current action sequence. The action sequence generated in this way not only has good scene adaptability (rhythmic consistency and style matching) and action coordination (coherence and coordination with the previous action sequence), but also can be coordinated with the expected action. At the same time, key actions can be triggered at the set time or trigger timing, which greatly meets personalized needs and settings, and can be closer to real usage scenarios, making the generated action sequence more flexible and realistic.

[0175] The method provided by the embodiment of the present invention obtains fused action features based on action probability distribution, lower body action features, style types, and action labels, and generates actions based on the fused action features and fused features. The generated action sequence can change with the change of input information, and can generate action sequences adapted to different inputs. It can be suitable for different scenes, different roles, different styles, different genders, etc. By effectively fusing input signals of different modes, it can generate natural and smooth action sequences that meet the semantic constraints of multimodal input information (scene adaptability constraints, coordination constraints with previous actions, consistency constraints with key actions, and coordination constraints between upper and lower body actions). This solves the problem that the current action generation method based on a preset action library cannot match the actual usage scenario and the generated actions lack authenticity and diversity.

[0176] Based on the above embodiment, obtaining the style type corresponding to the current scene includes:

[0177] Classify the audio data and / or the transcribed text corresponding to the audio data to obtain the emotion category and / or scene type;

[0178] Based on the emotion category and / or the scene type, the style type corresponding to the current scene is determined from a preset style type library.

[0179] Specifically, the process of obtaining the style type corresponding to the current scene may include:

[0180] Taking into account that the styles of different scenes may be quite different, for example, the style type of course teaching may tend to be quiet and rigorous, while the style type of shopping live broadcasts may be more noisy, enthusiastic, and exciting. Therefore, adding the style type of the scene in the process of action generation can make the generated action sequence more natural and appropriate, without being awkward or abrupt.

[0181] As for the style type of the scene, it can be pre-set according to actual needs, or it can be determined based on the information in the current scene. For example, the field corresponding to the current scene can be determined through audio data, text, background, etc., thereby determining the corresponding style type.

[0182] Specifically in the embodiments of the present invention, in order to enhance scene adaptability, the style type corresponding to the current scene can be determined through the information under the scene, that is, the audio data under the current scene and / or the transcribed text corresponding to the audio data can be classified to determine the emotion category and / or scene type. Specifically, the emotion classification can be performed based on the audio data to determine the emotion category, for example, happy, sad, excited, etc., and / or the audio data can be transcribed into text to transcribe the sound signal therein into text, thereby obtaining the transcribed text corresponding to the audio data, and then the scene classification can be performed according to the transcribed text, and the scene type of the current scene can be determined by the text type (live broadcast, teaching, product promotion, etc.) / text content (games, classes, driving, etc.) of the transcribed text.

[0183] Then, the style type corresponding to the current scene can be determined based on the emotion category obtained by emotion classification and / or the scene type obtained by scene classification. Specifically, the style type corresponding to the current scene can be selected from a preset style type library based on the scene type and / or emotion category. The style type library here contains multiple style types, as well as scenes, emotions, atmospheres, etc. that match each style type. That is, the style type corresponding to the scene type and / or emotion category is selected from the preset style type library as the style type corresponding to the current scene. The selected style type can be one or more, and the embodiments of the present invention do not specifically limit this.

[0184] Based on the above embodiment, in step 120, the audio features and action features obtained by feature extraction are fused to obtain fused features, including:

[0185] Performing time domain alignment processing on the audio features and the action features to obtain aligned audio features and action features;

[0186] Based on the correlation between the aligned audio features and the action features, the aligned audio features and the action features are fused to obtain fused features.

[0187] Specifically, the process of fusing the audio features and the action features obtained by feature extraction to obtain the fused features may specifically include:

[0188] After the audio features and action features are obtained by feature extraction, the extracted audio features and action features can be fused to obtain fused features. Since the audio data and the previous action sequence do not correspond in the time domain, the previous action sequence is the action before the current moment, and the audio data is real-time speech, the audio data must be ahead of the previous action sequence in the time domain. Therefore, in order to enable the subsequent action generation to be executed steadily and to ensure the scene adaptability and coherence and coordination of the generated action sequence, in an embodiment of the present invention, when fusing features of different modalities, they must first be aligned, that is, the audio features and action features must be aligned in the time domain so that the audio features and action features that do not correspond in the time domain can be aligned, thereby obtaining aligned audio features and action features.

[0189] Afterwards, the aligned audio features and action features can be fused, that is, the attention mechanism (such as the self-attention layer of Transformer) can be used to fuse features to obtain fused features. Since the feature fusion based on the attention mechanism essentially enables the fusion process to pay attention to the action-related information represented by different features, the fused features obtained by fusion include both the rhythmic rhythm of the audio and the key actions of the preceding action sequence, which can effectively ensure the vividness, realism and naturalness of the subsequent action generation.

[0190] It should be noted that the feature fusion based on the attention mechanism essentially considers the correlation between features. Therefore, it can be understood here that the aligned audio features and action features are fused according to the correlation between them to obtain the fused features. This fused feature not only contains the information of the action-related areas in the action features, but also includes the rhythmic rhythm represented by the audio features. Moreover, fusion based on correlation can add subtle features of the action-related areas in the current scene to the fused features. This subtle feature can more accurately distinguish between behavioral actions with similar action features, thereby improving the distinguishability, diversity and flexibility of the generated action sequences.

[0191] Figure 3 It is the overall framework diagram of the action generation method provided by the present invention, such as Figure 3 As shown, the present invention proposes a method for synthesizing actions based on multimodal input information. The main process includes feature extraction, feature fusion and action generation of information of different modalities to achieve cross-style, cross-role and cross-gender action generation. Compared with the traditional solution based on preset action library, the action generation method provided by the present invention greatly improves the authenticity and diversity of the generated action sequence, while ensuring timeliness. The specific process includes:

[0192] First, obtain the audio data of the current scene and the preceding action sequence.

[0193] Then, feature extraction is performed based on the audio data and the preceding action sequence respectively, and the audio features and action features obtained by the feature extraction are fused to obtain fused features.

[0194] The audio features and motion features obtained by feature extraction are fused to obtain fused features, including: performing time domain alignment processing on the audio features and motion features to obtain aligned audio features and motion features; based on the correlation between the aligned audio features and motion features, the aligned audio features and motion features are fused to obtain fused features.

[0195] Subsequently, action generation is performed based on the fused features to obtain the current action sequence in the current scene.

[0196] Among them, action generation is performed based on fusion features to obtain the current action sequence in the current scene, including: obtaining the style type corresponding to the current scene; determining the action probability distribution based on audio features and style types; and generating actions based on fusion features and action probability distribution to obtain the current action sequence in the current scene.

[0197] Obtaining the style type corresponding to the current scene includes: classifying based on the audio data and / or the transcribed text corresponding to the audio data to obtain the emotion category and / or scene type; and determining the style type corresponding to the current scene from a preset style type library based on the emotion category and / or scene type.

[0198] Here, based on the audio features and style types, the action probability distribution is determined, including: based on the audio features and style types, an action prediction model is applied to perform action prediction to obtain the action probability distribution corresponding to the audio data and style type; the action prediction model is trained on the basis of the initial model using sample audio data in the sample scenario and the corresponding sample style type.

[0199] The training steps of the action prediction model include: based on the initial model, determining the sample action probability distribution corresponding to the sample audio data and the sample style type; selecting positive samples and negative samples from the sample audio data and the corresponding sample style types, the positive samples are at least two sample audio data and their corresponding sample style types in the adjacent time domain under the same sample scene, and the negative samples are at least two sample audio data and their corresponding sample style types in different sample scenes and / or non-adjacent time domains; based on the similarity between the sample action probability distributions corresponding to each sample audio data and its corresponding sample style type in the positive sample, and the similarity between the sample action probability distributions corresponding to each sample audio data and its corresponding sample style type in the negative sample, iterating the parameters of the initial model to obtain the action prediction model.

[0200] Here, the initial model includes a downsampling module, a residual module and an upsampling module; the downsampling module is used to downsample the sample fusion features, and the sample fusion features are obtained by fusion of the sample audio features of the sample audio data and the corresponding sample style type; the residual module is used to determine the initial action probability distribution based on the features obtained by downsampling; the upsampling module is used to upsample the initial action probability distribution to obtain the sample action probability distribution corresponding to the sample audio data and the sample style type.

[0201] Among them, action generation is performed based on fusion features and action probability distribution to obtain a current action sequence in the current scene, including: obtaining a preset lower body action sequence in the current scene; performing feature extraction based on the lower body action sequence to obtain lower body action features; and performing action generation based on fusion features, action probability distribution, and lower body action features to obtain the current action sequence in the current scene.

[0202] Furthermore, action generation is performed based on the fused features, action probability distribution, and lower body action features to obtain a current action sequence in the current scene, including: based on the body action constraints in the current scene, applying the lower body action sequence to generate an upper body action sequence in the current scene; performing feature extraction based on the upper body action sequence to obtain upper body action features; performing action generation based on the fused features, action probability distribution, and lower body action features and upper body action features to obtain the current action sequence in the current scene.

[0203] Furthermore, based on the fusion features, action probability distribution, and lower body action features, action generation is performed to obtain a current action sequence in the current scene, including: obtaining action labels, the action labels include key actions in the current scene, and action attributes of the key actions; based on the fusion features, action probability distribution, lower body action features, style types and action labels, action generation is performed to obtain the current action sequence.

[0204] Among them, based on the fusion features, action probability distribution, lower body action features, style types and action labels, action generation is performed to obtain the current action sequence, including: based on the action probability distribution, lower body action features, style types, and action labels, the fusion action features are determined; based on the fusion features and the fusion action features, action generation is performed to obtain the current action sequence.

[0205] Compared with traditional action generation schemes, the action generation method based on multimodal input information provided by the present invention can better model the association between the output action sequence and the input information, so that the generated action sequence is more consistent with the semantic information in the real scene, which is extremely critical to improving the authenticity of virtual humans and 3D animations.

[0206] The present invention is used to generate virtual digital humans, which can improve the efficiency of generating virtual digital humans compared to manual modeling by modelers. In addition, the virtual digital humans have high-quality facial modeling. The hyper-realistic virtual humans thus obtained can be used for user interaction, virtual digital human videos, and virtual digital human services, so that users can visually experience the same experience as real-life interaction, real-life videos, and real-life services, thus avoiding the problems of serious homogeneity of character settings, low quality, and low efficiency of current virtual digital humans.

[0207] Moreover, in the embodiments of the present invention, the extensibility and universality of different styles, different scenes and semantic actions can be better guaranteed through multimodal joint modeling; through style types, different styles, genders, scenes and other diverse needs can be better customized; through lower body action sequences, actions within a specific motion range can be synthesized according to spatial constraints such as the current scene and stage; through action tags, specified key actions can be generated at any time, which improves the controllability and richness of action generation.

[0208] The method provided by the embodiment of the present invention, the action generation method provided by the present invention, extracts audio features and action features through the audio data and the previous action sequence in the current scene, and fuses the audio features and action features to obtain fused features, generates actions based on the fused features, and obtains the current action sequence, thereby achieving matching of the action sequence with the scene audio, allowing the generated action sequence to change with the change of input information, achieving the movement of the body with the sound, ensuring the diversity of action generation, and the continuity and coordination between the current action sequence and the previous action sequence, overcoming the defects of the lack of authenticity and diversity of the actions generated in the traditional scheme, and effectively improving the authenticity, timeliness and naturalness of the action generation.

[0209] The present invention also provides an action generation method. Figure 4 This is the second flow chart of the action generation method provided by the present invention. Figure 4 As shown, the method includes:

[0210] Step 410, obtaining first modal data and second modal data in the current scene, where the first modal data is used to reflect scene information of the current scene;

[0211] Step 420, extracting features from the first modal data to obtain first modal features, and extracting features from the second modal data to obtain second modal features;

[0212] Step 430, generating actions based on the first modal feature and the second modal feature to obtain a current action sequence in the current scene.

[0213] Specifically, in order to avoid the defects of traditional solutions based on preset action libraries in action generation, realize real, natural and diversified action generation, and ensure the adaptability of the generated action sequence to the scene, in the embodiments of the present invention, when generating actions, the information under the specific scene can be considered. By incorporating the scene information, it can be used as a reference in the subsequent action generation link to generate corresponding actions to ensure that they are consistent and compatible with the scene.

[0214] In view of this, in an embodiment of the present invention, before performing action generation, data that can characterize the scene information of the current scene, i.e., first modal data, can be first obtained. The first modal data here can be audio data, text data, background data, etc., and the embodiment of the present invention does not specifically limit this. Among them, the first modal data can be directly obtained from the scene background. For example, if the background-related data of the current scene is pre-set, the first modal data can be obtained from the background-related data; the first modal data can also be obtained by picking up a sound pickup device, for example, by picking up the current scene in real time through a sound pickup device (microphone, recorder, pickup, etc.) to obtain the first modal data; the first modal data can also be obtained by transcribing the picked-up audio data, or directly obtained from the background-related data of the current scene, for example, obtaining text data from the background-related data.

[0215] Here, the scene that needs to generate action can be various fields, for example, the scene corresponding to education, games, transportation, new media, etc., specifically, it can be course teaching, simulation games, rail transportation, film and television interpretation, shopping live broadcast, etc. Due to the differences in style and atmosphere of different scenes, in order to better adapt to specific scenes, so that the generated action can be accurately integrated with the scene without being awkward, etc., in the embodiment of the present invention, before generating the action, in addition to the first modal data representing the scene information, it is also necessary to obtain the second modal data in the current scene. The second modal data here is not only different from the first modal data in modality, but also has different emphasis on information feedback. The second modal data focuses on reflecting the type, style, action characteristics, etc. of the current scene. It can be the historical action in the current scene, the style characteristics of the current scene, the scene type, etc., and the embodiment of the present invention does not make specific limitations on this.

[0216] After the first modal data and the second modal data are obtained, feature extraction may be performed on them respectively to obtain first modal features of the first modal data and second modal features of the second modal data.

[0217] It is understandable that after obtaining the first modal data and the second modal data, before performing action generation based on them, feature extraction is required to be performed on them first, so as to respectively extract features in the first modal data that can be helpful for action generation, and features in the second modal data that are related to the style, type, action, etc. of the current scene, so as to ensure that the action sequence generated during subsequent action generation can be consistent with the scene information of the current scene, and be coherent and coordinated with the style, type, action, etc. of the current scene, thereby achieving real and natural action generation.

[0218] Specifically, the first modal data and the second modal data may be encoded respectively to extract features of the first modal data and features of the second modal data, thereby obtaining first modal features of the first modal data and second modal features of the second modal data. It is worth noting here that the encoder for encoding the first modal data and the encoder for encoding the second modal data may be the same or different. Preferably, in an embodiment of the present invention, in order to be able to extract features from different data more efficiently and accurately, different encoders are selected to encode the first modal data and the second modal data.

[0219] Specifically, an encoder adapted to the data characteristics of the first modal data is selected as the first modal encoder (such as selecting an encoder adapted to the rhythm, cadence, volume and other characteristics of the audio as the audio encoder), and an encoder adapted to the data characteristics of the second modal data is selected as the second modal encoder (such as selecting an encoder adapted to the coherence, coordination and other characteristics of the action as the action encoder). The first modal data is encoded by the first modal encoder to obtain the first modal features output by the first modal encoder, and the second modal data is encoded by the second modal encoder to obtain the second modal features output by the second modal encoder.

[0220] Afterwards, the extracted first modal features and second modal features can be used to generate actions to obtain the required action sequence, that is, the current action sequence in the current scene. Based on the generated actions at each moment, the complete action sequence in the current scene can be obtained. In this way, the action generation in the current scene is realized, which has good vividness, realism and real-time performance, and realizes real and natural action generation.

[0221] It is understandable that after obtaining the first modal feature and the second modal feature, when generating an action, the first modal feature and the second modal feature can be fused to obtain a fused feature. Since the modalities of the first modal data and the second modal data are different, the extracted first modal feature and the second modal feature also correspond to different modalities. Therefore, the process of fusing the two is essentially fusing information of different modalities. When fusing, attention can be paid to information of multiple modalities, thereby obtaining a fused feature containing information of multiple modalities.

[0222] Specifically, a multimodal encoder may be used here to fuse the first modal feature and the second modal feature, that is, the first modal feature and the second modal feature may be used as input, and the input first modal feature and the second modal feature may be processed by the multimodal encoder so that the first modal feature and the second modal feature can be aligned in the time domain, and both of them may be enhanced to obtain a fused multimodal feature, that is, a fused feature. The multimodal encoder here is essentially an encoder that can jointly model the input first modal feature and the second modal feature, and output the multimodal feature after the joint modeling.

[0223] After obtaining the fused multimodal features through feature fusion, action generation can be performed based on the fused features containing multimodal information to obtain an action sequence that matches the first modal data and the second modal data in the current scene, that is, the action sequence at the current moment in the current scene, which is called the current action sequence.

[0224] Specifically, after the fusion feature is obtained by fusion of the encoder, when the action is generated, the fusion feature can be decoded to decode the action, so as to obtain the current action sequence, that is, the fusion feature can be input into the decoder, so that the decoder decodes the fusion feature and outputs the action sequence, so as to obtain the current action sequence in the current scene. Here, the decoder that decodes the fusion feature can be an action decoder, that is, based on the fusion feature, the action can be generated by the action decoder to obtain the current action sequence in the current scene output by the action decoder.

[0225] In an embodiment of the present invention, the generated action sequence can change accordingly with the change of the input first modal data and the second modal data, thereby ensuring the diversity of action generation. At the same time, it can also make the generated action sequence coordinated and adapted to the current scene, avoiding the problems of limited action generation and lack of authenticity and diversity in traditional solutions based on preset action libraries, and improving the vividness and realism of the generated action sequence.

[0226] It should be noted that the method provided by the present invention can be applied to an artificial intelligence-based adolescent mental health diagnosis and treatment system. For example, if the present invention is used to generate a virtual digital person, the generated virtual person can provide companionship and psychological health treatment to adolescents, which greatly improves the current problem that the companionship and psychological health treatment of adolescents are completely dependent on human intervention, ensures the timeliness and frequency of psychological monitoring of adolescents, and can effectively improve the current situation of frequent psychological problems among adolescents and improve the efficiency of mental health diagnosis and treatment.

[0227] The action generation method provided by the present invention obtains the first modal feature and the second modal feature by respectively extracting features from the first modal data and the second modal data in the current scene, generates actions according to the first modal feature and the second modal feature, and obtains the current action sequence in the current scene, wherein the first modal data is used to reflect the scene information in the current scene, realizes the adaptation of the action sequence and the scene, and can make the generated action sequence change with the change of input information, thereby ensuring the diversity of action generation and the adaptability and coordination with the scene, overcoming the defects of lack of authenticity and diversity of the actions generated in the traditional scheme, and effectively improving the authenticity, timeliness and naturalness of the action generation.

[0228] Based on the above embodiment, the first modal data is audio data in the current scene, and the first modal feature is an audio feature of the audio data;

[0229] The second modal data is a preceding action sequence in the current scene, and the second modal feature is an action feature of the preceding action sequence; or, the second modal data is a style type corresponding to the current scene, and the second modal feature is a style feature.

[0230] Specifically, considering that in actual scenarios, the information represented by voice is usually more sufficient, comprehensive and accurate, and is more helpful in understanding the user's intentions and generating corresponding action sequences. For example, voice can more directly reflect the atmosphere of the current scene, user emotions, attitudes, etc.; therefore, in an embodiment of the present invention, in order to simplify the difficulty of information acquisition, ensure the simplicity and convenience of collection, and the timeliness of action generation, the first modal data used to reflect the scene information can be determined as audio data, that is, the audio data in the current scene is obtained.

[0231] The audio data here can be understood as real-time speech in the current scene, which can be obtained by real-time audio pickup through a sound pickup device. The sound pickup device here can be a microphone, a recorder, a pickup, etc.; it can be set / installed in the current scene to pick up sound in a specific area, range, etc. in the current scene, or it can be carried by the people in the current scene to pick up human voices better and more accurately. The embodiments of the present invention do not make specific limitations on this.

[0232] Correspondingly, when the first modal data is audio data, the first modal feature obtained by feature extraction of the first modal data is an audio feature; that is, the audio data can be encoded by an audio encoder to extract the audio features of the audio data. Specifically, the audio data can be taken as input and the audio encoder can be used to encode the input audio data to extract rhythm and tempo related features in the audio data. That is, information related to rhythm and tempo in the audio data can be encoded by the audio encoder to obtain the audio features output by the audio encoder.

[0233] It is worth noting that after obtaining the audio data, when generating actions, although the audio data can ensure the adaptation of the generated action sequence to the scene, it cannot guarantee its coherence and coordination with the previous actions. This will undoubtedly cause the newly generated action sequence to have a gap in connection, unable to be coherent with the previous actions, and poor vividness and naturalness.

[0234] Therefore, in an embodiment of the present invention, when generating actions, in order to ensure the continuity and coordination of the actions, the second modal data can be set as the previous action sequence in the current scene, that is, the action sequence before the current moment, which is called the previous action sequence of the current action sequence. This previous action sequence is essentially an action sequence composed of historical actions relative to the current moment, and the process of action generation is based on the historical action sequence and the audio data in the current scene. The process can obtain the scene information adapted to the current scene, and the action sequence that is coherent and coordinated with the previous action sequence in the current scene, thereby achieving efficient and accurate action generation and ensuring the authenticity, naturalness and timeliness of action generation.

[0235] Correspondingly, when the second modal data is a preceding action sequence, the second modal feature obtained by performing feature extraction on the second modal data is an action feature; that is, the preceding action sequence can be encoded by an action encoder to extract the action feature of the preceding action sequence. Specifically, the preceding action sequence can be taken as input, and the input preceding action sequence can be encoded by an action encoder to encode the action-related information in the preceding action sequence, thereby obtaining the action feature output by the encoder.

[0236] In an embodiment of the present invention, when the first modal data is audio data and the second modal data is a preceding action sequence, action generation is performed based on the first modal feature of the first modal data and the second modal feature of the second modal data. This ensures the scene adaptability of the generated action sequence, as well as the continuity and coordination with the preceding action sequence, thereby achieving real and natural action generation and ensuring the diversity of action generation.

[0237] In addition, it should be noted that, considering the actual situation that the styles required for different scenes are likely to be different, for example, the style type for course teaching may tend to be quiet, rigorous, etc., while the style type for shopping live broadcasts is more noisy, enthusiastic, etc. In an embodiment of the present invention, in order to more accurately adapt to specific scenes and make the generated action sequence compatible with the scene style, the second modal data can be set to the style type of the current scene.

[0238] That is, the second modal data can also be a style type, and this style type corresponds to the current scene. For example, when the current scene is a simulation game scene, the style type may be comic style, science fiction style, magic style, etc. When the current scene is course teaching, the corresponding style type may be experimental style, historical style, etc. When the current scene is cultural popularization, the corresponding style type may be humanistic style, natural style, etc., and when the current scene is a film and television interpretation, the corresponding style type may be comedy style, tragedy style, modern style, etc.

[0239] Here, the style type can be pre-set, and then when generating actions for the current scene, the style type that matches the current scene can be directly selected from the set various style types, and the selected style type can be one or more, and the embodiment of the present invention does not specifically limit this. For example, when the current scene is a film and television interpretation, the selected style type can be a comedy style or a tragedy style, or a comedy style and a modern style, or a tragedy style and a modern style. As a preference, when the set style type supports, multiple style types are preferred for the current scene, so that the generated action sequence is more realistic and closer to the real scene through the fusion of multiple styles. In addition, the style type can also be determined based on the information in the specific scene. For example, the field corresponding to the current scene can be determined based on the audio data, text, background, etc. in the current scene, thereby determining the corresponding style type.

[0240] When the second modal data is a style type, the corresponding second modal feature can be a style feature. Here, the style feature is not a feature directly extracted from the style type, but a feature obtained by feature extraction based on the style type and combined with other data in the current scene. For example, it can be an action probability distribution obtained by performing action prediction based on the features obtained by fusion based on the style type and feature extraction and fusion in combination with the audio data in the current scene; it can also be based on the action probability distribution, and the action features and / or action labels (including key actions in the current scene, and action attributes of key actions) of the preset lower body action sequence in the current scene are fused. The embodiments of the present invention do not specifically limit this.

[0241] The action generation device provided by the present invention is described below. The action generation device described below and the action generation method described above can be referenced to each other.

[0242] Figure 5 is one of the structural schematic diagrams of the action generation device provided by the present invention, such as Figure 5 As shown, the device comprises:

[0243] An acquisition unit 510 is used to acquire audio data in the current scene and a preceding action sequence;

[0244] A fusion unit 520, configured to extract features based on the audio data and the preceding action sequence, respectively, and fuse the audio features and action features obtained by the feature extraction to obtain fused features;

[0245] The generating unit 530 is used to generate actions based on the fusion features to obtain a current action sequence in the current scene.

[0246] The action generation device and the action generation method provided by the present invention extract audio features and action features through audio data and previous action sequences in the current scene, fuse the audio features and action features to obtain fused features, generate actions based on the fused features, and obtain the current action sequence, thereby achieving matching between the action sequence and the scene audio, allowing the generated action sequence to change with changes in input information, achieving body movement with sound, ensuring diversity in action generation, and continuity and coordination between the current action sequence and the previous action sequence, overcoming the defects of lack of authenticity and diversity in actions generated in traditional solutions, and effectively improving the authenticity, timeliness and naturalness of action generation.

[0247] Based on the above embodiment, the generating unit 530 is used for:

[0248] Get the style type corresponding to the current scene;

[0249] Determining an action probability distribution based on the audio feature and the style type;

[0250] Action generation is performed based on the fusion feature and the action probability distribution to obtain a current action sequence in the current scene.

[0251] Based on the above embodiment, the generating unit 530 is used for:

[0252] Based on the audio features and the style type, applying an action prediction model to perform action prediction to obtain an action probability distribution corresponding to the audio data and the style type;

[0253] The action prediction model is obtained by training on the basis of the initial model using sample audio data in a sample scene and corresponding sample style types; the initial model is constructed on the basis of a downsampling layer.

[0254] Based on the above embodiment, the device further includes a training unit, which is used to:

[0255] Based on the initial model, determining a sample action probability distribution corresponding to the sample audio data and the sample style type;

[0256] Selecting positive samples and negative samples from the sample audio data and their corresponding sample style types, wherein the positive samples are at least two sample audio data and their corresponding sample style types in adjacent time domains under the same sample scene, and the negative samples are at least two sample audio data and their corresponding sample style types in different sample scenes and / or non-adjacent time domains;

[0257] Based on the similarity between the sample action probability distribution corresponding to each sample audio data and its corresponding sample style type in the positive sample, and the similarity between the sample action probability distribution corresponding to each sample audio data and its corresponding sample style type in the negative sample, the initial model is iterated on parameters to obtain an action prediction model.

[0258] Based on the above embodiment, the initial model includes a downsampling module, a residual module and an upsampling module;

[0259] The downsampling module is used to downsample the sample fusion feature, where the sample fusion feature is obtained by fusion of the sample audio feature of the sample audio data and the corresponding sample style type;

[0260] The residual module is used to determine the initial action probability distribution based on the features obtained by downsampling; the upsampling module is used to upsample the initial action probability distribution to obtain the sample action probability distribution corresponding to the sample audio data and the sample style type.

[0261] Based on the above embodiment, the generating unit 530 is used for:

[0262] Obtaining a preset lower body action sequence in the current scene;

[0263] Perform feature extraction based on the lower body action sequence to obtain lower body action features;

[0264] Action generation is performed based on the fusion feature, the action probability distribution, and the lower body action feature to obtain a current action sequence in the current scene.

[0265] Based on the above embodiment, the generating unit 530 is used for:

[0266] Based on the body movement constraints in the current scene, applying the lower body movement sequence to generate an upper body movement sequence in the current scene;

[0267] Extract features based on the upper body action sequence to obtain upper body action features;

[0268] Action generation is performed based on the fusion feature, the action probability distribution, and the lower body action feature and the upper body action feature to obtain a current action sequence in the current scene.

[0269] Based on the above embodiment, the generating unit 530 is used for:

[0270] Acquire an action tag, where the action tag includes a key action in the current scene and an action attribute of the key action;

[0271] Action generation is performed based on the fusion feature, the action probability distribution, the lower body action feature, the style type and the action label to obtain a current action sequence.

[0272] Based on the above embodiment, the generating unit 530 is used for:

[0273] Determine a fused action feature based on the action probability distribution, the lower body action feature, the style type, and the action label;

[0274] Action generation is performed based on the fused features and the fused action features to obtain a current action sequence.

[0275] Based on the above embodiment, the generating unit 530 is used for:

[0276] Classify the audio data and / or the transcribed text corresponding to the audio data to obtain an emotion category and / or a scene type;

[0277] Based on the emotion category and / or the scene type, the style type corresponding to the current scene is determined from a preset style type library.

[0278] Based on the above embodiment, the fusion unit 520 is used for:

[0279] Performing time domain alignment processing on the audio feature and the action feature to obtain aligned audio feature and action feature;

[0280] Based on the correlation between the aligned audio features and the action features, the aligned audio features and the action features are fused to obtain fused features.

[0281] Figure 6This is the second structural diagram of the action generation device provided by the present invention. Figure 6 As shown, the device comprises:

[0282] The data acquisition unit 610 is used to acquire first modal data and second modal data in a current scene, wherein the first modal data is used to reflect scene information of the current scene;

[0283] A feature extraction unit 620, configured to perform feature extraction on the first modal data to obtain a first modal feature, and perform feature extraction on the second modal data to obtain a second modal feature;

[0284] The action generation unit 630 is used to generate an action based on the first modal feature and the second modal feature to obtain a current action sequence in the current scene.

[0285] The action generation device provided by the present invention obtains the first modal feature and the second modal feature by respectively extracting features from the first modal data and the second modal data in the current scene, generates actions according to the first modal feature and the second modal feature, and obtains the current action sequence in the current scene, wherein the first modal data is used to reflect the scene information in the current scene, thereby realizing the adaptation of the action sequence and the scene, and allowing the generated action sequence to change with the change of input information, thereby ensuring the diversity of action generation as well as the adaptability and coordination with the scene, overcoming the defects of lack of authenticity and diversity of the actions generated in traditional solutions, and effectively improving the authenticity, timeliness and naturalness of action generation.

[0286] Based on the above embodiment, the first modal data is audio data in the current scene, and the first modal feature is an audio feature of the audio data;

[0287] The second modal data is a preceding action sequence in the current scene, and the second modal feature is an action feature of the preceding action sequence; or, the second modal data is a style type corresponding to the current scene, and the second modal feature is a style feature.

[0288] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the action generation method, which includes: obtaining audio data and a preceding action sequence in the current scene; performing feature extraction based on the audio data and the preceding action sequence respectively, and fusing the audio features and action features obtained by the feature extraction to obtain a fused feature; performing action generation based on the fused feature to obtain the current action sequence in the current scene.

[0289] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0290] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the action generation method provided by the above-mentioned methods, and the method includes: obtaining audio data and a preceding action sequence in a current scene; performing feature extraction based on the audio data and the preceding action sequence, respectively, and fusing the audio features and action features obtained by the feature extraction to obtain fused features; performing action generation based on the fused features to obtain the current action sequence in the current scene.

[0291] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the action generation method provided by the above-mentioned methods, the method comprising: obtaining audio data in a current scene, and a preceding action sequence; performing feature extraction based on the audio data and the preceding action sequence, respectively, and fusing the audio features and action features obtained by the feature extraction to obtain fused features; performing action generation based on the fused features to obtain the current action sequence in the current scene.

[0292] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0293] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0294] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An action generation method, characterized in that: include: Acquire audio data in the current scene and the preceding action sequence; the current scene is a real usage scene, and the audio data is real-time voice collected by a sound pickup device; Extracting features based on the audio data and the preceding action sequence respectively, and fusing the audio features and action features obtained by the feature extraction to obtain fused features; Performing action generation based on the fusion features to obtain a current action sequence in the current scene; The step of generating an action based on the fusion feature to obtain a current action sequence in the current scene includes: Get the style type corresponding to the current scene; Determining an action probability distribution based on the audio feature and the style type; Performing action generation based on the fusion feature and the action probability distribution to obtain a current action sequence in the current scene; The obtaining of the style type corresponding to the current scene includes: Based on the audio data and / or the transcribed text corresponding to the audio data, classification is performed to obtain an emotion category and / or a scene type; the emotion category is an emotion category of the current user, and the scene type is a scene type of the actual usage scenario; Based on the emotion category and / or the scene type, the style type corresponding to the current scene is determined from a preset style type library.

2. The action generation method according to claim 1, characterized in that: The determining of the action probability distribution based on the audio feature and the style type comprises: Based on the audio features and the style type, applying an action prediction model to perform action prediction to obtain an action probability distribution corresponding to the audio data and the style type; The action prediction model is obtained by training on the basis of the initial model using sample audio data in sample scenes and corresponding sample style types.

3. The action generation method according to claim 2, characterized in that: The action prediction model is trained based on the following steps: Based on the initial model, determining a sample action probability distribution corresponding to the sample audio data and the sample style type; Selecting positive samples and negative samples from the sample audio data and their corresponding sample style types, wherein the positive samples are at least two sample audio data and their corresponding sample style types in adjacent time domains under the same sample scene, and the negative samples are at least two sample audio data and their corresponding sample style types in different sample scenes and / or non-adjacent time domains; Based on the similarity between the sample action probability distribution corresponding to each sample audio data and its corresponding sample style type in the positive sample, and the similarity between the sample action probability distribution corresponding to each sample audio data and its corresponding sample style type in the negative sample, the initial model is iterated on parameters to obtain an action prediction model.

4. The action generation method according to claim 3, characterized in that: The initial model includes a downsampling module, a residual module and an upsampling module; The downsampling module is used to downsample the sample fusion feature, where the sample fusion feature is obtained by fusion of the sample audio feature of the sample audio data and the corresponding sample style type; The residual module is used to determine the initial action probability distribution based on the features obtained by downsampling; the upsampling module is used to upsample the initial action probability distribution to obtain the sample action probability distribution corresponding to the sample audio data and the sample style type.

5. The motion generation method according to any one of claims 1 to 4, characterized in that: The generating of actions based on the fusion feature and the action probability distribution to obtain a current action sequence in the current scene includes: Obtaining a preset lower body action sequence in the current scene; Perform feature extraction based on the lower body action sequence to obtain lower body action features; Action generation is performed based on the fusion feature, the action probability distribution, and the lower body action feature to obtain a current action sequence in the current scene.

6. The action generation method according to claim 5, characterized in that: The generating of actions based on the fusion feature, the action probability distribution, and the lower body action feature to obtain a current action sequence in the current scene includes: Based on the body movement constraints in the current scene, applying the lower body movement sequence to generate an upper body movement sequence in the current scene; Perform feature extraction based on the upper body action sequence to obtain upper body action features; Action generation is performed based on the fusion feature, the action probability distribution, and the lower body action feature and the upper body action feature to obtain a current action sequence in the current scene.

7. The action generation method according to claim 5, characterized in that: The generating of actions based on the fusion feature, the action probability distribution, and the lower body action feature to obtain a current action sequence in the current scene includes: Acquire an action tag, where the action tag includes a key action in the current scene and an action attribute of the key action; Action generation is performed based on the fusion feature, the action probability distribution, the lower body action feature, the style type and the action label to obtain a current action sequence.

8. The action generation method according to claim 7, characterized in that: The step of generating an action based on the fusion feature, the action probability distribution, the lower body action feature, the style type and the action label to obtain a current action sequence includes: Determine a fused action feature based on the action probability distribution, the lower body action feature, the style type, and the action label; Action generation is performed based on the fused features and the fused action features to obtain a current action sequence.

9. The motion generation method according to any one of claims 1 to 4, characterized in that: The step of fusing the audio features and the action features obtained by feature extraction to obtain fused features includes: Performing time domain alignment processing on the audio feature and the action feature to obtain aligned audio feature and action feature; Based on the correlation between the aligned audio features and the action features, the aligned audio features and the action features are fused to obtain fused features.

10. An action generation method, characterized in that: include: Acquire first modal data and second modal data in a current scene, where the first modal data is used to reflect scene information of the current scene; Performing feature extraction on the first modal data to obtain a first modal feature, and performing feature extraction on the second modal data to obtain a second modal feature; Generating an action based on the first modal feature and the second modal feature to obtain a current action sequence in the current scene; The first modal data is audio data in the current scene, and the first modal feature is an audio feature of the audio data; The current scenario is a real usage scenario, and the audio data is real-time speech collected by a sound pickup device; The second modal data is a style type corresponding to the current scene, and the second modal data is determined based on the following steps: Determining a field corresponding to the current scene based on at least one of the audio data, text data, and background data in the current scene; Based on the field, determining the style type corresponding to the current scene from a preset style type library; The second modal feature is a style feature, and the style feature is determined based on the audio feature and the style type.

11. An action generating device, characterized in that: include: An acquisition unit, used to acquire audio data in a current scene and a preceding action sequence; the current scene is a real usage scene, and the audio data is real-time speech collected by a sound pickup device; A fusion unit, configured to extract features based on the audio data and the preceding action sequence, respectively, and fuse the audio features and action features obtained by the feature extraction to obtain fused features; A generating unit, configured to generate an action based on the fusion feature to obtain a current action sequence in the current scene; The generating unit is used for: Get the style type corresponding to the current scene; Determining an action probability distribution based on the audio feature and the style type; Performing action generation based on the fusion feature and the action probability distribution to obtain a current action sequence in the current scene; The generating unit is further configured to: Based on the audio data and / or the transcribed text corresponding to the audio data, classification is performed to obtain an emotion category and / or a scene type; the emotion category is an emotion category of the current user, and the scene type is a scene type of the actual usage scenario; Based on the emotion category and / or the scene type, the style type corresponding to the current scene is determined from a preset style type library.

12. An action generating device, characterized in that: include: A data acquisition unit, used to acquire first modal data and second modal data in a current scene, wherein the first modal data is used to reflect scene information of the current scene; a feature extraction unit, configured to perform feature extraction on the first modal data to obtain a first modal feature, and perform feature extraction on the second modal data to obtain a second modal feature; an action generation unit, configured to generate an action based on the first modal feature and the second modal feature to obtain a current action sequence in the current scene; The first modal data is audio data in the current scene, and the first modal feature is an audio feature of the audio data; The current scenario is a real usage scenario, and the audio data is real-time speech collected by a sound pickup device; The second modal data is a style type corresponding to the current scene, and the second modal data is determined based on the following steps: Determining a field corresponding to the current scene based on at least one of the audio data, text data, and background data in the current scene; Based on the field, determining the style type corresponding to the current scene from a preset style type library; The second modal feature is a style feature, and the style feature is determined based on the audio feature and the style type.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the action generation method according to any one of claims 1 to 10 is implemented.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the action generation method according to any one of claims 1 to 10 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the action generation method according to any one of claims 1 to 10 is implemented.

Images