Text and music multi-mode driving-based action sequence generation device and method

Abstract

The invention provides an action sequence generation device and method based on text and music multi-mode driving, and relates to the technical field of action sequence generation. The item comprises an action generation module and an application module. The working process is as follows: firstly, a user inputs to-be-processed text description and music in a client; then, a Web server deployed by adopting flash receives text description, action duration and audio files sent by the Avalia client, calls an action generation module, converts the data into npy action data and further generates an SMPL skeleton model and an MP4 preview video, and meanwhile, the Web server communicates with the client, compresses the generated files and sends the compressed files to the Avalia client; and finally, the user selects an export format, and the generated record is stored in a video table of the remote MySQL database, so that the generation of the action sequence is completed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of action sequence generation, in particular to an action sequence generation device and method based on text and music multi-modal driving. BACKGROUND

[0002] 3D animation design is a key technology for realizing visual effects and interactive experiences in the field of digital media. Currently, in the market, animation production requires high time cost, and the diversity of the produced actions is insufficient. Existing animation generation technologies are difficult to generate actions that conform to the scene according to the context. Traditional 3D model animation generation technologies have high thresholds, and the generated actions lack diversity.

[0003] In order to realize efficient and diversified action generation, the method of combining long text with multi-modal driving as a generation condition has good effect. Chinese patent "CN115631275A multi-modal driven human action sequence generation method and device" provides a multi-modal driven human action sequence generation method. The specific implementation is: obtaining an input text signal or an audio signal, outputting a hidden space representation conforming to a uniform probability distribution through a text encoder and an audio encoder, and then using an action decoder to decode the feature vector output by the encoder into an action sequence.

[0004] And "CN117808942A 3D digital human action generation method and system with strong semantic correlation" preprocesses the input text into corresponding audio, keyword sequence and word timestamp corresponding to each word in the text, generates a base action sequence corresponding to the audio based on a base action generation model; search and match the keywords in the keyword sequence with the semantics in the action sequence database to obtain a keyword-action dictionary; fuse the word timestamp of each word, the keyword-action dictionary and the base action sequence in time sequence to output a 3D action sequence with strong semantic correlation.

[0005] The technical solution in Chinese patent "CN115631275A multi-modal driven human action sequence generation method and device" has poor action diversity because the input data is processed by a text encoder and an audio encoder, and the generated action is processed by an action decoder, and noise is not introduced during training. The output action sequence lacks further processing and performs poorly in terms of physical laws and aesthetic requirements.

[0006] Chinese patent "CN117808942A 3D digital human action generation method and system with strong semantic correlation" selects the input text to be preprocessed into corresponding audio, keyword sequence and multiple word timestamps when processing actions and audio. The processed corresponding audio is strongly related to the input text, and it is difficult to obtain a matching action sequence for different scenarios. SUMMARY

[0007] In view of the deficiencies of the prior art, the present application provides an action sequence generation device and method based on text and music multi-modal driving; a software tool is provided, which can provide an efficient and convenient 3D animation generation scheme for users. By combining the Transformer and Diffusion model, the key action frame knowledge base is utilized to support text and music driven 3D action generation. The efficient action generation capability is provided to overcome the defects of the prior art that the action diversity is insufficient; the real-time optimization of the action is supported, and the mainstream development platform can be seamlessly integrated.

[0008] In one aspect, an action sequence generation device based on text and music multi-modal driving comprises an action generation module and an application module.

[0009] The action generation module is used for processing the input text description and music file to generate an action sequence.

[0010] The action generation module is implemented by an AI generation model and is used for action sequence generation; specifically, it includes a VAE text encoder, a Transformer encoding layer, and a Diffusion model; the VAE text encoder is used for text encoding and converting the generated action sequence code into a low-dimensional feature vector; the Transformer encoding layer is used for encoding word-level sequences and audio sequences, capturing long-distance dependency relationships and extracting semantic information; and the Diffusion model is used for providing a key frame action sequence knowledge base of key frame information and generating samples similar to the real data distribution.

[0011] The application module specifically includes a communication module, an agent module, and a format conversion module.

[0012] The communication module includes Avalonia client software, a Web service side webpage, and a MySQL database; the agent module includes an input prompt intelligent assistant of the Avalonia client and a project information intelligent assistant of the Web service side; and the format conversion module is used for format conversion of the generated action on the client side.

[0013] The Avalonia client software is used for receiving the input of the user, sending the input information to the Web service side, receiving the npy action data and MP4 preview video generated by the action generation module, and displaying them to the user; the input prompt intelligent assistant of the Avalonia client is used for giving optimization suggestions for the text description input in the corresponding Avalonia client software.

[0014] The Web service side webpage is used for providing operation guide / info route, and / infer route responsible for calling the action generation module to generate an action sequence; and the project information intelligent assistant of the Web service side is used for providing project information.

[0015] The MySQL database includes a users table for storing user information, an efmigrationshistory table for storing a merge record of product versions, and a videorecords table for storing a generation record;

[0016] In another aspect, a text and music multi-modal driven action sequence generation method is implemented by the aforementioned text and music multi-modal driven action sequence generation device, and includes the following steps:

[0017] Step 1: input the text description and music to be processed by the action generation module through the Avalonia client software of the communication module;

[0018] Step 1.1: the user inputs the text description of generating an action, and selects whether to add music, if the music is added, the audio file is uploaded, and the text description and the audio file are used as the input data of the action generation module; if the music is not added, only the text description is used as the input data of the action generation module;

[0019] Step 1.2: configure the basic parameters;

[0020] The basic parameters are the action duration;

[0021] Step 1.3: after clicking the action generation, the text description, the action duration and the audio file are sent to the Web service side of the communication module, and the Web service side calls the action generation module for processing;

[0022] Step 2: the Web service side receives the text description, the action duration and the audio file sent by the Avalonia client, calls the action generation module, converts the data into npy action data, generates an SMPL skeleton model and an MP4 preview video, and sends them to the Avalonia client of the communication module after compression:

[0023] Step 2.1: the text description, the audio file and the configured action duration input by the Avalonia client are sent to the / infer route of the Web service side through the POST mode for processing;

[0024] Specifically, the AI ​​generation model in the action generation module is called to process the input information: the CLIP text encoder is used to encode the text description, retaining word-level information to obtain a word-level sequence; at the same time, music information retrieval technology is used to analyze the audio file input by the user, extract music features, obtain an audio sequence, and combine it with the word-level sequence to generate actions;

[0025] Step 2.2: Encode the obtained word-level sequences and audio sequences using stacked Transformer encoder layers and a fully self-attention mechanism to capture long-distance dependencies and extract semantic information;

[0026] Step 2.3: The AI-generated model outputs global embedding and sequential embedding, representing the semantic information of the entire input sequence and the semantic information of each time step, respectively, and performs multimodal information fusion;

[0027] Step 2.4: By introducing the existing key action frame knowledge base, multiple actions are divided into multiple segments for processing, simplifying the actions into a series of key frames, each key frame containing the core information of the action;

[0028] Step 2.5: Control the keyframes and combine and adjust them. Gradually add noise through the Diffusion model pre-deployed in the action generation module to transform the keyframe data distribution into a prior distribution. Then gradually remove the noise to generate samples similar to the real data distribution.

[0029] Step 2.6: Generate action sequences step by step based on word-level sequences and musical features, and adjust the parameters in the action sequences using the DDPM diffusion denoising algorithm;

[0030] Step 2.7: Use the pre-deployed VAE encoder in the motion generation module to encode the generated motion sequence into a low-dimensional feature vector, and optimize the motion based on the low-dimensional feature vector; parse the optimized motion sequence into npy motion data, generate an SMPL skeleton model and an MP4 preview video, and send the obtained npy motion data and MP4 preview video back to the Avalonia client in the form of a compressed package.

[0031] Step 3: The web server communicates with the client;

[0032] In addition to calling the action generation module to provide action generation services for the Avalonia client, an intelligent agent is configured on the web server as an AI intelligent assistant. The overall information and usage methods of the action sequence generation device can be obtained by interacting with the AI ​​intelligent assistant or by accessing the / info route on the web server.

[0033] In the Avalonia client, call the Tongyi Qianwen API to configure the intelligent assistant. After entering a text description in the Avalonia client, you can interact with the intelligent assistant to get optimization suggestions for the text description entered in the Avalonia client;

[0034] Step 4: The user selects the export format, and the generated records are saved to the videorecords database in the MySQL database of the communication module. The user selects whether to bind the skeleton. If not, the motion sequence generation is completed; if so, the format conversion module is called to convert the obtained SMPL skeleton model into an FBX skeleton, name it according to the Aist++ specification, and bind it to the SMPL_model_for_unity.fbx model under fbx_path. The bound FBX format model is opened in the graphics engine to complete the motion sequence generation.

[0035] The beneficial effects of adopting the above technical solution are as follows:

[0036] This invention provides a device and method for generating action sequences based on text and music multimodal driving. By using long text to drive the generation, it overcomes the limitations of preset templates in existing AI action generation tools, generating narrative action sequences. Dynamic temporal modeling and physical rule adaptation solve the problem of the generated actions being disconnected from the scene. Automatic binding of skeletons and actions solves the problem of the complex process required by traditional skeleton binding. Users can obtain distributed visual guidance through natural language commands, significantly reducing the learning threshold for users. Attached Figure Description

[0037] Figure 1 This is a flowchart of the action sequence generation method based on text and music multimodal driving in an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the action generation module in an embodiment of the present invention;

[0039] Figure 3 This refers to the / info routing interface of the Web server in this embodiment of the invention;

[0040] Figure 4 This is a brief introduction to the project obtained by the / info route in this embodiment of the invention;

[0041] Figure 5 This is the intelligent assistant interface for project information on the Web server in this embodiment of the invention;

[0042] Figure 6 This is the input prompt intelligent assistant interface of the Avalonia client in this embodiment of the invention;

[0043] Figure 7This is a diagram illustrating the input generation action in an embodiment of the present invention;

[0044] Figure 8 This is an action diagram generated in an embodiment of the present invention;

[0045] Figure 9 This is an interactive effect diagram of the client-side intelligent agent in an embodiment of the present invention;

[0046] Figure 10 This describes the interaction effect of the server-side intelligent agent in an embodiment of the present invention.

[0047] Figure 11 This is an application diagram in Unity according to an embodiment of the present invention. Detailed Implementation

[0048] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0049] On the one hand, a text- and music-based multimodal motion sequence generation device includes: an motion generation module and an application module;

[0050] The action generation module is used to process the input text description and music file to generate an action sequence.

[0051] The action generation module is implemented through an AI generation model for generating action sequences. Specifically, it includes a VAE text encoder, a Transformer encoding layer, and a Diffusion model. The VAE text encoder is used for text encoding and converting the generated action sequences into low-dimensional feature vectors. The Transformer encoding layer is used for encoding word-level sequences and audio sequences, capturing long-distance dependencies, and extracting semantic information. The Diffusion model is used to provide a keyframe action sequence knowledge base with keyframe information and generate samples similar to the distribution of real data.

[0052] The application module specifically includes a communication module, an intelligent agent module, and a format conversion module;

[0053] The communication module includes Avalonia client software, a web server page, and a MySQL database; the intelligent agent module includes an input prompt intelligent assistant in the Avalonia client and a project information intelligent assistant in the web server; and the format conversion module is used to convert the format of the generated action on the client.

[0054] The Avalonia client software is used to receive user input, send the input information to the web server, and receive npy action data and MP4 preview video generated by the action generation module, and display them to the user; the input suggestion intelligent assistant of the Avalonia client is used to provide optimization suggestions for the text description entered in the corresponding Avalonia client software.

[0055] The web server page is used to provide the / info route for operation guidance and the / infer route for calling the action generation module to generate action sequences; the web server's intelligent project information assistant is used to provide project information;

[0056] The MySQL database includes a users table for storing user information, an efmigrationshistory table for storing merged records of product versions, and a videorecords table for storing generated records.

[0057] The user information in this embodiment includes username, password, email, role, and registration time; the merged record includes version number and merge information; the generated record includes the ID of the generated task, the generation time of the action sequence, the storage path of the video, the user ID of the user who performed the generation operation, the text description entered during generation, the configured action duration, whether it is the user's favorite generated record, which users have collected it, and whether it is in a shared state.

[0058] On the other hand, a method for generating action sequences based on text and music multimodal driving is implemented using the aforementioned action sequence generation device based on text and music multimodal driving, such as... Figure 1 As shown, it includes the following steps:

[0059] Step 1: Input the text description and music that the motion generation module needs to process through the Avalonia client software of the communication module;

[0060] Step 1.1: The user inputs a text description of the generated action and selects whether to add music. If music is added, an audio file is uploaded. In this embodiment, the audio file is an MP3 or WAV format music file. The text description and the audio file are used together as input data for the action generation module. If no music is added, only the text description is used as input data for the action generation module.

[0061] Step 1.2: Configure basic parameters;

[0062] The basic parameter is the action duration (in this embodiment, the action duration can be set to a range of 1-30 seconds).

[0063] Step 1.3: After clicking "Generate Action," the text description, action duration, and audio file are sent to the web server of the communication module. The web server then calls the action generation module for processing. In this embodiment, the interface of the action generation module is as follows: Figure 2 As shown:

[0064] Step 2: The web server receives the text description, motion duration, and audio file sent by the Avalonnia client, calls the motion generation module, converts this data into npy motion data, generates an SMPL skeleton model and an MP4 preview video, compresses them, and sends them to the Avalonnia client in the communication module.

[0065] Step 2.1: Send the text description, audio file, and configured action duration input by the Avalonia client to the / infer route on the web server via POST for processing;

[0066] Specifically, the AI ​​generation model in the action generation module is called to process the input information: the CLIP text encoder is used to encode the text description, retaining word-level information to obtain a word-level sequence; at the same time, music information retrieval technology is used to analyze the audio file input by the user, extracting music features such as rhythm, melody, and emotion to obtain an audio sequence, which is then combined with the word-level sequence to generate actions;

[0067] Step 2.2: Encode the obtained word-level sequences and audio sequences using stacked Transformer encoder layers and a fully self-attention mechanism to capture long-distance dependencies and extract semantic information;

[0068] Step 2.3: The AI-generated model outputs global embedding and sequential embedding, which represent the semantic information of the entire input sequence (action type, rhythm, style, etc.) and the semantic information of each time step (start, duration, end, etc. of the action), respectively, and performs multimodal information fusion.

[0069] Step 2.4: By introducing the existing key action frame knowledge base, multiple actions are divided into multiple segments for processing, simplifying the actions into a series of key frames, each key frame containing the core information of the action;

[0070] Step 2.5: Control the keyframes and combine and adjust them. Gradually add noise through the Diffusion model pre-deployed in the action generation module to transform the keyframe data distribution into a prior distribution. Then gradually remove the noise to generate samples similar to the real data distribution.

[0071] Step 2.6: Generate action sequences step by step based on word-level sequences and musical features, and adjust the parameters in the action sequences using the DDPM diffusion denoising algorithm;

[0072] Step 2.7: Use the pre-deployed VAE encoder in the motion generation module to encode the generated motion sequence into a low-dimensional feature vector, and optimize the motion based on the low-dimensional feature vector; parse the optimized motion sequence into npy motion data, generate an SMPL skeleton model and an MP4 preview video, and send the obtained npy motion data and MP4 preview video back to the Avalonia client in the form of a compressed package.

[0073] Step 3: The web server communicates with the client; in this embodiment, the following is the / info routing interface of the web server: Figure 3 As shown;

[0074] In addition to calling the action generation module to provide action generation services for the Avalonia client, an intelligent agent is configured on the web server as an AI assistant, such as... Figure 5 As shown, information about the action sequence generation device and its usage can be obtained through interaction with the AI ​​assistant or by accessing the / info route on the web server. The obtained project overview is as follows... Figure 4 As shown;

[0075] In the Avalonia client, call the Tongyi Qianwen API to configure the intelligent assistant. After entering a text description in the Avalonia client, you can interact with the intelligent assistant to obtain optimization suggestions for the text description entered in the Avalonia client, such as... Figure 6 As shown;

[0076] Step 4: The user selects the export format (SMPL / FBX), and the generated records are saved to the videorecords folder in the MySQL database of the communication module. The user chooses whether to bind bones. If not, the motion sequence generation is completed; if so, the format conversion module is called to convert the obtained SMPL bone model into an FBX bone, naming it according to Aist++ specifications, and binding it to the SMPL_model_for_unity.fbx model under fbx_path. The bound FBX format model can be opened in various graphics engines such as Unity, Unreal Engine, Blender, and 3ds Max to complete the motion sequence generation. The format conversion module performs format conversion via command line, and the conversion results are output to the output folder of this module.

[0077] Finally, this embodiment uses an actual action generation example to understand the specific process: First, enter "A person is walking" in the client, set the duration parameter to 100, click "Generate Action," and start the action generation process, as follows... Figure 7 As shown;

[0078] After processing by the action generation module, the generated action is obtained and sent to the client. The client then displays the generated action, such as... Figure 8 As shown: After generating the action, interaction with the client agent can be performed, such as... Figure 9 The image shows the interactive effect: In addition, we can also interact with the agent on the server-side webpage, such as... Figure 10 As shown: Finally, for the generated animation, clicking "Export" on the client allows you to select to bind bones, call the format conversion module, bind the bones, and obtain an FBX file, which can be directly applied in Unity, such as... Figure 11 As shown, the obtained FBX skeleton is applied in Unity; the process ends.

[0079] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A device for generating action sequences based on text and music multimodal driving, characterized in that, include: Action generation module and application module; The action generation module is implemented through an AI generation model and is used to generate action sequences. Specifically, it processes the input text description and music file to generate action sequences, which are then input into the application module. The application module specifically includes a communication module, an intelligent agent module, and a format conversion module; the communication module includes Avalonia client software, a web server webpage, and a MySQL database; the intelligent agent module includes an input prompt intelligent assistant in the Avalonia client and a project information intelligent assistant in the web server; and the format conversion module is used to convert the format of the generated actions on the client.

2. The action sequence generation device based on text and music multimodal driving according to claim 1, characterized in that, The action generation module includes a VAE text encoder, a Transformer encoding layer, and a Diffusion model. The VAE text encoder is used to encode text and convert the generated action sequences into low-dimensional feature vectors. The Transformer encoding layer is used to encode word-level sequences and audio sequences, capture long-distance dependencies, and extract semantic information. The Diffusion model is used to provide a keyframe action sequence knowledge base with keyframe information and generate samples similar to the distribution of real data.

3. The action sequence generation device based on text and music multimodal driving according to claim 1, characterized in that, The Avalonia client software is used to receive user input, send the input information to the web server, and receive npy action data and MP4 preview video generated by the action generation module, and display them to the user; the input suggestion intelligent assistant of the Avalonia client is used to provide optimization suggestions for the text description entered in the corresponding Avalonia client software. The web server page is used to provide the / info route for operation guidance and the / infer route for calling the action generation module to generate action sequences; the web server's intelligent project information assistant is used to provide project information; The MySQL database includes a users table for storing user information, an efmigrationshistory table for storing merged records of product versions, and a videorecords table for storing generated records.

4. A method for generating action sequences based on text and music multimodal driving, implemented by the action sequence generation device based on text and music multimodal driving as described in claim 1, characterized in that, Includes the following steps: Step 1: Input the text description and music that the motion generation module needs to process through the Avalonia client software of the communication module; Step 2: The web server receives the text description, motion duration, and audio file sent by the Avalonnia client, calls the motion generation module, converts this data into npy motion data, generates an SMPL skeleton model and an MP4 preview video, compresses them, and sends them to the Avalonnia client in the communication module. Step 3: The web server communicates with the client; Step 4: The user selects the export format, and the generated records are saved to the videorecords database in the MySQL database of the communication module. The user selects whether to bind the skeleton. If not, the motion sequence generation is completed; if so, the format conversion module is called to convert the obtained SMPL skeleton model into an FBX skeleton, name it according to the Aist++ specification, and bind it to the SMPL_model_for_unity.fbx model under fbx_path. The bound FBX format model is opened in the graphics engine to complete the motion sequence generation.

5. The method for generating action sequences based on text and music multimodal driving according to claim 4, characterized in that, Step 1 includes the following steps: Step 1.1: The user inputs a text description of the generated action and chooses whether to add music. If music is added, an audio file is uploaded, and the text description and audio file are used together as input data for the action generation module. If no music is added, only the text description is used as input data for the action generation module. Step 1.2: Configure basic parameters; The basic parameter is the action duration; Step 1.3: After clicking "Generate Action", the text description, action duration, and audio file are sent to the web server of the communication module. The web server then calls the action generation module for processing.

6. The method for generating action sequences based on text and music multimodal driving according to claim 4, characterized in that, Step 2 includes the following steps: Step 2.1: Send the text description, audio file, and configured action duration input by the Avalonia client to the / infer route on the web server via POST for processing; Specifically, the AI ​​generation model in the action generation module is called to process the input information: the CLIP text encoder is used to encode the text description, retaining word-level information to obtain a word-level sequence; at the same time, music information retrieval technology is used to analyze the audio file input by the user, extract music features, obtain an audio sequence, and combine it with the word-level sequence to generate actions; Step 2.2: Encode the obtained word-level sequences and audio sequences using stacked Transformer encoder layers and a fully self-attention mechanism to capture long-distance dependencies and extract semantic information; Step 2.3: The AI-generated model outputs global embedding and sequential embedding, representing the semantic information of the entire input sequence and the semantic information of each time step, respectively, and performs multimodal information fusion; Step 2.4: By introducing the existing key action frame knowledge base, multiple actions are divided into multiple segments for processing, simplifying the actions into a series of key frames, each key frame containing the core information of the action; Step 2.5: Control the keyframes and combine and adjust them. Gradually add noise through the Diffusion model pre-deployed in the action generation module to transform the keyframe data distribution into a prior distribution. Then gradually remove the noise to generate samples similar to the real data distribution. Step 2.6: Generate action sequences step by step based on word-level sequences and musical features, and adjust the parameters in the action sequences using the DDPM diffusion denoising algorithm; Step 2.7: Use the pre-deployed VAE encoder in the motion generation module to encode the generated motion sequence into a low-dimensional feature vector, and optimize the motion based on the low-dimensional feature vector; parse the optimized motion sequence into npy motion data, generate an SMPL skeleton model and an MP4 preview video, and send the obtained npy motion data and MP4 preview video back to the Avalonia client in the form of a compressed package.

7. The method for generating action sequences based on text and music multimodal driving according to claim 4, characterized in that, Step 3 specifically includes: calling the action generation module to provide action generation services for the Avalonia client, configuring an intelligent agent as an AI intelligent assistant on the Web server, and obtaining overall information and usage methods of the action sequence generation device through interaction with the AI ​​intelligent assistant or by accessing the / info route on the Web server; In the Avalonia client, you can call the Tongyi Qianwen API to configure the smart assistant; after entering a text description in the Avalonia client, you can get optimization suggestions for the text description entered in the Avalonia client by interacting with the smart assistant.

Images