Real-time semantic controllable action generation method and system based on information bottleneck regulation

Abstract

This invention discloses a real-time semantically controllable action generation method and system based on information bottleneck adjustment. The method includes: training an action representation module using a training action sequence to obtain a continuous action latent representation and obtaining an action decoder for action recovery; training a causal real-time action generation module based on information bottleneck adjustment, wherein action labels and prototype action segments are uniformly encoded to obtain semantic prior features; cross-modal control variables are generated based on the semantic prior features and current speech features; the action latent representation of the current block is obtained based on the current speech features, cross-modal control variables, and historical action context; the action latent representation of the current block is input into the action decoder to obtain the action sequence of the current block; and the trained real-time semantically controllable action generation network model is used to predict the action generation result from the speech to be generated. This invention can achieve action generation for real-time interactive scenarios without relying on future speech information.

Description

Technical Field

[0001] This invention relates to the fields of digital human interaction, multimodal information processing, action generation, and machine learning, and in particular to a real-time semantically controllable action generation method and system based on information bottleneck adjustment. Background Technology

[0002] In real-time interactive scenarios, digital humans not only need to output speech, but also generate body movements that match the current speech content, rhythm, and interactive state to enhance the realism, expressiveness, and immersion of the interaction. If the generated movements only maintain rhythmic consistency with the speech without clear semantic control and structural prior constraints, problems such as unclear meaning of movements, chaotic movement categories, and monotonous interactive expression can easily arise.

[0003] Most existing methods for generating communal speech actions are designed for offline scenarios, modeling entire segments and typically relying on future speech information or contextual information over a longer timeframe. This makes it difficult to meet the requirements of low latency, causal relationships, and rolling prediction in real-time interactive scenarios. On the other hand, existing semantic control methods either rely solely on discrete action labels, failing to fully express the local structural features of actions, or directly introduce example actions or retrieved actions as structural references. This can easily lead to the generated results over-reliance on fixed templates, weakening the ability of actions to adapt to dynamic changes in the current speech.

[0004] Therefore, how to uniformly model the category semantics provided by action tags, the structural priors provided by prototype action fragments, the current speech dynamics, and the historical action context under real-time interactive conditions, and output an action sequence that matches the current speech while ensuring the continuity of actions, has become a technical problem that needs to be solved in this field. Summary of the Invention

[0005] To address the problems of insufficient real-time performance, lack of semantic control capabilities, low quality of semantic action generation, and discontinuous connection between adjacent action blocks in existing action interaction methods, this invention proposes a real-time semantically controllable action generation method and system based on information bottleneck adjustment.

[0006] A real-time semantically controllable action generation method based on information bottleneck adjustment includes the following steps:

[0007] Step S1: Obtain the current speech segment, action tag, prototype action segment corresponding to the action tag, and historical action context;

[0008] Step S2: Train a real-time semantically controllable action generation network model using action samples from the training data; the real-time semantically controllable action generation network includes an action representation module and a causal real-time action generation module based on information bottleneck adjustment. Step S2 includes:

[0009] Step S21: Train the action representation module using the training action sequence. The action representation module includes an encoder and an action decoder. Input the training action sequence into the encoder of the action representation module to obtain a continuous action latent representation. Then, use the action decoder to recover and reconstruct the action sequence from the continuous action latent representation to obtain the action decoder for action latent representation recovery.

[0010] Step S22: Train the causal real-time action generation module based on information bottleneck adjustment. The causal real-time action generation module based on information bottleneck adjustment includes a semantic prior fusion submodule, an information bottleneck adjustment submodule, and a causal generation submodule. The action label, prototype action segment, current speech features, and historical action context are input into the causal real-time action generation module based on information bottleneck adjustment. The semantic prior fusion submodule encodes the action label and prototype action segment in a unified manner to obtain semantic prior features. The information bottleneck adjustment submodule generates cross-modal control variables based on the semantic prior features and current speech features. The causal generation submodule obtains the action latent representation of the current block based on the current speech features, cross-modal control variables, and historical action context.

[0011] Step S23: Input the action latent representation of the current block into the action decoder trained in step S21 to obtain the action sequence of the current block. Then, concatenate the actions of each block sequentially along the time dimension to obtain the final action generation result. ;

[0012] Step S3: Use the trained real-time semantically controllable action generation network model to predict the action generation result from the speech to be generated.

[0013] A real-time semantically controllable action generation system based on information bottleneck adjustment, used to execute the above-described real-time semantically controllable action generation method based on information bottleneck adjustment, includes:

[0014] The input acquisition module is used to acquire the current speech segment, action label, prototype action segment, and historical action context;

[0015] The action representation module is used to encode training action sequences into continuous action latent representations and recover action sequences based on the action latent representations;

[0016] The causal real-time action generation module based on information bottleneck adjustment is used to generate the current block action latent representation based on action tags, prototype action segments, current speech features and historical action context. The causal real-time action generation module based on information bottleneck adjustment includes a semantic prior fusion submodule, an information bottleneck adjustment submodule and a causal generation submodule.

[0017] The action decoding module in the action representation module is used to restore the current block action latent representation to the action generation result.

[0018] A computing device includes: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device performs the real-time semantically controllable action generation method based on information bottleneck adjustment.

[0019] A readable storage medium storing program instructions, which, when read and executed by a computing device, cause the computing device to perform the real-time semantically controllable action generation method based on information bottleneck adjustment.

[0020] A computer program product includes a computer program that, when executed by a processor, implements the aforementioned real-time semantically controllable action generation method based on information bottleneck adjustment.

[0021] The beneficial effects of this invention are as follows: Through the collaborative design of the action representation module and the causal real-time action generation module based on information bottleneck adjustment, this invention can realize action generation for real-time interactive scenarios without relying on future voice information; at the same time, through semantic prior fusion and information bottleneck adjustment, the semantic controllability and voice adaptability of action generation are improved; and through continuity constraints, the smooth connection between adjacent action blocks is improved. Attached Figure Description

[0022] Figure 1 This is a flowchart of the real-time semantically controllable action generation method based on information bottleneck adjustment in this invention. Detailed Implementation

[0023] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0024] To address the issues of ambiguous action semantics, structural prior templates, and insufficient matching between actions and current speech in the process of digital human action generation in real-time interactive scenarios, this invention proposes a real-time semantically controllable action generation method based on information bottleneck adjustment, thereby improving the real-time performance, controllability, and continuity of action generation in online interactive scenarios.

[0025] Figure 1 The present invention demonstrates a real-time semantically controllable action generation method and system flowchart based on information bottleneck adjustment, as shown in the figure. Figure 1As shown, the method includes the following steps:

[0026] Step S1: Obtain the current speech segment, action tag, prototype action segment corresponding to the action tag, and historical action context. The current speech segment is used to represent the local speech dynamics of the current block; the action tag is used to represent the semantic category of the action to be performed; the prototype action segment is used to provide structural priors on how the action usually moves, and the prototype action segment can be retrieved from a preset prototype action library based on the action tag; the historical action context is used to represent the temporal context of the actions of the preceding block.

[0027] Step S2: Train a real-time semantically controllable action generation network model using action samples from the training data. The real-time semantically controllable action generation network includes an action representation module and a causal real-time action generation module based on information bottleneck adjustment. Step S2 includes two stages: pre-training of the action representation module and training of the causal real-time action generation module based on information bottleneck adjustment. Specifically, S2 may include:

[0028] Step S21: Train the action representation module, which includes an encoder and an action decoder. Input the training action sequence into the encoder of the action representation module to obtain a continuous action latent representation. Then, use the action decoder to recover and reconstruct the action sequence from the continuous action latent representation, thereby training the action decoder for action latent representation recovery. The training action sequence for the action representation module can be derived from a publicly available action dataset, a self-built action dataset, or a combination of both.

[0029] First, during the pre-training phase of the action representation module, the training action sequences are mapped to continuous action latent representations suitable for real-time causal generation. For the first... The training action sequence of each block is denoted as . Preferably, it can be The body can be divided into at least two parts, such as the upper body, lower body, and hands, which are denoted as follows: , and For any body part Local action sequence input corresponds to encoder The local action latent representation is obtained: in, .

[0030] Subsequently, the local action latent representation is input into the corresponding decoder. The results of localized recovery for each body part were obtained: .

[0031] Then, the multiple local restoration results are fused and globally corrected to obtain the first result. Recovery results of the action sequence in each block:

[0032] ,

[0033] in, This indicates a fusion correction operation, used to restore the coordination between different body parts and correct inconsistencies in local boundaries. In this way, a stable action representation space and an action decoder for subsequent action recovery can be trained.

[0034] In one implementation, the training loss of the action representation module can be written as:

[0035] ,

[0036] in, Indicates the loss of motion reconstruction. Represents geometric consistency loss. and These are the corresponding weighting coefficients.

[0037] Furthermore, the action reconstruction loss can take the following form:

[0038] ,

[0039] The action reconstruction loss can be constrained by the reconstruction error between the currently recovered action and the actual action, as well as the difference between action changes at adjacent time points, in order to improve the action representation module's ability to recover action content and its temporal changes. Represents the actual sequence of actions. This represents a first-order difference operation. This represents a second-order difference operation. Indicates the sequence of actions after recovery. This represents a first-order difference operation. This represents a second-order difference operation. Indicates the first The number of frames in each block.

[0040] The geometric consistency loss is used to constrain the authenticity of the recovery action in the sense of joint rotation.

[0041] S22, train the causal real-time action generation module based on information bottleneck adjustment. This module includes a semantic prior fusion submodule, an information bottleneck adjustment submodule, and a causal generation submodule. Action labels, prototype action segments, current speech features, and historical action context are input into the causal real-time action generation module. The semantic prior fusion submodule uniformly encodes the action labels and prototype action segments to obtain semantic prior features. The information bottleneck adjustment submodule generates cross-modal control variables based on the semantic prior features and current speech features. The causal generation submodule obtains the action latent representation of the current block based on the current speech features, cross-modal control variables, and historical action context. Specifically, this includes the following steps:

[0042] S22-1, For the current block c, first set the action label. With prototype action clips Input semantic prior fusion submodule. Among them, action labels are semantically encoded to obtain category semantic features:

[0043] ,

[0044] in, This represents the action tag semantic encoding network. The semantic prior fusion submodule contains the action tag semantic encoding network, the prototype encoder, and the fusion mapping module.

[0045] Prototype action clip The body parts can be divided according to the same method as the action representation module, and the structural priors can be extracted by the corresponding encoder. For any body part p, we have:

[0046] ,

[0047] This represents the prototype encoder. By concatenating the structural prior features of multiple body parts, a structural prior representation of the prototype action is obtained. .

[0048] Subsequently, the category semantic features and structural prior representations are uniformly fused to obtain semantic prior features:

[0049] ,

[0050] in, and These represent the fusion mapping parameters.

[0051] S22-2, semantic prior features With current speech features Input information bottleneck adjustment submodule. To compress redundant template information that does not match the current speech dynamics and retain effective control information for generating the current action, the information bottleneck adjustment submodule outputs the distribution parameters of cross-modal control variables:

[0052] ,

[0053] in, Represents a bottleneck network. and These represent the mean and variance of the distribution of the control variables, respectively. This indicates a splicing operation.

[0054] Furthermore, cross-modal control variables are obtained by sampling using a reparameterization method. :

[0055] ,

[0056] in, This represents element-wise multiplication. This represents a standard Gaussian distribution.

[0057] S22-3, the current speech features Cross-modal control variables and historical action context Input causal generation submodule. Preferably, the historical action context can be composed of a preset number of historical block action latent representations, and processed by a historical state encoder. Aggregation into historical context representation:

[0058] ,

[0059] Based on this, the causal generation submodule predicts the action latent representation of the current block according to the current speech features, cross-modal control variables, and historical context:

[0060] ,

[0061] in, This indicates a causal real-time generation backbone. This backbone relies solely on the current speech segment, historical action context, and cross-modal control variables for block-by-block prediction, without depending on future speech information, thus meeting the requirements of low latency and causal generation in real-time interactive scenarios. Preferably, the... This can be achieved using conditional flow matching.

[0062] S23, input the predicted current block action latent representation into the action decoder trained in the aforementioned action representation module. The current action block sequence is obtained, and then the actions of each block are sequentially concatenated along the time dimension to obtain the final action generation result. .

[0063] By leveraging the differences between real and predicted action sequences, the bottleneck constraints of cross-modal control variables, and the continuity constraints between adjacent action blocks, the parameters in the real-time semantically controllable action generation network model are adjusted through gradient backpropagation. The above training process is iteratively executed until convergence, resulting in a well-trained real-time semantically controllable action generation network model.

[0064] Specifically, the objective function of the real-time semantically controllable action generation network model includes three parts: the first part is the generation loss function, which is used to constrain the difference between the predicted action latent representation or predicted action sequence of the current block and the actual action latent representation or actual action sequence; the second part is the information bottleneck loss function, which is used to constrain the difference between the posterior distribution corresponding to the cross-modal control variable output by the information bottleneck adjustment submodule and the preset prior distribution; the third part is the continuity loss function, which is used to constrain the smooth connection of actions at the boundaries of adjacent blocks.

[0065] In one implementation, the overall objective function can be written as:

[0066] ,

[0067] in, This represents the generation loss function. Represents the information bottleneck loss function. Represents the continuous loss function. , and These are the corresponding weighting coefficients.

[0068] Preferably, the generation loss can take the following form:

[0069] ,

[0070] The information bottleneck loss can take the following form:

[0071] ,

[0072] in, Indicates the Kullback-Leibler divergence. This represents the posterior distribution of the output of the information bottleneck adjustment submodule.

[0073] The continuity loss can take the following form:

[0074] ,

[0075] in, This indicates the predicted action for the first frame of the current block. This represents the predicted action in the last frame of the previous block. By introducing this constraint, action jumps at the boundaries of adjacent blocks can be reduced, improving the smoothness and naturalness of the overall action sequence.

[0076] Step S3: Using the trained real-time semantically controllable action generation network model, predict the action generation result from the speech to be generated. In practical applications, the current speech features corresponding to the speech to be generated can be extracted first, and then combined with the externally given action labels, prototype action fragments and historical action context, the action latent representation is generated block by block, and the action sequence output is recovered by the action decoder.

[0077] This invention also proposes a real-time semantically controllable action generation system based on information bottleneck adjustment, comprising:

[0078] The input acquisition module is used to acquire the current speech segment, action label, prototype action segment, and historical action context;

[0079] The action representation module is used to encode training action sequences into continuous action latent representations and recover action sequences based on the action latent representations;

[0080] The causal real-time action generation module based on information bottleneck adjustment is used to generate the current block action latent representation based on action tags, prototype action segments, current speech features and historical action context. The causal real-time action generation module based on information bottleneck adjustment includes a semantic prior fusion submodule, an information bottleneck adjustment submodule and a causal generation submodule.

[0081] The action decoding module in the action representation module is used to restore the current block action latent representation to the action generation result.

[0082] The present invention also provides a computing device, comprising: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device performs the real-time semantically controllable action generation method based on information bottleneck adjustment.

[0083] The present invention also provides a readable storage medium storing program instructions, which, when read and executed by a computing device, causes the computing device to perform the real-time semantically controllable action generation method based on information bottleneck adjustment.

[0084] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the aforementioned real-time semantically controllable action generation method based on information bottleneck adjustment.

[0085] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, applications, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time semantically controllable action generation method based on information bottleneck adjustment, characterized in that, Including the following steps: Step S1: Obtain the current speech segment, action tag, prototype action segment corresponding to the action tag, and historical action context; Step S2: Train a real-time semantically controllable action generation network model using action samples from the training data; the real-time semantically controllable action generation network includes an action representation module and a causal real-time action generation module based on information bottleneck adjustment. Step S2 includes: Step S21: Train the action representation module using the training action sequence. The action representation module includes an encoder and an action decoder. Input the training action sequence into the encoder of the action representation module to obtain a continuous action latent representation. Then, use the action decoder to recover and reconstruct the action sequence from the continuous action latent representation to obtain the action decoder for action latent representation recovery. Step S22: Train the causal real-time action generation module based on information bottleneck adjustment. The causal real-time action generation module based on information bottleneck adjustment includes a semantic prior fusion submodule, an information bottleneck adjustment submodule, and a causal generation submodule. The action label, prototype action segment, current speech features, and historical action context are input into the causal real-time action generation module based on information bottleneck adjustment. The semantic prior fusion submodule encodes the action label and prototype action segment in a unified manner to obtain semantic prior features. The information bottleneck adjustment submodule generates cross-modal control variables based on the semantic prior features and current speech features. The causal generation submodule obtains the action latent representation of the current block based on the current speech features, cross-modal control variables, and historical action context. Step S23: Input the action latent representation of the current block into the action decoder trained in step S21 to obtain the action sequence of the current block. Then, concatenate the actions of each block sequentially along the time dimension to obtain the final action generation result. ; Step S3: Use the trained real-time semantically controllable action generation network model to predict the action generation result from the speech to be generated.

2. The real-time semantically controllable action generation method based on information bottleneck adjustment according to claim 1, characterized in that, include: The training action sequences for the action representation module are derived from public action datasets, self-built action datasets, or a combination of public action datasets and self-built action datasets.

3. The real-time semantically controllable action generation method based on information bottleneck adjustment according to claim 1, characterized in that, include: By utilizing the differences between real action sequences and predicted action sequences, the bottleneck constraints of cross-modal control variables, and the continuity constraints between adjacent action blocks, the parameters in the real-time semantically controllable action generation network model are adjusted through gradient backpropagation. The training process is iteratively executed until convergence, resulting in a trained real-time semantically controllable action generation network model.

4. The real-time semantically controllable action generation method based on information bottleneck adjustment according to claim 1, characterized in that, The loss function of the real-time semantically controllable action generation network model includes three parts: the first part is the generation loss function, which is used to constrain the difference between the predicted action latent representation or predicted action sequence of the current block and the actual action latent representation or actual action sequence; the second part is the information bottleneck loss function, which is used to constrain the difference between the posterior distribution corresponding to the cross-modal control variable output by the information bottleneck adjustment submodule and the preset prior distribution; the third part is the continuity loss function, which is used to constrain the smooth connection of actions at the boundary of adjacent blocks.

5. The real-time semantically controllable action generation method based on information bottleneck adjustment according to claim 1, characterized in that, Step S21 includes: The training action sequence is divided into at least two local parts according to body parts, and each part is input into the corresponding encoder to obtain the local action latent representation; Multiple local action latent representations are input into their respective decoders for local action recovery. The results of the multiple local action recovery are then fused and corrected as a whole to obtain a complete action sequence.

6. The real-time semantically controllable action generation method based on information bottleneck adjustment as described in claim 1, characterized in that, In step S21, the training loss of the action representation module is written as: ， in, Indicates the loss of motion reconstruction. Represents geometric consistency loss. and These are the corresponding weighting coefficients.

7. A real-time semantically controllable action generation system based on information bottleneck adjustment, characterized in that, The method for generating real-time semantically controllable actions based on information bottleneck adjustment according to any one of claims 1-6 includes: The input acquisition module is used to acquire the current speech segment, action label, prototype action segment, and historical action context; The action representation module is used to encode training action sequences into continuous action latent representations and recover action sequences based on the action latent representations; The causal real-time action generation module based on information bottleneck adjustment is used to generate the current block action latent representation based on action tags, prototype action segments, current speech features and historical action context. The causal real-time action generation module based on information bottleneck adjustment includes a semantic prior fusion submodule, an information bottleneck adjustment submodule and a causal generation submodule. The action decoding module in the action representation module is used to restore the current block action latent representation to the action generation result.

8. A computing device, characterized in that, include: At least one processor and a memory storing program instructions; When the program instructions are read and executed by the processor, the computing device performs the real-time semantically controllable action generation method based on information bottleneck adjustment as described in any one of claims 1-6.

9. A readable storage medium storing program instructions, characterized in that, When the program instructions are read and executed by the computing device, the computing device performs the real-time semantically controllable action generation method based on information bottleneck adjustment as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the real-time semantically controllable action generation method based on information bottleneck adjustment as described in any one of claims 1-6.

Images