Digital human generation system and method based on visual autoregressive model

Abstract

The application provides a digital human generation system based on a visual autoregressive model, which introduces a BSQ multi-scale image tokenizer, a conditional image guided network and a timing attention layer based on the visual autoregressive model to construct a digital human generation model, adjusts the timing attention weight distribution by using a temperature map, realizes target digital human image sequence generation guided by a character identity label and a color coded image sequence, and improves the visual generation quality and timing consistency of the generated image sequence; the model training introduces an LPIPS perceptual loss to directly constrain the visual quality of the generated image, further improving the visual generation quality of the generated image; in the digital human image sequence generation stage, an interpolation smoothing inference method is used to further improve the timing consistency of the generated image sequence; the modules of the application cooperatively realize controllable target digital human image sequence generation, and significantly improve the image sequence generation speed under the premise of ensuring the quality of the generated image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of generative digital human, and particularly relates to a digital human generation system and method based on a visual autoregressive model. BACKGROUND

[0002] With the continuous development of computer graphics, artificial intelligence and other technologies and the rise of the metaverse, virtual digital humans as a new audiovisual medium have gradually entered the public eye. Virtual digital humans refer to virtual images with digitalized appearance and the ability to simulate human behavior and interaction created by using computer graphics, artificial intelligence, machine learning and other technologies.

[0003] Virtual digital human technology consists of four parts: modeling, driving, rendering and interaction. Modeling refers to building the structure and appearance of a digital human, driving refers to controlling the posture, expression and behavior of a digital human, rendering refers to displaying the digital human image to users in a visual way, and interaction refers to the interaction between the digital human and the user and the environment. Traditional digital human construction mainly relies on computer graphics technology and real action capture technology, which requires collecting a large amount of real data and conducting deep modeling, consuming a long time and being low in efficiency. With the rapid development of artificial intelligence technology, various neural network-based generative methods have been widely studied, such as generative adversarial networks, variational autoencoders and diffusion models. Such AIGC (Artificial Intelligence Generative Content) generative methods can efficiently generate target digital humans under the driving of data and algorithms. Such methods do not require professional modelers and animators, and do not require complex and expensive professional motion capture equipment, significantly reducing the production cost of digital humans and promoting the widespread application of digital human technology in various industries.

[0004] In recent years, AIGC-based digital human generation technology has been widely studied, and the quality, controllability and stability of digital human generation have been significantly improved. According to different technical routes, it can be divided into digital human generation technologies based on generative adversarial networks, diffusion models, neural radiation fields and three-dimensional Gaussians. Different technical routes have different characteristics. For example, the digital human generation technology based on diffusion model can generate high-quality digital humans and has the characteristic of strong generalization ability, but due to the need for multiple steps of reverse denoising in the inference stage of the diffusion model, it often leads to slow generation speed, which becomes one of the main bottlenecks of this technical route.

[0005] The generation technique based on the autoregressive model is widely used in the research work of large language models, and exhibits unprecedented generation capability on natural language tasks. In the field of image generation, autoregressive models also exhibit strong generation capability. In the tasks of category-guided and text-guided image generation, the related work based on visual autoregressive models has exceeded the related work based on diffusion models, and the generated images not only have higher quality, but also can significantly reduce image inference time. However, the generation of digital people based on visual autoregressive models has not been deeply studied and discussed, and the application method of visual autoregressive models in the field of generative digital people is unknown. SUMMARY

[0006] In view of the deficiencies of the prior art, the purpose of the present application is to provide a digital person generation system based on a visual autoregressive model, which can realize the generation of a digital person image sequence with multiple character identities, high fidelity and time sequence continuity, significantly improve the image sequence generation speed compared with diffusion models, and has a wide application prospect in many fields.

[0007] The digital person generation system based on the visual autoregressive model provided in the embodiment comprises the following modules:

[0008] A data acquisition module for acquiring character identity labels and color-coded images from real images;

[0009] A digital person visual autoregressive model construction module for constructing a digital person generation model based on a visual autoregressive model, introducing a BSQ multi-scale image tokenizer, a conditional image guide network and a time sequence attention layer, and realizing the guided generation of a target digital person image sequence by the character identity labels and the color-coded images; wherein the BSQ multi-scale image tokenizer is used for multi-scale encoding and decoding reconstruction of the color-coded images, the conditional image guide network is used for extracting action conditions from the color-coded images and injecting them into a visual autoregressive Transformer module, and the time sequence attention layer in the Transformer module is used for extracting time sequence features of the image sequence and adjusting the time sequence attention weight distribution by using a temperature map;

[0010] A model training module for training the constructed digital person generation model in multiple stages, wherein in the second stage of training, a LPIPS perceptual loss is introduced to obtain a trained digital person generation model;

[0011] A digital person image sequence generation module for inputting the character identity labels and the color-coded images into the trained digital person generation model to generate a target digital person image sequence.

[0012] In one embodiment, in the data acquisition module, the character identity labels are different digital labels used to represent character identity information.

[0013] The color-coded image is a conditional image drawn according to the face, hand and body key points of the character, and is used to represent the character's expression, gesture and body posture.

[0014] In an embodiment, in the digital human generation model construction module, the BSQ multi-scale image tokenizer includes a BSQ multi-scale image encoder and a BSQ multi-scale image decoder, which are used for multi-scale encoding and decoding reconstruction of the color-coded image.

[0015] The BSQ multi-scale image encoder uses an image encoder to encode the input real image to obtain an image feature map, extracts different scale image information from the image feature map, and obtains a multi-scale token sequence.

[0016] According to the multi-scale token sequence, an image feature map containing different scale image information is constructed, as shown in the following formula:

[0017] ,

[0018] wherein, is the image feature map containing different scale image information, is the discrete representation of the i-th scale image information, and represent the target resolution of the i-th scale, represents the up-sampling operation, and represent the total number of scales;

[0019] and the image feature map of the i-th scale image information is scaled to the i-th scale resolution according to the following formula, which is used to predict the i-th scale token sampling probability:

[0020]

[0021] wherein, represents the i-th scale image feature map after down-sampling, and represent the target resolution of the next scale; The BSQ multi-scale image decoder uses an image decoder

[0022] with the image feature map of the i-th scale image information as input, and decodes and reconstructs to obtain a generated image .

[0023] ​​​​​​​​In one example, in a digital human generation model building module, the conditional image guidance network includes an action condition extraction network and an action condition guidance network, which are used to extract action conditions based on color-coded images and inject them into a visual autoregressive Transformer module to achieve action-driven generation;

[0024] The action condition extraction network includes a conditional image encoder and an image feature encoder, which is used to extract conditional image features of corresponding scales from the color-coded image through the conditional image encoder, and extract features of corresponding scale feature maps from the downsampled corresponding scale image feature maps through the image feature encoder, and add the conditional image features and the features of the feature maps to obtain action conditions of corresponding scales;

[0025] The action condition guidance network includes multiple layers of bypass branch networks with the same structure. The first layer of bypass branch networks takes the action condition of the corresponding scale as input, and obtains the self-attention feature through adaptive normalized self-attention calculation. The output features of the Transformer module are input, and the self-attention features are concatenated with the output features of the Transformer module to obtain new features. The new features are used as the input of the next layer of Transformer module, and the self-attention features are used as the input of the next layer of bypass branch network to iteratively realize action-driven generation.

[0026] In one embodiment, in the digital human generation model construction module, the temporal attention layer extracts the temporal features of the image sequence by calculating the temporal correlation between consecutive multi-frame image sequences, specifically including:

[0027] Continuous frame Scaled image feature map As input, the feature is obtained after adaptive normalization self-attention layer and feedforward neural network And the features are obtained by dimension transformation , with characteristics The initial temporal features are obtained by temporal self-attention calculation for the input , after dimension transformation and residual addition, the time series features are output .

[0028] In one embodiment, in the digital human generation model construction module, the method of adjusting the temporal attention weight distribution using the temperature map includes:

[0029] Given Frame color-coded image sequence , the person foreground area mask is calculated based on the non-zero position in the color-coded image sequence. The calculation formula is as follows:

[0030] ,

[0031] ,

[0032] wherein, represents a person foreground region mask based on the person foreground region coordinates , represents a person foreground region mask corresponding to the color-coded image of the frame, represents a color-coded image of the frame, represents setting the background to a pure black RGB value;

[0033] The shortest distance of each pixel in the color-coded image sequence from the person foreground region is calculated based on the person foreground region mask, and the person foreground region temperature map is calculated according to the shortest distance, and the calculation formula is as follows:

[0034] ,

[0035] ,

[0036] wherein, represents the shortest distance, is a set of person foreground region pixel coordinates, represents a person foreground region temperature map, is used to adjust the influence degree of the temperature map on the temporal attention weight of the person foreground region, represents the image height, represents the image width;

[0037] Based on the resolution of the token sequence of different scales, the person foreground region temperature map is converted into a multi-scale temperature map , each scale temperature map is dimensionally transformed and spliced to obtain a temperature map , and the temperature map is used to adjust the temporal attention weight distribution.

[0038] In one embodiment, in the digital human generation model training module, the multi-stage training is divided into three stages of training, including:

[0039] The first stage of training is used to take the person identity label and the single-frame real image as the training data, establish a first stage of training loss function, train the digital human generation model without a conditional image guided network, and generate a target digital human image, wherein the first stage of training loss function is established by calculating the sum of cross-entropy of token sequences of all scales , all positions , and is represented as , is calculated as follows:

[0040] ,

[0041] wherein, represents the probability value of each discrete vector selected in the predicted discrete codebook, ， is a sampling probability matrix of the next scale token sequence, represents the discrete codebook index, represents the discrete codebook size, represents the resolution of the th scale, represents the real label of the th token sequence;

[0042] The second stage training is used to take the person identity label, the color coded image and the single frame real image as the training data, establish a second stage training loss function, train the conditional image guided network in the digital human generation model, and generate the target digital human image with controllable action, wherein the established second stage training loss function is represented as including the first stage training loss function and the LPIPS perceptual loss function, is calculated as follows:

[0043] ,

[0044] wherein, represents the single frame real image, represents the generated image, represents the single frame real image layer feature vector extracted by the pre-training network, represents the generated image layer feature vector extracted by the pre-training network, represents the number of feature layers of the pre-training network, represents the feature weight of the th layer feature vector, represents the LPIPS perceptual loss weight coefficient, represents the L2 distance;

[0045] The third stage training is used to take the person identity label, the continuous multi-frame color coded image sequence and the continuous multi-frame real image as the training data, establish a third stage training loss function, train the time sequence attention layer in the digital human generation model, and generate the target digital human image sequence with consistent action, wherein the established third stage training loss function is represented as , is calculated as follows:

[0046] ,

[0047] Where, Indicates the The first stage loss function corresponding to the frame, Indicates the total number of consecutive frames.

[0048] Furthermore, the pre-trained network used in the second stage of training is the VGG network.

[0049] In one embodiment, in the digital human image sequence generation module, generating a target digital human image sequence by adopting an interpolation smoothing inference method includes:

[0050] The character identity label and color-coded image are input into the trained digital human generation model, and the intermediate-scale image sequence is smoothed by the interpolation model RIFE, and the smoothed image is re-encoded to obtain the first Image feature map sequence at each scale , used for prediction Scale token sequence generation, scale-by-scale prediction until After the scale token sequence is generated, it is decoded and reconstructed through the BSQ multi-scale image decoder to generate the final target digital human image sequence.

[0051] On the other hand, the present invention also provides a method for generating a digital human based on a visual autoregressive model, using the digital human generation system based on a visual autoregressive model, comprising the following steps:

[0052] Obtain person identity labels and color-coded images from real images;

[0053] Based on a visual autoregressive model, a BSQ multi-scale image segmenter, a conditional image guidance network, and a temporal attention layer are introduced to construct a digital human generation model. This model enables the generation of target digital human image sequences guided by person identity labels and color-coded images. The BSQ multi-scale image segmenter is used to perform multi-scale encoding and decoding reconstruction on color-coded images. The conditional image guidance network is used to extract action conditions based on color-coded images and inject them into the visual autoregressive Transformer module. The temporal attention layer is used to extract temporal features of image sequences, and the temperature map is used to adjust the temporal attention weight distribution.

[0054] The constructed digital human generation model is trained in multiple stages. In the second stage of training, LPIPS perception loss is introduced to obtain the trained digital human generation model.

[0055] The person identity label and color-coded image are input into the trained digital human generation model to generate the target digital human image sequence.

[0056] Compared with the prior art, the present application has at least the following beneficial effects:

[0057] The digital human generation system and method based on the visual autoregressive model provided by the present application introduce a BSQ multi-scale image segmenter, a conditional image guided network and a timing attention layer to construct a digital human generation model based on the visual autoregressive model, adjust the timing attention weight distribution by using a temperature map, realize the generation of a target digital human image sequence guided by a character identity label and a color-coded image sequence, and improve the visual generation quality and timing consistency of the generated image sequence; during model training, an LPIPS perceptual loss is further introduced to directly constrain the visual quality of the generated image, further improving the visual generation quality of the generated image; during the digital human image sequence generation stage, an interpolation smoothing inference method is used to further improve the timing consistency of the generated image sequence; the modules of the present application cooperatively realize the generation of an action-controllable target digital human image sequence, and significantly improve the image sequence generation speed under the premise of ensuring the quality of the generated image. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced.

[0059] Figure 1 The structure schematic diagram of the digital human generation system based on the visual autoregressive model provided by the present application.

[0060] Figure 2 The color-coded image example diagram provided for the embodiments.

[0061] Figure 3 The structure schematic diagram of the visual autoregressive Transformer module provided for the embodiments.

[0062] Figure 4 The structure schematic diagram of the conditional image guided network provided for the embodiments.

[0063] Figure 5 The structure schematic diagram of the action condition extraction network in the conditional image guided network provided for the embodiments.

[0064] Figure 6 The structure schematic diagram of the action condition injection network in the conditional image guided network provided for the embodiments.

[0065] Figure 7 The example diagrams before and after the timing attention weight matrix adjustment provided for the embodiments.

[0066] Figure 8 The flow schematic diagram of the digital human generation method based on the visual autoregressive model provided by the present application. DETAILED DESCRIPTION

[0067] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and given in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0068] The inventive concept of the present application is that in the field of generative digital human technology, most of the existing methods are based on diffusion models and generative adversarial networks, wherein the diffusion model can generate high-quality digital human videos, but faces the problem of slow generation speed, and the visual autoregressive model can significantly improve the image generation speed compared to the diffusion model while ensuring the quality of the generated images, but the generative digital human technology based on the visual autoregressive model has not been thoroughly studied.

[0069] To solve this problem, the present application provides a digital human generation system based on a visual autoregressive model. The character identity label and the color-coded image are used to represent the character identity information and the character expression, gesture and body posture respectively, based on the visual autoregressive model VAR, the BSQ multi-scale image tokenizer, the conditional image guided network and the timing attention layer are introduced, the temperature map is used to adjust the timing attention weight distribution, the character identity label and the color-coded image sequence are used to guide the target digital human image sequence generation, and the visual generation quality and the timing consistency of the generated image sequence are improved. In the training stage of the digital human visual autoregressive model, the LPIPS perceptual loss is introduced to directly constrain the visual quality of the generated image, and the visual generation quality of the generated image is further improved. In the inference generation stage of the digital human visual autoregressive model, the frame interpolation smoothing inference method is used to further improve the timing consistency of the generated image sequence.

[0070] As shown in Figure 1 The digital human generation system based on the visual autoregressive model provided by the embodiment specifically includes the following modules: a data acquisition module, a digital human visual autoregressive model construction module, a model training module, and a digital human image sequence generation module. The first three modules complete the model development, and the fourth module realizes the model application.

[0071] 1. Data acquisition module: used for acquiring character identity labels and color-coded images from real images.

[0072] In the embodiment, the character identity label is a different digital label representing character identity information; the color-coded image is a conditional image drawn according to character face, hand and body key points, representing character expression, gesture and body posture, wherein the color-coded image is shown as Figure 2 .

[0073] 2. The digital human visual autoregressive model construction module is configured to introduce a BSQ multi-scale image tokenizer, a conditional image guided network and a time attention layer based on a visual autoregressive model to construct a digital human generation model, and realize generation of a target digital human image sequence guided by a character identity label and a color-coded image; wherein the BSQ multi-scale image tokenizer is configured to perform multi-scale encoding and decoding reconstruction on the color-coded image, the conditional image guided network is configured to extract action conditions according to the color-coded image and inject the visual autoregressive Transformer, and the time attention layer is configured to extract time sequence features of the image sequence and adjust the time attention weight distribution using a temperature map.

[0074] In the embodiment, the visual autoregressive model VAR is composed of a multi-scale image tokenizer and a visual autoregressive Transformer module, the multi-scale image tokenizer is configured to perform multi-scale encoding and decoding reconstruction on the image, and the visual autoregressive Transformer module is configured to predict high-scale image information according to low-scale image information. Based on the visual autoregressive model VAR, the BSQ multi-scale image tokenizer, the conditional image guided network and the time attention layer are introduced to realize generation of a target digital human image sequence guided by a character identity label and a color-coded image sequence.

[0075] In the model construction stage, the BSQ multi-scale image tokenizer is configured to perform multi-scale encoding and decoding reconstruction on the color-coded image; the conditional image guided network is configured to extract action conditions according to the color-coded image and inject the visual autoregressive Transformer module; the time attention layer is added in the visual autoregressive Transformer module, and the weight distribution of the time attention character foreground area is adjusted through the temperature map to realize time sequence modeling of the image sequence. After adding the time attention layer, the model structure of the visual autoregressive Transformer is shown as Figure 3 .

[0076] In the embodiment, the BSQ multi-scale image tokenizer is used to replace the multi-scale image tokenizer in the VAR to improve the multi-scale encoding and decoding reconstruction capability of the image, and further improve the visual generation quality of the digital human visual autoregressive model generated image. The BSQ multi-scale image tokenizer is composed of a BSQ multi-scale image encoder and a BSQ multi-scale image decoder, which can encode the image into a multi-scale token sequence and decode and reconstruct the image.

[0077] BSQ multi-scale image encoder utilizes an image encoder The input real image The encoded image feature map wherein is the real image resolution, is the image feature map resolution;

[0078] Then the different scale image information is extracted from the image feature map by the method of residual subtraction, the closest discrete vector in the discrete codebook is found for replacement to realize discrete quantization, and the multi-scale token sequence is obtained wherein is the discrete representation of scale image information, the scale token sequence resolution increases with the increase of scale , wherein is the dimension of each token vector. According to the multi-scale token sequence , the image feature map containing different scale image information can be constructed, as shown in the following formula:

[0079] ,

[0080] wherein, is the image feature map containing different scale image information, is the discrete representation of the th scale image information, and represent the target resolution of the th scale, represents the up-sampling operation, represents the total number of scales;

[0081] and the image feature map containing scale image information is scaled to scale resolution according to the following formula, which is used to predict scale token sampling probability:

[0082]

[0083] wherein, represents the th scale image feature map after down-sampling, and represent the target resolution of the next scale;

[0084] Again through the BSQ multi-scale image decoder, which utilizes an image decoder , to image feature map of scale image information for input, decoding reconstruction to obtain generated image .

[0085] In an embodiment, based on the conditional generation algorithm framework, a conditional image guided network composed of an action condition extraction network and an action condition guided network is introduced, the action condition is obtained according to the color-coded image, and the visual self-recurrent Transformer module is injected, realizing the color-coded image guided target digital human action driving, as shown in Figure 4 .

[0086] The action condition extraction network takes the corresponding scale image feature map and the color-coded image as input, and outputs the corresponding scale action condition, as shown in Figure 5 . Among them, the conditional image encoder and the image feature encoder are composed of two-dimensional convolution layers, which are respectively used to extract the features of the color-coded image and the corresponding scale image feature map. Taking scale as an example, taking scale color-coded image as input, image down-sampling and feature extraction are performed by using the conditional image encoder, and scale conditional image feature is output. Among them and represent the image down-sampling multiple. Taking scale image feature map as input, the image feature encoder is used to further extract the features of scale feature map . The scale conditional image feature and the feature map feature are added to obtain the scale action condition .

[0087] The action condition guided network is composed of multiple layers of bypass branch networks with the same structure, which is used to inject the action condition into the visual self-recurrent Transformer module, as shown in Figure 6 .

[0088] First, the scale conditional action is input into the adaptive normalization self-attention layer, and after the adaptive normalization operation, it is used as the query matrix (Query), the key-value matrix (Key, Value) in the self-attention operation, and the self-attention calculation is performed to obtain the self-attention feature , and the Transformer module output is spliced in the last dimension to obtain a new feature , as shown below:

[0089] ,

[0090] wherein, represents concatenation in feature dimension.

[0091] The new features are input into a linear layer to adjust the dimension to obtain as the input of the next layer of the Transformer module. as the input of the next layer of the bypass branch network, and the action condition is injected into the subsequent Transformer module. The action condition injection is realized by the conditional image guiding network through the above method, so that the action driving of the color-coded image is realized. The application adds an action condition guiding network in the rear layer of the visual self-recurrent Transformer.

[0092] In the embodiment, a time attention layer is added in the visual self-recurrent Transformer module to extract the time sequence features of the image sequence, the visual self-recurrent model is injected through the method of residual addition, and a temperature map is obtained according to the color-coded image sequence, which is used to adjust the weight distribution of the foreground region of the person in the time attention, improve the attention degree to the foreground region of the person, and realize the time sequence modeling of the image sequence.

[0093] The time attention layer is composed of a time Transformer, which realizes the time sequence modeling of the image sequence by calculating the time sequence correlation between continuous multiple frame image information. Specifically, the continuous frame scale image feature map is input, and the feature is obtained after passing through an adaptive normalization self-attention layer and a feedforward neural network. Before inputting into the time attention layer, the dimension of the feature matrix is adjusted, the time dimension is adjusted to the second last dimension, and the feature is obtained as the query matrix (Query) in the time self-attention operation, the key-value matrix (Key, Value), and the time sequence feature is obtained by time self-attention calculation. The time dimension is adjusted back to the first dimension to obtain , and the output of the current layer of the Transformer module is obtained by the method of residual addition, as the input of the next layer of the Transformer module.

[0094] The temperature map is used to adjust the weight matrix distribution of the time attention, and the weight distribution of the foreground region of the person in the time attention is adjusted. Specifically, given a color-coded image sequence , the foreground region mask of the person is calculated according to the non-zero position in the color-coded image sequence as shown in the following formula:

[0095] ,

[0096] ,

[0097] In the formula, represents a person foreground region mask based on the person foreground region coordinates , represents a person foreground region mask corresponding to the color-coded image of the frame, represents the color-coded image of the frame, represents the RGB value of setting the background to pure black;

[0098] Assuming is a set of pixel coordinates with a value of 1 in the person foreground mask, the distance map is calculated according to the following formula represents the shortest distance of each pixel in the color-coded image sequence to the person foreground region, and the person foreground region temperature map is calculated according to the shortest distance, and the calculation formula is as follows:

[0099] ,

[0100] ,

[0101] In the formula, represents the shortest distance, represents the person foreground region temperature map, is used to adjust the influence degree of the temperature map on the time sequence attention weight of the person foreground region, represents the image height, represents the image width; the person foreground region in the person foreground region temperature map is kept as 1, and the background region is set to different temperature values according to the distance from the foreground. The farther the distance, the greater the temperature value.

[0102] For the multi-scale generation problem of the visual self-recurrence model, the person foreground region temperature map needs to be scaled according to the resolution of the token sequence of different scales to obtain a multi-scale temperature map , each scale temperature map is dimensionally flattened, and then spliced in the 0th dimension to obtain a temperature map , wherein represents the total number of tokens, is the batch size. According to the temperature map , the time sequence attention weight matrix is adjusted to increase the weight of the person foreground region, as shown below:

[0103] ,

[0104] ,

[0105] ,

[0106] ,

[0107] wherein, , and denote the query matrix, key matrix and value matrix respectively, denotes attention score rearrangement, which enables each token to independently possess a continuous frame of the time sequence attention matrix; denotes temperature map rearrangement, which enables the temperature weight of each token to be extended to a continuous frame matrix by adding two dimensions to the flattened temperature map; denotes attention score adjustment, which enables token-level temperature adjustment; based on the standard attention mechanism, the time sequence feature is obtained by using the temperature-adjusted score for time sequence feature fusion , so as to enhance the time sequence feature of the foreground region of the character. Figure 7 The color-coded image, distance map and time sequence attention weight map before and after temperature map adjustment are shown in FIGS. 1-3, wherein the brighter the color, the greater the weight.

[0108] 3. A model training module configured to perform multi-stage training on the constructed digital human generation model, wherein during the second stage of training, a LPIPS perceptual loss is introduced to obtain the trained digital human generation model.

[0109] In the embodiment, a three-stage model training method is adopted, and the training targets, required training data and loss function definitions are different in different training stages. Each stage of training is further trained on the basis of the model trained in the previous stage, and the specific stages of training are introduced as follows:

[0110] First stage training: the first stage training target is to realize, based on the digital human visual self-recurrent model, the generation of a target digital human image conditioned on a character identity label; the first stage training does not temporarily introduce a conditional image guided network, and the training data thereof is a character identity label and a single-frame real image; the first stage training loss is represented as:

[0111] ,

[0112] wherein, represents a probability value of selecting each discrete vector in the predicted discrete codebook, ， sampling probability matrix for the next scale token sequence, denotes a discrete codebook index, denotes a discrete codebook size, denotes the resolution of the scale, denotes the resolution of the token sequence, only the real class corresponding position is 1, and the rest is 0.

[0113] The second stage training target is to realize, based on the digital human visual autoregressive model, generating action controllable target digital human image conditioned on the person identity label and the color coded image; the second stage training introduces a conditional image guided network, and freezes the parameters of other modules, only updating the parameters of the conditional image guided network, and the training data thereof is the person identity label, single frame real image and single frame color coded image; wherein the second stage training loss function is represented as including the first stage training loss function and the LPIPS perceptual loss function, and the calculation is as follows:

[0114] ,

[0115] In the formula, denotes a single frame real image, denotes a generated image, denotes the single frame real image feature vector of the layer extracted by the VGG network, denotes the generated image feature vector of the layer extracted by the VGG network, denotes the number of feature layers of the VGG network, denotes the feature weight of the layer feature vector, denotes the LPIPS perceptual loss weight coefficient, denotes the L2 distance. By introducing the LPIPS perceptual loss, the direct constraint on the visual quality of the generated image is realized.

[0116] The third stage training is to realize, based on the digital human visual autoregressive model, generating action consistent target digital human image sequence conditioned on the person identity label and the color coded image sequence; in the model training stage, a timing attention layer is introduced, and the timing attention layer in the digital human generation model is trained, and the parameters of other modules are frozen, only the network parameters of the timing attention layer are updated, and the training data thereof is the person identity label, continuous multi-frame real image sequence and continuous multi-frame color coded image sequence; the third stage training loss is represented as:

[0117] ,

[0118] In the formula, represents the first frame corresponding to the first stage loss function, represents the total number of consecutive frames. This stage does not join the LPIPS perceptual loss, and continues the training of the time attention layer module on the basis of the trained model in the second stage, which is used to reduce the memory requirement during training.

[0119] 4. Digital human image sequence generation module: which is used to input the character identity label and color coded image into the trained digital human generation model to generate the target digital human image sequence.

[0120] In the embodiment, in the digital human image sequence generation inference stage, a scale-by-scale prediction method is adopted to generate the target digital human image sequence according to the character identity label and the color coded image sequence; the action condition of the color coded image sequence is extracted through the conditional image guided network and injected into the visual self-recurrent Transformer module; the time sequence modeling of the image sequence is realized through the time attention layer and the temperature map; the interpolation smoothing inference method is adopted to introduce the video interpolation model RIFE to perform interpolation smoothing on the generated intermediate scale image sequence; and the generated image sequence is obtained by decoding and reconstructing through the BSQ multi-scale image tokenizer.

[0121] The RIFE model is a lightweight video interpolation model, which designs a video interpolation method based on intermediate flow estimation. Given adjacent image frames and , and then synthesize the intermediate frame . Taking the prediction of the scale token sequence as an example, the action condition of the scale color coded image sequence is extracted through the conditional image guided network , and the scale image feature map sequence is input into the digital human visual self-recurrent model, and the scale token sequence is output . According to the token sequence of no more than scale , scale feature map sequence is reconstructed , and the scale image sequence is obtained by decoding and reconstructing through the BSQ multi-scale image decoder , and the RIFE model is used to perform interpolation smoothing using adjacent frames to obtain the smoothed scale image sequence , as shown below:

[0122]

[0123] Then the BSQ multi-scale image encoder is used to encode the smoothed image sequence scale image sequence to the latent space, obtaining scale image feature map sequence for subsequent scale token sequence generation. Repeat the above process until scale token sequence generation is completed, and finally use the BSQ multi-scale image decoder to decode and reconstruct to obtain the generated image sequence , thereby improving the temporal consistency of the generated image sequence and generating the target digital human image sequence.

[0124] Based on the same inventive concept, the embodiments also provide a digital human generation method based on a visual autoregressive model, which uses the digital human generation system based on the visual autoregressive model, as shown in Figure 8 , comprising the following steps:

[0125] obtaining a character identity label and a color-coded image from a real image;

[0126] Based on the visual autoregressive model, a BSQ multi-scale image tokenizer, a conditional image guide network, and a temporal attention layer are introduced to construct a digital human generation model, and the character identity label and the color-coded image are used to guide the generation of the target digital human image sequence; wherein the BSQ multi-scale image tokenizer is used for multi-scale encoding and decoding reconstruction of the color-coded image, the conditional image guide network is used for extracting action conditions from the color-coded image and injecting them into the visual autoregressive Transformer module, and the temporal attention layer is used for extracting the temporal features of the image sequence and adjusting the temporal attention weight distribution using the temperature map.

[0127] The constructed digital human generation model is trained in multiple stages, wherein in the second stage of training, a LPIPS perceptual loss is introduced to obtain the trained digital human generation model.

[0128] The character identity label and the color-coded image are input into the trained digital human generation model to generate the target digital human image sequence.

[0129] In addition, the terms "upper", "lower", "inner", "outer", "front", "back" are only used for description purposes and cannot be understood as indicating or implying relative importance. Unless otherwise specified, the relative steps, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present application. Of course, the above only describes specific embodiments of the present application and is not intended to limit the scope of the present application. Any equivalent changes or modifications made in accordance with the structure, features, and principles described in the patent application scope of the present application should be included in the patent application scope of the present application.

[0130] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present application, and are used to illustrate the technical solutions of the present application, but are not limitations thereof. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features within the technical range disclosed by the present application. The modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A digital human generation system based on a visual auto-regressive model, characterized by, The method comprises the following modules: a data acquisition module for acquiring a character identity label and a color-coded image from a real image; a digital human visual self-recurrence model construction module for constructing a digital human generation model based on a visual self-recurrence model, introducing a BSQ multi-scale image tokenizer, a conditional image guide network and a time sequence attention layer, and realizing generation of a target digital human image sequence guided by the character identity label and the color-coded image; wherein the BSQ multi-scale image tokenizer is used for multi-scale encoding and decoding reconstruction of the color-coded image, the conditional image guide network is used for extracting action conditions from the color-coded image and injecting the visual self-recurrence Transformer module, and the time sequence attention layer in the Transformer module is used for extracting time sequence features of the image sequence and adjusting the time sequence attention weight distribution by using a temperature map; a model training module for training the constructed digital human generation model in multiple stages, wherein in the second stage of training, a LPIPS perceptual loss is introduced to obtain a trained digital human generation model; a digital human image sequence generation module for inputting the character identity label and the color-coded image into the trained digital human generation model to generate a target digital human image sequence.

2. The digital human generation system based on visual autoregressive model according to claim 1, wherein, In the data acquisition module, the character identity label is a different digital label, which is used to represent the character identity information; The color-coded image is a conditional image drawn according to the character face, hand and body key points, which is used to represent the character expression, gesture and body posture.

3. The digital human generation system based on visual autoregressive model according to claim 1, wherein, In the digital human generation model construction module, the BSQ multi-scale image tokenizer comprises a BSQ multi-scale image encoder and a BSQ multi-scale image decoder, which are used for multi-scale encoding and decoding reconstruction of the color-coded image; The BSQ multi-scale image encoder encodes the input real image by using an image encoder to obtain an image feature map, extracts different scale image information from the image feature map, and obtains a multi-scale token sequence; An image feature map containing different scale image information is constructed according to the multi-scale token sequence, as shown in the following formula: ， wherein, is an image feature map comprising image information of different scales, is a discrete representation of the image information of the th scale, and denotes a target resolution of the th scale, denotes an up-sampling operation, denotes the total number of scales; and according to the following formula image feature map of scale image information scaled to scale resolution, for predicting scale token sampling probability: ， wherein, denotes the down-sampled feature map of the and denotes the target resolution of the next scale.​ The BSQ multi-scale image decoder utilizes an image decoder To The image feature map of the scale image information As input, the decoded reconstruction obtains the generated image .

4. The digital human generation system based on visual autoregressive model according to claim 3, wherein, In the digital human generation model construction module, the conditional image guide network comprises an action condition extraction network and an action condition guide network, which are used for extracting action conditions from the color-coded image and injecting the visual self-recurrence Transformer module to realize action-driven generation; The action condition extraction network comprises a conditional image encoder and an image feature encoder, which are used for extracting a corresponding scale conditional image feature from the color-coded image by the conditional image encoder, extracting a corresponding scale feature map feature from the down-sampled corresponding scale image feature map by the image feature encoder, adding the conditional image feature and the feature map feature, and obtaining the corresponding scale action condition; The action-conditioned guiding network comprises a plurality of bypass branch networks with the same structure, a first bypass branch network taking an action condition of a corresponding scale as input, performing self-adaptive normalization self-attention calculation to obtain a self-attention feature, and simultaneously taking a feature map of the corresponding scale as input The self-attention feature and the feature output by the Transform module are spliced to obtain a new feature, the new feature is taken as input of a next Transform module, and the self-attention feature is taken as input of a next bypass branch network, so as to iteratively generate an action drive.

5. The digital human generation system based on visual autoregressive model according to claim 4, wherein, In the digital human generation model construction module, the time sequence attention layer extracts the time sequence features of the image sequence by calculating the time sequence correlation between continuous multiple frame image sequences, and specifically comprises: Continuous frame Scaled image feature map As input, the features are obtained through adaptive normalized self-attention layer and feedforward neural network And the features are obtained by dimension transformation , with characteristics The initial temporal features are obtained by temporal self-attention calculation for the input , after dimension transformation and residual addition, the time series features are output .

6. The digital human generation system based on visual autoregressive model according to claim 5, wherein, In the digital human generation model construction module, the temperature map is used to adjust the time sequence attention weight distribution, which comprises: Given frame color-coded image sequence According to the non-zero position in the color-coded image sequence, the foreground region mask of the person is calculated, and the calculation formula is as follows: ， ， wherein, represents a person foreground region mask based on the person foreground region coordinates represents a first frame color-coded image corresponding to the person foreground region mask, represents a second frame color-coded image, represents an RGB value with the background set to pure black;​ The shortest distance of each pixel in the color-coded image sequence to the character foreground region is calculated based on the character foreground region mask, and a character foreground region temperature map is calculated according to the shortest distance, and the calculation formula is as follows: ， ， In the formula, denotes the shortest distance, is a set of pixel coordinates of the human foreground region, denotes a temperature map of the human foreground region, is used to adjust the degree of influence of the temperature map on the temporal attention weight of the human foreground region, denotes the image height, denotes the image width; based on resolutions of different scale token sequences, the character foreground region temperature map is converted into a multi-scale temperature map , dimensional transformation and splicing are performed on the temperature maps of different scales to obtain a temperature map , and the temperature map is used to adjust the timing attention weight distribution.

7. The digital human generation system based on visual autoregressive model according to claim 6, wherein, In the digital human generation model training module, the multi-stage training is divided into three stages, including: The first stage training is used to take the person identity label and the single-frame real image as the training data, establish a first stage training loss function, train the digital human generation model without the conditional image guided network, and generate the target digital human image, wherein the first stage training loss function is established by calculating the sum of token sequence cross-entropies of all scales , all positions , and is expressed as , and is calculated as follows: ， In the formula, represents the probability value of each discrete vector selected in the predicted discrete codebook, ， is a sampling probability matrix of the next scale token sequence, represents the discrete codebook index, represents the discrete codebook size, represents the resolution of the first scale, represents the resolution of the first token sequence, and the real label of the token sequence. The second stage training is used to take the person identity label, the color coded image and the single frame real image as the training data, establish a second stage training loss function, train the conditional image guided network in the digital human generation model, and generate the target digital human image controllable in action, wherein the established second stage training loss function is represented as The first stage training loss function and the LPIPS perceptual loss function are included, and the calculation is as follows: ， In the formula, represents a single-frame real image, represents a generated image, represents a first layer feature vector of the single-frame real image extracted by the pre-training network, represents a first layer feature vector of the generated image extracted by the pre-training network, represents a number of feature layers of the pre-training network, represents a first layer feature vector, represents an LPIPS perceptual loss weight coefficient, represents an L2 distance; The third stage training is used to take the person identity label, the continuous multi-frame color coding image sequence and the continuous multi-frame real image as the training data, establish a third stage training loss function, train the time sequence attention layer in the digital human generation model, and generate the target digital human image sequence consistent with the action, wherein the third stage training loss function is expressed as , and is calculated as follows: ， In the formula, represents the first frame corresponding to the first stage loss function, represents the total number of consecutive frames.

8. The digital human generation system based on visual autoregressive model according to claim 7, wherein, In the digital human image sequence generation module, the target digital human image sequence is generated by using an interpolation smoothing inference method, including: The character identity label and color-coded image are input into the trained digital human generation model, and the intermediate-scale image sequence is smoothed by the interpolation model RIFE, and the smoothed image is re-encoded to obtain the first Image feature map sequence at each scale , used for prediction Scale token sequence generation, scale-by-scale prediction until After the scale token sequence is generated, it is decoded and reconstructed through the BSQ multi-scale image decoder to generate the final target digital human image sequence.

9. A digital human generation method based on a visual autoregressive model, characterized by, The digital human generation method uses the digital human generation system based on the visual self-recurrence model in any one of claims 1-8, including the following steps: Obtain the character identity label and the color-coded image from the real image; Based on the visual self-recurrence model, a BSQ multi-scale image tokenizer, a conditional image guide network and a timing attention layer are introduced to construct a digital human generation model, and the generation of the target digital human image sequence is guided by the character identity label and the color-coded image; wherein the BSQ multi-scale image tokenizer is used for multi-scale encoding and decoding reconstruction of the color-coded image, the conditional image guide network is used for extracting action conditions according to the color-coded image and injecting the visual self-recurrence Transformer module, and the timing attention layer is used for extracting the timing features of the image sequence, and adjusting the timing attention weight distribution by using the temperature map; The constructed digital human generation model is trained in multiple stages, wherein in the second stage of training, an LPIPS perceptual loss is introduced to obtain the trained digital human generation model; The character identity label and the color-coded image are input into the trained digital human generation model to generate the target digital human image sequence.

Images