Digital human generation system and method based on visual autoregression model

Abstract

The invention provides a digital human generation system based on a visual autoregression model, and the system introduces a BSQ multi-scale image word segmentation device, a condition image guide network and a time sequence attention layer based on the visual autoregression model to construct a digital human generation model, and adjusts the time sequence attention weight distribution through a temperature map. Target digital human image sequence generation guided by a human identity tag and a color coding image sequence is realized, and the visual generation quality and the time sequence consistency of the generated image sequence are improved; lPIPS perception loss is introduced during model training, direct constraint on the visual quality of the generated image is realized, and the visual generation quality of the generated image is further improved; in the digital human image sequence generation stage, a frame insertion smooth reasoning method is adopted, and the time sequence consistency of the generated image sequence is further improved; according to the invention, the modules cooperate to realize the generation of the motion-controllable target digital human image sequence, and the image sequence generation speed is obviously improved on the premise of ensuring the quality of the generated image.

Description

Technical Field

[0001] The present invention belongs to the technical field of generative digital humans, and specifically relates to a digital human generation system and method based on a visual autoregressive model. Background Art

[0002] With the continuous development of technologies such as computer graphics and artificial intelligence, and the rise of the metaverse, virtual humans, as a new audiovisual medium, have gradually entered the public eye. Virtual humans are avatars with digital appearances that can simulate human behavior and interactions, created using computer graphics, artificial intelligence, and machine learning.

[0003] Virtual human technology consists of four components: modeling, actuation, presentation, and interaction. Modeling involves constructing the structure and appearance of the digital human; actuation involves controlling the digital human's posture, expression, and behavior; presentation involves visually displaying the digital human image to the user; and interaction involves enabling interaction between the digital human, the user, and the environment. Traditional digital human construction relies primarily on computer graphics and real-life motion capture technology. This requires the collection of large amounts of real-life data and in-depth modeling, which is time-consuming and inefficient. With the rapid development of artificial intelligence (AI), various generative methods based on neural networks, such as generative adversarial networks, variational autoencoders, and diffusion models, have been widely researched. These generative methods, known as AIGC (Artificial Intelligence Generative Content), can efficiently generate target digital humans driven by data and algorithms. These methods eliminate the need for professional modelers and animators, and require complex and expensive specialized motion capture equipment, significantly reducing the cost of digital human production and promoting the widespread application of digital human technology across various industries.

[0004] In recent years, AIGC-based digital human generation technology has been widely researched, significantly improving the quality, controllability, and stability of digital human generation. Based on different technical approaches, these technologies can be categorized as those based on generative adversarial networks, diffusion models, neural radiation fields, and three-dimensional Gaussian models. Each approach has distinct characteristics. For example, diffusion-based digital human generation technology can generate high-quality digital humans with strong generalization capabilities. However, because diffusion models require multiple steps of reverse denoising during inference, generation speed is often slow, becoming a major bottleneck for this approach.

[0005] Generative techniques based on autoregressive models have been widely used in research on large language models, demonstrating unprecedented generative capabilities in natural language tasks. Autoregressive models have also demonstrated strong generative capabilities in image generation. In category-guided and text-guided image generation tasks, work based on visual autoregressive models has surpassed work based on diffusion models, generating higher-quality images while significantly reducing image inference time. However, generative digital human technology based on visual autoregressive models has not yet been thoroughly researched and explored, and the application of visual autoregressive models in the field of generative digital humans is currently unknown. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the purpose of the present invention is to provide a digital human generation system based on a visual autoregressive model. The system can realize the generation of digital human image sequences with multiple identities, high fidelity, and temporal continuity. Compared with the diffusion model, it significantly improves the image sequence generation speed and has broad application prospects in multiple fields.

[0007] The embodiment provides a digital human generation system based on a visual autoregressive model, comprising the following modules: A data acquisition module, which is used to obtain person identity labels and color-coded images from real images; The digital human visual autoregressive model construction module is used to build a digital human generation model based on the visual autoregressive model, introducing the BSQ multi-scale image segmenter, the conditional image guidance network, and the temporal attention layer to achieve the generation of target digital human image sequences guided by character identity labels and color-coded images. Among them, the BSQ multi-scale image segmenter is used to perform multi-scale encoding and decoding reconstruction on the color-coded images, the conditional image guidance network is used to extract action conditions based on the color-coded images and inject them into the visual autoregressive Transformer module, and the temporal attention layer in the Transformer module is used to extract the temporal features of the image sequence and use the temperature map to adjust the temporal attention weight distribution. The model training module is used to perform multi-stage training on the constructed digital human generation model. In the second stage of training, LPIPS perception loss is introduced to obtain the trained digital human generation model; The digital human image sequence generation module is used to input the character identity label and color-coded image into the trained digital human generation model to generate the target digital human image sequence.

[0008] In one embodiment, in the data acquisition module, the person identity tags are different digital tags used to represent the person identity information; The color-coded image is a conditional image drawn based on the key points of the character's face, hands, and body, and is used to represent the character's expression, gestures, and body posture.

[0009] In one embodiment, in the digital human generation model construction module, the BSQ multi-scale image segmenter includes a BSQ multi-scale image encoder and a BSQ multi-scale image decoder, which are used to perform multi-scale encoding and decoding reconstruction on the color-coded image; The BSQ multi-scale image encoder uses an image encoder to encode the input real image to obtain an image feature map, extracts image information of different scales from the image feature map, and obtains a multi-scale token sequence; The image feature map containing image information of different scales is constructed based on the multi-scale token sequence, as shown in the following formula: , in, is an image feature map containing image information of different scales, For the Discrete representation of image information at different scales, and Indicates the Target resolution at each scale, represents the upsampling operation, Indicates the total number of scales; And according to the following formula Image feature map of scale image information Zoom to Scale resolution, used for prediction Scale token sampling probability:

[0010] in, Represents the first scale image feature maps, and Indicates the target resolution of the next scale; The BSQ multi-scale image decoder, which uses an image decoder ,by Image feature map of scale image information As input, decode and reconstruct to generate the image .

[0011] In one example, in a digital human generation model building module, the conditional image guidance network includes an action condition extraction network and an action condition guidance network, which are used to extract action conditions based on color-coded images and inject them into a visual autoregressive Transformer module to achieve action-driven generation; The action condition extraction network includes a conditional image encoder and an image feature encoder, which is used to extract conditional image features of corresponding scales from the color-coded image through the conditional image encoder, and extract features of corresponding scale feature maps from the downsampled corresponding scale image feature maps through the image feature encoder, and add the conditional image features and the features of the feature maps to obtain action conditions of corresponding scales; The action condition guidance network includes multiple layers of bypass branch networks with the same structure. The first layer of bypass branch networks takes the action condition of the corresponding scale as input, and obtains the self-attention feature through adaptive normalized self-attention calculation. The output features of the Transformer module are input, and the self-attention features are concatenated with the output features of the Transformer module to obtain new features. The new features are used as the input of the next layer of Transformer module, and the self-attention features are used as the input of the next layer of bypass branch network to iteratively realize action-driven generation.

[0012] In one embodiment, in the digital human generation model construction module, the temporal attention layer extracts the temporal features of the image sequence by calculating the temporal correlation between consecutive multi-frame image sequences, specifically including: Continuous frame Scaled image feature map As input, the feature is obtained after adaptive normalization self-attention layer and feedforward neural network And the features are obtained by dimension transformation , with characteristics The initial temporal features are obtained by temporal self-attention calculation for the input , after dimension transformation and residual addition, the time series features are output .

[0013] In one embodiment, in the digital human generation model construction module, the method of adjusting the temporal attention weight distribution using the temperature map includes: Given Frame color-coded image sequence , the person foreground area mask is calculated based on the non-zero position in the color-coded image sequence. The calculation formula is as follows: , , Where, Indicates the coordinates of the foreground area based on the character The character foreground area mask, Indicates the The character foreground area mask corresponding to the frame color-coded image, Indicates the frame color-coded image, Indicates the RGB value that sets the background to pure black; Based on the person foreground area mask, the shortest distance between each pixel in the color-coded image sequence and the person foreground area is calculated. The temperature map of the person foreground area is calculated based on the shortest distance. The calculation formula is as follows: , , Where, Indicates the shortest distance, is the pixel coordinate set of the character foreground area, Represents the temperature map of the character foreground area, Used to adjust the influence of the temperature map on the temporal attention weight of the character foreground area. Indicates the image height, Indicates the image width; Based on the resolution of token sequences of different scales, the temperature map of the foreground area of ​​the character is Convert to multi-scale temperature map , transform and splice the temperature maps of each scale to obtain the temperature map , using the temperature map Adjust temporal attention weight distribution.

[0014] In one embodiment, in the digital human generation model training module, the multi-stage training is divided into three stages of training, including: The first stage of training is used to establish the first stage training loss function using the person identity label and a single frame of real image as training data, and to train the digital human generation model that does not contain the conditional image guidance network to generate the target digital human image. The first stage training loss function is established by calculating all scales. , all locations The sum of the cross entropy of the token sequence is expressed as , calculated as follows: , Where, Represents the probability value of each discrete vector selected in the predicted discrete codebook, ， is the sampling probability matrix of the next scale token sequence, represents the discrete codebook index, represents the discrete codebook size, Indicates the The resolution of the scale, Indicates the The true label of the token sequence; The second stage training is used to establish the second stage training loss function, which uses the character identity label, color-coded image and single-frame real image as training data, trains the conditional image guidance network in the digital human generation model, and generates the target digital human image with controllable action. The second stage training loss function is expressed as Including the first stage training loss function and LPIPS perception loss function, the calculation is as follows: , Where, represents a single frame of real image, Indicates the generated image, Represents a single frame of real image extracted using the pre-trained network Layer feature vector, Represents the generated image extracted using the pre-trained network Layer feature vector, Indicates the number of pre-trained network feature layers, Indicates the The feature weights of the layer feature vectors, represents the LPIPS perception loss weight coefficient, represents the L2 distance; The third stage training is used to establish the third stage training loss function, which uses the character identity label, continuous multi-frame color-coded image sequence and continuous multi-frame real image as training data, trains the temporal attention layer in the digital human generation model, and generates a target digital human image sequence with consistent movements. The established third stage training loss function is expressed as , calculated as follows: , Where, Indicates the The first stage loss function corresponding to the frame, Indicates the total number of consecutive frames.

[0015] Furthermore, the pre-trained network used in the second stage of training is the VGG network.

[0016] In one embodiment, in the digital human image sequence generation module, generating a target digital human image sequence by adopting an interpolation smoothing inference method includes: The character identity label and color-coded image are input into the trained digital human generation model, and the intermediate-scale image sequence is smoothed by the interpolation model RIFE, and the smoothed image is re-encoded to obtain the first Image feature map sequence at each scale , used for prediction Scale token sequence generation, scale-by-scale prediction until After the scale token sequence is generated, it is decoded and reconstructed through the BSQ multi-scale image decoder to generate the final target digital human image sequence.

[0017] On the other hand, the present invention also provides a method for generating a digital human based on a visual autoregressive model, using the digital human generation system based on a visual autoregressive model, comprising the following steps: Obtain person identity labels and color-coded images from real images; Based on a visual autoregressive model, a BSQ multi-scale image segmenter, a conditional image guidance network, and a temporal attention layer are introduced to construct a digital human generation model. This model enables the generation of target digital human image sequences guided by person identity labels and color-coded images. The BSQ multi-scale image segmenter is used to perform multi-scale encoding and decoding reconstruction on color-coded images. The conditional image guidance network is used to extract action conditions based on color-coded images and inject them into the visual autoregressive Transformer module. The temporal attention layer is used to extract temporal features of image sequences, and the temperature map is used to adjust the temporal attention weight distribution. The constructed digital human generation model is trained in multiple stages. In the second stage of training, LPIPS perception loss is introduced to obtain the trained digital human generation model. The person identity label and color-coded image are input into the trained digital human generation model to generate the target digital human image sequence.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The digital human generation system and method based on a visual autoregressive model provided by the present invention introduces a BSQ multi-scale image segmenter, a conditional image guidance network and a temporal attention layer based on the visual autoregressive model to construct a digital human generation model, uses a temperature map to adjust the temporal attention weight distribution, realizes the generation of a target digital human image sequence guided by a character identity label and a color-coded image sequence, and improves the visual generation quality and temporal consistency of the generated image sequence; during model training, LPIPS perception loss is further introduced to directly constrain the visual quality of the generated image, further improving the visual generation quality of the generated image; in the digital human image sequence generation stage, an interpolation smoothing inference method is adopted to further improve the temporal consistency of the generated image sequence; the various modules of the present invention cooperate to realize the generation of a target digital human image sequence with controllable motion, and significantly improve the image sequence generation speed while ensuring the quality of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0020] Figure 1 This is a structural diagram of the digital human generation system based on the visual autoregressive model provided by the present invention.

[0021] Figure 2 An example of a color-coded image provided for an embodiment.

[0022] Figure 3 Schematic diagram of the structure of the visual autoregressive Transformer module provided in the embodiment.

[0023] Figure 4 A schematic diagram of the structure of the conditional image guidance network provided in the embodiment.

[0024] Figure 5 A schematic diagram of the structure of the action condition extraction network in the conditional image guidance network provided in the embodiment.

[0025] Figure 6 A schematic diagram of the structure of an action condition injection network in a conditional image guidance network provided in an embodiment.

[0026] Figure 7 Example diagrams before and after adjustment of the temporal attention weight matrix provided in the embodiment.

[0027] Figure 8 This is a flow chart of the digital human generation method based on the visual autoregressive model provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and given in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0029] The inventive concept of the present invention is: in the field of generative digital human technology, most existing methods are based on diffusion models and generative adversarial networks. Among them, the diffusion model can generate high-quality digital human videos, but faces the problem of slow generation speed. The visual autoregressive model can significantly improve the image generation speed compared to the diffusion model while ensuring the quality of the generated image. However, the generative digital human technology based on the visual autoregressive model has not yet been deeply studied.

[0030] To solve this problem, the present invention provides a digital human generation system based on a visual autoregressive model. Character identity labels and color-coded images are used to represent character identity information and character expressions, gestures and body postures respectively. Based on the visual autoregressive model VAR, a BSQ multi-scale image segmenter, a conditional image guidance network and a temporal attention layer are introduced. The temperature map is used to adjust the temporal attention weight distribution to achieve the generation of target digital human image sequences guided by character identity labels and color-coded image sequences, and improve the visual generation quality and temporal consistency of the generated image sequences. In the training stage of the digital human visual autoregressive model, the LPIPS perceptual loss is introduced to directly constrain the visual quality of the generated images, thereby further improving the visual generation quality of the generated images. In the inference generation stage of the digital human visual autoregressive model, the interpolation smoothing inference method is used to further improve the temporal consistency of the generated image sequence.

[0031] like Figure 1 As shown, the embodiment provides a digital human generation system based on a visual autoregressive model, which specifically includes the following modules: a data acquisition module, a digital human visual autoregressive model construction module, a model training module, and a digital human image sequence generation module. The first three modules complete the model development, and the fourth module realizes the model application.

[0032] 1. Data acquisition module: It is used to obtain person identity labels and color-coded images from real images.

[0033] In the embodiment, the character identity tags are different digital tags, indicating the character identity information; the color-coded images are conditional images drawn based on the key points of the character's face, hands and body, indicating the character's expression, gestures and body posture, wherein the color-coded images are exemplified as follows: Figure 2 shown.

[0034] 2. Digital human visual autoregressive model construction module: It is used to build a digital human generation model based on the visual autoregressive model, introducing the BSQ multi-scale image segmenter, the conditional image guidance network and the temporal attention layer, to realize the generation of target digital human image sequences guided by character identity labels and color-coded images; among them, the BSQ multi-scale image segmenter is used to perform multi-scale encoding and decoding reconstruction on color-coded images, the conditional image guidance network is used to extract action conditions based on color-coded images and inject them into the visual autoregressive Transformer, and the temporal attention layer is used to extract the temporal features of the image sequence, and the temperature map is used to adjust the temporal attention weight distribution.

[0035] In this embodiment, the visual autoregressive model (VAR) consists of a multi-scale image segmenter and a visual autoregressive Transformer module. The multi-scale image segmenter is used to perform multi-scale encoding and decoding reconstruction of images, and the visual autoregressive Transformer module predicts high-scale image information based on low-scale image information. Based on the visual autoregressive model (VAR), a BSQ multi-scale image segmenter, a conditional image guidance network, and a temporal attention layer are introduced to achieve the generation of target digital human image sequences guided by character identity labels and color-coded image sequences.

[0036] In the model construction phase, the BSQ multi-scale image segmenter is used to perform multi-scale encoding and decoding reconstruction on the color-coded image; the conditional image guidance network is used to extract action conditions based on the color-coded image and inject them into the visual autoregressive Transformer module; a temporal attention layer is added to the visual autoregressive Transformer module, and the weight distribution of the temporal attention figure foreground area is adjusted through the temperature map to achieve temporal modeling of the image sequence. After adding the temporal attention layer, the model structure of the visual autoregressive Transformer is as follows: Figure 3 shown.

[0037] In this embodiment, a BSQ multi-scale image tokenizer replaces the multi-scale image tokenizer in VAR, improving the multi-scale image encoding and decoding reconstruction capabilities, thereby improving the visual generation quality of images generated by the digital human visual autoregressive model. The BSQ multi-scale image tokenizer consists of a BSQ multi-scale image encoder and a BSQ multi-scale image decoder. It encodes an image into a multi-scale token sequence and decodes it to reconstruct the image.

[0038] BSQ multi-scale image encoder uses image encoder The real image input Encode to obtain image feature map ,in is the actual image resolution, is the image feature map resolution; Then, the residual subtraction method is used to extract the image feature map Extract image information of different scales, find the closest discrete vector in the discrete codebook for replacement, implement discrete quantization, and obtain a multi-scale token sequence .in for Discrete representation of scale image information, Standard token sequence resolution With the scale Increase and increase, among which For each token vector dimension. According to the multi-scale token sequence An image feature map containing image information of different scales can be constructed as shown in the following formula: , in, is an image feature map containing image information of different scales, For the Discrete representation of image information at different scales, and Indicates the Target resolution at each scale, represents the upsampling operation, Indicates the total number of scales; And according to the following formula, Image feature map of scale image information Zoom to Scale resolution, used for prediction Scale token sampling probability:

[0039] in, Represents the first scale image feature maps, and Indicates the target resolution of the next scale; Then through the BSQ multi-scale image decoder, which uses the image decoder ,by Image feature map of scale image information As input, decode and reconstruct to generate the image .

[0040] In the embodiment, based on the conditional generation algorithm framework, a conditional image guidance network consisting of an action condition extraction network and an action condition guidance network is introduced to obtain action conditions according to the color-coded image and inject a visual autoregressive Transformer module to achieve the target digital human action driving guided by the color-coded image, such as Figure 4 shown.

[0041] The action condition extraction network takes the corresponding scale image feature map and color-coded image as input and outputs the corresponding scale action condition, such as Figure 5 As shown in Figure 2, the conditional image encoder and image feature encoder are composed of two-dimensional convolutional layers, which are used to extract the features of the color-coded image and the corresponding scale image feature map respectively. For example, take the scale Scaled color-coded image As input, the conditional image encoder is used for image downsampling and feature extraction, and the output Scale-Conditional Image Features .in and Indicates the image downsampling multiple. Scaled image feature map As input, the image feature encoder is used to further extract Features of scale feature maps .Will The scale-condition image features and feature map features are added to obtain Scale action conditions .

[0042] The action condition guidance network consists of multiple layers of bypass branch networks with the same structure, which is used to inject action conditions into the visual autoregressive Transformer module, such as Figure 6 shown.

[0043] First, Scale Conditional Action Input the adaptive normalized self-attention layer, perform the adaptive normalization operation, and use it as the query matrix (Query) and key value matrix (Key, Value) in the self-attention operation to perform self-attention calculation and obtain the self-attention feature. , and the Transformer module output Splicing in the last dimension to get new features , as shown below: , in, Represents concatenation on the feature dimension.

[0044] The new features Input linear layer is used to adjust the dimension and get As the input of the next layer Transformer module. As the input of the next layer of bypass branch network, the action condition is injected into the following Transformer module. The above method guides the network through the conditional image to achieve the action condition injection, thereby realizing the action drive of the color-coded image. The layer adds action conditions to guide the network.

[0045] In an embodiment, a temporal attention layer is added to the visual autoregressive Transformer module to extract the temporal features of the image sequence, which are injected into the visual autoregressive model through the residual addition method, and a temperature map is obtained based on the color-coded image sequence to adjust the weight distribution of the person foreground area in the temporal attention, increase the degree of attention to the person foreground area, and realize temporal modeling of the image sequence.

[0046] The temporal attention layer is composed of a temporal transformer, which realizes temporal modeling of image sequences by calculating the temporal correlation between consecutive multi-frame image information. frame Scaled image feature map As input, the feature is obtained after the adaptive normalization self-attention layer and feedforward neural network Before entering the temporal attention layer, adjust the features Matrix dimensions, adjust the time dimension to the second to last dimension, and get , as the query matrix (Query) and key-value matrix (Key, Value) in the temporal self-attention operation, perform temporal self-attention calculation to obtain temporal features , adjust the time dimension back to the first dimension to obtain , by adding the residuals, we get the output of the Transformer module of this layer , as the input of the next layer Transformer module.

[0047] The temperature map is used to adjust the temporal attention weight matrix distribution and the weight distribution of the character foreground area in the temporal attention. Specifically, given Frame color-coded image sequence , the character foreground area mask is calculated based on the non-zero position in the color-coded image sequence , as shown below: , , Where, Indicates the coordinates of the foreground area based on the character The character foreground area mask, Indicates the The character foreground area mask corresponding to the frame color-coded image, Indicates the frame color-coded image, Indicates the RGB value that sets the background to pure black; Assumptions is the pixel coordinate set with a median value of 1 in the person foreground mask, and the distance map is calculated according to the following formula , represents the shortest distance between each pixel in the color-coded image sequence and the person's foreground area. The temperature map of the person's foreground area is calculated based on the shortest distance. The calculation formula is as follows: , , Where, Indicates the shortest distance, Represents the temperature map of the character foreground area, Used to adjust the influence of the temperature map on the temporal attention weight of the character foreground area. Indicates the image height, Indicates image width; temperature map of the character foreground area The foreground area of ​​the middle figure is kept at 1, and the background area is set to different temperature values ​​according to the distance from the foreground. The farther the distance, the greater the temperature value.

[0048] To solve the multi-scale generation problem of the visual autoregressive model, it is necessary to transform the temperature map of the foreground area of ​​the character into Scaling according to the resolution of token sequences of different scales to obtain a multi-scale temperature map , flatten the temperature maps of each scale, and then splice them in the 0th dimension to obtain the temperature map ,in Indicates the total number of tokens, is the batch size. According to the temperature map Adjust the temporal attention weight matrix to increase the weight of the person's foreground area, as shown below: , , , , Where, 、 and represent the query matrix, key matrix and value matrix respectively, Indicates the rearrangement of attention scores so that each token has independent continuous frames The temporal attention matrix of Represents the temperature map rearrangement, by adding two dimensions to the flattened temperature map, so that the temperature weight of each token can be extended to consecutive frames matrix; Indicates attention score adjustment, which can achieve token-level temperature adjustment; based on the standard attention mechanism, the temperature-adjusted score is used to fuse the time series features to obtain the time series features , so that the temporal characteristics of the character foreground area are enhanced. Figure 7 Color-coded images and distance maps are shown in And the temporal attention weight map before and after adjustment using the temperature map. The brighter the color, the greater the weight.

[0049] 3. Model training module, which is used to perform multi-stage training on the constructed digital human generation model. In the second stage of training, LPIPS perception loss is introduced to obtain the trained digital human generation model.

[0050] In the embodiment, a three-stage model training method is adopted. The training objectives, required training data and loss function definitions of different training stages are different. Each stage of training is further trained on the basis of the model trained in the previous stage. The specific introduction of each stage of training is as follows: Phase 1 training: The goal of the first phase training is to generate the target digital human image based on the digital human visual autoregressive model with the person identity label as the condition; the first phase training does not temporarily introduce the conditional image guidance network, and its training data is the person identity label and a single frame of real image; the first phase training loss Expressed as: , Where, Represents the probability value of each discrete vector selected in the predicted discrete codebook, ， is the sampling probability matrix of the next scale token sequence, represents the discrete codebook index, represents the discrete codebook size, Indicates the The resolution of the scale, Indicates the The true label of the token sequence is 1 only for the position corresponding to the true category, and the rest are 0.

[0051] Second stage training: The second stage training goal is to achieve, with the person identity label and color-coded image as conditions, generate the action-controllable target digital human image based on the digital human visual autoregressive model; the second stage training introduces the conditional image guidance network, freezes the parameters of other modules, and only updates the conditional image guidance network parameters. Its training data is the person identity label, a single-frame real image, and a single-frame color-coded image; the second stage training loss function established is expressed as Including the first stage training loss function and LPIPS perception loss function, the calculation is as follows: , Where, represents a single frame of real image, Indicates the generated image, Represents a single frame of real image extracted using the VGG network Layer feature vector, Represents the generated image extracted by VGG network Layer feature vector, Indicates the number of feature layers of the VGG network, Indicates the The feature weights of the layer feature vectors, represents the LPIPS perception loss weight coefficient, Represents the L2 distance. By introducing LPIPS perceptual loss, a direct constraint on the visual quality of the generated image is achieved.

[0052] The third stage of training: based on the digital human visual autoregressive model, the target digital human image sequence with consistent movements is generated with the character identity label and color-coded image sequence as conditions; in the model training stage, the temporal attention layer is introduced to train the temporal attention layer in the digital human generation model, and the parameters of other modules are frozen, and only the network parameters of the temporal attention layer are updated. The training data are the character identity label, the continuous multi-frame true value image sequence and the continuous multi-frame color-coded image sequence; the third stage training loss Expressed as: , Where, Indicates the The first stage loss function corresponding to the frame, Indicates the total number of consecutive frames. LPIPS perception loss is not added to this stage of training. Temporal attention layer training continues based on the model trained in the second stage to reduce video memory requirements during training.

[0053] 4. Digital human image sequence generation module: It is used to input the character identity label and color-coded image into the trained digital human generation model to generate the target digital human image sequence.

[0054] In the embodiment, in the digital human image sequence generation and reasoning stage, a scale-by-scale prediction method is adopted to generate a target digital human image sequence based on the character identity label and the color-coded image sequence; the action conditions of the color-coded image sequence are extracted through the conditional image guidance network and injected into the visual autoregressive Transformer module; the temporal modeling of the image sequence is realized through the temporal attention layer and the temperature map; the interpolation smoothing inference method is adopted, and the video interpolation model RIFE is introduced to perform interpolation smoothing on the generated intermediate-scale image sequence; the generated image sequence is obtained by decoding and reconstructing the BSQ multi-scale image segmenter.

[0055] The RIFE model is a lightweight video interpolation model. It designs a video interpolation method based on intermediate flow estimation. Given adjacent image frames and , and then synthesize the intermediate frames . Taking scale token sequence prediction as an example, the network is extracted by conditional image guidance. Action Conditioning for Scaled Color-Coded Image Sequences ,and Scaled image feature map sequence Input digital human visual autoregressive model, output Standard token sequence , according to no more than Standard token sequence , Reconstructing feature map sequences at different scales , which is decoded and reconstructed by the BSQ multi-scale image decoder scaled image sequence , use the RIFE model to use adjacent frames for interpolation smoothing, and get the smoothed scaled image sequence , as shown below:

[0056] Then the BSQ multi-scale image encoder is used to smooth the scaled image sequence Encoding back to the latent space, we get Image feature map sequence under different scales , for subsequent The scale token sequence is generated. Repeat the above process until The scale token sequence is generated, and finally the BSQ multi-scale image decoder is used to decode and reconstruct the generated image sequence. , thereby improving the temporal consistency of the generated image sequence and generating the target digital human image sequence.

[0057] Based on the same inventive concept, the embodiment also provides a digital human generation method based on a visual autoregressive model, using the digital human generation system based on a visual autoregressive model, such as Figure 8 As shown, the following steps are included: Obtain person identity labels and color-coded images from real images; Based on a visual autoregressive model, a BSQ multi-scale image segmenter, a conditional image guidance network, and a temporal attention layer are introduced to construct a digital human generation model. This model enables the generation of target digital human image sequences guided by person identity labels and color-coded images. The BSQ multi-scale image segmenter is used to perform multi-scale encoding and decoding reconstruction on color-coded images. The conditional image guidance network is used to extract action conditions based on color-coded images and inject them into the visual autoregressive Transformer module. The temporal attention layer is used to extract temporal features of image sequences, and the temperature map is used to adjust the temporal attention weight distribution. The constructed digital human generation model is trained in multiple stages. In the second stage of training, LPIPS perception loss is introduced to obtain the trained digital human generation model. The person identity label and color-coded image are input into the trained digital human generation model to generate the target digital human image sequence.

[0058] In addition, the terms "upper," "lower," "inner," "outer," "front," and "back" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the present invention. Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes or modifications made based on the structures, features, and principles described in the scope of the present invention should be included in the scope of the present invention.

[0059] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A digital human generation system based on a visual autoregressive model, characterized in that: Includes the following modules: A data acquisition module, which is used to obtain person identity labels and color-coded images from real images; The digital human visual autoregressive model construction module is used to build a digital human generation model based on the visual autoregressive model, introducing the BSQ multi-scale image segmenter, the conditional image guidance network, and the temporal attention layer to achieve the generation of target digital human image sequences guided by character identity labels and color-coded images. Among them, the BSQ multi-scale image segmenter is used to perform multi-scale encoding and decoding reconstruction on the color-coded images, the conditional image guidance network is used to extract action conditions based on the color-coded images and inject them into the visual autoregressive Transformer module, and the temporal attention layer in the Transformer module is used to extract the temporal features of the image sequence and use the temperature map to adjust the temporal attention weight distribution. The model training module is used to perform multi-stage training on the constructed digital human generation model. In the second stage of training, LPIPS perception loss is introduced to obtain the trained digital human generation model; The digital human image sequence generation module is used to input the character identity label and color-coded image into the trained digital human generation model to generate the target digital human image sequence.

2. The digital human generation system based on the visual autoregressive model according to claim 1 is characterized in that: In the data acquisition module, the person identity tags are different digital tags used to represent the person's identity information; The color-coded image is a conditional image drawn based on the key points of the character's face, hands, and body, and is used to represent the character's expression, gestures, and body posture.

3. The digital human generation system based on the visual autoregressive model according to claim 1 is characterized in that: In the digital human generation model construction module, the BSQ multi-scale image segmenter includes a BSQ multi-scale image encoder and a BSQ multi-scale image decoder, which are used to perform multi-scale encoding and decoding reconstruction on color-coded images; The BSQ multi-scale image encoder uses an image encoder to encode the input real image to obtain an image feature map, extracts image information of different scales from the image feature map, and obtains a multi-scale token sequence; The image feature map containing image information of different scales is constructed based on the multi-scale token sequence, as shown in the following formula: ， in, is an image feature map containing image information of different scales, For the Discrete representation of image information at different scales, and Indicates the Target resolution at each scale, represents the upsampling operation, Indicates the total number of scales; And according to the following formula Image feature map of scale image information Zoom to Scale resolution, used for prediction Scale token sampling probability: ， Where, Represents the first scale image feature maps, and Indicates the target resolution of the next scale; The BSQ multi-scale image decoder, which uses an image decoder ,by Image feature map of scale image information As input, decode and reconstruct to generate the image .

4. The digital human generation system based on the visual autoregressive model according to claim 3 is characterized in that: In the digital human generation model construction module, the conditional image guidance network includes an action condition extraction network and an action condition guidance network, which are used to extract action conditions based on color-coded images and inject them into the visual autoregressive Transformer module to achieve action-driven generation; The action condition extraction network includes a conditional image encoder and an image feature encoder, which is used to extract conditional image features of corresponding scales from the color-coded image through the conditional image encoder, and extract features of corresponding scale feature maps from the downsampled corresponding scale image feature maps through the image feature encoder, and add the conditional image features and the features of the feature maps to obtain action conditions of corresponding scales; The action condition guidance network includes multiple layers of bypass branch networks with the same structure. The first layer of bypass branch networks takes the action condition of the corresponding scale as input, and obtains the self-attention feature through adaptive normalized self-attention calculation. The output features of the Transformer module are input, and the self-attention features are concatenated with the output features of the Transformer module to obtain new features. The new features are used as the input of the next layer of Transformer module, and the self-attention features are used as the input of the next layer of bypass branch network to iteratively realize action-driven generation.

5. The digital human generation system based on the visual autoregressive model according to claim 4 is characterized in that: In the digital human generation model construction module, the temporal attention layer extracts the temporal features of the image sequence by calculating the temporal correlation between consecutive multi-frame image sequences, including: Continuous frame Scaled image feature map As input, the features are obtained through adaptive normalized self-attention layer and feedforward neural network And the features are obtained by dimension transformation , with characteristics The initial temporal features are obtained by temporal self-attention calculation for the input , after dimension transformation and residual addition, the time series features are output .

6. The digital human generation system based on the visual autoregressive model according to claim 5 is characterized in that: In the digital human generation model construction module, the method of adjusting the temporal attention weight distribution using the temperature map includes: Given Frame color-coded image sequence , the person foreground area mask is calculated based on the non-zero position in the color-coded image sequence. The calculation formula is as follows: ， ， Where, Indicates the coordinates of the foreground area based on the character The character foreground area mask, Indicates the The character foreground area mask corresponding to the frame color-coded image, Indicates the frame color-coded image, Indicates the RGB value that sets the background to pure black; Based on the person foreground area mask, the shortest distance between each pixel in the color-coded image sequence and the person foreground area is calculated. The temperature map of the person foreground area is calculated according to the shortest distance. The calculation formula is as follows: ， ， Where, Indicates the shortest distance, is the pixel coordinate set of the character foreground area, Represents the temperature map of the character foreground area, Used to adjust the influence of the temperature map on the temporal attention weight of the character foreground area. Indicates the image height, Indicates the image width; Based on the resolution of token sequences of different scales, the temperature map of the foreground area of ​​the character is Convert to multi-scale temperature map , transform and splice the temperature maps of each scale to obtain the temperature map , using the temperature map Adjust temporal attention weight distribution.

7. The digital human generation system based on the visual autoregressive model according to claim 6 is characterized in that: In the digital human generation model training module, multi-stage training is divided into three stages, including: The first stage of training is used to establish the first stage training loss function using the person identity label and a single frame of real image as training data, and to train the digital human generation model that does not contain the conditional image guidance network to generate the target digital human image. The first stage training loss function is established by calculating all scales. , all locations The sum of the cross entropy of the token sequence is expressed as , calculated as follows: ， Where, Represents the probability value of each discrete vector selected in the predicted discrete codebook, ， is the sampling probability matrix of the next scale token sequence, represents the discrete codebook index, represents the discrete codebook size, Indicates the The resolution of the scale, Indicates the The true label of the token sequence; The second stage training is used to establish the second stage training loss function, which uses the character identity label, color-coded image and single-frame real image as training data, trains the conditional image guidance network in the digital human generation model, and generates the target digital human image with controllable action. The second stage training loss function is expressed as Including the first stage training loss function and LPIPS perception loss function, the calculation is as follows: ， Where, represents a single frame of real image, Indicates the generated image, Represents a single frame of real image extracted using the pre-trained network Layer feature vector, Represents the generated image extracted using the pre-trained network Layer feature vector, Indicates the number of pre-trained network feature layers, Indicates the The feature weights of the layer feature vectors, represents the LPIPS perception loss weight coefficient, represents the L2 distance; The third stage training is used to establish the third stage training loss function, which uses the character identity label, continuous multi-frame color-coded image sequence and continuous multi-frame real image as training data, trains the temporal attention layer in the digital human generation model, and generates a target digital human image sequence with consistent movements. The established third stage training loss function is expressed as , calculated as follows: ， Where, Indicates the The first stage loss function corresponding to the frame, Indicates the total number of consecutive frames.

8. The digital human generation system based on the visual autoregressive model according to claim 7 is characterized in that: In the digital human image sequence generation module, the target digital human image sequence is generated by adopting the interpolation smoothing inference method, including: The character identity label and color-coded image are input into the trained digital human generation model, the intermediate-scale image sequence is smoothed by the interpolation model RIFE, and the smoothed image is re-encoded to obtain the first Image feature map sequence at each scale , used for prediction Scale token sequence generation, scale-by-scale prediction until After the scale token sequence is generated, it is decoded and reconstructed through the BSQ multi-scale image decoder to generate the final target digital human image sequence.

9. A digital human generation method based on a visual autoregressive model, characterized in that: The digital human generation method uses the digital human generation system based on the visual autoregressive model according to any one of claims 1 to 8, comprising the following steps: Obtain person identity labels and color-coded images from real images; Based on a visual autoregressive model, a BSQ multi-scale image segmenter, a conditional image guidance network, and a temporal attention layer are introduced to construct a digital human generation model. This model enables the generation of target digital human image sequences guided by person identity labels and color-coded images. The BSQ multi-scale image segmenter is used to perform multi-scale encoding and decoding reconstruction on color-coded images. The conditional image guidance network is used to extract action conditions based on color-coded images and inject them into the visual autoregressive Transformer module. The temporal attention layer is used to extract temporal features of image sequences, and the temperature map is used to adjust the temporal attention weight distribution. The constructed digital human generation model is trained in multiple stages. In the second stage of training, LPIPS perception loss is introduced to obtain the trained digital human generation model. The person identity label and color-coded image are input into the trained digital human generation model to generate the target digital human image sequence.

Images