Visual content generation method and device

Abstract

The embodiment of the invention provides a visual content generation method and device.The method comprises the steps that user instruction information and a generated image data set sequence are obtained, the user instruction information is used for indicating generation of visual content, and the image data set sequence comprises at least one image data set; the image data set comprises a plurality of image data, and each image data set corresponds to the visual content; based on the user instruction information and the image data set sequence, visual distribution information corresponding to a next image data set is obtained through prediction by an autoregression large model, and the visual distribution information is probability distribution information in a continuous domain; and performing continuous domain sampling based on the visual distribution information corresponding to the next image data set to obtain the next image data set.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular, to a method and apparatus for generating visual content. Background Art

[0002] In the field of artificial intelligence, large models generally refer to machine learning models with large numbers of parameters (e.g., tens or hundreds of billions of parameters) and complex structures. These models are trained on massive amounts of data and are capable of performing a variety of complex tasks.

[0003] Video content generation technology, which generates visual content such as images and videos based on user-provided text, images, and videos, has enormous commercial and research value. Currently, existing video generation technologies use autoregressive models (AR models) and sample in a discrete domain. However, these technologies are limited by quantization loss caused by the discrete process, resulting in low-quality and inefficient generated visual content. Summary of the Invention

[0004] The embodiments of this specification provide a visual content generation solution that uses a large model as the basis of an autoregressive model and introduces a continuous sampling method, which can avoid information loss caused by vector quantization and improve the quality and efficiency of the generated visual content.

[0005] In a first aspect, an embodiment of the present specification provides a method for generating visual content, comprising: obtaining user instruction information and a sequence of generated image data sets, wherein the user instruction information is used to indicate the generation of visual content, the sequence of image data sets includes at least one image data set, the image data set includes multiple image data, and each image data corresponds to the visual content; based on the user instruction information and the sequence of image data sets, an autoregressive large model is used to predict visual distribution information corresponding to the next image data set, wherein the visual distribution information is probability distribution information in a continuous domain; based on the visual distribution information corresponding to the next image data set, continuous domain sampling is performed to obtain the next image data set.

[0006] In some embodiments, when the visual content is an image, each image data corresponds to an image block after the image is segmented, and for an image block corresponding to any image data in the image data set, there is at least one image block corresponding to image data in the same image data set adjacent to it.

[0007] In some embodiments, when the visual content is an image, the image data set includes multiple image data, and the number of image data in each image data set of the image data set sequence is positively correlated with the generation order of the image data sets.

[0008] In some embodiments, when the visual content is an image, the visual distribution information corresponding to the next image data set is predicted by the autoregressive large model based on the user instruction information and the image data set sequence, including: constructing a target sequence based on the user instruction information and the image data set sequence, the target sequence including: a user instruction sequence, an image start identifier before the image set sequence, and the image set sequence; inputting the target sequence into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

[0009] In some embodiments, in response to the next image data set to be predicted being the last image data set, the target sequence further includes an image end marker following the image set sequence.

[0010] In some embodiments, when the visual content is a video, a first video frame of the video corresponds to a plurality of first image data sets in the image data sets, and the first video frame is the first video frame in the video.

[0011] In some embodiments, each target video frame of the video corresponds one-to-one to each target image data set in the video content set sequence, the target video frame is other video frames other than the first video frame in the video, and the target image data set is other image data sets other than the first image data set in the video content set sequence.

[0012] In some embodiments, the autoregressive large model predicts visual distribution information corresponding to the next image data set based on the user instruction information and the image data set sequence, including: in response to the image data set to be predicted being the image data set corresponding to the first video frame, constructing a target sequence based on the user instruction information and the image data set sequence, the target sequence including: a user instruction sequence, an image start identifier before the first frame set sequence, and the first frame set sequence; inputting the target sequence into the autoregressive large model to predict visual distribution information corresponding to the next image data set.

[0013] In some embodiments, the visual distribution information corresponding to the next image data set is predicted by the autoregressive large model based on the user instruction information and the image data set sequence, including: in response to the image data set to be predicted being the image data set corresponding to the target video frame, a target sequence is constructed based on the user instruction information and the image data set sequence, the target sequence including: a user instruction sequence, an image start identifier before the first frame set sequence, the first frame set sequence, an image end identifier after the first frame set sequence, a frame start identifier before the target video frame, the target video frame, and a frame end identifier after the target video frame; the target sequence is input into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

[0014] In some embodiments, the user instruction information includes at least one of the following information: text information, image information and video information.

[0015] In some embodiments, the autoregressive large model consists of a decoder.

[0016] In some embodiments, the visual distribution information corresponding to the next image data set is predicted by a large autoregressive model based on the user instruction information and the image data set sequence, including: encoding the user instruction information by a target encoder corresponding to different information in the user instruction information to obtain a user instruction sequence; and predicting the visual distribution information corresponding to the next image data set by a large autoregressive model based on the user instruction sequence and the image data set sequence.

[0017] In some embodiments, when the user instruction information includes image information or video information, the target encoder corresponding to the image information or video information is a two-dimensional variational autoencoder.

[0018] In some embodiments, after obtaining the next image data set, the method further includes: decoding the next image data set to obtain visual content corresponding to the next image data set.

[0019] In some embodiments, the visual distribution information corresponding to the next image data set is predicted by the autoregressive large model based on the user instruction information and the image data set sequence, including: decoding to obtain the generated visual content based on the image data set sequence; inputting the user instruction information and the generated visual content into the encoder for encoding to obtain a continuous field vector; inputting the continuous field vector into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

[0020] In some embodiments, the continuous domain sampling is performed based on the visual distribution information corresponding to the next image data set to obtain the next image data set, including: a continuous domain sampler performs continuous domain sampling based on the visual distribution information corresponding to the next image data set to obtain the next image data set.

[0021] In some embodiments, the autoregressive large model is trained by: obtaining user instruction samples and a generated image data set sample sequence; based on the user instruction samples and the image data set sample sequence, the autoregressive large model predicts the visual distribution sample information corresponding to the next visual content sample set, and the visual distribution sample information is the probability distribution information in the continuous domain; based on the visual distribution sample information corresponding to the next visual content sample set, continuous domain sampling is performed to obtain the next visual content sample set; based on the difference between the obtained next visual content label set and the next visual content sample set, the parameters of the autoregressive large model are adjusted.

[0022] In a second aspect, an embodiment of this specification provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any implementation manner in the first aspect is implemented.

[0023] In the solution provided in the above embodiments of this specification, visual content such as pictures and videos is generated by an autoregressive large model constructed based on a large model, which fully taps the generation capability of the large model, and obtains an image data set by sampling the generated visual information in a continuous domain, thereby avoiding information loss caused by vector quantization, thereby improving the quality of generated visual content, and generating an image data set each time instead of a single token, which speeds up the generation of visual content. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 is a schematic diagram of the prediction method of the autoregressive model shown in the embodiment of this specification;

[0026] Figure 2 is a flow chart of the process of generating visual content in an embodiment of this specification;

[0027] Figure 3 is a flow chart of a method for generating visual content according to an embodiment of this specification;

[0028] Figure 4 is a flowchart of another visual content generation process in an embodiment of this specification;

[0029] Figure 5 is a schematic diagram of image block division in an embodiment of this specification;

[0030] Figure 6 This is a schematic diagram of the architecture for generating visual content in the embodiments of this specification;

[0031] Figure 7 It is a structural diagram of the visual content generation device in the embodiment of this specification. DETAILED DESCRIPTION

[0032] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0033] As mentioned earlier, visual content generation technology refers to the automatic generation of high-fidelity visual content such as images or videos based on given single-modal or multi-modal data such as text, images, and videos. Current visual content generation strategies generally use two types of technologies: autoregressive models + discrete domain sampling, and diffusion models + continuous domain sampling. Among them, visual content generation technology based on autoregressive models is limited by the quantization loss caused by the discrete process, resulting in poor quality of the generated visual content. Visual content generation technology based on diffusion models, while generating high-quality visual content, is very computationally intensive and computationally expensive.

[0034] In order to better explain the solutions of the embodiments of this specification, the generation solution based on the autoregressive model + discrete domain sampling in the prior art is first described below.

[0035] When performing visual content generation tasks, for given user instruction information such as text, images or videos, when inputting this information into the autoregressive model, it is necessary to convert it into an input c = {x} that the autoregressive model can understand, where x is a token. For text, a token generally refers to the smallest unit or basic element used to represent the processed text in the natural language processing process. A text segmenter can be used to convert text into tokens. For images or videos, the understanding of tokens can be as follows: after dividing the input image (or video frame) into patches of fixed size, each patch is mapped to a vector representation. These vector representations are tokens. Each token represents a local area in the image and can be regarded as an abstract representation of the image. A visual segmenter can be used to convert the image or video into a visual token.

[0036] For autoregressive models, the generation of visual content such as images or videos can be consistently defined as generating a series of frames {f t} t＝1:T , for image generation, T=1; for video generation, T>1. The t-th frame can be represented as a sequence of multiple tokens n is the number of tokens in the frame.

[0037] The autoregressive model is a generative model that gradually generates each element in the sequence. When generating an image, the autoregressive model generates the probability distribution of the image token each time, and has a one-way dependency, that is, each time the probability distribution of the next token is generated, it only considers the previously generated tokens, not the future tokens. For example, when generating the t+1th token, it only depends on the previous t tokens. In order to generate Sequence, the autoregressive model needs to be conditioned on the probability distribution of the previously generated token when generating the probability distribution of the next token. Taking image generation as a simple example, only the tokens generated by the decoder’s autoregressive model The probability distribution can be expressed as:

[0038]

[0039] In order to express it concisely, the above formula omits the frame identifier t. From this formula, it can be seen that each token generated The generation of the probability distribution of depends on All before However, the above generation process generates a discrete next token each time, which cannot effectively utilize the spatial and temporal dependencies between visual tokens.

[0040] Moreover, visual data is essentially a continuous analog signal. The current autoregressive model + discrete domain sampling requires the use of VQ (Vector Quantization) to map the continuous visual data to a finite set (code book) for compression and representation. In this process, due to the finiteness of the code book, some data points cannot be accurately represented, which will cause quantization errors. Figure 1 As shown in the figure, the dark gray grid represents the codebook space, the orange dashed line represents the continuous video signal, the gray circle represents random noise, the pink circle represents the generated token obtained by sampling, and the orange circle represents the sampled token to be predicted. It can be seen that after quantization, the video data can only be represented in the space where the grid (codebook) is located. This is different from the position of the continuous video signal, which is the quantization error. In addition, due to the discrete nature of the codebook, the generated visual content will lose its smoothness or continuity.

[0041] Based on this, an embodiment of this specification proposes a visual content generation method, which is an improvement to the current generation scheme based on the autoregressive model, so as to improve the generation quality of the visual content while speeding up the generation speed and reducing the computational cost. Figure 2 A schematic diagram of a method for generating visual content according to an embodiment of the present specification is shown. First, user instruction information and a sequence of generated image data sets are obtained, wherein the user instruction information is used to indicate the generation of visual content, for example, it can be text information, picture information, or video information provided by the user, and the image data set sequence includes at least one previously generated image data set, and the image data set is composed of multiple image data. In the embodiment of the present specification, image data refers to the feature representation of a patch, which corresponds to a patch in an image or video frame, and can be a token or other embedded vector obtained by encoding. Then, the user instruction information and the image data set sequence are fed into an autoregressive large model, and the autoregressive large model predicts the visual distribution information corresponding to the next image data set, wherein the visual distribution information is the probability distribution information in the continuous domain. Finally, based on the visual distribution information corresponding to the next image data set, continuous domain sampling is performed to obtain the next image data set. This image data set can be added to the image data set sequence and continued to be used for the generation of subsequent image data sets. The above steps are iterated and looped. When the generation is completed, the complete generated visual content can be obtained based on the image data set sequence.

[0042] Figure 3 301 - 303.

[0043] like Figure 3 As shown, in step 301, user instruction information and a generated image data set sequence are obtained.

[0044] The user instruction information is used to instruct the generation of visual content. For example, it can be one or more of text information, image information, or video information provided by the user. This embodiment does not limit the specific form and content of the user instruction information. For example, when the visual content generation task is to generate a picture from text, the given text information can be: a red apple covered with water droplets on a black background; when the visual content generation task is to generate a video from text, the given text information can be: when fireworks explode in the night sky, the cathedral is illuminated; when the visual content generation task is to generate a video from a picture, the given image information can be a picture of two huskies.

[0045] In this embodiment, the image data set sequence includes at least one image data set, each image data set includes multiple image data, and each image data corresponds to a visual content. For ease of description, the following will be described using image data as a token as an example.

[0046] Considering that the generation process of existing autoregressive models only generates a discrete next token each time, and cannot effectively utilize the spatial and temporal dependencies between visual tokens, this embodiment groups related tokens into a group, namely an image data set, and performs the next set of predictions under the AR framework to achieve better generation quality and speed. Reasonable division of tokens helps reduce the difficulty of model generation and improve generation quality. This embodiment does not restrict the division of image data sets. When the visual content is images or videos, different grouping strategies can be adopted. The different grouping strategies will be briefly described below, and the specific grouping strategies will be described in detail later.

[0047] When the visual content to be generated is an image, the target token sequence to be predicted {x n} n＝1:N Divided into K image data sets, the image data set sequence is {s k} k＝1:K , where each image data set s k Consists of multiple tokens. The next image data set s k The prediction can be expressed as:

[0048]

[0049] Where C is the user instruction information that can be understood by the model described above, s kThe generation of the corresponding probability distribution depends on the generated image data set sequence s <k .

[0050] When the visual content to be generated is a video, different grouping strategies are used for the first frame and other frames of the video. When generating , the generation process can be regarded as the generation process of the image, and its grouping strategy can be the same as the grouping strategy of the aforementioned image, and the first video frame can be generated according to formula (2). Then, when generating the remaining T-1 video frames, the tokens corresponding to each video frame can be divided into the same group, that is, the tokens corresponding to each video frame are processed as an image data set. In this way, each video frame except the first frame corresponds to the image data set in the image data set sequence, and the tokens contained in each image data set are the tokens corresponding to the same video frame. The prediction of the next image data set is the next frame f t The prediction can be expressed as:

[0051]

[0052] Among them, f t The generation of the corresponding probability distribution depends on the generated image data set sequence f <t .

[0053] In this way, grouping related tokens into a separate image data set and predicting their probability distribution at the same time can better utilize their spatial and temporal dependencies, and such set prediction can greatly reduce the number of inference steps and speed up the generation speed. The existing autoregressive model's token-by-token generation method requires the same number of inferences to be performed as the length of the token sequence, while the scheme of this embodiment requires the same number of inference steps as the number of token sets.

[0054] Next, in step 302, based on the user instruction information and the image data set sequence, the autoregressive large model is used to predict the visual distribution information corresponding to the next image data set.

[0055] In practice, the original representation of the user command information and the image data set sequence can be directly input into the autoregressive model for prediction. Alternatively, the user command information, such as text, image, or video, can be converted into an input C = {x} that the model can understand before inputting it into the autoregressive model. This embodiment does not limit the encoding method for converting the user command information into C.

[0056] As an implementation method, user instruction information and a sequence of image data sets are input into an encoder, and the encoder can extract useful features from the sequence of user instruction information and image data sets, and then form a representation in the latent space to obtain a vector representation, and then input the vector representation into the autoregressive large model for prediction.

[0057] As another implementation method, this step may be, first, decoding based on the sequence of image data sets to obtain the generated visual content, or, after each time the next image data set is obtained, decoding the next image data set to obtain the visual content corresponding to the next image data set. This embodiment does not limit the decoding method. When decoding, various types of image decoders may be used, such as VAE (Variational Autoencoder) decoders or Transformers decoders to map the image data back to the original image space. Then, the user instruction information and the generated visual content are input into the encoder for encoding to obtain a continuous domain vector, such as Figure 4 As shown, the encoder is a network that transforms through a continuous domain feature space. Both its input and output are continuous domain vectors. It receives user instruction information and generated visual content, and then converts this information into a continuous feature vector. This embodiment does not limit the network architecture used by the encoder; for example, a convolutional neural network, a recurrent neural network, or a transformer can be used. Finally, the continuous domain vector is input into a large autoregressive model to predict the visual distribution information corresponding to the next image data set.

[0058] As another implementation, this step may be: first, target encoders corresponding to different information in the user command information encode the user command information to obtain a user command sequence. For example, for text information, the corresponding target encoder is a text segmenter. This embodiment does not limit the specific text segmenter used. For example, a CLIP (Contrastive Language-Image Pre-training) text encoder may be used to encode the text information into a 1024-dimensional latent space. For another example, when the user command information includes image information or video information, the corresponding target encoder is a visual segmenter. Unlike the VQ segmenter used in previous AR work, this embodiment uses a variational autoencoder (VAE) without VQ to map each patch in the image or video frame into a continuous representation of the user command sequence. This continuous feature preserves the inherent visual continuity in the image and can enhance subtle details and motion cues during the generation process. For example, a 2D (2D) VAE or a 3D (3D) VAE can be used. In one example, considering that 2D VAE has high computational efficiency and is faster than 3D VAE, the target encoder corresponding to the image information or video information can use 2D VAE, and the output based on 2D VAE is directly input into the large autoregressive model. The above process of encoding user instruction information can be expressed as:

[0059] {x}=E(I) (4)

[0060] Here, E(·) is the VAE encoder and I represents an image or video frame in pixel space.

[0061] Then, based on the user command sequence and the image dataset sequence, the autoregressive model predicts the visual distribution information corresponding to the next image dataset. For example, the tokens in the user command sequence and the tokens in the image dataset sequence can be concatenated and input into the autoregressive model for prediction.

[0062] In addition, the next image data set to be predicted can be input into the autoregressive model in the form of a mask to prompt the model to predict the visual distribution information corresponding to the next image data set. Taking the visual content to be generated as an image as an example, in the kth step of image generation, that is, the autoregressive model G θIn the kth inference, in addition to the user instruction information and the generated image data set sequence, the input to the autoregressive model can also include a set of special mask image data (position embedding codes can also be added to the image data to identify the position of the corresponding image block in the image), or the mask image data set, which can be specifically represented as a mask token set {p}, the number of mask tokens contained in {p} is equal to the next image data set s to be predicted. k The number of tokens in the .

[0063] In this step, the visual distribution information corresponding to the next image data set predicted by the large autoregressive model is the probability distribution information in the continuous domain, as shown in the above formulas (2) and (3). Specifically, it can be a vector in the continuous domain. This vector describes the various possibilities of the currently predicted visual content and is used to represent the potential visual space of the visual content corresponding to the next image data set. Specifically, the visual distribution information can be represented as a continuous probability distribution token set {z}.

[0064] In this embodiment, the autoregressive large model G θ There is no restriction on the network structure of the large model used, where θ is a trainable parameter and an existing pre-trained large model can be used, such as the Llama (Large Language Model Meta AI) model.

[0065] In one embodiment, the autoregressive large model consists of a decoder, that is, the large model used is a decoder-only autoregressive network. In this architecture, the network only contains a decoder, which can be composed of multiple Transformer blocks without an encoder. The role of the decoder is to gradually predict the next element in the sequence by using the previously generated output. The autoregressive large model containing only the decoder can be pre-trained, and training is performed in the continuous domain without vector quantization. The training process of the autoregressive large model will be described in detail later.

[0066] Then, in step 303, continuous domain sampling is performed based on the visual distribution information corresponding to the next image data set to obtain the next image data set.

[0067] In this step, continuous sampling is used instead of discrete sampling in existing methods to maintain the continuity of the generated visual content. Continuous sampling refers to sampling in a continuous space directly using the original feature information without vector quantization. This embodiment does not limit the continuous sampling method; for example, random sampling or uniform sampling can be used.

[0068] In one embodiment, this step may involve a continuous domain sampler D performing continuous domain sampling based on the visual distribution information corresponding to the next image data set to obtain the next image data set. The continuous domain sampler D is configured to sample from the predicted visual distribution information, and this embodiment does not impose any restrictions on the network structure of the continuous domain sampler D.

[0069] For example, the continuous domain sampler D can be implemented using a small network composed of a multilayer perceptron (MLP), which samples from visual distribution information by repeatedly adding random noise and denoising it. The continuous domain sampler D can also use a small denoising network, such as a denoising autoencoder, to introduce sampling randomness through iterative noise adjustment.

[0070] Specifically, taking the visual content to be generated as an image as an example, each time the next image data set s is sampled from the visual distribution information {z} k The process can be expressed as:

[0071] s k =D(G θ ({p}|s <k )) (5)

[0072] After the next image data set is sampled and generated, the generated image data set is sent to the sequence of generated image data sets in step 301, and the above steps are iterated until all image data sets are generated.

[0073] It should be noted that each time the next image data set is obtained, it can also be visualized. Specifically, the next image data set is decoded to obtain the visual content corresponding to the next image data set. For example, when the visual content to be generated is an image, each time the next image data set is decoded, a partial image block in the image can be obtained; when the visual content to be generated is an image, each time the next image data set is decoded, a video frame in the video can be obtained. In some embodiments, Figure 4 As shown, the next visual content may be generated after sampling, and the generated visual content may be fed into the first step of the framework, and iterated until all visual contents are generated.

[0074] The visual content generation method provided in the embodiments of this specification introduces continuous domain sampling into the visual generation framework of the autoregressive large language model, which can avoid the information loss caused by vector quantization, thereby improving the quality of generated visual content. Moreover, the visual information corresponding to the next image data set is generated each time to improve the efficiency of the model, speeding up the reasoning speed. In terms of generation efficiency, the method proposed in this embodiment can shorten the time to generate 16 frames of video at a resolution of 256*256 to 1.1 seconds. In addition, compared to the diffusion model, the large model as the backbone network can fully utilize the autoregressive generation characteristics and multi-task capabilities of the existing large model.

[0075] Properly dividing the image dataset helps reduce the difficulty of model generation and improve generation quality. The following describes the grouping methods for image datasets in two different situations: when the visual content to be generated is images and when it is videos. It should be understood that the following grouping methods are exemplary, and other grouping methods may be used in other embodiments.

[0076] 1) When the visual content is an image

[0077] When the visual content is an image, each image data corresponds to each image block after the image is segmented. For an image block corresponding to any image data in an image data set, there is at least one image block corresponding to image data in the same image data set adjacent to it.

[0078] Considering the local nature of visual content, adjacent regions are related and share common visual cues. For image generation, this embodiment can group spatially adjacent image data as a set. Taking image data as tokens as an example, this embodiment groups adjacent tokens into a set to utilize this spatial dependency and better generate images.

[0079] For example, an image can be divided into multiple patches of the same size from top to bottom and from left to right. For each patch, a patch embedding (a vector representation of an image block) can be encoded, i.e., image data, which is referred to as a token in this specification. To facilitate the distinction of patches at different positions, each patch can be encoded from top to bottom and from left to right to obtain patch 1, patch 2…patchN. The image data corresponding to each patch is token {x n} n＝1:N It is understandable that each patch has at least one adjacent patch in the image, for example, patch 1 is adjacent to patch 2. When grouping, adjacent patches can be divided into the same image data set, such as Figure 5 As shown, assuming N is 16, for example, patch 1 to patch 3 can be divided into the same group, and patch 4 to patch 8 can be divided into the same group; for another example, patch 1, patch 2, patch 5, and patch 6 can be divided into the same group, and patch 3, patch 4, patch 7, and patch 8 can be divided into the same group.

[0080] In addition, considering that the generation difficulty varies at different generation stages, there are few visual clues available at the beginning, and generating too many tokens at the same time may lead to semantic conflicts between visual contents. Setting fewer tokens in the image data set that is just generated can help reduce the generation difficulty and improve the generation quality in the initial AR generation step.

[0081] In one embodiment, the number of image data in each image data set of the image data set sequence is positively correlated with the generation order of the image data sets. This partitioning method is also called spatial progressive partitioning (SPP). That is, the number of tokens in each generated image data set is iteratively increased. This embodiment does not limit the positive correlation. For example, it can be a function curve that conforms to the positive correlation, or it can be randomly set. For example, in the first image data set s 1 Set 2 tokens in the second image data set s 2 Set up 3 tokens.

[0082] In one example, the positive correlation can refer to a sinusoidal function relationship, where the change in the number of image data sets conforms to a sinusoidal function curve. Specifically, adjacent image tokens are divided into a set, and then the number of tokens in each set increases as the sinusoidal curve changes. Gradually increasing the size of the set in a sinusoidal manner can also prevent the generation of too many tokens in the later steps. Each image data set s k The number of tokens in can be determined by the following formula:

[0083]

[0084] in, Represents the rounding function, α is a constant determined by the number of tokens N and the number of image data sets K, and is calculated as follows:

[0085]

[0086] Since K is smaller than N, for example, K=32 and N=196, this formula can ensure the increasing trend of the sine function and the sum of tokens in all sets is equal to N.

[0087] In the case where the visual content is an image, the above step 302 can be to construct a target sequence based on the user instruction information and the image data set sequence, where the target sequence includes: the user instruction sequence, the image start identifier before the image set sequence, and the image set sequence; the target sequence is input into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

[0088] Specifically, user instruction information can be converted into a user instruction sequence C = {x} through a corresponding word segmenter. To help the model distinguish different information types, in this embodiment, a begin of image identifier (BOI) can be used as a delimiter in the target sequence of the input model. This delimiter can be regarded as a special token. This delimiter can be set by technicians according to actual needs or used as a learnable parameter, which is used to adjust during the training process of the large autoregressive model.

[0089] In addition, in response to the fact that the next image data set to be predicted is the last image data set, the target sequence also includes an end of image marker (EOI) after the image set sequence. The end of image marker serves as a separator, similar to the above-mentioned start of image marker, and can be regarded as a special token. The separator can be set by technicians according to actual needs, or it can be used as a learnable parameter, which is used to adjust during the training process of the autoregressive large model. The role of the end of image marker is to help the model distinguish different types of information and prompt the model that the image generation has ended. In this way, when the last image data set is generated, the image set sequence will be surrounded by a specific start of image marker [BOI] and end of image marker [EOI].

[0090] Once the [BOI] token and subsequent mask tokens are input into the model, the generation process of the next set of image data corresponding to the mask tokens begins. Similarly, the input of the [EOI] token indicates the end of the image generation process.

[0091] 2) When the visual content is a video

[0092] When the visual content is a video, each target video frame of the video corresponds one-to-one to each target image data set in the video content set sequence. The target video frames are other video frames other than the first video frame in the video, and the target image data sets are other image data sets other than the first image data set in the video content set sequence.

[0093] For video generation, considering that humans usually perceive the world in a linear time sequence and pay attention to what will happen next, predicting image data sets in chronological order is consistent with the way humans observe and predict the dynamics of the physical world. In this embodiment, when performing grouping, the temporal next-frame partitioning (TNP) method can be adopted. That is, except for the first frame, the tokens corresponding to each frame in other frames are regarded as an image data set, and the tokens corresponding to the next frame are generated each time.

[0094] Considering that the generation of the first frame is at the beginning of video generation, the first frame of video lacks the previous frame as visual context, and the generation of the first frame can be regarded as an image generation task. This embodiment does not limit the generation method of the first frame. For example, the image set can be grouped and generated in the same way as the above-mentioned case where the visual content is an image. The first video frame of the video and the multiple first image data sets in the image data set (i.e., s 1 , s 2 ...s K ), the first video frame is the first video frame in the video, and the first video frame can be finally expressed as:

[0095] f 1 =s 1 ∪s 2 ∪...∪s K (8)

[0096] Among them, ∪ represents the union of the image data sets.

[0097] In other embodiments, the image may be generated by other existing image generation methods.

[0098] In the case where the visual content is a video, step 302 may be:

[0099] In response to the image data set to be predicted being the image data set corresponding to the first video frame, a target sequence is constructed based on user instruction information and the image data set sequence. The target sequence includes: the user instruction sequence, the image start identifier before the first frame set sequence, and the first frame set sequence. The target sequence is input into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

[0100] The process of generating the image data set corresponding to the first video frame is the same as the process of generating the visual content as an image. Please refer to the relevant description in the previous article and will not be repeated here.

[0101] In response to the image data set to be predicted being the image data set corresponding to the target video frame, a target sequence is constructed based on user instruction information and the image data set sequence, wherein the target sequence includes: a user instruction sequence, an image start identifier before the first frame set sequence, the first frame set sequence, an image end identifier after the first frame set sequence, a frame start identifier before the target video frame, the target video frame, and a frame end identifier after the target video frame; the target sequence is input into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

[0102] Among them, the image start marker, image end marker, frame start marker (begin of frame, abbreviated as [BOF]) and frame end marker (end of frame, abbreviated as [EOF]) are used as delimiters, which can be regarded as special tokens. The values ​​of these delimiters can be set by technicians according to actual needs, or they can be used as learnable parameters, which are used to adjust during the training process of the autoregressive large model. The role of the above delimiters is to help the model distinguish different types of information, thereby establishing the relationship between different information, such as user command information, images and video frames, and prompting the model to start and end the generation of images and video frames, reducing the difficulty of model reasoning.

[0103] Specifically, given the generated frame f <t and understandable user instruction information C, G θ The next frame f can be predicted by the spatial context of the generated frame t Corresponding visual distribution information. Among them, model G θ The generated image set sequence, i.e., the frame sequence f <t As a basis, when generating the image data set corresponding to the next frame at each step, similar to when the visual content is an image, each video frame is surrounded by [BOF] and [EOF], and the visual distribution information corresponding to the tth frame is The generation process can be expressed as:

[0104]

[0105] like Figure 6 As shown, Figure 6 The framework of visual content generation in this embodiment when generating the second frame is shown, which consists of two main parts: the autoregressive large model G θ and continuous domain sampler D. G θ Predict the visual distribution information of the next image data set based on the given user instruction information and the generated visual content N = 4, and D visually distributes information in the predicted continuous domain Random sampling is performed within the frame to obtain the image data set corresponding to the second frame. The image data set is input into the VAE decoder for decoding to obtain the image of the second frame. Among them, the user instruction information is text information: a man holding a dog. The text information is segmented by the text segmenter to obtain the token sequence C; the generated visual content is the first video frame. The first video frame can be encoded by the VAE encoder to generate the image data set sequence {s k} k＝1:K Alternatively, in other embodiments, the previously generated image data set sequence {s k} k＝1:K , in the image data set sequence {s k} k＝1:K There are separators [BOI] and [EOI] before and after. In addition, the input of the model also includes the mask corresponding to the next frame to be predicted. exist There are delimiters [BOF] and [EOF] before and after respectively.

[0106] In this way, the image data is grouped to obtain multiple image data sets, and each time the next image data set is generated. For image generation, the proposed method divides the image content into multiple sets and generates the image information of one set at a time. For video generation, the proposed method generates one frame of the video at a time, which can greatly reduce the number of inference steps, reducing the number of inference steps from the total number of tokens to the number of token sets.

[0107] The following describes how to train a large autoregressive model.

[0108] In one embodiment, it can be trained in the following manner: obtaining a user instruction sample and a sample sequence of a generated image data set; based on the user instruction sample and the sample sequence of the image data set, predicting the visual distribution sample information corresponding to the next visual content sample set by the autoregressive large model, where the visual distribution sample information is the probability distribution information in the continuous domain; performing continuous domain sampling based on the visual distribution sample information corresponding to the next visual content sample set to obtain the next visual content sample set; and adjusting the parameters of the autoregressive large model based on the difference between the obtained next visual content label set and the next visual content sample set.

[0109] It is understandable that the autoregressive model G θ Used to: predict each target tokenx in the image data set n Visual distribution information z n The training goal is to optimize the conditional probability of generating a single tokenx in each generated image dataset in the image dataset sequence. The loss function It can be expressed as:

[0110]

[0111] Where x is each s k A single token in .

[0112] Considering the lack of direct training G θ Visual distribution information z n Supervision information, this embodiment uses the sampling result x n The labels are used as supervision information to train G θ The continuous domain sampler D used for continuous domain sampling is obtained from the predicted z n Each token x is sampled one by one n The process can be expressed as:

[0113] x n =D(z n ) (11)

[0114] For the set s generated at each step k , we can sample each token one by one and calculate G θ and the loss function of D. In this embodiment, G θ The gradient of is calculated by back-propagating the denoising sampling loss, and the loss function can be expressed as:

[0115]

[0116] in, is the loss function of the continuous domain sampler D, which measures the degree of match between the visual distribution information and the original continuous value. This embodiment does not limit the calculation method of the loss function. For example, it can be calculated using the following formula:

[0117]

[0118] in, is a noise vector sampled from N(0,1), x′ i is defined as The noise corrosion vector, represents the noise schedule, i is the time step in the schedule. The noise estimator ε φ Parameterized by φ.

[0119] In practice, when training an autoregressive large model, first, a user instruction sample and a generated image dataset sample sequence are obtained. The user instruction sample can be text information, image information, or video information. The generated image dataset sample sequence contains multiple image dataset samples.

[0120] The user command sample and the image dataset sample sequence are then input into the autoregressive model to predict the visual distribution sample information corresponding to the next visual content sample set. The visual distribution sample information represents the probability distribution information in a continuous domain. The encoding method for the user command sample and the image dataset sample sequence is described in the previous embodiment and will not be further elaborated here.

[0121] Next, continuous domain sampling is performed based on the visual distribution sample information corresponding to the next visual content sample set to obtain the next visual content sample set. For the method of continuous sampling, please refer to the above embodiment and will not be described in detail here.

[0122] Finally, based on the difference between the next visual content label set and the next visual content sample set obtained, the parameters of the autoregressive large model are adjusted.

[0123] For example, the next set of visual content labels may be pre-acquired tokens labels, which may be obtained by encoding image blocks in an image or video frame. This embodiment does not limit the training method used when adjusting the parameters. The gap between the model's prediction results and the labels can be measured by the loss function shown above, and the network parameters can be adjusted to minimize the loss function to improve the prediction accuracy. This process can be fine-tuned by using supervised learning. When the network iteration end condition is reached, the network training is terminated. This embodiment does not limit the preset stop training condition. Among them, the condition can be that the iteration reaches a certain number of times, or the loss value of the loss function is less than a certain threshold.

[0124] It should be noted that when adjusting the network parameters of the large autoregressive model with the goal of minimizing the loss function, the values ​​of the above-mentioned learnable parameters can also be adjusted at the same time, and the network parameters of the continuous domain sampler D can also be adjusted.

[0125] Taking video generation as an example, during the training phase, the complete image set sequence that the model needs to predict can be expressed as {s 1 , s 2 ,...,s K ,f 2 ,...,f T There are two types of spatial and temporal prediction involved. Spatial prediction (i.e., the generation of the first frame) has K steps, and temporal prediction (i.e., the generation of frames beyond the first frame) has T-1 steps. In other words, generating a video of T frames requires (K+T-1) steps.

[0126] Figure 7This is a schematic diagram of the structure of the visual content generation device in the embodiment of this specification. The device can be applied to any device, platform or device cluster with computing and processing capabilities. The device includes:

[0127] The acquisition module 71 is used to obtain user instruction information and a generated image data set sequence, where the user instruction information is used to indicate the generation of visual content. The image data set sequence includes at least one image data set, and the image data set includes multiple image data, each of which corresponds to visual content.

[0128] Prediction module 72, for predicting visual distribution information corresponding to the next image data set based on user instruction information and the image data set sequence using an autoregressive large model, where the visual distribution information is probability distribution information in a continuous domain;

[0129] The sampling module 73 is used to perform continuous domain sampling based on the visual distribution information corresponding to the next image data set to obtain the next image data set.

[0130] In some embodiments, when the visual content is an image, each image data corresponds to an image block after the image is segmented. For an image block corresponding to any image data in an image data set, there is at least one image block corresponding to image data in the same image data set that is adjacent to it.

[0131] In some embodiments, when the visual content is an image, the image data set includes multiple image data, and the number of image data in each image data set of the image data set sequence is positively correlated with the generation order of the image data sets.

[0132] In some embodiments, when the visual content is an image, the prediction module 72 is specifically used to: construct a target sequence based on user instruction information and an image data set sequence, where the target sequence includes: a user instruction sequence, an image start identifier before the image set sequence, and an image set sequence; input the target sequence into the autoregressive model to predict the visual distribution information corresponding to the next image data set.

[0133] In some embodiments, in response to the next image data set to be predicted being the last image data set, the target sequence further includes an image end marker following the image set sequence.

[0134] In some embodiments, when the visual content is a video, the first video frame of the video corresponds to a plurality of first image data sets in the image data sets, and the first video frame is the first video frame in the video.

[0135] In some embodiments, each target video frame of the video corresponds one-to-one to each target image data set in the video content set sequence, the target video frame is other video frames other than the first video frame in the video, and the target image data set is other image data sets other than the first image data set in the video content set sequence.

[0136] In some embodiments, the prediction module 72 is specifically used to: in response to the image data set to be predicted being the image data set corresponding to the first video frame, construct a target sequence based on user instruction information and the image data set sequence, the target sequence including: the user instruction sequence, the image start identifier before the first frame set sequence, and the first frame set sequence; input the target sequence into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

[0137] In some embodiments, the prediction module 72 is specifically used to: in response to the image data set to be predicted being the image data set corresponding to the target video frame, construct a target sequence based on user instruction information and the image data set sequence, the target sequence including: the user instruction sequence, the image start identifier before the first frame set sequence, the first frame set sequence, the image end identifier after the first frame set sequence, the frame start identifier before the target video frame, the target video frame, and the frame end identifier after the target video frame; input the target sequence into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

[0138] In some embodiments, the user instruction information includes at least one of the following information: text information, image information, and video information.

[0139] In some embodiments, the autoregressive large model consists of a decoder.

[0140] In some embodiments, the prediction module 72 is specifically used to: encode the user instruction information by a target encoder corresponding to different information in the user instruction information to obtain a user instruction sequence; based on the user instruction sequence and the image data set sequence, predict the visual distribution information corresponding to the next image data set by the autoregressive large model.

[0141] In some embodiments, when the user instruction information includes image information or video information, the target encoder corresponding to the image information or video information is a two-dimensional variational autoencoder.

[0142] In some embodiments, after obtaining the next image data set, the method further includes: decoding the next image data set to obtain visual content corresponding to the next image data set.

[0143] In some embodiments, the prediction module 72 is specifically used to: decode the generated visual content based on the image data set sequence; input the user instruction information and the generated visual content into the encoder for encoding to obtain a continuous field vector; input the continuous field vector into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

[0144] In some embodiments, the sampling module 73 is specifically configured to: perform continuous domain sampling by a continuous domain sampler based on visual distribution information corresponding to the next image data set to obtain the next image data set.

[0145] In some embodiments, the autoregressive large model is trained by: obtaining user instruction samples and a sample sequence of a generated image data set; based on the user instruction samples and the sample sequence of the image data set, the autoregressive large model predicts the visual distribution sample information corresponding to the next visual content sample set, where the visual distribution sample information is the probability distribution information in the continuous domain; based on the visual distribution sample information corresponding to the next visual content sample set, continuous domain sampling is performed to obtain the next visual content sample set; based on the difference between the obtained next visual content label set and the next visual content sample set, the parameters of the autoregressive large model are adjusted.

[0146] The embodiment of the present specification also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, the computer is caused to execute the following Figure 3 Described method.

[0147] The embodiment of this specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the following is achieved: Figure 3 Described method.

[0148] The embodiments of this specification also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the following Figure 3 Describe the steps of the method.

[0149] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described in the various embodiments disclosed in this specification may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0150] In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. Additionally, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0151] The specific implementation methods described above further illustrate in detail the purposes, technical solutions and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above description is only the specific implementation methods of the multiple embodiments disclosed in this specification, and is not intended to limit the protection scope of the multiple embodiments disclosed in this specification. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the multiple embodiments disclosed in this specification should be included in the protection scope of the multiple embodiments disclosed in this specification.

Claims

1. A method for generating visual content, the method comprising: Obtaining user instruction information and a generated sequence of image data sets, wherein the user instruction information is used to instruct generation of visual content, the sequence of image data sets includes at least one image data set, the image data set includes a plurality of image data, each of the image data corresponding to the visual content; Based on the user instruction information and the image data set sequence, the autoregressive large model is used to predict visual distribution information corresponding to the next image data set, where the visual distribution information is probability distribution information in a continuous domain; Based on the visual distribution information corresponding to the next image data set, continuous domain sampling is performed to obtain the next image data set.

2. The method according to claim 1, wherein When the visual content is an image, each image data corresponds to an image block after the image is segmented. For an image block corresponding to any image data in the image data set, there is at least one image block corresponding to image data in the same image data set adjacent to it.

3. The method according to claim 1, wherein In the case where the visual content is an image, the number of image data in each image data set of the image data set sequence is positively correlated with the generation order of the image data sets.

4. The method according to claim 1, wherein In the case where the visual content is an image, the predicting of visual distribution information corresponding to the next image data set by the autoregressive large model based on the user instruction information and the image data set sequence includes: constructing a target sequence based on the user instruction information and the image data set sequence, wherein the target sequence includes: a user instruction sequence, an image start identifier before the image set sequence, and the image set sequence; The target sequence is input into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

5. The method according to claim 4, wherein In response to the next image data set to be predicted being the last image data set, the target sequence further includes an image end marker following the image set sequence.

6. The method according to claim 1, wherein In the case where the visual content is a video, a first video frame of the video corresponds to a plurality of first image data sets in the image data sets, and the first video frame is the first video frame in the video.

7. The method according to claim 6, wherein: Each target video frame of the video corresponds one-to-one to each target image data set in the video content set sequence. The target video frame is other video frames in the video other than the first video frame, and the target image data set is other image data sets other than the first image data set in the video content set sequence.

8. The method according to claim 7, wherein: The method of predicting visual distribution information corresponding to a next image data set based on the user instruction information and the image data set sequence by the autoregressive large model includes: In response to the image data set to be predicted being the image data set corresponding to the first video frame, constructing a target sequence based on the user instruction information and the image data set sequence, the target sequence including: the user instruction sequence, an image start identifier preceding the first frame set sequence, and the first frame set sequence; The target sequence is input into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

9. The method according to claim 7, wherein: The method of predicting visual distribution information corresponding to a next image data set based on the user instruction information and the image data set sequence by the autoregressive large model includes: In response to the image data set to be predicted being the image data set corresponding to the target video frame, constructing a target sequence based on the user instruction information and the image data set sequence, the target sequence including: the user instruction sequence, an image start identifier before a first frame set sequence, the first frame set sequence, an image end identifier after the first frame set sequence, a frame start identifier before the target video frame, the target video frame, and a frame end identifier after the target video frame; The target sequence is input into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

10. The method according to claim 1, wherein The user instruction information includes at least one of the following information: text information, image information and video information.

11. The method according to claim 1, wherein The autoregressive large model consists of a decoder.

12. The method according to claim 1, wherein The method of predicting visual distribution information corresponding to a next image data set based on the user instruction information and the image data set sequence by the autoregressive large model includes: The target encoder corresponding to different information in the user instruction information encodes the user instruction information to obtain a user instruction sequence; Based on the user instruction sequence and the image data set sequence, the visual distribution information corresponding to the next image data set is predicted by the autoregressive large model.

13. The method according to claim 12, wherein: In a case where the user instruction information includes image information or video information, the target encoder corresponding to the image information or video information is a two-dimensional variational autoencoder.

14. The method according to claim 1, wherein After obtaining the next image data set, the method further includes: The next image data set is decoded to obtain visual content corresponding to the next image data set.

15. The method according to claim 1, wherein The method of predicting visual distribution information corresponding to a next image data set based on the user instruction information and the image data set sequence by the autoregressive large model includes: Decoding the image data set sequence to obtain generated visual content; Inputting the user instruction information and the generated visual content into an encoder for encoding to obtain a continuous field vector; The continuous domain vector is input into the autoregressive large model to predict the visual distribution information corresponding to the next image data set.

16. The method according to claim 1, wherein The performing continuous domain sampling based on the visual distribution information corresponding to the next image data set to obtain the next image data set includes: The continuous domain sampler performs continuous domain sampling based on the visual distribution information corresponding to the next image data set to obtain the next image data set.

17. The method according to claim 1, wherein The autoregressive large model is trained in the following way: Obtaining user instruction samples and generated image data set sample sequences; Based on the user instruction sample and the image data set sample sequence, the autoregressive large model predicts visual distribution sample information corresponding to the next visual content sample set, wherein the visual distribution sample information is probability distribution information in a continuous domain; performing continuous domain sampling based on visual distribution sample information corresponding to the next visual content sample set to obtain the next visual content sample set; Based on the difference between the next visual content label set obtained and the next visual content sample set, the parameters of the autoregressive large model are adjusted.

18. A computing device comprising a memory and a processor, wherein: The memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 17 is implemented.

Images