Auto-regression image generation method, computer equipment and computer program product

Abstract

The invention relates to an autoregression image generation method, computer equipment and a computer program product. The method comprises the following steps: generating text sequence representation according to an image description text; inputting the text sequence representation into a frequency autoregression generation model, and gradually predicting a plurality of frequency domain sequence representations corresponding to the to-be-generated image according to a sequence from low frequency to high frequency; decoding the frequency domain sequence representation to generate a target image; and the quality of the generated image is improved.

Description

Technical Field

[0001] The present application relates to the field of image generation, and in particular to an autoregressive image generation method, a computer device, and a computer program product. Background Art

[0002] The autoregressive image generation model is a deep learning method for generating images. It converts images into discrete region representations and uses these representations to perform raster scan image generation predictions. Specifically, the autoregressive image generation model first divides the image into a series of discrete regions (usually pixels). The model then generates feature representations of these regions one by one in a raster scan (from left to right and from top to bottom). By gradually generating various parts of the image, it achieves modeling and generation of complex images.

[0003] However, discrete region representation discretizes image partitions, resulting in a focus on local areas while ignoring the overall image information. Raster scanning image generation prediction methods represent an artificially defined one-dimensional image generation order, but image information is arranged in two dimensions and is non-sequential. The artificially introduced sequence dependencies can interfere with the image generation process and reduce image quality.

[0004] Currently, no effective solution has been provided for the technical problem of low quality of generated images in related technologies. Summary of the Invention

[0005] Based on this, it is necessary to provide an autoregressive image generation method, computer device and computer program product that can improve the quality of generated images to address the above technical problems.

[0006] In a first aspect, the present application provides a method for generating an autoregressive image. The method comprises:

[0007] Generate text sequence representation based on image description text;

[0008] Inputting the text sequence representation into a frequency autoregressive generative model, and gradually predicting multiple frequency domain sequence representations corresponding to the image to be generated in order from low frequency to high frequency;

[0009] The frequency domain sequence representation is decoded to generate a target image.

[0010] In some embodiments, the text sequence representation is input into a frequency autoregressive generative model, and multiple frequency domain sequence representations corresponding to the image to be generated are gradually predicted in order from low frequency to high frequency, including:

[0011] In the initial prediction stage, the text sequence representation is input into the frequency autoregressive generation model to predict the frequency domain sequence representation of the initial prediction stage;

[0012] In each prediction stage after the initial prediction stage, obtaining the first frequency domain sequence representation predicted in the previous prediction stage, and using the first frequency domain sequence representation and the text sequence representation as inputs of the current prediction stage to predict a second frequency domain sequence representation corresponding to the current prediction stage;

[0013] The frequency represented by the first frequency domain sequence is lower than the frequency represented by the second frequency domain sequence.

[0014] In some embodiments, in each prediction stage after the initial prediction stage, obtaining the first frequency domain sequence representation predicted in the previous prediction stage, and using the first frequency domain sequence representation and the text sequence representation as inputs of the current prediction stage to predict a second frequency domain sequence representation corresponding to the current prediction stage includes:

[0015] Concatenate the text sequence representation with the first frequency domain sequence representation to obtain an input sequence for this prediction stage;

[0016] Performing feature encoding on the input sequence to obtain an implicit representation sequence;

[0017] Inputting the implicit representation sequence into the frequency autoregressive generative model, the frequency autoregressive generative model outputs an activation function probability distribution based on the image hidden state at the current prediction position; wherein the activation function probability distribution represents the probability that the image hidden state at the current prediction position belongs to each token in the discrete token library;

[0018] Based on the probability distribution of the activation function, the token corresponding to the highest probability is selected from the discrete token library as the target token for this prediction position;

[0019] The second frequency domain sequence representation is obtained based on the target token.

[0020] In some embodiments, decoding the frequency domain sequence representation to generate a target image includes:

[0021] Inputting the frequency domain sequence representation into a trained decoder, and restoring the frequency domain sequence representation to frequency domain sequence features via the decoder;

[0022] Perform inverse Fourier transform on the frequency domain sequence features to generate the target image.

[0023] In some embodiments, before inputting the frequency domain sequence representation into the trained decoder, the method further comprises:

[0024] The encoder and decoder are constructed based on the neural network modules of multi-layer integrated self-attention mechanism;

[0025] Acquire a spectrogram of the first image, and extract multi-level frequency domain features from low frequency to high frequency based on the spectrogram;

[0026] Inputting the multi-level frequency domain features into the encoder, and compressing the multi-level frequency domain features into a frequency domain sequence representation from low frequency to high frequency via the encoder;

[0027] Inputting the frequency domain sequence representation into the decoder, and restoring the frequency domain sequence representation into multi-level frequency domain features from low frequency to high frequency via the decoder;

[0028] Performing inverse Fourier transform on the multi-level frequency domain features to generate a second image;

[0029] The encoder and the decoder are trained with the goal of minimizing the image space reconstruction difference between the first image and the second image.

[0030] In some embodiments, obtaining a spectrogram of the first image, and extracting multi-level frequency domain features from low frequency to high frequency based on the spectrogram include:

[0031] Performing Fourier transform on the first image to obtain a frequency spectrum of the first image;

[0032] Dividing the spectrum graph according to a preset frequency range to obtain multiple bandpass spectrum features;

[0033] Based on the multi-segment bandpass spectrum characteristics, the multi-level frequency domain characteristics are obtained.

[0034] In some embodiments, decoding the frequency domain sequence representation to generate a target image includes:

[0035] Decoding each of the frequency domain sequence representations to generate a target image corresponding to each of the frequency domain sequence representations; or,

[0036] The frequency domain sequence representation is decoded in order from low frequency to high frequency intervals to generate a target image corresponding to a portion of the frequency domain sequence representation.

[0037] In some embodiments, decoding the frequency domain sequence representation to generate a target image includes:

[0038] Determine whether the prediction progress of the current frequency domain sequence representation reaches a preset progress; the preset progress is greater than 0% and less than 100%;

[0039] If the predicted progress of the current frequency domain sequence representation reaches the preset progress, decoding the currently generated frequency domain sequence representation and outputting a preview image of the target image;

[0040] An operation instruction input by a user is received, and whether to continue predicting the frequency domain sequence representation is determined according to the operation instruction.

[0041] In a second aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0042] In a third aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0043] The aforementioned autoregressive image generation method, computer device, and computer program product for generating high-quality images combine image description text with a frequency autoregressive generation model to gradually generate a frequency domain sequence representation from low frequency to high frequency in an autoregressive manner, and decode the target image based on the frequency domain sequence representation. This low-to-high frequency generation sequence enables the model to first construct the overall framework of the image and then gradually refine the details. This captures local details while retaining the overall image information, and the overall image processing can alleviate image quality issues caused by pixel discontinuities between generated image regions due to block processing. Furthermore, the frequency domain representation obtained based on frequency decomposition has a higher upper limit on the representation domain than the block representation domain, allowing it to express more image details and address the blurring and distortion issues of generated images caused by block representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 FIG1 is a hardware structure block diagram of a terminal of an autoregressive image generation method according to an embodiment;

[0045] Figure 2 1 is a flow chart of an autoregressive image generation method according to an embodiment;

[0046] Figure 3 An example diagram of an autoregressive image generation process in one embodiment;

[0047] Figure 4 An example diagram of an autoregressive image generation result in one embodiment;

[0048] Figure 5 Schematic diagram of a process for predicting frequency domain sequence representation in one embodiment;

[0049] Figure 6 1 is a schematic diagram of a process for training an encoder / decoder in one embodiment;

[0050] Figure 7 A schematic diagram of feature reconstruction training of a discrete autoencoder in one embodiment;

[0051] Figure 8 A schematic diagram of image feature extraction based on frequency feature decomposition in one embodiment;

[0052] Figure 9 Schematic diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0055] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a block diagram of the hardware structure of a terminal of the autoregressive image generation method according to an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1Only one is shown) a processor 101 and a memory 102 for storing data, wherein the processor 101 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 103 for communication functions and an input and output device 104. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0056] Memory 102 can be used to store computer programs, such as software programs and modules for application software, such as the computer program corresponding to the autoregressive image generation method in this embodiment. Processor 101 executes the computer programs stored in memory 102 to perform various functional applications and data processing, thereby implementing the above-mentioned method. Memory 102 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 102 may further include memory remotely located relative to processor 101, and such remote memory may be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0057] Transmission device 103 is used to receive or transmit data via a network. This network may include a wireless network provided by the terminal's communications provider. In one embodiment, transmission device 103 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 103 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0058] In one embodiment, Figure 2 As shown, an autoregressive image generation method is provided, which is applied to Figure 1 The following steps are used as an example to illustrate the terminal in the figure:

[0059] Step S101: Generate a text sequence representation based on the image description text.

[0060] Text sequence representation is a structured representation of text sequences that can be understood and processed by frequency-dependent autoregressive generative models. A text sequence representation can be a sequence of characters, words, or subword units (such as tokens in BERT).

[0061] Figure 3 An example diagram of the autoregressive image generation process is provided. The terminal receives user-entered image description text (for example, "seven-piece puzzle") and processes the image description text using a pretrained text converter. The text converter preprocesses and tokenizes / encodes the image description text. During tokenization / encoding, a tokenization tool (such as the BERT Tokenizer) is used to break the text into basic units (tokens). These basic units are converted into corresponding numerical codes, and these encoded tokens are arranged in sequence to form a continuous text sequence, which is the text sequence representation.

[0062] In step S102 , the text sequence representation is input into a frequency autoregressive generative model, and a plurality of frequency domain sequence representations corresponding to the image to be generated are gradually predicted in order from low frequency to high frequency.

[0063] The frequency autoregressive generative model is a deep learning model that can be composed of multiple layers of neural network modules with integrated self-attention mechanisms.

[0064] Frequency domain sequence representation refers to a series of ordered frequency domain coefficients obtained after converting an image from the time domain to the frequency domain. These coefficients are used to describe the components of the image at different frequencies, including low-frequency and high-frequency components.

[0065] Continue to refer Figure 3 In this step, the terminal inputs the text sequence representation generated in the previous step into the frequency autoregressive generative model, which predicts the frequency domain coefficients of the image one by one in an autoregressive manner. Specifically, the model first initializes some parameters and prepares to begin generating frequency domain coefficients. It generates frequency domain coefficients one by one in order from low frequency to high frequency. Each time a new frequency domain coefficient is generated, it uses the previously generated frequency domain coefficients as context information, ultimately generating a series of ordered frequency domain coefficients that constitute the frequency domain sequence representation.

[0066] Step S103: decode the frequency domain sequence representation to generate a target image.

[0067] Continue to refer Figure 3 In this step, the terminal inputs the generated frequency domain sequence representation into the trained decoder, and the decoder restores the frequency domain sequence representation to frequency domain sequence features; performs inverse Fourier transform on the frequency domain sequence features to generate the target image.

[0068] In some embodiments, the terminal can decode each frequency domain sequence representation to generate a target image corresponding to each frequency domain sequence representation. This means outputting all intermediate target images and the final target image. These target images correspond to frequency domain sequences ranging from low to high frequencies, and their image quality varies from low to high.

[0069] Figure 4 An example of an autoregressive image generation result is provided, such as Figure 4 As shown in the figure, the user enters "Generate a cartoon-style koala" in the terminal, and the terminal generates intermediate images and the final image. It can be seen that these images gradually become clearer and blurrier as they progress from the beginning to the end, and the image quality improves.

[0070] In some embodiments, the terminal may decode the frequency domain sequence representation in order from low frequency to high frequency, generating a target image corresponding to a portion of the frequency domain sequence representation. This means outputting a portion of the intermediate target images and a final target image. These target images correspond to the frequency domain sequence from low frequency to high frequency, and the image quality varies from low to high.

[0071] In some embodiments, the terminal may decode only the frequency domain sequence representation generated in the last prediction stage to generate a target image corresponding to the final frequency domain sequence representation. In other words, only the final target image is output, and intermediate target images are not output.

[0072] In related technologies, the regional representation of the autoregressive image generation model is discrete, that is, it focuses on local details and ignores the overall information of the image, which leads to low image quality, specifically manifested in blurred generated images, discontinuous pixels between generated image regions, and distortion of generated images.

[0073] In the above steps S101 to S103, by combining the image description text and the frequency autoregressive generation model, the frequency domain sequence representation from low frequency to high frequency is gradually generated in an autoregressive mode, and the target image is obtained by decoding based on the frequency domain sequence representation. This generation order from low frequency to high frequency enables the model to first construct the overall framework of the image and then gradually refine the details. While obtaining local details, the overall information of the image is retained, and the overall processing of the image can alleviate the image quality problem of pixel discontinuity between the generated image areas caused by block processing. At the same time, the representation domain upper limit of the frequency domain representation obtained based on frequency decomposition is higher than the representation domain upper limit of the block representation, which can express more image details and solve the problem of blurring and distortion of the generated image caused by block representation.

[0074] This embodiment realizes the generation process from outline construction to detail rendering by using a low-frequency to high-frequency image generation method, which complies with the image drawing logic process, gradually improves the quality of the generated image during the prediction process, and ultimately alleviates the problems of image generation blur, block discontinuity, and image distortion.

[0075] In one embodiment, the text sequence representation is input into a frequency autoregressive generative model, and multiple frequency domain sequence representations corresponding to the image to be generated are gradually predicted in order from low frequency to high frequency. This can be achieved by the following method:

[0076] In the initial prediction stage, the text sequence representation is input into the frequency autoregressive generation model to predict the frequency domain sequence representation of the initial prediction stage; in each prediction stage after the initial prediction stage, the first frequency domain sequence representation predicted in the previous prediction stage is obtained, and the first frequency domain sequence representation and the text sequence representation are used as inputs of this prediction stage to predict the second frequency domain sequence representation corresponding to this prediction stage; wherein, the frequency of the first frequency domain sequence representation is lower than the frequency of the second frequency domain sequence representation.

[0077] In the initial prediction phase, only the text sequence representation is input. The frequency autoregressive generative model predicts a frequency domain sequence representation based on the input text sequence representation. In each subsequent prediction phase, the text sequence representation is combined with the first frequency domain sequence representation predicted in the previous prediction phase and used as the input for the current prediction phase. Similarly, the autoregressive model predicts frequency domain sequence representations from low frequency to high frequency. It should be noted that the first frequency domain sequence representation and the second frequency domain sequence representation are relative. For example, assume there are four prediction phases, where the second prediction phase generates frequency domain sequence representation A, the third prediction phase generates frequency domain sequence representation B, and the fourth prediction phase generates frequency domain sequence representation C. For the second and third prediction phases, frequency domain sequence representation A belongs to the first frequency domain sequence representation, and frequency domain sequence representation B belongs to the second frequency domain sequence representation. For the third and fourth prediction phases, frequency domain sequence representation B belongs to the first frequency domain sequence representation, and frequency domain sequence representation C belongs to the second frequency domain sequence representation.

[0078] Among them, for each prediction stage after the initial prediction stage, you can use Figure 5 The process shown is implemented as Figure 5 As shown, the process includes the following steps:

[0079] Step S201 : concatenate the text sequence representation and the first frequency domain sequence representation to obtain an input sequence for this prediction stage.

[0080] The text sequence representation can be composed of text tokens, and the first frequency domain sequence representation can be composed of image tokens. The concatenated input sequence can be represented as [TEXT_1, TEXT_2, ..., TEXT_M, IMG_1, IMG_2, ..., IMG_(t-1)] , where TEXT is a text unit (i.e., a text token), M is the number of text tokens, IMG is an image unit (i.e., an image token), t-1 is the number of generated image tokens, and t is the number of the image token to be generated in the current prediction phase, which also represents the prediction position.

[0081] Step S202: feature encode the input sequence to obtain an implicit representation sequence.

[0082] The input sequence is encoded by the corresponding encoder (text encoder / image encoder) to obtain the corresponding implicit representation sequence (hidden state list).

[0083] In step S203, the implicit representation sequence is input into the frequency autoregressive generation model, and the frequency autoregressive generation model outputs the activation function probability distribution according to the image hidden state of the current prediction position; wherein the activation function probability distribution represents the probability that the image hidden state of the current prediction position belongs to each token in the discrete token library.

[0084] The implicit representation sequence enters the frequency autoregressive generative model to predict the implicit representation of the image at position t. Specifically, the model uses causal masking to ensure that the prediction of the current position t depends only on the previous representation. Based on the predicted hidden state of the image at position t, the model outputs an activation function (softmax) probability distribution, which represents the probability that the predicted hidden state of the image at position t belongs to each token in the discrete token library. The activation function probability distribution expression is P(IMG_t | TEXT, IMG_ <t) = softmax(W h_t + b). Where h_t is the hidden state vector at predicted position t, and W and b are the projection layer parameters.

[0085] In step S204, based on the probability distribution of the activation function, the token corresponding to the highest probability is selected from the discrete token library as the target token for this predicted position.

[0086] In this step, a greedy sampling method can be used to select the token corresponding to the highest probability from the discrete token library based on the probability distribution of the activation function as the target token for this prediction position.

[0087] Step S205: Obtain a second frequency domain sequence representation based on the target token.

[0088] The target token at this predicted position is added to the previous frequency domain sequence to obtain the second frequency domain sequence representation: IMG_1:t = [IMG_1, IMG_2, ..., IMG_(t-1), IMG_t]. This setting ensures that the next model input contains the latest token for predicting the next image token.

[0089] In step S206 , it is determined whether the end symbol (eos token) is predicted or the maximum frequency domain sequence representation length is reached; if so, the process ends; if not, the process returns to step S201 to continue predicting the next image token.

[0090] In one embodiment, the method further comprises jointly training an encoder and a decoder, and decoding the frequency domain sequence representation based on the trained decoder to generate a target image. Figure 6 This is a flow chart of the encoder / decoder training process in this embodiment. Figure 7 This is a diagram of feature reconstruction training based on discrete autoencoder in this implementation. Figure 8 Schematic diagram of image feature extraction based on frequency feature decomposition in this implementation.

[0091] like Figure 6 As shown, the encoder / decoder training process includes the following steps:

[0092] Step S301: construct an encoder and a decoder based on a neural network module with a multi-layer integrated self-attention mechanism.

[0093] refer to Figure 7 The VQ-VAE (Vein-the-Field-Encoder) consists of an encoder and a decoder. Both structures are identical and constructed from multi-layer neural network modules with integrated self-attention mechanisms. This self-attention mechanism captures global dependencies between input features, and through these multi-layered modules, it extracts higher-level abstract features layer by layer. The encoder and decoder are symmetrically designed, with opposite functions: the encoder compresses information, while the decoder restores it.

[0094] Step S302 : obtaining a spectrogram of the first image, and extracting multi-level frequency domain features from low frequency to high frequency based on the spectrogram.

[0095] Unlike the previous method of dividing the image represented by RGB pixels into multiple image blocks, the image is represented as a combination of multiple local blocks. Figure 8 In this embodiment, the original image (i.e., the first image) is subjected to a Fourier transform (specifically, a two-dimensional fast Fourier transform) as a whole to obtain a spectrogram corresponding to the original image. This method captures local details while preserving the overall image information. It is important to emphasize that the Fourier transform causes negligible information loss to the image, and the spectrogram can be inversely transformed to obtain the original image.

[0096] After obtaining the original image's spectrogram, its spectrum is divided into N equal parts according to a preset frequency range, resulting in multi-bandpass spectrum features. This is essentially a multi-level frequency domain representation of the image, from low to high frequencies. Because frequency domain features lack frequency band overlap and are naturally orthogonal and independent, whereas typical image pixel block features exhibit similarities and overlap, the upper limit of the representation domain for frequency domain features is higher than that for image blocks.

[0097] In step S303 , the multi-level frequency domain features are input to an encoder, and the encoder compresses the multi-level frequency domain features into a frequency domain sequence representation from low frequency to high frequency.

[0098] The encoder compresses multi-level frequency domain features to obtain a frequency domain sequence representation with higher information density.

[0099] In step S304 , the frequency domain sequence representation is input to a decoder, and the decoder restores the frequency domain sequence representation into multi-level frequency domain features from low frequency to high frequency.

[0100] The decoder restores the frequency domain sequence representation to the original multi-level frequency domain features.

[0101] Step S305: Perform inverse Fourier transform on the multi-level frequency domain features to generate a second image.

[0102] Step S306 : training the encoder and the decoder with the goal of minimizing the image space reconstruction difference between the first image and the second image.

[0103] In this step, a loss function can be designed based on the difference in image space reconstruction between the first and second images. The discrete autoencoder is iteratively trained with the goal of minimizing the loss function. Specifically, the image space reconstruction difference can be the mean squared error in image space. After training, the encoder is able to significantly compress multi-level frequency domain features without losing the original information. The decoder, on the other hand, is able to restore the frequency domain sequence representation as closely as possible, obtaining the original multi-level frequency domain features.

[0104] This embodiment is based on the VQ-VAE training process of frequency domain analysis and modeling. It first converts the image into frequency domain features, then compresses and restores it through self-attention modeling, and finally optimizes the entire model through backpropagation of image reconstruction errors, thereby learning an efficient and compact image representation.

[0105] In one embodiment, when decoding the frequency domain sequence representation to generate a target image, it is also possible to first determine whether the prediction progress of the current frequency domain sequence representation has reached a preset progress; wherein the preset progress is greater than 0% and less than 100%; if the prediction progress of the current frequency domain sequence representation has reached the preset progress, the currently generated frequency domain sequence representation is decoded and a preview image of the target image is output; an operation instruction input by the user is received, and it is determined whether to continue predicting the frequency domain sequence representation according to the operation instruction.

[0106] In this embodiment, by setting a preset progress (between 0% and 100%), a decoding preview can be performed at an early stage of the image generation process, allowing the user to see the approximate appearance of the generated image in advance. If the preliminary image generated according to the preset progress is good enough or clearly does not meet expectations, the user can choose to stop the further prediction process, thereby saving potentially unnecessary computing resources.

[0107] For example, if the user is dissatisfied with the currently generated image and wishes to observe subsequent images, the user is instructed to continue predicting the frequency-domain sequence representation to generate a new target image. If the user is satisfied with the currently generated image, or if the currently generated image clearly does not meet expectations, the user is instructed to stop predicting the frequency-domain sequence representation. This arrangement improves generation efficiency and enhances the user experience while maintaining a certain level of image quality.

[0108] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication. The wireless communication can be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements an autoregressive image generation method. The display screen of the computer device can be a liquid crystal display or an electronic ink display. The input device of the computer device can be a touch layer covering the display screen, or keys, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse.

[0109] Those skilled in the art will understand that Figure 9The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0110] Based on the same inventive concept, an embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned autoregressive image generation method when executed by a processor.

[0111] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0112] Generate text sequence representation based on image description text;

[0113] The text sequence representation is input into the frequency autoregressive generative model, and multiple frequency domain sequence representations corresponding to the image to be generated are gradually predicted in order from low frequency to high frequency;

[0114] The frequency domain sequence representation is decoded to generate the target image.

[0115] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0116] In the initial prediction stage, the text sequence representation is input into the frequency autoregressive generative model to predict the frequency domain sequence representation of the initial prediction stage;

[0117] In each prediction stage after the initial prediction stage, the first frequency domain sequence representation predicted in the previous prediction stage is obtained, and the first frequency domain sequence representation and the text sequence representation are used as inputs of the current prediction stage to predict the second frequency domain sequence representation corresponding to the current prediction stage;

[0118] The frequency represented by the first frequency domain sequence is lower than the frequency represented by the second frequency domain sequence.

[0119] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0120] Concatenate the text sequence representation with the first frequency domain sequence representation to obtain the input sequence for this prediction stage;

[0121] Perform feature encoding on the input sequence to obtain an implicit representation sequence;

[0122] The implicit representation sequence is input into the frequency autoregressive generative model, which then outputs an activation function probability distribution based on the image hidden state at the current prediction position. The activation function probability distribution represents the probability that the image hidden state at the current prediction position belongs to each token in the discrete token library.

[0123] Based on the probability distribution of the activation function, the token with the highest probability is selected from the discrete token library as the target token for this prediction position;

[0124] A second frequency domain sequence representation is obtained based on the target token.

[0125] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0126] The frequency domain sequence representation is input into the trained decoder, and the decoder restores the frequency domain sequence representation to frequency domain sequence features;

[0127] Perform inverse Fourier transform on the frequency domain sequence features to generate the target image.

[0128] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0129] The encoder and decoder are constructed based on the neural network modules of multi-layer integrated self-attention mechanism;

[0130] Acquire a spectrogram of the first image, and extract multi-level frequency domain features from low frequency to high frequency based on the spectrogram;

[0131] The multi-level frequency domain features are input into the encoder, and the encoder compresses the multi-level frequency domain features into a frequency domain sequence representation from low frequency to high frequency;

[0132] The frequency domain sequence representation is input into the decoder, and the decoder restores the frequency domain sequence representation into multi-level frequency domain features from low frequency to high frequency;

[0133] Performing inverse Fourier transform on the multi-level frequency domain features to generate a second image;

[0134] The encoder and decoder are trained with the goal of minimizing the image space reconstruction difference between the first image and the second image.

[0135] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0136] Performing Fourier transform on the first image to obtain a frequency spectrum of the first image;

[0137] Divide the spectrum graph according to the preset frequency range to obtain multiple bandpass spectrum characteristics;

[0138] Based on the multi-bandpass spectrum characteristics, multi-level frequency domain features are obtained.

[0139] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0140] Decoding each frequency domain sequence representation to generate a target image corresponding to each frequency domain sequence representation; or,

[0141] The frequency domain sequence representation is decoded in order from low frequency to high frequency intervals to generate a target image corresponding to part of the frequency domain sequence representation.

[0142] In one embodiment, when the processor executes the computer program, the following steps are implemented:

[0143] Determine whether the prediction progress of the current frequency domain sequence representation has reached the preset progress; the preset progress is greater than 0% and less than 100%;

[0144] If the prediction progress of the current frequency domain sequence representation reaches the preset progress, the currently generated frequency domain sequence representation is decoded and a preview image of the target image is output;

[0145] Receive an operation instruction input by the user, and determine whether to continue predicting the frequency domain sequence representation based on the operation instruction.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0147] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0148] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An autoregressive image generation method, characterized in that: include: Generate text sequence representation based on image description text; Inputting the text sequence representation into a frequency autoregressive generative model, and gradually predicting multiple frequency domain sequence representations corresponding to the image to be generated in order from low frequency to high frequency; The frequency domain sequence representation is decoded to generate a target image.

2. The autoregressive image generation method according to claim 1, characterized in that The text sequence representation is input into the frequency autoregressive generative model, and multiple frequency domain sequence representations corresponding to the image to be generated are gradually predicted in order from low frequency to high frequency, including: In the initial prediction stage, the text sequence representation is input into the frequency autoregressive generation model to predict the frequency domain sequence representation of the initial prediction stage; In each prediction stage after the initial prediction stage, obtaining the first frequency domain sequence representation predicted in the previous prediction stage, and using the first frequency domain sequence representation and the text sequence representation as inputs of the current prediction stage to predict a second frequency domain sequence representation corresponding to the current prediction stage; The frequency represented by the first frequency domain sequence is lower than the frequency represented by the second frequency domain sequence.

3. The autoregressive image generation method according to claim 2, characterized in that In each prediction stage after the initial prediction stage, obtaining the first frequency domain sequence representation predicted in the previous prediction stage, and using the first frequency domain sequence representation and the text sequence representation as inputs of the current prediction stage, and predicting a second frequency domain sequence representation corresponding to the current prediction stage, including: Concatenate the text sequence representation with the first frequency domain sequence representation to obtain an input sequence for this prediction stage; Performing feature encoding on the input sequence to obtain an implicit representation sequence; Inputting the implicit representation sequence into the frequency autoregressive generative model, the frequency autoregressive generative model outputs an activation function probability distribution based on the image hidden state at the current prediction position; wherein the activation function probability distribution represents the probability that the image hidden state at the current prediction position belongs to each token in the discrete token library; Based on the probability distribution of the activation function, the token corresponding to the highest probability is selected from the discrete token library as the target token for this prediction position; The second frequency domain sequence representation is obtained based on the target token.

4. The autoregressive image generation method according to claim 1, wherein: Decoding the frequency domain sequence representation to generate a target image includes: Inputting the frequency domain sequence representation into a trained decoder, and restoring the frequency domain sequence representation to frequency domain sequence features via the decoder; Perform inverse Fourier transform on the frequency domain sequence features to generate the target image.

5. The autoregressive image generation method according to claim 4, characterized in that: Before inputting the frequency domain sequence representation into the trained decoder, the method further includes: The encoder and decoder are constructed based on the neural network modules of multi-layer integrated self-attention mechanism; Acquire a spectrogram of the first image, and extract multi-level frequency domain features from low frequency to high frequency based on the spectrogram; Inputting the multi-level frequency domain features into the encoder, and compressing the multi-level frequency domain features into a frequency domain sequence representation from low frequency to high frequency via the encoder; Inputting the frequency domain sequence representation into the decoder, and restoring the frequency domain sequence representation into multi-level frequency domain features from low frequency to high frequency via the decoder; Performing inverse Fourier transform on the multi-level frequency domain features to generate a second image; The encoder and the decoder are trained with the goal of minimizing the image space reconstruction difference between the first image and the second image.

6. The autoregressive image generation method according to claim 5, characterized in that: Acquiring a spectrogram of the first image, and extracting multi-level frequency domain features from low frequency to high frequency based on the spectrogram, including: Performing Fourier transform on the first image to obtain a frequency spectrum of the first image; Dividing the spectrum graph according to a preset frequency range to obtain multiple bandpass spectrum features; Based on the multi-segment bandpass spectrum characteristics, the multi-level frequency domain characteristics are obtained.

7. The autoregressive image generation method according to claim 1, wherein: Decoding the frequency domain sequence representation to generate a target image includes: Decoding each of the frequency domain sequence representations to generate a target image corresponding to each of the frequency domain sequence representations; or, The frequency domain sequence representation is decoded in order from low frequency to high frequency intervals to generate a target image corresponding to a portion of the frequency domain sequence representation.

8. The autoregressive image generation method according to claim 1, wherein: Decoding the frequency domain sequence representation to generate a target image includes: Determine whether the prediction progress of the current frequency domain sequence representation reaches a preset progress; the preset progress is greater than 0% and less than 100%; If the predicted progress of the current frequency domain sequence representation reaches the preset progress, decoding the currently generated frequency domain sequence representation and outputting a preview image of the target image; An operation instruction input by a user is received, and whether to continue predicting the frequency domain sequence representation is determined according to the operation instruction.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images