Image generation method and device, storage medium and processor
By combining image resolution binning strategy and relative position encoding, the high computational complexity and resolution scaling distortion problems of image generation in existing technologies are solved, and efficient and accurate multi-resolution image generation is achieved.
Patent Information
- Application Number
- CN202511523081.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-23
- Publication Date
- 2026-01-09
AI Technical Summary
Existing diffusion and autoregressive models suffer from high computational complexity, slow generation speed, and image distortion and detail loss due to resolution scaling in image generation, making them difficult to meet the needs of real-time and large-scale applications.
An image resolution bucketing strategy is adopted to convert the initial image into an image of the corresponding bucket resolution. A feature sequence is generated by combining a pre-set image conversion model and a text conversion model. The image is generated by a mask autoregressive model and an MLP diffusion denoising model. Relative position encoding is introduced to ensure the consistency of pixel relationship under different resolutions.
It improves the accuracy and efficiency of image generation, avoids information loss and distortion caused by forced scaling, enhances the model's ability to generate multi-resolution images, and ensures the consistency of details in high-resolution images.
Smart Images
Figure CN121304451A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, specifically to an image generation method, an image generation device, a computer program product, a machine-readable storage medium, and a processor. Background Technology
[0002] In the field of image generation, the current mainstream technical approaches mainly include diffusion models and autoregressive models. Diffusion models are a class of methods that generate images by progressively denoising. Specifically, diffusion models first add noise to the data in the forward process using a Markov chain model, transforming it into Gaussian noise, and then progressively remove the noise in the reverse process to recover the original image. The advantage of this method is that it can approximate the complex distribution of the data, thereby generating high-quality images. However, the generation process of diffusion models usually requires multiple iterations, resulting in high computational complexity and slow generation speed, which cannot meet the needs of some real-time or large-scale applications.
[0003] On the other hand, autoregressive models, by progressively predicting the next element in a sequence, have demonstrated superior performance in language and vision tasks. Autoregressive models can progressively generate high-quality images and maintain strict dependence on the input sequence during generation, ensuring the coherence and consistency of the generated images. However, in existing techniques, to adapt to a fixed-size model input, images typically need to be scaled to a specific resolution scale. This scaling process often causes image distortion, especially when the original image resolution differs significantly from the target resolution, resulting in more pronounced loss of detail and deformation. This approach not only affects the quality of the generated images but also increases the difficulty of ensuring consistency in image processing across different resolutions. Summary of the Invention
[0004] The purpose of this application is to provide an image generation method, an image generation apparatus, a computer program product, a machine-readable storage medium, and a processor.
[0005] To achieve the above objectives, the first aspect of this application provides an image generation method, the image generation method comprising: Obtain the initial image and text description; Based on the image resolution binning strategy, the initial image is converted into the resolution image of the corresponding bin to obtain the image to be processed; The image to be processed is converted into an image token sequence using a preset image conversion model; The text description is converted into a semantic vector sequence using a pre-defined text conversion model; Based on the image token sequence and the semantic vector sequence, a feature sequence is obtained; The image tokens in the feature sequence are encoded by relative position to obtain the input feature sequence; The input feature sequence is input into a preset image generation model to obtain a new image.
[0006] In this embodiment of the application, the step of converting the initial image into a corresponding bucket resolution image based on the image resolution bucketing strategy to obtain the image to be processed includes: The bucket corresponding to the initial image is obtained by matching among a number of preset image resolution buckets, which are pre-constructed using an image resolution bucketing strategy; Based on the resolution of the bucket corresponding to the initial image, the initial image is converted into an image to be processed.
[0007] In this embodiment of the application, the step of encoding the relative positions of the image tokens in the feature sequence to obtain the input feature sequence includes: Convert the pixel coordinates of each image token in the feature sequence into a high-dimensional space vector; The high-dimensional space vector is rotated using a position-based rotation matrix to achieve relative position encoding, thereby obtaining the input feature sequence.
[0008] In this embodiment of the application, the preset image generation model includes: a mask autoregressive model and an MLP diffusion denoising model; The step of inputting the input feature sequence into a preset image generation model to obtain a new image includes: The input feature sequence is randomly masked to obtain a masked sequence; The masked sequence is input into the masked autoregressive model to obtain a new image token vector; Using the new image token vector as a diffusion denoising condition, the MLP diffusion denoising model is used for denoising to obtain a new image.
[0009] In this embodiment of the application, the mask autoregressive model is used to predict the token vector at the masked position using a bidirectional attention mechanism.
[0010] In this embodiment of the application, the training process of the masked autoregressive model includes: Obtain multiple input sequences, each with a different length; A mask for each input sequence is generated based on the length of each input sequence and the random mask ratio, respectively. Based on each input sequence and its corresponding mask, a pre-set autoregressive prediction model is trained to obtain a masked autoregressive model.
[0011] In this embodiment of the application, the training process of the preset image conversion model includes: Obtain the original image dataset, which includes original images of multiple sizes; Based on the image resolution binning strategy, each original image is converted into the resolution image of the corresponding bin, resulting in multiple original images to be processed corresponding to the bins. Based on the original images to be processed corresponding to each bucket, a pre-set image tokenizer model is trained to obtain a pre-set image transformation model.
[0012] A second aspect of this application provides an image generating apparatus, the image generating apparatus comprising: The acquisition module is used to acquire the initial image and text description; The bucketing module is used to convert the initial image into the resolution image of the corresponding bucket based on the image resolution bucketing strategy, so as to obtain the image to be processed; The first conversion module is used to convert the image to be processed into an image token sequence using a preset image conversion model; The second conversion module is used to convert the text description into a semantic vector sequence using a preset text conversion model; The combination module is used to obtain a feature sequence based on the image token sequence and the semantic vector sequence; The encoding module is used to encode the relative positions of the image tokens in the feature sequence to obtain the input feature sequence; The generation module is used to input the input feature sequence into a preset image generation model to obtain a new image.
[0013] A third aspect of this application provides a processor configured to perform the image generation method described above.
[0014] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the image generation method described above.
[0015] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the image generation method described above.
[0016] The above technical solution involves acquiring an initial image and text description; converting the initial image to the resolution of the corresponding bucket based on an image resolution binning strategy to obtain the image to be processed; converting the image to be processed into an image token sequence using a preset image conversion model; converting the text description into a semantic vector sequence using a preset text conversion model; obtaining a feature sequence based on the image token sequence and the semantic vector sequence; encoding the relative positions of the image tokens in the feature sequence to obtain an input feature sequence; and inputting the input feature sequence into a preset image generation model to obtain a new image. By introducing an image resolution binning strategy, the input initial image is grouped according to resolution range and adjusted to the corresponding bucket, avoiding information loss and distortion caused by forced uniform scaling, enhancing the model's ability to generate multi-resolution images. By introducing position encoding into autoregressive image generation, and embedding the relative position information of pixels, combined with the binning strategy, the consistency of pixel relationships at different resolutions is ensured, thereby ensuring the detail consistency of high-resolution images and improving the accuracy of image generation.
[0017] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description
[0018] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 This illustration schematically shows an application environment diagram of the image generation method according to an embodiment of this application; Figure 2 A schematic flowchart of an image generation method according to an embodiment of this application is shown. Figure 3 This illustration schematically shows a vectorless autoregressive image generation process according to an embodiment of this application; Figure 4 A schematic diagram illustrating the MLP diffusion denoising process according to an embodiment of this application is shown. Figure 5 This schematic diagram illustrates a structural block diagram of an image generation apparatus according to an embodiment of the present application; Figure 6 The diagram illustrates the internal structure of a computer device according to an embodiment of this application.
[0019] Explanation of reference numerals in the attached figures 410 - Acquisition Module; 420 - Bucketing Module; 430 - First Conversion Module; 440 - Second Conversion Module; 450 - Combination Module; 460 - Encoding Module; 470 - Generation Module; A01 - Processor; A02 - Network Interface; A03 - Internal Memory; A04 - Display Screen; A05 - Input Device; A06 - Non-Volatile Storage Medium; B01 - Operating System; B02 - Computer Program. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0021] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.
[0022] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.
[0023] The acquisition, transmission, storage, use, and processing of data in this application comply with relevant national laws and regulations. Furthermore, it should be noted that existing industry solutions such as software, components, and models may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.
[0024] The image generation method provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. By introducing an image resolution bucketing strategy, the initial input image is grouped according to resolution range and adjusted to the corresponding bucket, avoiding information loss and distortion caused by forced uniform scaling, thus enhancing the model's ability to generate multi-resolution images. By introducing positional encoding into autoregressive image generation, and embedding the relative position information of pixels, combined with the bucketing strategy, the consistency of pixel relationships at different resolutions is ensured, thereby ensuring the detail consistency of high-resolution images and improving the accuracy of image generation. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.
[0025] Figure 2 A schematic flowchart of an image generation method according to an embodiment of this application is shown. Figure 2 As shown, in one embodiment of this application, an image generation method is provided. This embodiment mainly applies this method to the above-mentioned... Figure 1 Taking terminal 102 (or server 104) as an example, the following steps are included: Step 210: Obtain the initial image and text description; In this embodiment, the initial image can be an image of any resolution, and the text description can be textual information describing the initial image.
[0026] Step 220: Based on the image resolution binning strategy, convert the initial image into an image of the resolution corresponding to the bin to obtain the image to be processed; In this embodiment, the image resolution binning strategy refers to dividing the image into predefined "buckets" with specific resolution ranges based on its width and height. Each bucket represents a resolution range or a specific target resolution. The buckets can be pre-set, and then the corresponding buckets can be determined based on the initial image resolution.
[0027] In some embodiments, the method of converting the initial image into an image of the corresponding bucket resolution based on the image resolution bucketing strategy to obtain the image to be processed includes: First, the bucket corresponding to the initial image is obtained by matching multiple preset image resolution buckets, which are pre-constructed using an image resolution bucketing strategy. In this embodiment, the buckets can be pre-set, specifically generated by a model. For example, first calculate the aspect ratio of all images in the dataset, then use a clustering algorithm (such as K-means) to find several main centroids of the aspect ratios. Based on these clustering results, recalculate the initial anchor box size for the model, and divide the images into corresponding buckets according to the cluster categories. Buckets corresponding to the resolution range can be matched based on the resolution of the initial image.
[0028] Then, based on the resolution of the bucket corresponding to the initial image, the initial image is converted into an image to be processed.
[0029] In this embodiment, each bucket corresponds to the resolution of the output image. After the bucket is determined, the initial image is scaled according to the resolution of the output image of the bucket (e.g., [128, 196]) as the target size to obtain the image to be processed.
[0030] By matching the initial image to a set of multiple preset image resolution buckets, the resolution of the initial image can be quickly converted to the resolution of the output image corresponding to each bucket, based on the resolution of the bucket corresponding to the initial image, so as to facilitate subsequent processing.
[0031] Step 230: Convert the image to be processed into an image token sequence using a preset image conversion model; In this embodiment, the image conversion model described above is used to convert continuous, dense pixel information into discrete, compact token sequences. It can be pre-trained, for example, the preset image conversion model can be a pre-trained image tokenizer model.
[0032] In some embodiments, the training process of the preset image conversion model includes: First, obtain the original image dataset, which includes original images of multiple sizes; Then, based on the image resolution binning strategy, each original image is converted into the resolution image of the corresponding bin, resulting in multiple original images to be processed corresponding to the bins; Finally, based on the original images to be processed corresponding to each bucket, the pre-set image tokenizer model is trained to obtain the pre-set image transformation model.
[0033] In this embodiment, to enhance the image conversion model's ability to generate images at different resolutions, an image resolution binning strategy can be introduced during training. Specifically, the original images in the original image dataset are assigned to different resolution bins based on their resolution. Each bin corresponds to a fixed resolution range. All images within this range are uniformly adjusted to the resolution corresponding to the bin during training. During training, a specified batch size of images can be selected from the image resolution bins as the training data for the current batch, and then the image tokenizer model is trained to obtain the pre-defined image conversion model. The above-described process of training the image tokenizer model is existing technology and will not be described in detail here.
[0034] The image conversion model trained using this bucketing strategy can effectively handle inputs of various resolutions without forcibly scaling all images to the same fixed resolution. This greatly preserves the original information of the images, reduces distortion caused by scaling, and makes the image conversion model more accurate.
[0035] Step 240: Use a preset text conversion model to convert the text description into a semantic vector sequence; In this embodiment, the aforementioned pre-built text conversion model is used to decompose the original text string into smaller units (called tokens) that the model can understand. These tokens can be words, subwords, characters, etc., and these tokens are mapped to identifiers in the model's vocabulary. The aforementioned pre-built text conversion model can be pre-trained and can be a text tokenizer + embedding model, where the tokenizer defines the model's basic vocabulary and segmentation rules, and the embedding is responsible for assigning mathematical meaning to each word in this vocabulary. The above conversion process is existing technology and will not be described in detail here.
[0036] Step 250: Based on the image token sequence and the semantic vector sequence, obtain the feature sequence; In this embodiment, a feature sequence is obtained by combining the image token sequence and the semantic vector sequence.
[0037] Step 260: Encode the relative positions of the image tokens in the feature sequence to obtain the input feature sequence; In this embodiment, the feature sequence contains multiple image tokens and multiple semantic vectors. The input feature sequence is obtained by positionally encoding the image tokens. Positional encoding injects prior knowledge of the original spatial location of each image token, which facilitates subsequent image processing.
[0038] In some embodiments, the relative position encoding described above can be rotation position encoding (RoPE), which embeds position information by mapping each position in the input sequence to a vector in the latent space and applying a rotation transformation to these vectors. The process of performing relative position encoding on the image tokens in the feature sequence to obtain the input feature sequence includes: First, the pixel coordinates of each image token in the feature sequence are converted into a high-dimensional space vector; Then, the high-dimensional space vector is rotated based on the position rotation matrix to achieve relative position encoding, thereby obtaining the input feature sequence.
[0039] In this embodiment, positional encoding converts the coordinates of each pixel in the image into a vector in a high-dimensional space. Then, these vectors are rotated using a position-based rotation matrix to generate a relative positional encoding.
[0040] In multi-scale image generation tasks, RoPE ensures that the relative positional relationships between pixels are preserved at different resolutions. By embedding positional codes of low-resolution images into high-resolution images, the model can better understand and process image data of different resolutions.
[0041] Step 270: Input the input feature sequence into a preset image generation model to obtain a new image.
[0042] In this embodiment, the aforementioned preset image generation model can be pre-trained and can be an autoregressive image generation model, such as the LlamaGen model, which achieves high-quality image generation by adopting the next-token prediction paradigm of a large-scale language model, combined with vector quantization (VQ-VAE) and an autoregressive model.
[0043] Please refer to Figure 3 , Figure 3 The illustration schematically depicts a vector-quantization-free autoregressive image generation process according to an embodiment of this application. In some embodiments, the preset image generation model includes: a masked autoregressive model and an MLP diffusion denoising model; correspondingly, the step of inputting the input feature sequence into the preset image generation model to obtain a new image includes: First, the input feature sequence is randomly masked to obtain a masked sequence; In this embodiment, a portion of the positions can be randomly selected for masking, and the masking ratio (e.g., 50%) determines the number of tokens predicted. Masking positions can be set to 0 or special values to generate mask sequences and attention masks.
[0044] Then, the masked sequence is input into the masked autoregressive model to obtain a new image token vector; In this embodiment, the mask autoregressive model can be pre-trained. The input of the model is a text sequence plus a masked image sequence. The mask autoregressive model predicts and generates the image token corresponding to the mask.
[0045] In some embodiments, the masked autoregressive model is used to predict the token vector at the masked location using a bidirectional attention mechanism.
[0046] In this embodiment, the bidirectional attention mechanism allows the model to simultaneously attend to elements at all positions in the sequence, including those before and after it, when processing an element (e.g., a word), thereby obtaining the most comprehensive contextual information. The masked sequence is input into the MAR model, and the Transformer's bidirectional attention mechanism computes the context of all masked positions in parallel, outputting a new image token vector.
[0047] The introduction of a bidirectional attention mechanism enables the model to utilize contextual information simultaneously during generation. This not only improves generation quality but also makes non-sequential generation possible, further optimizing the overall generation process. Specifically, when predicting image tokens, a non-sequential generation approach can be adopted, predicting all tokens at masked locations at once instead of generating them sequentially. This non-sequential generation method not only improves generation efficiency but also reduces potential cumulative errors during the generation process, ensuring the accuracy and consistency of the generated results. This mechanism enhances the coherence of image generation, especially when dealing with long sequences or complex structures, significantly improving the quality of the generated images.
[0048] In some embodiments, the training process of the masked autoregressive model includes: The first step is to obtain multiple input sequences, each with a different length; In this embodiment, the input sequence can be the output of the image conversion model used as the input sequence during training. Since the image conversion model is trained by acquiring images from different buckets, the output image token sequences have different lengths.
[0049] The second step is to generate a mask for each input sequence based on the length of each input sequence and the random mask ratio. In this embodiment, the corresponding mask can be generated based on the length of the token sequence generated after the image passes through the tokenizer and the ratio of the random mask, thereby realizing dynamic adjustment of the masking strategy.
[0050] The third step is to train the preset autoregressive prediction model based on each input sequence and the corresponding mask to obtain the masked autoregressive model.
[0051] In this embodiment, during training, each sample input into the model for forward and backward propagation is adjusted to the corresponding resolution according to its respective resolution bucket. Due to the change in resolution, the length of the input sequence is different between each sample, allowing the trained masked autoregressive model to dynamically adapt to changes in input. The impact of sequence length at different resolutions on the model is incorporated into the training, improving the model's flexibility. The training of the masked autoregressive model is prior art and will not be described in detail here.
[0052] By dynamically adjusting the masking strategy to adapt to input sequences of different lengths, the model learns how to effectively perform autoregressive predictions at different input lengths for each resolution bucket, while simultaneously processing pixel values at multiple mask locations. This process ensures that the model can generate consistent and high-quality images at different resolutions during the inference phase.
[0053] Finally, the new image token vector is used as the diffusion denoising condition, and the MLP diffusion denoising model is used for denoising to obtain a new image.
[0054] In this embodiment, the process of generating an image using the MLP diffusion denoising model involves first initializing Gaussian noise. Specifically, this can be achieved by using a token vector generated by MAR as a conditional input, iteratively denoising to generate image features, and then reconstructing the image using a VAE decoder. Specifically, a Markov chain can be used to model the noise distribution. During the forward process, noise is gradually added to convert the image data into Gaussian noise; during the reverse process, the model gradually removes noise to generate high-probability pixel values for the image.
[0055] Please refer to Figure 4 , Figure 4 The diagram illustrates an MLP diffusion denoising process according to an embodiment of this application. Each step of the MLP diffusion denoising model uses a specially designed neural network (typically a U-Net architecture, but for speed, a 3-layer MLP network with fewer parameters) to perform denoising operations. During training, the network parameters are optimized by minimizing the diffusion loss function to approximate the distribution of the real image. In some embodiments, a diffusion loss function can be introduced to replace the traditional cross-entropy loss. This loss function can more accurately reflect the complex details and structural information in the image. During training, the loss function measures the difference between the image generated by the model and the original image, while considering the probability distribution of each pixel to ensure the quality and realism of the generated image.
[0056] By introducing a probabilistic modeling technique based on diffusion processes, image data in continuous space is directly processed, thereby eliminating the errors introduced by vector quantization. This not only significantly improves the detail fidelity and clarity of the generated images but also simplifies the overall structure of the model and enhances its computational efficiency.
[0057] In the above implementation process, an initial image and text description are acquired; based on an image resolution binning strategy, the initial image is converted into an image of the corresponding bin resolution to obtain the image to be processed; a preset image conversion model is used to convert the image to be processed into an image token sequence; a preset text conversion model is used to convert the text description into a semantic vector sequence; based on the image token sequence and the semantic vector sequence, a feature sequence is obtained; relative position encoding is performed on the image tokens in the feature sequence to obtain an input feature sequence; the input feature sequence is input to a preset image generation model to obtain a new image. By introducing an image resolution binning strategy, the input initial image is grouped according to resolution range and adjusted to the corresponding bin, avoiding information loss and distortion caused by forced uniform scaling, enhancing the model's ability to generate multi-resolution images. By introducing position encoding into autoregressive image generation, by embedding the relative position information of pixels, combined with the binning strategy, the consistency of pixel relationships under different resolutions is ensured, thereby ensuring the detail consistency of high-resolution images and improving the accuracy of image generation.
[0058] Figure 2 This is a flowchart illustrating an image generation method in one embodiment. It should be understood that, although... Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0059] Please refer to Figure 5 , Figure 5 A schematic block diagram of an image generation apparatus according to an embodiment of this application is shown. This embodiment provides an image generation apparatus, which includes an acquisition module 410, a binning module 420, a first conversion module 430, a second conversion module 440, a combination module 450, an encoding module 460, and a generation module 470, wherein: Module 410 is used to acquire the initial image and text description; Bucketing module 420 is used to convert the initial image into a corresponding bucket resolution image based on an image resolution bucketing strategy to obtain the image to be processed; The first conversion module 430 is used to convert the image to be processed into an image token sequence using a preset image conversion model; The second conversion module 440 is used to convert the text description into a semantic vector sequence using a preset text conversion model; The combination module 450 is used to obtain a feature sequence based on the image token sequence and the semantic vector sequence; Encoding module 460 is used to encode the relative positions of the image tokens in the feature sequence to obtain the input feature sequence; The generation module 470 is used to input the input feature sequence into a preset image generation model to obtain a new image.
[0060] The image generation device includes a processor and a memory. The acquisition module 410, the binning module 420, the first conversion module 430, the second conversion module 440, the combination module 450, the encoding module 460, and the generation module 470 are all stored in the memory as program units. The processor executes the program modules stored in the memory to implement the corresponding functions.
[0061] The bucketing module 420 includes: A matching unit is used to match the bucket corresponding to the initial image among a plurality of preset image resolution buckets, wherein the plurality of preset image resolution buckets are pre-constructed using an image resolution bucketing strategy; The conversion unit is used to convert the initial image into an image to be processed based on the resolution of the bucket corresponding to the initial image.
[0062] The encoding module 460 includes: A coordinate transformation unit is used to convert the pixel coordinates of each image token in the feature sequence into a high-dimensional space vector. A rotation unit is used to rotate the high-dimensional space vector based on a position rotation matrix to achieve relative position encoding and obtain the input feature sequence.
[0063] The preset image generation model includes: a mask autoregressive model and an MLP diffusion denoising model; the generation module 470 includes: A masking unit is used to randomly mask the input feature sequence to obtain a masked sequence; The prediction unit is used to input the masked sequence into the masked autoregressive model to obtain a new image token vector; The denoising unit is used to take the new image token vector as the diffusion denoising condition and perform denoising using the MLP diffusion denoising model to obtain a new image.
[0064] The mask autoregressive model is used to predict the token vector at the masked position using a bidirectional attention mechanism.
[0065] This also includes: The sequence acquisition module is used to acquire multiple input sequences, each with a different length. A mask generation module is used to generate masks for each input sequence based on the length of each input sequence and the random mask ratio, respectively. The first model training module is used to train a preset autoregressive prediction model based on each input sequence and the corresponding mask, respectively, to obtain a masked autoregressive model.
[0066] This also includes: The original image acquisition module is used to acquire the original image dataset, which includes original images of multiple sizes; The bucketing module is used to convert each original image into a resolution image corresponding to the bucket based on the image resolution bucketing strategy, so as to obtain multiple original images to be processed corresponding to the buckets. The second model training module is used to train the preset image tokenizer model based on the original images to be processed corresponding to each bucket, so as to obtain the preset image conversion model.
[0067] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the image generation method can be implemented by adjusting the kernel parameters.
[0068] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0069] This application provides a machine-readable storage medium storing a program that, when executed by a processor, implements the above-described image generation method.
[0070] This application provides a processor for running a program, wherein the program executes the image generation method described above.
[0071] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown in the figure, the computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A06. The network interface A02 is used for communication with external terminals via a network connection. When the computer program is executed by the processor A01, it implements an image generation method. The display screen A04 can be a liquid crystal display (LCD) or an e-ink display. The input device A05 can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.
[0072] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0073] In one embodiment, the image generation apparatus provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 6 It runs on the computer device shown. The computer device's memory can store the various program modules that make up the image generation apparatus, for example, Figure 5 The diagram shows an acquisition module 410, a binning module 420, a first conversion module 430, a second conversion module 440, a combination module 450, an encoding module 460, and a generation module 470. The computer program comprised of these modules causes the processor to execute the steps of the image generation methods in the various embodiments of this application described in this specification.
[0074] Figure 6 The computer device shown can be used as follows Figure 5The image generation apparatus shown executes step 210 via acquisition module 410. The computer device can execute step 220 via binning module 420. The computer device can execute step 230 via first conversion module 430. The computer device can execute step 240 via second conversion module 440. The computer device can execute step 250 via combination module 450. The computer device can execute step 260 via encoding module 460. The computer device can execute step 270 via generation module 470.
[0075] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes the following method steps: Obtain the initial image and text description; Based on the image resolution binning strategy, the initial image is converted into the resolution image of the corresponding bin to obtain the image to be processed; The image to be processed is converted into an image token sequence using a preset image conversion model; The text description is converted into a semantic vector sequence using a pre-defined text conversion model; Based on the image token sequence and the semantic vector sequence, a feature sequence is obtained; The image tokens in the feature sequence are encoded by relative position to obtain the input feature sequence; The input feature sequence is input into a preset image generation model to obtain a new image.
[0076] In one embodiment, the step of converting the initial image into an image of the corresponding bucket resolution based on the image resolution bucketing strategy to obtain the image to be processed includes: The bucket corresponding to the initial image is obtained by matching among a number of preset image resolution buckets, which are pre-constructed using an image resolution bucketing strategy; Based on the resolution of the bucket corresponding to the initial image, the initial image is converted into an image to be processed.
[0077] In one embodiment, the step of encoding the relative positions of the image tokens in the feature sequence to obtain the input feature sequence includes: Convert the pixel coordinates of each image token in the feature sequence into a high-dimensional space vector; The high-dimensional space vector is rotated using a position-based rotation matrix to achieve relative position encoding, thereby obtaining the input feature sequence.
[0078] In one embodiment, the preset image generation model includes: a mask autoregressive model and an MLP diffusion denoising model; The step of inputting the input feature sequence into a preset image generation model to obtain a new image includes: The input feature sequence is randomly masked to obtain a masked sequence; The masked sequence is input into the masked autoregressive model to obtain a new image token vector; Using the new image token vector as a diffusion denoising condition, the MLP diffusion denoising model is used for denoising to obtain a new image.
[0079] In one embodiment, the masked autoregressive model is used to predict the token vector at the masked location using a bidirectional attention mechanism.
[0080] In one embodiment, the training process of the masked autoregressive model includes: Obtain multiple input sequences, each with a different length; A mask for each input sequence is generated based on the length of each input sequence and the random mask ratio, respectively. Based on each input sequence and its corresponding mask, a pre-set autoregressive prediction model is trained to obtain a masked autoregressive model.
[0081] In one embodiment, the training process of the preset image conversion model includes: Obtain the original image dataset, which includes original images of multiple sizes; Based on the image resolution binning strategy, each original image is converted into the resolution image of the corresponding bin, resulting in multiple original images to be processed corresponding to the bins. Based on the original images to be processed corresponding to each bucket, a pre-set image tokenizer model is trained to obtain a pre-set image transformation model. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0082] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0083] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0084] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0085] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0086] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0087] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0088] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0089] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. An image generation method, characterized in that, The image generation method includes: Obtain the initial image and text description; Based on the image resolution binning strategy, the initial image is converted into the resolution image of the corresponding bin to obtain the image to be processed; The image to be processed is converted into an image token sequence using a preset image conversion model; The text description is converted into a semantic vector sequence using a pre-defined text conversion model; Based on the image token sequence and the semantic vector sequence, a feature sequence is obtained; The image tokens in the feature sequence are encoded by relative position to obtain the input feature sequence; The input feature sequence is input into a preset image generation model to obtain a new image.
2. The method according to claim 1, characterized in that, The image resolution-based bucketing strategy converts the initial image into an image of the corresponding bucket resolution to obtain the image to be processed, including: The bucket corresponding to the initial image is obtained by matching among a number of preset image resolution buckets, which are pre-constructed using an image resolution bucketing strategy; Based on the resolution of the bucket corresponding to the initial image, the initial image is converted into an image to be processed.
3. The method according to claim 1, characterized in that, The step of encoding the relative positions of the image tokens in the feature sequence to obtain the input feature sequence includes: Convert the pixel coordinates of each image token in the feature sequence into a high-dimensional space vector; The high-dimensional space vector is rotated using a position-based rotation matrix to achieve relative position encoding, thereby obtaining the input feature sequence.
4. The method according to claim 1, characterized in that, The preset image generation model includes: a mask autoregressive model and an MLP diffusion denoising model; The step of inputting the input feature sequence into a preset image generation model to obtain a new image includes: The input feature sequence is randomly masked to obtain a masked sequence; The masked sequence is input into the masked autoregressive model to obtain a new image token vector; Using the new image token vector as a diffusion denoising condition, the MLP diffusion denoising model is used for denoising to obtain a new image.
5. The method according to claim 4, characterized in that, The masked autoregressive model is used to predict the token vector at the masked position using a bidirectional attention mechanism.
6. The method according to claim 4, characterized in that, The training process of the masked autoregressive model includes: Obtain multiple input sequences, each with a different length; A mask for each input sequence is generated based on the length of each input sequence and the random mask ratio, respectively. Based on each input sequence and its corresponding mask, a pre-set autoregressive prediction model is trained to obtain a masked autoregressive model.
7. The method according to claim 1, characterized in that, The training process of the preset image conversion model includes: Obtain the original image dataset, which includes original images of multiple sizes; Based on the image resolution binning strategy, each original image is converted into the resolution image of the corresponding bin, resulting in multiple original images to be processed corresponding to the bins. Based on the original images to be processed corresponding to each bucket, a pre-set image tokenizer model is trained to obtain a pre-set image transformation model.
8. An image generation apparatus, characterized in that, The image generation device includes: The acquisition module is used to acquire the initial image and text description; The bucketing module is used to convert the initial image into the resolution image of the corresponding bucket based on the image resolution bucketing strategy, so as to obtain the image to be processed; The first conversion module is used to convert the image to be processed into an image token sequence using a preset image conversion model; The second conversion module is used to convert the text description into a semantic vector sequence using a preset text conversion model; The combination module is used to obtain a feature sequence based on the image token sequence and the semantic vector sequence; The encoding module is used to encode the relative positions of the image tokens in the feature sequence to obtain the input feature sequence; The generation module is used to input the input feature sequence into a preset image generation model to obtain a new image.
9. A processor, characterized in that, It is configured to perform the image generation method according to any one of claims 1 to 7.
10. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the image generation method according to any one of claims 1 to 7.
11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image generation method according to any one of claims 1 to 7.