An image generation method, apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure disclose an image generation method and device, electronic equipment and storage medium, the method comprising: obtaining a first prompt word; generating a first image according to the first prompt word through a self-correcting image generation model; wherein the self-correcting image generation model is constructed based on a self-correcting image feature sequence; wherein the self-correcting image feature sequence is determined based on an image tokenizer; the image tokenizer is constructed based on synchronous learning of a first task and a second task; the first task includes a task of reconstructing an image based on a correct image feature sequence, and the second task includes a task of reconstructing an image based on a noise image feature sequence. The self-correcting image feature sequence can be determined based on the image tokenizer, so that the self-correcting image generation model constructed based on the self-correcting image feature sequence has the ability to correct in the process of predicting the image feature sequence, thereby ensuring the image generation effect.

Description

Technical Field

[0001] This disclosure relates to the field of machine learning technology, and in particular to an image generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In related technologies, one-dimensional image segmenters can embed image information into a one-dimensional token sequence, which is then quantized into a discrete token sequence. This discrete token sequence can be used to construct an autoregressive image generation model, enabling the model to predict the current token based on preceding tokens, thus efficiently completing the image generation task based on the predicted token sequence.

[0003] However, the currently constructed one-dimensional image segmenter can only provide a correct discrete token sequence. Based on this, the autoregressive image generation model can only predict the current token based on the correct token. However, in actual inference, there are cases where the autoregressive image generation model predicts an incorrect token. In such cases, the model cannot guarantee the correctness of subsequent token predictions, thus compromising the image generation quality. Summary of the Invention

[0004] This disclosure provides an image generation method, apparatus, electronic device, and storage medium that can determine a self-correcting image feature sequence based on an image segmenter, so that the autoregressive image generation model constructed based on the self-correcting image feature sequence has the ability to correct errors in the process of predicting the image feature sequence, thereby ensuring the image generation effect.

[0005] In a first aspect, embodiments of this disclosure provide an image generation method, including:

[0006] Get the first prompt word;

[0007] A first image is generated based on the first prompt word using an autoregressive image generation model;

[0008] The autoregressive image generation model is constructed based on a self-correcting image feature sequence; the self-correcting image feature sequence is determined based on an image word segmenter; the image word segmenter is constructed based on simultaneous learning of a first task and a second task; the first task includes the task of reconstructing an image based on a correct image feature sequence, and the second task includes the task of reconstructing an image based on a noisy image feature sequence.

[0009] Secondly, embodiments of this disclosure also provide an image generation apparatus, comprising:

[0010] The acquisition module is used to acquire the first prompt word;

[0011] The generation module is used to generate a first image based on the first prompt word using an autoregressive image generation model;

[0012] The autoregressive image generation model is constructed based on a self-correcting image feature sequence; the self-correcting image feature sequence is determined based on an image word segmenter; the image word segmenter is constructed based on simultaneous learning of a first task and a second task; the first task includes the task of reconstructing an image based on a correct image feature sequence, and the second task includes the task of reconstructing an image based on a noisy image feature sequence.

[0013] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0014] One or more processors;

[0015] Storage device for storing one or more programs.

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the image generation method as described in any of the embodiments of this disclosure.

[0017] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the image generation method as described in any of the embodiments of this disclosure.

[0018] Fifthly, this disclosure also provides a computer program product, characterized in that the computer program product includes a computer program, which, when executed by a processor, implements the image generation method as described in any of the embodiments of this disclosure.

[0019] In the technical solution of this embodiment, a first prompt word can be obtained; a first image is generated based on the first prompt word through an autoregressive image generation model; wherein, the autoregressive image generation model is constructed based on a self-correcting image feature sequence; wherein, the self-correcting image feature sequence is determined based on an image segmenter; the image segmenter is constructed based on synchronous learning of a first task and a second task; the first task includes the task of reconstructing an image based on a correct image feature sequence, and the second task includes the task of reconstructing an image based on a noisy image feature sequence.

[0020] In the process of building an image word segmenter, by learning the first task, the image word segmenter can acquire the ability to embed image information into a correct one-dimensional image feature sequence, and the ability to reconstruct images based on the correct image feature sequence. Simultaneously, by learning the second task, the image word segmenter can acquire the ability to output subsequent data with error correction capabilities even when noisy data appears in the feature sequence embedded in the image information; that is, it can output a self-correcting image feature sequence, thereby enabling image reconstruction based on noisy image feature sequences.

[0021] The constructed image segmenter can generate a self-correcting image feature sequence. By constructing an autoregressive image generation model based on the self-correcting image feature sequence, the autoregressive image generation model can predict and correct data after predicting erroneous data during the prediction of image feature sequence, thereby ensuring the image generation effect. Attached Figure Description

[0022] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0023] Figure 1 This is a schematic flowchart of an image generation method provided in an embodiment of the present disclosure;

[0024] Figure 2 This is a flowchart illustrating the process of determining the self-correcting image feature sequence in an image generation method provided in this embodiment of the present disclosure.

[0025] Figure 3 A data flow diagram illustrating the process of determining a self-correcting image feature sequence in an image generation method provided in this embodiment of the present disclosure;

[0026] Figure 4 This is a flowchart illustrating the process of determining a noise sequence segment in an image generation method provided in this embodiment of the present disclosure.

[0027] Figure 5 This is a schematic block diagram of the data flow in the image segmenter construction process of an image generation method provided in this embodiment of the disclosure;

[0028] Figure 6 This is a schematic diagram of the structure of an image generation apparatus provided in an embodiment of the present disclosure;

[0029] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0030] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0031] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0032] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0033] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0034] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0035] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0036] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0037] Before describing the image generation method provided in this disclosure, the image segmenter involved in the embodiments of this disclosure will be introduced. The image segmenter involved in the embodiments of this disclosure may include an encoder, a quantizer, and a decoder.

[0038] The encoder can include an image encoder based on a Vision Transformer (ViT). The image can be processed into an image token, which can be concatenated with a one-dimensional query token and input into the encoder for encoding. Within the encoder, information can be integrated between tokens to embed image information into a one-dimensional, continuous, implicit token sequence.

[0039] The quantizer can include a codebook, which may contain several vectors. A continuous implicit token sequence corresponding to a query token can be input into the quantizer. The quantizer calculates the distance between the input continuous features and each vector in the codebook, and the closest vector is used as the quantized feature. After processing by the quantizer, the continuous token sequence can be quantized into a discrete token sequence.

[0040] The decoder can include a ViT-based image decoder. The decoder can be used to decode the quantized discrete implicit token sequence back into an image.

[0041] Figure 1 This is a schematic flowchart illustrating an image generation method provided in an embodiment of this disclosure. This disclosure applies to situations where images are generated using an autoregressive image generation model. The method can be executed by an image generation apparatus, which can be implemented in software and / or hardware and can be configured in an electronic device, such as a computer device.

[0042] like Figure 1 As shown, the image generation method provided in this embodiment may include:

[0043] S110, Obtain the first prompt word;

[0044] S120. Generate a first image based on the first prompt word using an autoregressive image generation model;

[0045] The autoregressive image generation model is constructed based on a self-correcting image feature sequence; the self-correcting image feature sequence is determined based on an image segmenter; the image segmenter is constructed based on simultaneous learning of the first and second tasks; the first task includes the task of reconstructing an image based on a correct image feature sequence, and the second task includes the task of reconstructing an image based on a noisy image feature sequence.

[0046] The image segmenter, through learning the first task, acquires the ability to embed image information into a correct one-dimensional image feature sequence and to reconstruct an image based on that correct feature sequence. Simultaneously, through learning the second task, it acquires the ability to output subsequent data with error correction capabilities even when noisy data appears in the embedded feature sequence, thus enabling it to output a self-correcting image feature sequence and ultimately reconstruct an image from the noisy image feature sequence.

[0047] The constructed image segmenter generates a self-correcting image feature sequence for the second image. Correspondingly, an autoregressive image generation model can be constructed based on the second image and the self-correcting image feature sequence. For example, the self-correcting image feature sequence can be used as a supervision signal to construct the token sequence prediction module in the autoregressive image generation model, enabling the prediction module to predict the next token with correction information even when an incorrect token is predicted. The second image can be used as a supervision signal to construct the image generation module in the autoregressive image generation model, allowing the image generation module to reconstruct the second image based on the self-correcting image feature sequence. Thus, the autoregressive image generation model, during the prediction of image feature sequences, possesses the ability to predict correction data after predicting incorrect data.

[0048] In this embodiment of the disclosure, the first prompt word may include descriptive text of the first image to be generated. For example, the first prompt word could be "Please generate an image of a cat," etc. The first prompt word can be received by the user through a user interface provided by the image generation device. The first prompt word can be input into the constructed autoregressive image generation model, so that the prediction module in the autoregressive image generation model sequentially predicts the token sequence used to generate the first image; and the image generation module in the autoregressive image generation model generates the first image based on the predicted token sequence. Since the autoregressive image generation model has the ability to predict and correct error data after predicting incorrect data during the prediction of the image feature sequence, the generation effect of the first image can be guaranteed.

[0049] In the technical solution of this embodiment, a first prompt word can be obtained; a first image is generated based on the first prompt word through an autoregressive image generation model; wherein, the autoregressive image generation model is constructed based on a self-correcting image feature sequence; wherein, the self-correcting image feature sequence is determined based on an image segmenter; the image segmenter is constructed based on synchronous learning of a first task and a second task; the first task includes the task of reconstructing an image based on a correct image feature sequence, and the second task includes the task of reconstructing an image based on a noisy image feature sequence.

[0050] The constructed image segmenter can generate a self-correcting image feature sequence. By constructing an autoregressive image generation model based on the self-correcting image feature sequence, the autoregressive image generation model can predict and correct data after predicting erroneous data during the prediction of image feature sequence, thereby ensuring the image generation effect.

[0051] This embodiment can be combined with various optional schemes in the image generation method provided in the above embodiments. The image generation method provided in this embodiment describes in detail the process of determining the self-correcting image feature sequence.

[0052] For example, Figure 2 This is a flowchart illustrating the process of determining a self-correcting image feature sequence in an image generation method provided in this embodiment of the disclosure. See also... Figure 2 In the image generation method provided in this embodiment, the image segmenter includes an encoder and a quantizer; the process of determining the self-correcting image feature sequence may include:

[0053] S210. Obtain a second image and a noise sequence segment of the image features of the second image; wherein the noise sequence segment is obtained by random quantization of a portion of the first encoded sequence of the second image.

[0054] For example, Figure 3 This is a data flow diagram illustrating the process of determining a self-correcting image feature sequence in an image generation method provided in this embodiment of the present disclosure. See also... Figure 3 The process of obtaining the self-correcting image feature sequence may include:

[0055] A second image is acquired and divided into multiple image blocks to obtain a second image token. The second image token is then concatenated with a query token and input into an encoder. The dimensions of both the second image token and the query token must meet the encoder's input requirements, and the sequence length of the query token in this disclosure must meet a preset length (e.g., 32 tokens). The encoder encodes the second image token and the query token, and retains the encoding result corresponding to the query token, thus obtaining the first encoded sequence of the second image. This first encoded sequence can be considered a one-dimensional, continuous, implicit token sequence of the second image.

[0056] like Figure 3 As shown, a partial sequence can be selected from the first encoded sequence. Furthermore, at least a portion of the data in this partial sequence can be randomly quantized using a quantizer to obtain a noisy sequence segment containing erroneously quantized data. Here, random quantization can be considered as an operation based on a random vector. The noisy sequence segment can be considered as a discrete, noisy token sequence.

[0057] S220. The second image and noise sequence segment are encoded by the encoder to obtain the second encoded sequence.

[0058] The noise sequence segment can be mapped to the dimension required by the encoder to obtain the first data sequence. A second data sequence can be randomly obtained and concatenated with the first and second data sequences to form a new query token. The length of the new query token must also meet a preset length, meaning the total sequence length of the first and second data sequences must satisfy the preset length.

[0059] See Figure 3 The second image token and a new query token composed of the first and second data sequences can be input into the encoder to encode the input data sequence, obtaining a second encoded sequence. The second encoded sequence corresponds to the second data sequence; that is, the second encoded sequence is the encoded result of the second data sequence. The second encoded sequence can be considered as a continuous implicit token sequence.

[0060] S230. The second encoded sequence is matched and quantized by a quantizer to obtain the error-corrected sequence segment corresponding to the noise sequence segment.

[0061] Matching quantization can be considered as a quantization operation based on the matching vector. Since image word segmenters have the ability to output self-correcting image feature sequences, see also... Figure 2 The second encoded sequence can be input into a quantizer. The quantizer calculates the distance between the second encoded sequence and each vector in the codebook, determines the matching vector based on the distance, and uses the matching vector to quantize the second encoded sequence to obtain the error-corrected sequence segment. This error-corrected sequence segment is also a feature of the second image and may contain error-correction information from the noisy sequence segment. The error-corrected sequence segment can be considered a discrete token sequence containing error-correction information.

[0062] S240. Determine the self-correcting image feature sequence based on the noise sequence segment and the error correction sequence segment.

[0063] like Figure 3 As shown, a noisy sequence segment containing noise and an error-correcting sequence segment with error-correcting information can be concatenated to obtain a self-correcting image feature sequence. The self-correcting image feature sequence can be considered as a discrete token sequence with self-correcting capabilities related to the second image.

[0064] In the technical solution of this disclosure, the process of determining the self-correcting image feature sequence is described in detail. By partially selecting and randomly quantizing the first encoded sequence after encoding the second image, a noise sequence segment containing noise data can be obtained. Through the encoder and quantizer in the constructed image segmenter, an error-correcting sequence segment can be generated to correct the noise sequence segment, which can then be concatenated to form a self-correcting image feature sequence. Furthermore, the image generation method provided in this disclosure belongs to the same concept as the image generation method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments. Various optional solutions in the image generation methods provided in this disclosure and the above embodiments can be combined. The image generation method provided in this embodiment describes the process of determining the noise sequence segment in detail.

[0065] For example, Figure 4 This is a flowchart illustrating the process of determining a noise sequence segment in an image generation method provided by an embodiment of this disclosure. Figure 4 As shown, the process of determining the noise sequence segment in the image generation method provided in this embodiment may include:

[0066] S410. The second image is encoded by the encoder to obtain the first encoded sequence.

[0067] Among them, see Figure 3 The first encoding sequence process shown involves encoding the second image token and the query token using an encoder to obtain the first encoding sequence.

[0068] S420. Obtain a random number of unit data starting from the first unit data in the first encoded sequence to obtain a partial sequence.

[0069] In this embodiment of the disclosure, the number of tokens predicted in a single iteration by the autoregressive image generation model can be considered as a unit, and each unit can contain at least one piece of data. The first encoded sequence can be divided into multiple units of data, and the data contained in each unit can be referred to as unit data. Figure 3 In the first encoded sequence, each unit of data can contain two tokens.

[0070] In this embodiment, m units of data can be randomly selected starting from the first unit data (i.e. the first unit data) in the first encoding sequence to obtain a partial sequence in the first encoding sequence; where m can include a positive integer greater than or equal to 1.

[0071] S430. Using a quantizer, the last unit data in a portion of the sequence is randomly quantized, and other unit data is matched to obtain a noise sequence segment; wherein, other unit data includes unit data in the portion of the sequence excluding the last unit data.

[0072] In this embodiment, other unit data includes unit data in a partial sequence of the first coding sequence, excluding the last unit data. By obtaining the first m unit data in the first coding sequence to obtain a partial sequence, and by randomly quantizing the last unit data (i.e., the last unit data) in the partial sequence and matching quantizing the other unit data, a noise sequence segment can be obtained. This noise sequence segment can be used to simulate the token sequence when the autoregressive image generation model predicts incorrect data for the first time during the sequential prediction of tokens.

[0073] In some optional implementations, random quantization of the last unit data in a portion of the sequence can be performed using a quantizer. This can include: obtaining a preset number of first vectors that match the last unit data from the codebook corresponding to the quantizer; and randomly determining a second vector from the first vectors to perform random quantization of the last unit data.

[0074] In these optional implementations, a quantizer can calculate the distance between the final unit data in a portion of the first encoded sequence and each vector in the codebook, and select a preset number of vectors from the closest to the furthest distances as the first preset number of vectors to match the final unit data. The preset number can be pre-set based on empirical or experimental values, for example, 50. A second vector can be randomly selected from the first vectors to introduce noise into the quantization result of the final unit data.

[0075] For example, Figure 3 A portion of the first encoded sequence may include two unit data; wherein, the first unit data ( Figure 3 (represented by c1 and c2) are used for matching and quantization to obtain the correct token. Figure 3 (represented by Z1 and Z2); the second unit data ( Figure 3 Random matching is performed on tokens (represented by c3 and c4) to obtain tokens containing noise. Figure 3 Chinese and express).

[0076] In this embodiment, the correct token obtained by quantizing a portion of the first encoded sequence and the noise token can be concatenated to obtain a noise sequence segment. For example... Figure 3 As shown, the fully connected layer can be used as a mapping layer to map the noise sequence segments to the dimension (i.e., input size) required by the encoder, so as to obtain the first data sequence.

[0077] The technical solution of this disclosure describes in detail the process of determining the noise sequence segment. By selecting a random number of unit data from the correct encoded sequence (i.e., the first encoded sequence) of the first image, a partial sequence in the first encoded sequence can be obtained. By randomly quantizing the last unit data in the partial sequence, noise data can be introduced to simulate the situation where the autoregressive image generation model predicts incorrect data, thus obtaining the noise sequence segment.

[0078] Furthermore, the image generation method provided in this embodiment belongs to the same concept as the image generation method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.

[0079] This disclosure embodiment can be combined with various optional schemes in the image generation method provided in the above embodiments. The image generation method provided in this embodiment details the construction process of the image word segmenter.

[0080] In the image generation method provided in this embodiment, the image segmenter includes an encoder, a quantizer, and a decoder; the process of constructing the image segmenter based on the first task may include:

[0081] The process involves: acquiring a third image; encoding the third image using an encoder to obtain a third encoded sequence; performing matching quantization on the third encoded sequence using a quantizer to obtain a correct image feature sequence; decoding the correct image feature sequence using a decoder to obtain a fourth image; constructing a first loss value based on the third and fourth images; and constructing an image segmenter based on the first loss value.

[0082] For example, Figure 5 This is a schematic block diagram of the data flow in the image segmenter construction process of an image generation method provided in this embodiment of the disclosure. Figure 5 As shown, the process of constructing an image word segmenter based on the first task may include:

[0083] The process involves: acquiring a third image and dividing it into multiple image blocks to obtain a third image token; concatenating the third image token with a query token and inputting the concatenation into an encoder; encoding both the third image token and the query token using the encoder, and retaining the encoding result corresponding to the query token to obtain the third encoded sequence; performing match quantization on the third encoded sequence using a quantizer to obtain the correct image feature sequence; and decoding the correct image feature sequence back to the fourth image using a decoder. Figure 5 (Represented by the fourth image token in the text).

[0084] Specifically, the first loss value between the third and fourth images can be calculated using an existing inter-image loss calculation function. For example... Figure 5 In this process, a first loss value can be calculated between the third image token and the fourth image token. This first loss value can then be backpropagated to construct the image segmenter. Due to the non-differentiability of quantization operations, some implementations utilize existing techniques (such as pass-through estimation) to allow gradients to travel directly from the decoder to the encoder during backpropagation, thereby enabling parameter tuning of the decoder and encoder in the image segmenter.

[0085] The first task learning can be achieved by embedding image information from a third image into a one-dimensional discrete sequence of correct image features, reconstructing the third image based on the correct image feature sequence, and determining a first loss value based on the reconstructed fourth image. By learning the first task, the image segmenter can acquire the ability to embed image information into a one-dimensional sequence of correct image features and the ability to reconstruct images based on the correct image feature sequence.

[0086] In the image generation method provided in this embodiment, the process of constructing an image segmenter based on the second task may include:

[0087] A sample noise sequence segment is obtained; wherein the sample noise sequence segment is obtained by random quantization of a portion of the third encoded sequence; the third image and the sample noise sequence segment are encoded by an encoder to obtain a fourth encoded sequence; the fourth encoded sequence is matched and quantized by a quantizer to obtain a sample error correction sequence segment corresponding to the sample noise sequence segment; the noisy image feature sequence is determined based on the sample noise sequence segment and the sample error correction sequence segment; the noisy image feature sequence is decoded by a decoder to obtain a fifth image; a second loss value is constructed based on the third image and the fifth image, and an image segmenter is constructed based on the second loss value.

[0088] In this embodiment, a portion of the third coding sequence can be obtained by referring to the method of obtaining a portion of the first coding sequence; and a sample noise sequence segment can be determined based on a portion of the third coding sequence by referring to the method of determining the noise sequence segment.

[0089] For example, such as Figure 5 As shown, the process of determining the sample noise sequence segment may include: obtaining a random number of unit data starting from the first unit data of the third coding sequence to obtain a partial sequence in the third coding sequence; randomly quantizing the last unit data in the partial sequence of the third coding sequence through a quantizer, and matching quantizing the other unit data to obtain the sample noise sequence segment.

[0090] See you again Figure 5The sample noise sequence segment can be mapped to the dimension required by the encoder to obtain the third data sequence. A fourth data sequence can be randomly obtained to concatenate the third and fourth data sequences into a query token, ensuring the total sequence length of the third and fourth data sequences meets a preset length. Referring to the method for determining the second encoding sequence, the encoder encodes the third image, the third data sequence, and the fourth data sequence to obtain the fourth encoding sequence. Referring to the method for error correction sequence segments, the quantizer performs matched quantization on the fourth encoding sequence to obtain the sample error correction sequence segment. The sample noise sequence segment and the sample error correction sequence segment can be concatenated to obtain the noisy image feature sequence. The decoder can then decode the noisy image feature sequence back to the fifth image. Figure 5 (Represented by the fifth image token in the text).

[0091] In this embodiment, a second loss value between the third and fifth images can be calculated using an existing inter-image loss calculation function. For example... Figure 5 In this process, a second loss value can be calculated between the third image token and the fifth image token. Furthermore, the second loss value can be backpropagated using the same method as the first loss value to construct the image tokenizer.

[0092] By introducing noise into a one-dimensional data sequence containing image information of the third image, a noisy image feature sequence is obtained. A fifth image is then reconstructed based on this noisy image feature sequence, and a second loss value is determined based on the reconstructed fifth image, thus enabling the learning of the second task. Through learning this second task, the image segmenter can be equipped to output subsequent data with error-correcting capabilities even when noisy data appears in the data sequence containing the image information, i.e., it can output a self-correcting image feature sequence.

[0093] In some alternative implementations, the image segmenter is constructed based on a first probability using a first loss value, or based on a second probability using a second loss value; wherein the sum of the first probability and the second probability is one.

[0094] In these optional implementations, the first probability and the second probability can vary with the construction time step of the image word segmenter. For example, in the initial construction phase, the second probability can approach 1, enabling the image word segmenter to learn to embed image information into a correct one-dimensional image feature sequence and to reconstruct the image based on the correct image feature sequence. Furthermore, the second probability can gradually decrease with time steps, while the first probability can gradually increase, allowing the image word segmenter to learn to output a self-correcting image feature sequence based on its ability to output a correct image feature sequence. By constructing the image word segmenter based on the first probability using a first loss value, or based on the second probability using a second loss value, simultaneous learning and construction of the image word segmenter based on the first and second tasks can be achieved.

[0095] The technical solutions of this disclosure provide a detailed description of the construction process of an image word segmenter. A first task learning is achieved by embedding image information of a third image into a one-dimensional correct image feature sequence, reconstructing the image based on the correct image feature sequence, and determining a first loss value based on the reconstructed image. By learning the first task, the image word segmenter gains the ability to embed image information into a one-dimensional correct image feature sequence and to reconstruct the image based on the correct image feature sequence. A second task learning is achieved by introducing noise data into the one-dimensional data sequence containing the image information of the third image, reconstructing the image based on the noise image feature sequence, and determining a second loss value based on the reconstructed image. By learning the second task, the image word segmenter gains the ability to output subsequent data with error correction functionality even when noise data appears in the data sequence containing the embedded image information; that is, it gains the ability to output a self-correcting image feature sequence.

[0096] Furthermore, the image generation method provided in this embodiment belongs to the same concept as the image generation method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.

[0097] Figure 6 This is a schematic diagram of an image generation apparatus provided in an embodiment of this disclosure. The image generation apparatus provided in this embodiment is suitable for generating images using an autoregressive image generation model.

[0098] like Figure 6 As shown, the image generation apparatus provided in this embodiment may include:

[0099] Module 610 is used to obtain the first prompt word;

[0100] The generation module 620 is used to generate a first image based on a first prompt word using an autoregressive image generation model.

[0101] The autoregressive image generation model is constructed based on a self-correcting image feature sequence; the self-correcting image feature sequence is determined based on an image segmenter; the image segmenter is constructed based on simultaneous learning of the first and second tasks; the first task includes the task of reconstructing an image based on a correct image feature sequence, and the second task includes the task of reconstructing an image based on a noisy image feature sequence.

[0102] In some alternative implementations, the image segmenter includes an encoder and a quantizer; the image generation apparatus may include:

[0103] The feature sequence determination module can determine the self-correcting image feature sequence through the following process:

[0104] A second image is obtained, along with a noise sequence segment of the image features of the second image; wherein the noise sequence segment is obtained by random quantization of a portion of the first encoded sequence of the second image.

[0105] The second image and the noise sequence segment are encoded by an encoder to obtain the second encoded sequence;

[0106] The second encoded sequence is matched and quantized using a quantizer to obtain the error-corrected sequence segment corresponding to the noise sequence segment.

[0107] Based on the noise sequence segment and the error correction sequence segment, the self-correcting image feature sequence is determined.

[0108] In some alternative implementations, the feature sequence determination module can also be used to determine noisy sequence segments based on the following process:

[0109] The second image is encoded using an encoder to obtain the first encoded sequence;

[0110] A random number of unit data points are obtained starting from the first unit data in the first encoded sequence to obtain a partial sequence;

[0111] The quantizer performs random quantization on the last unit data in a portion of the sequence and performs matching operations on the other unit data to obtain a noisy sequence segment; wherein, the other unit data includes the unit data in the portion of the sequence excluding the last unit data.

[0112] In some alternative implementations, the feature sequence determination module can be used to:

[0113] From the codebook corresponding to the quantizer, obtain the first preset number of first vectors that match the data of the last cell;

[0114] A second vector is randomly determined from the first vector to perform random quantization on the data in the last unit.

[0115] In some alternative implementations, the image segmenter includes an encoder, a quantizer, and a decoder; the image generation apparatus may include:

[0116] The building module is used to construct an image word segmenter based on the first task through the following process:

[0117] Obtain the third image;

[0118] The third image is encoded using an encoder to obtain a third encoded sequence;

[0119] By using a quantizer, the third encoded sequence is matched and quantized to obtain the correct image feature sequence;

[0120] The fourth image is obtained by decoding the correct image feature sequence using a decoder.

[0121] A first loss value is constructed based on the third and fourth images, and an image segmenter is constructed based on the first loss value.

[0122] In some alternative implementations, a building module is used to construct an image word segmenter based on the second task through the following process:

[0123] Obtain sample noise sequence segments; wherein, the sample noise sequence segments are obtained by randomly quantizing a portion of the third encoded sequence;

[0124] The third image and the sample noise sequence segment are encoded by an encoder to obtain the fourth encoded sequence;

[0125] The fourth encoded sequence is matched and quantized using a quantizer to obtain the sample error correction sequence segment corresponding to the sample noise sequence segment.

[0126] Based on the sample noise sequence segment and the sample error correction sequence segment, determine the feature sequence of the noisy image;

[0127] The fifth image is obtained by decoding the feature sequence of the noisy image using a decoder.

[0128] A second loss value is constructed based on the third and fifth images, and an image word segmenter is constructed based on the second loss value.

[0129] In some alternative implementations, the image segmenter is constructed based on a first probability using a first loss value, or based on a second probability using a second loss value; wherein the sum of the first probability and the second probability is one.

[0130] The image generation apparatus provided in this disclosure can execute the image generation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0131] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0132] The following is for reference. Figure 7 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 7The diagram below shows the structure of the terminal device or server 700. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0133] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0134] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays, speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0135] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined above in the image generation method of embodiments of this disclosure.

[0136] The electronic device provided in this embodiment and the image generation method provided in the above embodiments belong to the same disclosed concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0137] This disclosure provides a storage medium for computer-executable instructions, which, when executed by a computer processor, can be used to perform the image generation method provided in the above embodiments.

[0138] It should be noted that the storage medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory (FLASH), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores executable instructions that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable executable instructions. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit executable instructions for use by or in connection with an instruction execution system, apparatus, or device. Executable instructions contained on the storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0139] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0140] The aforementioned storage medium may be included in the aforementioned electronic device; or it may exist independently and not be assembled into the electronic device.

[0141] The aforementioned storage medium carries one or more executable instructions, which, when executed by the electronic device, cause the electronic device to:

[0142] Obtain the first prompt word; generate the first image based on the first prompt word using an autoregressive image generation model; wherein the autoregressive image generation model is constructed based on a self-correcting image feature sequence; wherein the self-correcting image feature sequence is determined based on an image segmenter; the image segmenter is constructed based on simultaneous learning of the first task and the second task; the first task includes the task of reconstructing the image based on the correct image feature sequence, and the second task includes the task of reconstructing the image based on the noisy image feature sequence.

[0143] Executable instructions for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The executable instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the image generation method provided in any embodiment of this disclosure.

[0145] The computer program product includes a computer program carried on a non-transitory computer-readable medium, which contains program code for performing an image generation method. The program code can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0147] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units and modules do not, in certain circumstances, constitute a limitation on the unit or module itself.

[0148] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0149] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0150] According to one or more embodiments of this disclosure, an image generation method is provided, the method comprising:

[0151] Get the first prompt word;

[0152] A first image is generated based on the first prompt word using an autoregressive image generation model;

[0153] The autoregressive image generation model is constructed based on a self-correcting image feature sequence; the self-correcting image feature sequence is determined based on an image word segmenter; the image word segmenter is constructed based on simultaneous learning of a first task and a second task; the first task includes the task of reconstructing an image based on a correct image feature sequence, and the second task includes the task of reconstructing an image based on a noisy image feature sequence.

[0154] According to one or more embodiments of this disclosure, an image generation method is provided, further comprising:

[0155] In some optional implementations, the image segmenter includes an encoder and a quantizer; the process of determining the self-correcting image feature sequence includes:

[0156] A second image is obtained, along with a noise sequence segment of image features of the second image; wherein the noise sequence segment is obtained by random quantization of a portion of a first encoded sequence of the second image.

[0157] The encoder encodes the second image and the noise sequence segment to obtain a second encoded sequence.

[0158] The second encoded sequence is matched and quantized using the quantizer to obtain an error-corrected sequence segment corresponding to the noise sequence segment.

[0159] The self-correcting image feature sequence is determined based on the noise sequence segment and the error correction sequence segment.

[0160] According to one or more embodiments of this disclosure, an image generation method is provided, further comprising:

[0161] In some optional implementations, the process of determining the noise sequence segment includes:

[0162] The second image is encoded using the encoder to obtain the first encoded sequence;

[0163] A random number of unit data are obtained from the first unit data in the first encoded sequence to obtain the partial sequence;

[0164] The quantizer performs random quantization on the last unit data in the partial sequence and matching operation on the other unit data to obtain the noise sequence segment; wherein, the other unit data includes unit data in the partial sequence excluding the last unit data.

[0165] According to one or more embodiments of this disclosure, an image generation method is provided, further comprising:

[0166] In some optional implementations, the random quantization of the last unit data in the partial sequence using the quantizer includes:

[0167] From the codebook corresponding to the quantizer, obtain the first preset number of first vectors that match the data of the last unit;

[0168] A second vector is randomly determined from the first vector to perform random quantization on the final unit data.

[0169] According to one or more embodiments of this disclosure, an image generation method is provided, further comprising:

[0170] In some optional implementations, the image segmenter includes an encoder, a quantizer, and a decoder; the process of constructing the image segmenter based on the first task includes:

[0171] Obtain the third image;

[0172] The encoder encodes the third image to obtain a third encoded sequence;

[0173] The correct image feature sequence is obtained by matching quantization of the third encoded sequence using the quantizer.

[0174] The fourth image is obtained by decoding the correct image feature sequence using the decoder.

[0175] A first loss value is constructed based on the third image and the fourth image, and the image segmenter is constructed based on the first loss value.

[0176] According to one or more embodiments of this disclosure, an image generation method is provided, further comprising:

[0177] In some alternative implementations, the process of constructing the image segmenter based on the second task includes:

[0178] Obtain a sample noise sequence segment; wherein, the sample noise sequence segment is obtained based on random quantization of a portion of the third encoded sequence;

[0179] The encoder encodes the third image and the sample noise sequence segment to obtain a fourth encoded sequence;

[0180] The fourth encoded sequence is matched and quantized using the quantizer to obtain a sample error correction sequence segment corresponding to the sample noise sequence segment.

[0181] The noise image feature sequence is determined based on the sample noise sequence segment and the sample error correction sequence segment;

[0182] The fifth image is obtained by decoding the noisy image feature sequence using the decoder.

[0183] A second loss value is constructed based on the third image and the fifth image, and the image segmenter is constructed based on the second loss value.

[0184] According to one or more embodiments of this disclosure, an image generation method is provided, further comprising:

[0185] In some alternative implementations, the image segmenter is constructed based on a first probability using the first loss value, or based on a second probability using the second loss value; wherein the sum of the first probability and the second probability is one.

[0186] According to one or more embodiments of the present disclosure, an image generation apparatus is provided, the apparatus comprising:

[0187] The acquisition module is used to acquire the first prompt word;

[0188] The generation module is used to generate a first image based on the first prompt word using an autoregressive image generation model;

[0189] The autoregressive image generation model is constructed based on a self-correcting image feature sequence; the self-correcting image feature sequence is determined based on an image word segmenter; the image word segmenter is constructed based on simultaneous learning of a first task and a second task; the first task includes the task of reconstructing an image based on a correct image feature sequence, and the second task includes the task of reconstructing an image based on a noisy image feature sequence.

[0190] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0191] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0192] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image generation method, characterized in that, include: Get the first prompt word; A first image is generated based on the first prompt word using an autoregressive image generation model; The autoregressive image generation model is constructed based on a self-correcting image feature sequence; the self-correcting image feature sequence is determined based on an image word segmenter; the image word segmenter is constructed based on simultaneous learning of a first task and a second task; the first task includes the task of reconstructing an image based on a correct image feature sequence, and the second task includes the task of reconstructing an image based on a noisy image feature sequence.

2. The method according to claim 1, characterized in that, The image segmenter includes an encoder and a quantizer; the process of determining the self-correcting image feature sequence includes: A second image is obtained, along with a noise sequence segment of image features of the second image; wherein the noise sequence segment is obtained by random quantization of a portion of a first encoded sequence of the second image. The encoder encodes the second image and the noise sequence segment to obtain a second encoded sequence. The second encoded sequence is matched and quantized using the quantizer to obtain an error-corrected sequence segment corresponding to the noise sequence segment. The self-correcting image feature sequence is determined based on the noise sequence segment and the error correction sequence segment.

3. The method according to claim 2, characterized in that, The process of determining the noise sequence segment includes: The second image is encoded using the encoder to obtain the first encoded sequence; A random number of unit data are obtained from the first unit data in the first encoded sequence to obtain the partial sequence; The quantizer performs random quantization on the last unit data in the partial sequence and matching operation on the other unit data to obtain the noise sequence segment; wherein, the other unit data includes unit data in the partial sequence excluding the last unit data.

4. The method according to claim 3, characterized in that, The step of performing random quantization on the last unit data in the partial sequence using the quantizer includes: From the codebook corresponding to the quantizer, obtain the first preset number of first vectors that match the data of the last unit; A second vector is randomly determined from the first vector to perform random quantization on the final unit data.

5. The method according to claim 1, characterized in that, The image segmenter includes an encoder, a quantizer, and a decoder; the process of constructing the image segmenter based on the first task includes: Obtain the third image; The encoder encodes the third image to obtain a third encoded sequence; The correct image feature sequence is obtained by matching quantization of the third encoded sequence using the quantizer. The fourth image is obtained by decoding the correct image feature sequence using the decoder. A first loss value is constructed based on the third image and the fourth image, and the image segmenter is constructed based on the first loss value.

6. The method according to claim 5, characterized in that, The process of constructing the image segmenter based on the second task includes: Obtain a sample noise sequence segment; wherein, the sample noise sequence segment is obtained based on random quantization of a portion of the third encoded sequence; The encoder encodes the third image and the sample noise sequence segment to obtain a fourth encoded sequence; The fourth encoded sequence is matched and quantized using the quantizer to obtain a sample error correction sequence segment corresponding to the sample noise sequence segment. The noise image feature sequence is determined based on the sample noise sequence segment and the sample error correction sequence segment; The fifth image is obtained by decoding the noisy image feature sequence using the decoder. A second loss value is constructed based on the third image and the fifth image, and the image segmenter is constructed based on the second loss value.

7. The method according to claim 6, characterized in that, The image segmenter is constructed based on a first probability using the first loss value, or based on a second probability using the second loss value; wherein the sum of the first probability and the second probability is one.

8. An image generation apparatus, characterized in that, include: The acquisition module is used to acquire the first prompt word; The generation module is used to generate a first image based on the first prompt word using an autoregressive image generation model; The autoregressive image generation model is constructed based on a self-correcting image feature sequence; the self-correcting image feature sequence is determined based on an image word segmenter; the image word segmenter is constructed based on simultaneous learning of a first task and a second task; the first task includes the task of reconstructing an image based on a correct image feature sequence, and the second task includes the task of reconstructing an image based on a noisy image feature sequence.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the image generation method as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the image generation method as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the image generation method as described in any one of claims 1-7.

Images