An image generation method, apparatus, device, medium, product

Abstract

This application discloses an image generation method, apparatus, device, medium, and product. The method includes a prediction process for token sequences at multiple scales, and the prediction process for the token sequence at each scale is obtained through multiple rounds of sampling prediction. This ensures that at the same scale, the subsequent round of sampling prediction is based on the results obtained from the previous round of sampling prediction, so that the subsequent round of sampling prediction can learn the correlation between different local regions (such as different local regions represented by different tokens) from the results. This makes the results obtained from the subsequent round of sampling prediction more accurate, thereby improving the image generation effect.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an image generation method, apparatus, device, medium, and product. Background Technology

[0002] In some scenarios, there is a requirement to generate an image based on text provided by the user, so that the image meets the constraints described by the text.

[0003] However, how to implement the above image generation process is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides an image generation method, apparatus, device, medium, and product, which helps to improve the image generation effect.

[0005] To achieve the above objectives, the technical solution provided in this application is as follows:

[0006] This application provides an image generation method, the method comprising: acquiring target text and a dictionary, the dictionary including multiple visual features and indices of each visual feature; for any scale among multiple scales, predicting the probability of each index at that scale based on reference data corresponding to that scale, the reference data including the target text; determining an index prediction result for that scale based on sampling parameters and the probability of each index at that scale, the sampling range indicated by the sampling parameters including the index prediction result for that scale; updating the reference data corresponding to that scale based on the index prediction result for that scale, the updated reference data including the index prediction result; and continuing to execute the step of predicting the probability of each index at that scale based on the reference data corresponding to that scale until a preset stopping condition is reached; and generating an image described by the target text based on the visual features indicated in the dictionary by the index prediction results of the multiple scales.

[0007] In one possible implementation, after determining the index prediction result for the scale, the method further includes: updating the sampling parameters, wherein the sampling range indicated by the updated sampling parameters is smaller than the sampling range indicated by the sampling parameters before the update.

[0008] In one possible implementation, the preset stop condition includes: the sampling range indicated by the sampling parameters before the update is not greater than a preset range threshold.

[0009] In one possible implementation, the preset stopping condition includes: the number of times the reference data corresponding to the scale is updated reaches a preset number threshold.

[0010] In one possible implementation, updating the reference data corresponding to the scale based on the index prediction result of the scale includes: for any scale among multiple scales, if the scale is greater than a preset scale threshold, and / or the index prediction result of the scale includes at least two indices, then updating the reference data corresponding to the scale based on the index prediction result of the scale.

[0011] In one possible implementation, the plurality of scales includes a first scale and a second scale, wherein the first scale is arranged adjacent to the second scale in the plurality of scales, and the first scale is arranged earlier than the second scale in the plurality of scales; the initial value of the reference data corresponding to the second scale is determined based on the index prediction results of the target text and the first scale.

[0012] In one possible implementation, the image is generated using a decoder; the decoder and the plurality of visual features are determined using the same training process.

[0013] This application provides an image generation apparatus, comprising: an acquisition unit for acquiring target text and a dictionary, the dictionary including multiple visual features and indices of each visual feature; a processing unit for, for any scale among multiple scales, predicting the probability of each index at that scale based on reference data corresponding to that scale, the reference data including the target text; determining an index prediction result for that scale based on sampling parameters and the probability of each index at that scale, the sampling range indicated by the sampling parameters including the index prediction result for that scale; updating the reference data corresponding to that scale based on the index prediction result for that scale, the updated reference data including the index prediction result; and continuing to execute the step of predicting the probability of each index at that scale based on the reference data corresponding to that scale until a preset stop condition is reached; and a generation unit for generating an image described by the target text based on the visual features indicated in the dictionary by the index prediction results of the multiple scales.

[0014] This application provides an electronic device, the device comprising: a processor and a memory; the memory for storing instructions or computer programs; the processor for executing the instructions or computer programs in the memory, so that the electronic device performs the image generation method provided in this application.

[0015] This application provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform the image generation method provided in this application.

[0016] This application provides a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the image generation method provided in this application.

[0017] Compared with related technologies, this application has at least the following advantages:

[0018] The image generation scheme provided in this application includes a prediction process for index (token) sequences at multiple scales. The prediction process for each scale is obtained through multiple rounds of sampling prediction. This ensures that at the same scale, the subsequent round of sampling prediction is based on the results of the previous round of sampling prediction. This allows the subsequent round of sampling prediction to learn the correlation between different local regions (such as different local regions represented by different tokens) from the results, thereby making the results obtained from the subsequent round of sampling prediction more accurate and thus improving the image generation effect. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating an image generation method provided in this application embodiment;

[0021] Figure 2 A schematic diagram illustrating a multi-round sampling prediction at the same scale, provided as an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of an image generation process provided in an embodiment of this application;

[0023] Figure 4 A schematic diagram of a decoder training process provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] Research has revealed that for some text-based image generation schemes, such as autoregressive image generation schemes based on next-token prediction, the number of steps in the image generation process is very large, resulting in very slow inference speed, because these schemes generate images one token at a time, such as through raster scans from left to right and from top to bottom.

[0027] The study also found that, in order to improve the reasoning speed, an image generation process based on next-scale prediction can be used. This allows the scheme to generate images in a coarse-to-fine manner, which not only conforms to the painting logic of the real world, but also greatly speeds up the reasoning process.

[0028] Further research revealed the following drawbacks of the scheme described above: When the scheme employs a top-k top-p sampling strategy, multiple tokens need to be predicted at certain scales (e.g., 4 tokens at a 2×2 scale, 9 tokens at a 3×3 scale, etc.). Moreover, the prediction processes for different tokens are completely independent, ensuring that the samples obtained through top-k top-p sampling when predicting different tokens are independent and unrelated. This can lead to different tokens potentially selecting the same index (e.g., the index used to represent the head), which can cause problems such as image corruption and repetitive localities (e.g., multiple heads or hands appearing in one image), thus affecting the image generation quality.

[0029] It should be noted that the top-k top-p sampling strategy works as follows: When predicting a token, if the predicted probabilities of each candidate (such as feature vectors of various image information) are obtained, the candidates are first sorted according to these predicted probabilities to obtain a sorting result; then, the top k candidates are selected from the sorting result, and multiple candidates whose sum of probabilities reaches the threshold p are selected from the sorting result; next, the intersection between the top k candidates and the multiple candidates is used as the sampling result, so that a sample can be randomly selected from the sampling result to determine the prediction result for the token.

[0030] It should also be noted that a token is a symbol used to represent a basic unit of data (such as a word in text or a block of images in an image). For example, the token can be implemented using an index similar to that in a dictionary (such as a vocabulary or a codebook) so that the token can represent the entry (such as a word or discrete feature) indicated by the token in the dictionary in some data processing process.

[0031] Based on the above research, in order to better improve the image generation effect, this application provides an optimization scheme for multi-round sampling prediction for image generation schemes based on next-scale prediction. The scheme includes: for each token sequence at each scale, the prediction process of the token sequence is obtained through multi-round sampling prediction, so that at the same scale, the subsequent round of sampling prediction is based on the result obtained by the previous round of sampling prediction. This allows the subsequent round of sampling prediction to learn from the result the correlation between different local regions represented by different tokens, thereby effectively overcoming the defects caused by the independent sampling prediction processes of different tokens, and making the result obtained by the subsequent round of sampling prediction more accurate, which is beneficial to improving the image generation effect.

[0032] As can be seen, the image generation method provided in this application includes: acquiring target text and a dictionary, the dictionary including multiple visual features and indices of each visual feature; for any scale among multiple scales, predicting the probability of each index at that scale based on reference data corresponding to that scale, the reference data including the target text; determining the index prediction result at that scale based on sampling parameters and the probability of each index at that scale, the sampling range indicated by the sampling parameters including the index prediction result at that scale; updating the reference data corresponding to that scale based on the index prediction result at that scale, the updated reference data including the index prediction result; and continuing to execute the step of predicting the probability of each index at that scale based on the reference data corresponding to that scale, until a preset stopping condition is reached; and generating an image described by the target text based on the visual features indicated in the dictionary by the index prediction results at multiple scales.

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0034] To better understand the technical solution provided in this application, the image generation method provided in this application will be explained below with reference to some accompanying drawings. For example... Figure 1 As shown, the image generation method provided in this application includes the following S1-S6.

[0035] S1: Obtain the target text and dictionary, which includes multiple visual features and an index for each visual feature.

[0036] Among them, the target text (such as Figure 2 or Figure 3 The target text shown is used to describe what the final generated image will look like, so that the text can describe the user's image generation requirements. For example, the text could be the string "A puppy is playing in the grass".

[0037] A dictionary is a dataset used when tokenizing any data (such as text, images, or image patches) so that it can provide usable data for the tokenization process, such as vocabulary, discrete features for representing vocabulary, and discrete features for representing image information (such as semantic-level information and / or pixel-level information).

[0038] As can be seen, a dictionary can include at least multiple visual features and indices for each visual feature, so that the dictionary can describe as diverse image information as possible. Here, the j-th visual feature refers to the discrete feature identified by index j, used to describe the j-th type of image information, where j is a positive integer, j≤J, and J is a positive integer representing the number of visual features in the dictionary.

[0039] Furthermore, this application does not limit the implementation of the j-th visual feature. For example, in some scenarios, such as image generation scenarios implemented by a pixel-based reconstruction token converter, the j-th visual feature includes an entry indicated (or identified) by index j in the codebook corresponding to the token converter, so that the j-th visual feature is used to characterize some pixel-level image information.

[0040] For example, in some scenarios, such as using... Figure 4 In the image generation scenario implemented by the indexing scheme shown, the j-th visual feature can include the index j in... Figure 4 The entries indicated in the semantic-level codebook shown, and / or, by index j in Figure 4The entries indicated in the pixel-level codebook are shown. It should be noted that the semantic-level codebook and the pixel-level codebook must at least satisfy the following constraints: the semantic-level codebook and the pixel-level codebook share an index; the semantic-level codebook includes multiple entries, and one entry in the semantic-level codebook describes image information at a semantic level; the pixel-level codebook includes multiple entries, and one entry in the pixel-level codebook describes image information at a pixel level, so that the shared index j can jointly represent multiple types of image information (such as semantic-level information and pixel-level information).

[0041] Furthermore, this application does not limit the method of obtaining the dictionary. For example, when the image generation method provided in this application is implemented by using a Large Language Model (LLM), the process of obtaining the dictionary may include: expanding the original vocabulary in the LLM using all entries in the semantic-level codebook and all entries in the pixel-level codebook to obtain the dictionary, so that the dictionary not only includes the content in the vocabulary (such as each word or the discrete features of each word), but also includes the content in these two codebooks (such as the discrete features of image information at each pixel level and the discrete features of image information at each semantic level), thereby enabling the dictionary to serve subsequent processes (such as text feature extraction and decoding processes).

[0042] S2: Based on the reference data corresponding to the i-th scale, predict the probability of each index at the i-th scale, where i is a positive integer, i≤N, N is a positive integer, and N represents the number of scales. The reference data includes the target text.

[0043] Here, the reference data corresponding to the i-th scale refers to the data required when performing index prediction processing at that i-th scale, such as the input data required when using LLM to implement index prediction processing; and the reference data corresponding to the i-th scale includes at least the target text. It should be noted that i is used to represent any scale.

[0044] In addition, in order to improve the image generation effect, the initial value of the reference data corresponding to the i-th scale can at least satisfy the following constraints: when i = 1, the initial value of the reference data corresponding to the i-th scale is the target text; however, when i ≥ 2, the initial value of the reference data corresponding to the i-th scale is determined based on the target text and the index prediction result of the (i-1)-th scale.

[0045] Therefore, in one possible implementation, when multiple scales include a first scale and a second scale, and the first scale's position in the multiple scales is adjacent to the second scale's position in the multiple scales, but the first scale's position in the multiple scales is earlier than the second scale's position in the multiple scales, the initial value of the reference data corresponding to the second scale is determined based on the target text and the index prediction result of the first scale. Here, the second scale represents a scale that is not the first in the multiple scales, and the first scale represents the preceding scale corresponding to the scale that is not the first in the multiple scales.

[0046] It should be noted that when i≥2, this application does not limit the process of determining the initial value of the reference data corresponding to the i-th scale (such as the second scale). For example, it can be: after obtaining the index prediction result of the (i-1)-th scale, the visual features indicated by the index prediction result in the dictionary are upsampled to obtain the upsampled result, so that the scale of the upsampled result is consistent with the i-th scale, so that the reference data corresponding to the i-th scale can be initialized using the upsampled result and the target text, so that the initial value of the reference data corresponding to the i-th scale includes the target text and the upsampled result. In this way, the defect caused by scale inconsistency can be overcome by adjusting (e.g., resizing) the representation features of the (i-1)-th scale to the i-th scale, thereby improving the image generation effect.

[0047] Based on the above two paragraphs, it can be seen that after obtaining the target text, the reference data corresponding to the first scale (such as the 1×1 scale) is initialized according to the text, so that subsequent multi-round sampling prediction processing for the first scale can be completed based on the initial value of the reference data corresponding to the first scale, so as to obtain the index prediction result of the first scale, so that the index prediction result can represent what information the image described by the text carries at the first scale;

[0048] The index prediction result of the first scale is then adjusted to the second scale (e.g., 2×2 scale) based on the visual features indicated in the dictionary. The reference data corresponding to the second scale is initialized based on the text and the data adjusted to the second scale. This allows for subsequent multi-round sampling prediction processing for the second scale based on the initial values ​​of the reference data corresponding to the second scale, resulting in the index prediction result of the second scale. This enables the index prediction result of the second scale to represent what information the image described by the text carries at the second scale.

[0049] ... (and so on)

[0050] Then, the index prediction result of the (N-1)th scale is adjusted to the visual features indicated in the dictionary to the Nth scale (e.g., N×N scale). Based on the text and the data adjusted to the Nth scale, the reference data corresponding to the Nth scale is initialized so that subsequent multi-round sampling prediction processing for the Nth scale can be completed based on the initial value of the reference data corresponding to the Nth scale, so as to obtain the index prediction result of the Nth scale. This allows the index prediction result of the Nth scale to represent what information the image described by the text carries at the Nth scale.

[0051] It should be noted that N is used to describe the scale of the image described by the target text. For example, if the image is represented using 256×256 tokens (or image blocks), then N is 256. Furthermore, this application does not limit the method of obtaining N; for example, N can be set according to the actual application scenario. Or, N can be set by the user.

[0052] The probability corresponding to index j at the i-th scale represents the likelihood that the image described by the target text carries image information represented by index j at the i-th scale. Here, the image information represented by index j refers to the image information (such as semantic-level information and / or pixel-level information) represented by the visual features indicated (or identified) by index j in the dictionary, where j is a positive integer and j ≤ J.

[0053] Furthermore, this application does not limit the implementation of the probability corresponding to index j at the i-th scale. For example, when the information carried by the image described by the target text at the i-th scale (e.g., 1×1 scale) is represented by a token (or an image patch), the probability corresponding to index j at the i-th scale can represent the probability of using index j as the token. As another example, when the information carried by the image described by the target text at the i-th scale (e.g., 2×2 scale) is represented by a token sequence (or multiple image patches), the probability corresponding to index j at the i-th scale is also represented by a probability sequence, so that each probability in the probability sequence can represent the probability of using index j as the token at the corresponding position in the token sequence.

[0054] Furthermore, this application does not limit the implementation of the above-described S2. For example, S2 can be implemented using some modules in an LLM. For example, when the reference data corresponding to the i-th scale includes at least the target text, S2 can include at least the following: first, the text encoder in the LLM processes the target text to obtain a token sequence corresponding to the target text, so that each token in the token sequence is used to indicate the corresponding entry in the dictionary, such as the corresponding entry in the vocabulary; then, the prediction module in the LLM predicts the probability of each index at the i-th scale based on the token sequence (and other data, such as the index prediction result of the (i-1)-th scale), and determines the index prediction result of the i-th scale based on these probabilities.

[0055] It should be noted that this application does not limit the implementation method of the above-described text encoder; for example, it can be implemented using Byte Pair Encoding (BPE). Furthermore, this application does not limit the implementation method of the above-described prediction module; for example, it can be implemented using a Transformer.

[0056] S3: Based on the sampling parameters and the probability of each index at the i-th scale, determine the index prediction result at the i-th scale. The sampling range indicated by the sampling parameters includes the index prediction result at the i-th scale.

[0057] The sampling parameters are used to limit the sampling range under the top-k and top-p sampling strategies. These parameters include a first parameter (k parameter) and a second parameter (p parameter). The first parameter constrains the sampling range of the top-k sampling strategy; for example, if the first parameter is 3, the top-k sampling strategy samples the top 3 candidate items. The second parameter constrains the sampling range of the top-p sampling strategy; for example, if the second parameter is 0.8, the top-p sampling strategy samples multiple candidate items with the highest probabilities (summing up to 0.8).

[0058] In addition, the sampling parameters can be preset.

[0059] The index prediction result at the i-th scale is used to characterize the image information carried by the image described by the target text at the i-th scale.

[0060] Furthermore, this application does not limit the implementation method of the index prediction result at the i-th scale. For example, when the information carried by the image described by the target text at the i-th scale (e.g., 1×1 scale) is represented by one token, the index prediction result at the i-th scale includes one index. As another example, when the information carried by the image described by the target text at the i-th scale (e.g., 2×2 scale) is represented by a token sequence, the index prediction result at the i-th scale is an index sequence (e.g., a 2×2 sequence).

[0061] Furthermore, this application does not limit the implementation of the above-mentioned S3. For example, it may specifically include: firstly, based on the sampling parameters and the probability corresponding to each index at the i-th scale, sampling at least one index from the indexes of the above-mentioned multiple visual features so that the at least one index can represent the sampling range indicated by the sampling parameters; and then obtaining the index prediction result of the i-th scale by randomly selecting the at least one index.

[0062] As can be seen, when the information carried by the image described by the target text at the i-th scale (e.g., 2×2 scale) is represented using a token sequence, and the probability corresponding to index j at the i-th scale is a probability sequence with the same dimension as the token sequence, and j is a positive integer, j≤J, firstly, based on the sampling parameters and the q-th data in the probability sequence corresponding to each index at the i-th scale, at least one index is sampled from the indices of the aforementioned multiple visual features as the sampling result corresponding to the q-th data in the token sequence, and an index is randomly selected from the sampling result corresponding to the q-th data in the token sequence as the matching index for the q-th data in the token sequence, where q is a positive integer, q≤Q, Q is a positive integer, and Q represents the dimension of the token sequence; then, based on the matching indices of each data in the token sequence, the index prediction result at the i-th scale is determined so that the index prediction result includes these matching indices.

[0063] Furthermore, the aforementioned S3 can also be implemented using the prediction module in LLM.

[0064] S4: Determine whether the preset stop condition has been met. If yes, proceed to S6 below; otherwise, proceed to S5 below.

[0065] The preset stopping condition refers to the condition required to stop multiple rounds of sampling prediction at the same scale, such as the number of rounds of sampling prediction for the i-th scale reaching a preset round threshold (e.g., 3 rounds).

[0066] It should be noted that this application does not limit the timing of the judgment of the preset stop condition; it can be executed before or after the following update.

[0067] S5: Based on the index prediction result of the i-th scale, update the reference data corresponding to the i-th scale. The updated reference data includes the index prediction result, and return to continue executing S2 and its subsequent steps.

[0068] It should be noted that this application does not limit the implementation of the above S5. For example, if the current round is the first round of sampling processing for the i-th scale, it can be determined that the reference data corresponding to the i-th scale does not contain the index prediction result for the i-th scale. Therefore, the index prediction result for the i-th scale determined in the current round can be directly added to the reference data corresponding to the i-th scale. However, if the current round is a non-first round of sampling processing for the i-th scale, it can be determined that the reference data corresponding to the i-th scale includes the index prediction result for the i-th scale obtained through the previous round of sampling prediction processing. Therefore, the index prediction result for the i-th scale determined in the current round can be used to directly replace the existing index prediction result in the reference data corresponding to the i-th scale. In this way, it is possible to realize that in multiple rounds of sampling prediction at the same scale, the subsequent round of sampling prediction is based on the result obtained from the previous round of sampling prediction, which is beneficial to improving the image generation effect.

[0069] S6: Generate an image described by the target text based on the visual features indicated in the dictionary according to the index prediction results at each scale.

[0070] It should be noted that this application does not limit the implementation method of S6, for example, as Figure 3 As shown, S6 specifically includes: after obtaining the index prediction results at all scales, the index prediction results at all scales can be summed (or concatenated) according to the visual features indicated in the dictionary to obtain the processed result; then the decoder performs decoding processing on the processed result to obtain the image described by the target text (such as...). Figure 3 Image 1 shown is used to make the image meet the image generation requirements described in the text.

[0071] Furthermore, this application does not limit the implementation method of the decoder, as long as it is compatible with multiple visual features in the dictionary. For example, when each visual feature in the dictionary is used to represent pixel-level image information, the decoder can be implemented using any decoder capable of decoding discrete features at the pixel level (such as the decoder in image generation implemented by a pixel-based token converter). Similarly, when each visual feature in the dictionary is used to represent semantic-level image information, the decoder can be implemented using any decoder capable of decoding discrete features at the semantic level (such as the decoder in image generation implemented by a semantic-based token converter). Moreover, when each visual feature in the dictionary is used to jointly represent semantic-level information and pixel-level information, the decoder can be implemented using... Figure 4 The pixel-level decoder shown is implemented.

[0072] Furthermore, to better improve image generation results, the decoder and multiple visual features in the dictionary can utilize the same training process (e.g., Figure 4 The training process shown is determined to make the two more coordinated.

[0073] Furthermore, this application does not limit the implementation method of the training process described above. For example, it can be specifically described in steps one to four below.

[0074] Step 1, acquire sample images (e.g.) Figure 4 Image 2 shown, and the sample image is divided into multiple image blocks;

[0075] Step 2: Use a semantic-level encoder to determine the semantic-level features of each image patch, and use a pixel-level encoder to determine the pixel-level features of each image patch.

[0076] Step 3: For any image block, determine the matching index of the image block based on the distance between the semantic-level features of the image block and each entry in the semantic-level codebook, and the distance between the pixel-level features of the image block and each entry in the pixel-level codebook. This ensures that the sum of the distances obtained by adding the distances between the entry indicated by the matching index in the semantic-level codebook and the semantic-level feature, and the distances between the entry indicated by the matching index in the pixel-level codebook and the pixel-level feature, reaches a minimum value. This allows the semantic-level decoder to process the entry indicated by the matching index in the semantic-level codebook to obtain a semantic prediction result, and the pixel-level decoder to process the entry indicated by the matching index in the pixel-level codebook to obtain a pixel prediction result (such as reconstructed image).

[0077] Step four, based on the difference between the semantic prediction result and its corresponding semantic truth value (e.g. Figure 4The loss shown is 1) the difference between the pixel prediction result and the sample image (e.g., Figure 4 The loss shown in step 2) is calculated based on the difference between the semantic-level feature and the entry indicated by the matching index in the semantic-level codebook, and the difference between the pixel-level feature and the entry indicated by the matching index in the pixel-level codebook. This is used to update the parameters in the two encoders, the entries in the two codebooks, and the parameters in the two decoders. The process then returns to continue executing step 1 and subsequent steps, iterating in this way until a pre-set training stopping condition is met (e.g., the loss is below a preset loss threshold, the rate of change of the loss is below a preset rate of change threshold, or the number of updates reaches a preset number of times threshold, etc.). The loss is determined based on the difference between the semantic prediction result and its semantic ground truth (e.g., L2 distance), the difference between the pixel prediction result and the sample image (e.g., pixel reconstruction loss, perceptual loss, adversarial loss), the difference between the semantic-level feature and the entry indicated by the matching index in the semantic-level codebook (e.g., codebook learning loss), and the difference between the pixel-level feature and the entry indicated by the matching index in the pixel-level codebook (e.g., codebook learning loss). This application does not limit the calculation process of this loss.

[0078] It should be noted that the semantic ground truth value corresponding to the semantic prediction result of any image patch is obtained by the teacher model of the semantic level encoder extracting semantic features from the image patch. The semantic level encoder is initialized using the teacher model. Furthermore, this application does not limit the implementation method of the above codebook learning loss; for example, it can be implemented using the formula (1) below.

[0079]

[0080] In the formula, Indicates codebook learning loss; represents the original feature (such as the semantic-level feature or pixel-level feature of an image patch); z represents the discrete feature found in the codebook that is closest to the original feature (such as the entry in the semantic-level codebook indicated by the index matched by the image patch, or the entry in the pixel-level codebook indicated by the matched index); sg[·] represents the stop-gradient operation.

[0081] Based on the relevant content in S1 to S6 above, the image generation method provided in this application includes a prediction process for token sequences at multiple scales. Moreover, the prediction process for token sequences at each scale is obtained through multiple rounds of sampling prediction. This ensures that at the same scale, the subsequent round of sampling prediction is based on the result obtained from the previous round of sampling prediction. This allows the subsequent round of sampling prediction to learn from the result the correlation between different local regions represented by different tokens, thereby making the result obtained from the subsequent round of sampling prediction more accurate and thus improving the image generation effect.

[0082] Furthermore, this application does not limit the executing entity of the image generation method. For example, the method can be applied to a terminal device or a server. Alternatively, the method can be implemented through data interaction between the terminal device and the server. The terminal device can be a smartphone, computer, personal digital assistant (PDA), tablet computer, etc. The server can be a standalone server, a cluster server, or a cloud server.

[0083] Research has shown that multi-round sampling prediction, achieved by gradually narrowing the sampling range, can achieve better image generation results.

[0084] Based on the above research, this application also provides an implementation of the index prediction process at the i-th scale, which may include steps 21 to 24 below.

[0085] Step 21: Obtain the initial value of the reference data corresponding to the i-th scale. If i = 1, the initial value is determined based on the target text. If i ≥ 2, the initial value is determined based on the target text and the index prediction result of the (i-1)-th scale.

[0086] Step 22: Based on the reference data corresponding to the i-th scale, predict the probability of each index at the i-th scale, where i is a positive integer.

[0087] It should be noted that for details regarding step 22, please refer to the relevant content of S2 above.

[0088] Step 23: Based on the sampling parameters and the probability corresponding to each index at the i-th scale, determine the index prediction result at the i-th scale. The sampling range indicated by the sampling parameters includes the index prediction result at the i-th scale.

[0089] It should be noted that for details regarding step 23, please refer to the relevant content of S3 above.

[0090] It should also be noted that the initial value of the sampling parameter at the i-th scale is preset. Furthermore, the initial values ​​of the sampling parameter at different scales can be the same or different; this application does not impose specific limitations on this.

[0091] Step 24: Based on the index prediction result of the i-th scale, update the reference data corresponding to the i-th scale so that the updated reference data includes the index prediction result, update the sampling parameters so that the sampling range indicated by the updated sampling parameters is smaller than the sampling range indicated by the sampling parameters before the update, and continue to execute the above step 22 and subsequent steps until the preset stopping condition is reached.

[0092] It should be noted that this application does not limit the method of updating the sampling parameters. For example, when the sampling parameters before the update include k1 and p1, and the updated sampling parameters are k2 and p2, the update method can satisfy the following constraints: k2 < k1, and p2 < p1.

[0093] It should also be noted that this application does not limit the implementation of the preset stop condition in step 24 above. For example, the preset stop condition may include the sampling range indicated by the sampling parameters before the update not being greater than a preset range threshold (such as the sampling parameters used to indicate that only one candidate is sampled), which is beneficial to improving the flexibility of multi-round sampling prediction.

[0094] For example, the aforementioned preset stopping condition may include: the number of times the reference data corresponding to the i-th scale is updated reaches a preset number threshold (such as 2 times), which is beneficial to improving the efficiency of multi-round sampling prediction.

[0095] Based on the relevant content of steps 21 to 24 above, it can be seen that for multi-round sampling prediction at the same scale, the subsequent round of sampling prediction is based on the result of the previous round of sampling prediction, and the sampling range involved in the subsequent round of sampling prediction is smaller than the sampling range involved in the previous round of sampling prediction. In this way, the local consistency of the image can be ensured as much as possible by gradually reducing the sampling range, which is beneficial to improving the image generation effect.

[0096] Research has found that multi-round sampling prediction is used to overcome the defects caused by the independent prediction processes of different tokens in the token sequence. Therefore, in order to improve the inference speed as much as possible, the i-th scale suitable for index prediction processing using multi-round sampling prediction satisfies the following constraints: the i-th scale is greater than a preset scale threshold (such as a 1×1 scale), and / or the index prediction result of the i-th scale includes at least two indices.

[0097] As can be seen, in one possible implementation, for any scale among multiple scales, the index prediction process for that scale includes: first, predicting the probability of each index at that scale based on the reference data corresponding to that scale; then, determining the index prediction result for that scale based on the sampling parameters and the probability of each index at that scale, so that when the scale is determined to be greater than a preset scale threshold, and / or the index prediction result for that scale includes at least two indices, the reference data corresponding to that scale is updated based on the index prediction result for that scale, the updated reference data including the index prediction result, and the aforementioned step of "predicting the probability of each index at that scale based on the reference data corresponding to that scale" and its subsequent steps are continued until a preset stopping condition is reached.

[0098] Based on this, in some scenarios, after obtaining the target text, the index prediction result at the 1×1 scale is first determined based on the target text, so that the index prediction process at the 1×1 scale only includes one round of sampling prediction; then, based on the target text and the index prediction result at the (m-1)th scale (e.g., 2×2 scale), the index prediction result at the mth scale is determined, so that the index prediction result at the mth scale includes at least two rounds of sampling prediction, where m is a positive integer, m≥2, and m≤N; then, based on the visual features indicated in the dictionary by the index prediction results at each scale, an image described by the target text is generated, which helps to improve efficiency.

[0099] In addition, in order to better improve the image generation effect, this application also provides a training process for the above-mentioned LLM model, which specifically includes steps 31 to 34 below.

[0100] Step 31: Obtain a pre-constructed data pair, which includes text and an image, with the text describing the image.

[0101] It should be noted that this application does not limit the implementation of step 31.

[0102] Step 32: Tokenize the images in the above data pairs to obtain the indexed results at each scale (such as token sequences).

[0103] It should be noted that this application does not limit the implementation of step 32; for example, it can be achieved by means of... Figure 4 The tokenization process shown is implemented. Figure 4 The symbol "N" is used to represent normalization.

[0104] Therefore, in one possible implementation, after acquiring the image from the above data pair, step 32 may specifically include the following process:

[0105] First, the semantic-level encoder is used to obtain the semantic-level features of the image (such as the semantic-level features of each image patch), and then the pixel-level encoder is used to obtain the pixel-level features of the image (such as the pixel-level features of each image patch).

[0106] The semantic-level features of the image are then downsampled to obtain semantic-level features at the first scale (e.g., 1×1 scale), so that the information described by the semantic-level features at the first scale is equivalent to the semantic-level information described by 1×1 image blocks. The pixel-level features of the image are then downsampled to obtain pixel-level features at the first scale, so that the information described by the pixel-level features at the first scale is equivalent to the pixel-level information described by 1×1 image blocks.

[0107] Then, based on the distance between the semantic-level features at the first scale and each entry in the semantic-level codebook, and the distance between the pixel-level features at the first scale and each entry in the pixel-level codebook, the indexing result at the first scale is determined so that the indexing result can represent the image information actually carried by the image at the first scale.

[0108] Then calculate the difference feature between the entry indicated by the indexing result at the first scale in the semantic-level codebook and the semantic-level features of the image. sem1 So that the difference feature sem1 It can represent the difference between the semantic level information indicated by the indexing result and the semantic level information carried by the image, and calculate the difference feature between the entry indicated by the first-scale indexing result in the pixel-level codebook and the pixel-level features of the image. pix1 So that the difference feature pix1 It can represent the difference between the pixel-level information indicated by the indexing result and the pixel-level information carried by the image;

[0109] Then the difference feature sem1 Downsampling is performed to obtain semantic-level features at a second scale (e.g., 2×2 scale), so that the information described by the semantic-level features at this second scale is equivalent to the semantic-level information described by 2×2 image patches. This difference feature is then used to... pix1 Downsampling is performed to obtain pixel-level features at the second scale, so that the information described by the pixel-level features at the second scale is equivalent to the pixel-level information described by 2×2 image blocks;

[0110] Then, based on the distance between the semantic-level features and each entry in the semantic-level codebook at the second scale, and the distance between the pixel-level features and each entry in the pixel-level codebook at the second scale, the indexing result at the second scale is determined so that the indexing result can represent the image information actually carried by the image at the second scale.

[0111] Then calculate the difference feature between the indexed result of the second scale and the entry indicated in the semantic-level codebook. sem1 Difference characteristics between sem2 So that the difference feature sem2 It can represent the semantic level information indicated by the indexing result and the information from the difference features. sem1 The gap between the semantic-level information carried, and the difference features between the entries indicated by the indexing results at the first scale in the pixel-level codebook and the pixel-level features of the image. pix2 So that the difference feature pix2 It can represent the pixel-level information indicated by the indexing result and the difference features. pix1 The difference between the pixel-level information carried; ... (and so on), until the indexing result of the Nth scale is determined, then it ends.

[0112] It should be noted that, in order to better unify the feature space, after obtaining the indexed results of each scale shown in the above process, it can be processed through two layers of multilayer perceptron (MLP) to avoid the defects caused by the inconsistency of the feature space, thereby improving the results.

[0113] It should also be noted that this application does not limit the execution time of step 32, but only needs to ensure that the execution time of step 32 is earlier than the execution time of step 34 below.

[0114] It should also be noted that this application does not limit the way the indexed results at each scale are used. For example, they can be concatenated with the token sequence of the text in the above data pair; then the concatenation result can be input into the LLM so that the LLM can obtain the token sequence from the concatenation result for index prediction processing, and obtain the indexed results at these scales from the concatenation result to guide the index prediction processing.

[0115] Step 33: Input the text from the above data pair into the LLM to obtain the index prediction results for each scale of the LLM.

[0116] It should be noted that the implementation of step 32 above is similar to the target text processing process described above, and will not be repeated here for the sake of brevity.

[0117] Step 34: Based on the difference between the index prediction results and the indexing results at each scale, update the LLM and return to continue executing Step 31 and subsequent steps until the iterative training process for the LLM ends when the preset iteration stopping conditions are met (such as the model loss being lower than a preset loss threshold, the rate of change of the model loss being lower than a preset rate of change threshold, or the number of updates of the LLM reaching a preset number threshold, etc.).

[0118] It should be noted that the model loss shown in the above paragraph is used to characterize the performance of LLM, and the loss is determined based on the difference between the index prediction results at each scale and the indexing results at each scale (such as cross-entropy loss). This application does not limit the calculation method of the loss.

[0119] Based on the relevant content in steps 31 to 34 above, it is known that in some scenarios, text images are used to train an LLM, so that the trained LLM (such as...) Figure 2 or Figure 3 The LLM shown can achieve high-quality image generation in the next-scale prediction paradigm, thus giving it good text-to-image performance.

[0120] In addition, during the training of LLM, the conditional text is randomly replaced with an empty string with a probability of 0.1 to support classifier-free guidance (CFG) inference.

[0121] In addition, QK-normalization and norm re-ordering are introduced during the training process of LLM to stabilize the training process and improve the stability of model training.

[0122] Based on the above-mentioned image processing methods, it is known that the image generation scheme based on the next-scale prediction paradigm provided in this application has the advantages shown in ①-④ below.

[0123] ① Compared with some image generation schemes based on encoder-decoder architecture (such as diffusion model), the decoder-only architecture adopted in this application is easier to integrate with LLM.

[0124] ② This application realizes the image generation process from coarse to fine (i.e. from blurry to clear) through the progressive generation method of the next-scale prediction paradigm, which significantly reduces the computational complexity and thus achieves a faster inference speed.

[0125] ③ The multi-stage sampling prediction scheme proposed in this application effectively solves problems such as image collapse and repetitive patterns, and improves the visual quality and diversity of the generated images.

[0126] ④ The overall solution of this application maintains the ability to generate high-quality images while reducing the demand for computing resources, thus improving the practicality of the solution.

[0127] Based on the image generation method provided in the embodiments of this application, the embodiments of this application also provide an image generation apparatus, which will be discussed below. Figure 5 Explanation and clarification will be provided. Among them, Figure 5 This is a schematic diagram of an image generation apparatus provided in an embodiment of this application. It should be noted that for technical details of the image generation apparatus provided in this application embodiment, please refer to the relevant content of the image generation method above.

[0128] like Figure 5 As shown, the image generation apparatus 500 provided in this application embodiment includes:

[0129] The acquisition unit 501 is used to acquire target text and a dictionary, wherein the dictionary includes multiple visual features and an index of each visual feature;

[0130] Processing unit 502 is configured to, for any one of multiple scales, predict the probability of each index at that scale based on reference data corresponding to that scale, wherein the reference data includes the target text; determine the index prediction result for that scale based on sampling parameters and the probability of each index at that scale, wherein the sampling range indicated by the sampling parameters includes the index prediction result for that scale; update the reference data corresponding to that scale based on the index prediction result for that scale, wherein the updated reference data includes the index prediction result; and continue to execute the step of predicting the probability of each index at that scale based on the reference data corresponding to that scale until a preset stopping condition is reached.

[0131] The generation unit 503 is used to generate an image described by the target text based on the visual features indicated in the dictionary according to the index prediction results of the multiple scales.

[0132] In one possible implementation, the processing unit 502 is specifically configured to: update the reference data corresponding to the scale based on the index prediction result of the scale, wherein the updated reference data includes the index prediction result; update the sampling parameters, wherein the sampling range indicated by the updated sampling parameters is smaller than the sampling range indicated by the sampling parameters before the update; and continue to execute the step of predicting the probability corresponding to each index at the scale based on the reference data corresponding to the scale.

[0133] In one possible implementation, the preset stop condition includes: the sampling range indicated by the sampling parameters before the update is not greater than a preset range threshold.

[0134] In one possible implementation, the preset stopping condition includes: the number of times the reference data corresponding to the scale is updated reaches a preset number threshold.

[0135] In one possible implementation, the processing unit 502 is specifically used to: for any scale among multiple scales, if the scale is greater than a preset scale threshold, and / or the index prediction result of the scale includes at least two indices, then update the reference data corresponding to the scale based on the index prediction result of the scale.

[0136] In one possible implementation, the plurality of scales includes a first scale and a second scale, wherein the first scale is arranged adjacent to the second scale in the plurality of scales, and the first scale is arranged earlier than the second scale in the plurality of scales; the initial value of the reference data corresponding to the second scale is determined based on the index prediction results of the target text and the first scale.

[0137] In one possible implementation, the image is generated using a decoder; the decoder and the plurality of visual features are determined using the same training process.

[0138] Based on the aforementioned content of the image generation apparatus 500, it can be understood that the working principle of the apparatus 500 includes: performing a prediction process of token sequences at multiple scales, and the prediction process of the token sequence at each scale is obtained through multiple rounds of sampling prediction, so that at the same scale, the subsequent round of sampling prediction is based on the result obtained by the previous round of sampling prediction, so that the subsequent round of sampling prediction can learn from the result the correlation between different local regions represented by different tokens, thereby making the result obtained by the subsequent round of sampling prediction more accurate, which in turn helps to improve the image generation effect.

[0139] In addition, this application embodiment also provides an electronic device, the device including a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device performs any implementation of the image generation method provided in this application embodiment.

[0140] See Figure 6This diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0141] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0142] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0143] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0144] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0145] This application also provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform any implementation of the image generation method provided in this application.

[0146] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0147] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0148] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0149] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0150] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0153] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0154] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0155] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0156] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0157] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0158] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0159] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image generation method, characterized in that, The method includes: Obtain the target text and a dictionary, wherein the dictionary includes multiple visual features and an index of each visual feature; For any one of the multiple scales, based on the reference data corresponding to that scale, predict the probability of each index at that scale, where the reference data includes the target text; Based on the sampling parameters and the probability corresponding to each index at that scale, the index prediction result at that scale is determined, wherein the sampling range indicated by the sampling parameters includes the index prediction result at that scale; Based on the index prediction results at this scale, update the reference data corresponding to this scale. The updated reference data includes the index prediction results. Continue to execute the step of predicting the probability of each index at this scale based on the reference data corresponding to this scale, until a preset stopping condition is reached. Based on the visual features indicated in the dictionary by the index prediction results of the multiple scales, an image described by the target text is generated.

2. The method according to claim 1, characterized in that, After determining the index prediction result at this scale, the method further includes: The sampling parameters are updated, and the sampling range indicated by the updated sampling parameters is smaller than the sampling range indicated by the sampling parameters before the update.

3. The method according to claim 2, characterized in that, The preset stop condition includes: the sampling range indicated by the sampling parameters before the update is not greater than a preset range threshold.

4. The method according to any one of claims 1-3, characterized in that, The preset stopping condition includes: the number of times the reference data corresponding to this scale is updated reaches a preset number threshold.

5. The method according to claim 1, characterized in that, The step of updating the reference data corresponding to the scale based on the index prediction results of that scale includes: For any scale among multiple scales, if the scale is greater than a preset scale threshold, and / or the index prediction result of the scale includes at least two indices, then the reference data corresponding to the scale is updated based on the index prediction result of the scale.

6. The method according to claim 1, characterized in that, The plurality of scales includes a first scale and a second scale, wherein the first scale is arranged adjacent to the second scale in the plurality of scales, and the first scale is arranged earlier than the second scale in the plurality of scales. The initial value of the reference data corresponding to the second scale is determined based on the target text and the index prediction results of the first scale.

7. The method according to claim 1, characterized in that, The image was generated using a decoder; The decoder and the plurality of visual features are determined using the same training process.

8. An image generation apparatus, characterized in that, include: An acquisition unit is used to acquire target text and a dictionary, wherein the dictionary includes multiple visual features and an index of each visual feature; The processing unit is configured to predict the probability of each index at any of the multiple scales, based on the reference data corresponding to that scale, wherein the reference data includes the target text. Based on the sampling parameters and the probability corresponding to each index at that scale, the index prediction result at that scale is determined, wherein the sampling range indicated by the sampling parameters includes the index prediction result at that scale; based on the index prediction result at that scale, the reference data corresponding to that scale is updated, wherein the updated reference data includes the index prediction result; and the step of predicting the probability corresponding to each index at that scale based on the reference data corresponding to that scale is continued until a preset stopping condition is reached. A generation unit is configured to generate an image described by the target text based on the visual features indicated in the dictionary by the index prediction results of the multiple scales.

9. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory to cause the electronic device to perform the method according to any one of claims 1-7.

10. A computer-readable medium, characterized in that, The computer-readable medium stores instructions or computer programs that, when executed on the device, cause the device to perform the method according to any one of claims 1-7.

11. A computer program product, characterized in that, It includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method of any one of claims 1-7.

Images