Image generation method and device, terminal, electronic equipment and storage medium

Abstract

The invention relates to an image generation method and device, a terminal, electronic equipment and a storage medium, and the method comprises the steps: obtaining text prompt information and a style code, the text prompt information is used for indicating the image content of a target image, and the style code is used for indicating the image style of the target image; generating target index information based on the style code through a style generator, and determining a target style quantization code corresponding to the style code based on a style codebook and the target index information; decoding the target style quantization code through a decoder corresponding to the style codebook to obtain a target style feature; and inputting the text prompt information and the target style feature into an image generation model to obtain a target image. According to the method, the style can be automatically deduced only through one style code, a reference image or a character does not need to be input into the model to express the style, style loss does not exist, and style exploration is achieved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an image generation method, apparatus, terminal, electronic device, and storage medium. Background Technology

[0002] In the field of AI-generated images, how to generate images with specific styles has always been a core issue of common concern to both academia and industry. Traditional methods can generate images based on existing styles, but they cannot currently create styles, that is, they cannot create images with novel styles, which greatly limits the imagination space of using AI to generate images. Summary of the Invention

[0003] This disclosure provides an image generation method, apparatus, terminal, electronic device, and storage medium to solve the problems in related technologies. The technical solution of this disclosure is as follows: According to a first aspect of the present disclosure, an image generation method is provided, the method comprising: Obtain text prompt information and style code, wherein the text prompt information is used to indicate the image content of the target image and the style code is used to indicate the image style of the target image; Using a style generator, target index information is generated based on the style code; and based on the style codebook and the target index information, the target style quantization code corresponding to the style code is determined. The target style quantization encoding is decoded using the decoder corresponding to the style codebook to obtain the target style features; The text prompt information and the target style features are input into the image generation model to obtain the target image.

[0004] In one exemplary embodiment, the target index information is represented by a target index sequence, and the generation of target index information based on the style code by the style generator includes: Based on the style code, the style codebook is sampled to determine an initial index value, which is the first index value of the target index sequence; Using the style generator, the target index sequence is derived autoregressively based on the initial index value.

[0005] In one exemplary implementation, the step of using the style generator to autoregressively derive the target index sequence based on the initial index value includes: The initial index value is determined as the index value corresponding to the current index number, where the current index number points to the first index of the target index sequence; Input the index value corresponding to the current index number into the style generator to obtain the prediction distribution corresponding to the next index number; Based on the predicted distribution corresponding to the next index number, determine the index value corresponding to the next index number; Update the current index number using the next index number, and continue to predict the index value until the prediction is complete.

[0006] In one exemplary implementation, the prediction distribution includes multiple index values ​​and a prediction probability corresponding to each index value; determining the index value corresponding to the next index number based on the prediction distribution corresponding to the next index number includes: For each index value in the predicted distribution, if the statistical probability corresponding to the index value is higher than or equal to a preset threshold, the reference weight corresponding to the index value is determined as the first weight; if the statistical probability corresponding to the index value is lower than the preset threshold, the reference weight corresponding to the index value is determined as the second weight, wherein the first weight is less than the second weight. Based on the predicted probability corresponding to the index value and the reference weight corresponding to the index value, the recommendation probability corresponding to the index value is obtained; The index value with the highest recommendation probability is determined as the index value corresponding to the next index number; The statistical probability is used to indicate the statistical value of the probability that the corresponding index value is sampled.

[0007] In one exemplary embodiment, the method further includes: Obtain a preset image set, which includes images of various styles; Determine the index sequence corresponding to each image in the preset image set; The probability of each index value appearing in the index set formed by the index sequences is calculated, and the probability of appearance is used as the statistical probability corresponding to the index value.

[0008] In one exemplary embodiment, the decoder is trained using the following method: Obtain multiple first sample images and the style label corresponding to each first sample image; Obtain the encoder, the style codebook, and the decoder; For each first sample image, extract the first visual feature corresponding to the first sample image, input the first visual feature into the encoder, so that the encoder outputs the corresponding first prediction index sequence based on the style codebook; input the first prediction index sequence into the decoder, so that the decoder outputs the corresponding prediction style feature; Positive sample pairs are constructed based on the predicted style features corresponding to two first sample images with the same style label; negative sample pairs are constructed based on the predicted style features corresponding to two first sample images with different style labels. The parameters of the decoder are adjusted by comparing and learning from the positive and negative sample pairs.

[0009] In one exemplary implementation, after performing comparative learning based on the positive and negative sample pairs and adjusting the parameters of the decoder, the image generation model is trained using the following method: Acquire a second sample image and a third sample image, wherein the second sample image and the third sample image have different content but the same style; Extract the second visual features from the second sample image, input the second visual features into the encoder, so that the encoder outputs the corresponding sample index sequence based on the style codebook; input the sample index sequence into the decoder, so that the decoder outputs the corresponding sample style features; The sample style features and the text description corresponding to the third sample image are input into the image generation model to obtain the predicted image; The image generation model is trained based on the difference between the predicted image and the third sample image.

[0010] In one exemplary implementation, after performing comparative learning based on the positive and negative sample pairs and adjusting the parameters of the decoder, the style generator is trained using the following method: Obtain the fourth sample image; The third visual feature of the fourth sample image is extracted and input into the encoder so that the encoder outputs the corresponding reference index sequence based on the style codebook. Using the style generator, a second predicted index sequence is generated autoregressively based on the reference index sequence; the style generator is trained based on the difference between the second predicted index sequence and the reference index sequence.

[0011] In one exemplary implementation, the step of using the style generator to autoregressively generate a second predicted index sequence based on the reference index sequence, and training the style generator based on the difference between the second predicted index sequence and the reference index sequence, includes: Initialize the current index number in the second predicted index sequence; Using the style generator, predict the predicted index value corresponding to the current index number; based on the difference between the predicted index value and the reference index value obtained by sampling the reference index sequence based on the current index number, determine the loss term corresponding to the current index number; Update the current index number using the next index number of the current index number, and repeatedly predict the predicted index value corresponding to the current index number and the loss term corresponding to the current index number until the current index number is the last index number of the second predicted index sequence; The style generator is trained based on each of the aforementioned loss terms.

[0012] In one exemplary implementation, using the style generator to predict the predicted index value corresponding to the current index number includes: If the current index number is not the first index number of the second predicted index sequence, a reference subsequence is sampled from the reference index sequence based on the current index number. The reference subsequence is a sequence formed by reference index values ​​whose index numbers are less than the current index number. Using the style generator, based on the reference subsequence, predict the predicted index value corresponding to the current index number.

[0013] In one exemplary embodiment, after obtaining the target image, the method further includes: Determine style description information, which is used to describe the image style of the target image; Record the correspondence between the style code and the style description information; This demonstrates the correspondence between the style code and the style description information.

[0014] According to a second aspect of the present disclosure, an image generation apparatus is provided, the apparatus comprising: The content style acquisition module is configured to acquire text prompt information and style code, wherein the text prompt information is used to indicate the image content of the target image, and the style code is used to indicate the image style of the target image. The style generation module is configured to execute the following steps: generating target index information based on the style code using a style generator; determining the target style quantization code corresponding to the style code based on the style codebook and the target index information; and decoding the target style quantization code using a decoder corresponding to the style codebook to obtain the target style feature. The image generation module is configured to input the text prompt information and the target style features into the image generation model to obtain the target image.

[0015] In one exemplary implementation, the style generation module is configured to perform: Based on the style code, the style codebook is sampled to determine an initial index value, which is the first index value of the target index sequence; Using the style generator, the target index sequence is derived autoregressively based on the initial index value.

[0016] In one exemplary implementation, the style generation module is configured to perform: The initial index value is determined as the index value corresponding to the current index number, where the current index number points to the first index of the target index sequence; Input the index value corresponding to the current index number into the style generator to obtain the prediction distribution corresponding to the next index number; Based on the predicted distribution corresponding to the next index number, determine the index value corresponding to the next index number; Update the current index number using the next index number, and continue to predict the index value until the prediction is complete.

[0017] In one exemplary implementation, the prediction distribution includes multiple index values ​​and a prediction probability corresponding to each index value; the style generation module is configured to execute: For each index value in the predicted distribution, if the statistical probability corresponding to the index value is higher than or equal to a preset threshold, the reference weight corresponding to the index value is determined as the first weight; if the statistical probability corresponding to the index value is lower than the preset threshold, the reference weight corresponding to the index value is determined as the second weight, wherein the first weight is less than the second weight. Based on the predicted probability corresponding to the index value and the reference weight corresponding to the index value, the recommendation probability corresponding to the index value is obtained; The index value with the highest recommendation probability is determined as the index value corresponding to the next index number; The statistical probability is used to indicate the statistical value of the probability that the corresponding index value is sampled.

[0018] In one exemplary implementation, the style generation module is configured to perform: Obtain a preset image set, which includes images of various styles; Determine the index sequence corresponding to each image in the preset image set; The probability of each index value appearing in the index set formed by the index sequences is calculated, and the probability of appearance is used as the statistical probability corresponding to the index value.

[0019] In one exemplary implementation, the style generation module is configured to perform: Obtain multiple first sample images and the style label corresponding to each first sample image; Obtain the encoder, the style codebook, and the decoder; For each first sample image, extract the first visual feature corresponding to the first sample image, input the first visual feature into the encoder, so that the encoder outputs the corresponding first prediction index sequence based on the style codebook; input the first prediction index sequence into the decoder, so that the decoder outputs the corresponding prediction style feature; Positive sample pairs are constructed based on the predicted style features corresponding to two first sample images with the same style label; negative sample pairs are constructed based on the predicted style features corresponding to two first sample images with different style labels. The parameters of the decoder are adjusted by comparing and learning from the positive and negative sample pairs.

[0020] In one exemplary implementation, the style generation module is configured to perform: Acquire a second sample image and a third sample image, wherein the second sample image and the third sample image have different content but the same style; Extract the second visual features from the second sample image, input the second visual features into the encoder, so that the encoder outputs the corresponding sample index sequence based on the style codebook; input the sample index sequence into the decoder, so that the decoder outputs the corresponding sample style features; The sample style features and the text description corresponding to the third sample image are input into the image generation model to obtain the predicted image; The image generation model is trained based on the difference between the predicted image and the third sample image.

[0021] In one exemplary implementation, the style generation module is configured to perform: Obtain the fourth sample image; The third visual feature of the fourth sample image is extracted and input into the encoder so that the encoder outputs the corresponding reference index sequence based on the style codebook. Using the style generator, a second predicted index sequence is generated autoregressively based on the reference index sequence; the style generator is trained based on the difference between the second predicted index sequence and the reference index sequence.

[0022] In one exemplary implementation, the style generation module is configured to perform: Initialize the current index number in the second predicted index sequence; Using the style generator, predict the predicted index value corresponding to the current index number; based on the difference between the predicted index value and the reference index value obtained by sampling the reference index sequence based on the current index number, determine the loss term corresponding to the current index number; Update the current index number using the next index number of the current index number, and repeatedly predict the predicted index value corresponding to the current index number and the loss term corresponding to the current index number until the current index number is the last index number of the second predicted index sequence; The style generator is trained based on each of the aforementioned loss terms.

[0023] In one exemplary implementation, the style generation module is configured to perform: If the current index number is not the first index number of the second predicted index sequence, a reference subsequence is sampled from the reference index sequence based on the current index number. The reference subsequence is a sequence formed by reference index values ​​whose index numbers are less than the current index number. Using the style generator, based on the reference subsequence, predict the predicted index value corresponding to the current index number.

[0024] In one exemplary embodiment, the content style acquisition module is configured to perform: determining style description information, the style description information being used to describe the image style of the target image; recording the correspondence between the style code and the style description information; and displaying the correspondence between the style code and the style description information.

[0025] According to a third aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the image generation method as described in any of the above embodiments.

[0026] According to a fourth aspect of the present disclosure, a computer storage medium is provided, wherein when instructions in the computer storage medium are executed by a processor of an electronic device, the electronic device performs the image generation method described in any of the above embodiments.

[0027] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the image generation method described in any of the above embodiments.

[0028] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: The image generation method provided in this disclosure requires only a style code, which can be arbitrarily entered by the user. Based on this style code, a corresponding target style feature can be derived. Using the target style feature and user-inputted text prompts, a target image can be generated. This target image possesses a style corresponding to the target style feature. Since this style is created based on the user-inputted style code, it can be a completely new style that does not belong to any existing style. In other words, by converting any style code into a corresponding image style, a target image adapted to that style is created for the user, thereby realizing style exploration and expanding the imaginative space for artificial intelligence in image creation.

[0029] Furthermore, this disclosure can automatically derive the style using only this style code, without needing to input reference images or text into the model to express the style, thus eliminating style loss. Therefore, the image style generated by this disclosure does not have style bias, significantly improving style accuracy.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0032] Figure 1 This is a flowchart illustrating an image generation method according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating an autoregressive derivation method for a target index sequence according to an exemplary embodiment; Figure 3 This is a schematic diagram illustrating a decoder generation method according to an exemplary embodiment; Figure 4 This is a schematic diagram illustrating the training process of an image generation model according to an exemplary embodiment; Figure 5 This is a schematic diagram illustrating the style generator training process according to an exemplary embodiment; Figure 6 This is a schematic diagram illustrating an implementation process of an image generation method according to an exemplary embodiment; Figure 7 This is a block diagram of an image generation apparatus according to an exemplary embodiment; Figure 8 This is a structural block diagram of a computer device according to an exemplary embodiment. Figure 1 ; Figure 9 This is a structural block diagram of a computer device according to an exemplary embodiment. Figure 2 . Detailed Implementation

[0033] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0034] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0035] In the field of AI-generated images, how to generate images with specific styles has always been a core issue of common concern to both academia and industry. Traditional methods rely on reference images, long text prompts, or fine-tuning modules such as LoRA to represent certain style information. These methods are cumbersome to operate and have weak style preservation capabilities. Moreover, although AI has demonstrated powerful capabilities in the fields of humanities and artistic creation, there is currently no method that can create novel image styles, which greatly limits the imagination space of using AI to generate images.

[0036] One type of related technology can generate images based on reference images. This method inputs a user-provided style image along with text prompts into a diffusion model. Through a specific network structure (such as an adapter or feature injection), the visual features of the reference style image are used as conditional signals to guide the diffusion model to generate a new image with a style consistent with the reference image. This approach relies on existing reference style images: it can only imitate the style of existing images and cannot create entirely new visual styles that have not yet appeared in the world. Furthermore, the style representation is not compact; style information exists in the form of the entire image, occupying a large storage space and making it difficult to manage and retrieve as a unique and reproducible identifier.

[0037] Another approach to image generation utilizes parameter-efficient fine-tuning techniques, such as LoRA. LoRA, short for Low-Rank Adaptation, achieves efficient parameter fine-tuning through low-rank matrix factorization. This method significantly reduces the number of parameters requiring adjustment by introducing a low-rank factorization matrix into the pre-trained model, thereby lowering computational costs and storage requirements. However, this approach requires users to train a separate LoRA module for each style. When generating images, the corresponding LoRA module is loaded and combined with the base text-to-image model to generate images of that style. This approach requires a large amount of image data of the same style to train the LoRA module. Data collection is difficult, and the need to train a dedicated LoRA module for each style results in significant resource consumption. Furthermore, the style representation is not compact; LoRA-based solutions require storing style information within the LoRA module, consuming significant storage space and hindering propagation.

[0038] To address these technical problems, this disclosure provides an image generation method that allows users to freely input style codes, enabling style generation based on these codes. This allows for the creation of new styles, not limited to existing ones, thus generating images with novel styles. Furthermore, the style code can be any arbitrary number input by the user, yet a complete style can be derived from that number. This allows the number to be mapped to all style information without information loss. Moreover, the number occupies little space, is easy to distribute, and can be managed and invoked as a unique and reproducible identifier. The image generation method, apparatus, terminal, electronic device, and storage medium of this disclosure are described in detail below.

[0039] Figure 1 This is a flowchart illustrating an image generation method according to an exemplary embodiment. The image generation method can be applied to an electronic device, which can be implemented independently by a server or a terminal, or jointly by a terminal and a server. The terminal can be, but is not limited to, physical devices such as smartphones, tablets, laptops, desktop computers, smart speakers, smart wearable devices, digital assistants, augmented reality devices, and virtual reality devices, and can also include software such as applications running on the physical device. The server can be, but is not limited to, a standalone server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms, etc. (Refer to...) Figure 1 As shown, the method includes the following steps.

[0040] In S110, text prompt information and style code are obtained. The text prompt information is used to indicate the image content of the target image, and the style code is used to indicate the image style of the target image.

[0041] This disclosure does not limit the text prompts, as long as they can indicate the content of the target image. For example, text prompts may include keywords, phrases, or sentences describing objects, scenes, or specific elements in the target image. For instance, it could be a textual description of people, animals, buildings, natural landscapes, or other visual content. Furthermore, text prompts may include more specific details, such as color, shape, texture, or spatial layout, to further clarify the content characteristics of the target image.

[0042] The style code is used to deduce the overall visual style of the target image. In this disclosure, the style code can be a simple numeric code, such as an integer, or a graphic code, which can be used to represent the numeric code. The style code is essentially the seed used for style deduction in this disclosure. Therefore, compared to technical solutions that require users to directly input reference images or text information describing the style, this disclosure only requires users to provide the seed used for style deduction—the style code—significantly reducing the difficulty for users to provide the style. The style code obviously occupies less space and is easier to manage and provide than reference images or text information describing the style.

[0043] In S120, a style generator generates target index information based on the style code, and determines the target style quantization code corresponding to the style code based on the style codebook and the target index information.

[0044] The style codebook is a publicly disclosed codebook used for style derivation. Its size K can be set according to actual conditions; for example, K=8192, then the codebook includes integers from 0 to 8191. The style codebook works in conjunction with index information to determine style quantization encoding, which is a quantized expression of style. For example, if the style codebook determines the index information for a certain image as {A1,A2,A3,……,AT}, where T is a preset integer, such as 20, and A1,A2,A3,……,AT are the index values ​​in this index sequence, then querying the style codebook based on this index information {A1,A2,A3,……,AT} yields the code value B1 corresponding to A1, the code value B2 corresponding to A2, and so on, ultimately obtaining the complete style quantization encoding sequence {B1,B2,B3,……,BT}. This method of combining index information with the codebook efficiently compresses complex style information into a compact quantized expression, significantly reducing storage and computational overhead. Meanwhile, since the style codebook is predefined, different styles correspond to different index information. This approach not only improves the flexibility of style generation but also ensures the consistency and controllability of the generated results.

[0045] The style generator uses style codes as seeds to generate target index information. By using this target index information to query the style codebook, the target style quantization code corresponding to the style code can be determined.

[0046] In S130, the target style quantization encoding is decoded by the decoder corresponding to the style codebook to obtain the target style features.

[0047] Both the style generator and decoder use the same style codebook during training. Therefore, the decoder corresponding to the style codebook can decode the target style quantization encoding to obtain the target style features. These target style features are the features that the image generation model can understand and use to guide image style generation.

[0048] S140. Input the text prompt information and the target style features into the image generation model to obtain the target image.

[0049] This disclosure does not limit the image generation model; it can be a model based on a diffusion model. Diffusion models generate images through progressive denoising, offering high generation quality and flexibility. Furthermore, this model can effectively capture complex style features and combine them with textual prompts to generate the desired target image. Alternatively, the image generation model can also be based on a generative adversarial network (GAN). GANs, through adversarial training between the generator and discriminator, can generate highly realistic images. This approach excels in handling complex textures and details while better adapting to diverse style requirements. Whether using a diffusion model or a GAN, the core objective is to effectively combine textual prompts with target style features, ensuring that the generated image both conforms to semantic description and accurately reflects the desired style characteristics.

[0050] This disclosure allows users to freely input style codes, enabling style generation based on these codes. This allows for the creation of new styles beyond existing ones, resulting in images with novel styles. Furthermore, the style code can be any seed input by the user, yet a complete style can be derived from that seed. This ensures that the seed maps to all style information without information loss. Moreover, the seed has a small footprint, is easy to distribute, and can be managed and invoked as a unique and reproducible identifier.

[0051] In one exemplary implementation, the target index information is represented by a target index sequence. The step of generating the target index information based on the style code using a style generator includes: sampling the style codebook based on the style code to determine an initial index value, wherein the initial index value is the first index value of the target index sequence; and using the style generator, deriving the target index sequence autoregressively based on the initial index value.

[0052] Using style codes to sample the style codebook yields initial index values. This means the style code is essentially only used to directly determine the first index value of the target index sequence. Other index values ​​in the target sequence can be automatically generated by the style generator based on this first index value. The information or capabilities used to infer other index values ​​are carried by the style generator, and the user only needs to provide the seed (style code) for style generation. This design greatly simplifies the user's workflow while improving the flexibility and efficiency of style generation. Because the style generator has autoregressive prediction capabilities, it can progressively derive the complete index sequence from the first index value without requiring the user to input complex parameters or perform tedious configurations. This approach not only reduces the user's learning curve but also makes the entire system more intelligent and user-friendly. Furthermore, by centralizing the inference capabilities in the style generator, modularity is improved, facilitating subsequent optimization and expansion.

[0053] In one exemplary implementation, please refer to Figure 2 This diagram illustrates the autoregressive derivation method for the target index sequence in this disclosure. The step of using the style generator to autoregressively derive the target index sequence based on the initial index value includes: S210. Initialize the current index number.

[0054] For example, if the target index sequence includes 10 index values ​​with corresponding index numbers 0-9, then the current index number can be initialized to 0.

[0055] S220. The initial index value is determined as the index value corresponding to the current index number.

[0056] S230. Input the index value corresponding to the current index number into the style generator to obtain the prediction distribution corresponding to the next index number.

[0057] If the current index number is 0, the initial index value can be input into the style generator to obtain the prediction distribution corresponding to the next index number. If the current index number is not 0, the index values ​​corresponding to the current index number and the previous index numbers can be input into the style generator to obtain the prediction distribution corresponding to the next index number.

[0058] S240. Based on the predicted distribution corresponding to the next index number, determine the index value corresponding to the next index number.

[0059] In one exemplary implementation, the prediction distribution includes multiple index values ​​and a prediction probability corresponding to each index value; determining the index value corresponding to the next index number based on the prediction distribution corresponding to the next index number includes: A1. For each index value in the predicted distribution, if the statistical probability corresponding to the index value is higher than or equal to a preset threshold, the reference weight corresponding to the index value is determined as a first weight; if the statistical probability corresponding to the index value is lower than the preset threshold, the reference weight corresponding to the index value is determined as a second weight, wherein the first weight is less than the second weight; wherein the statistical probability is used to indicate the statistical value of the probability that the corresponding index value is sampled.

[0060] The statistical probability refers to the probability of an index value being adopted in a big data scenario. If the probability of an index value being adopted is very high, it means that the index value is likely to point to a useless placeholder code value in the style codebook. This is similar to large-scale language generation models, where the statistical probability of punctuation marks in a segment predicted by the large-scale language generation model is much higher than the statistical probability of a single character, but punctuation marks have no actual semantic meaning. Similarly, the code values ​​corresponding to some index values ​​with high statistical probabilities may only serve as placeholders and contain little or no style information. To improve the content and quality of style information in the index sequence, this disclosure reduces the decrease in style information content caused by high-frequency placeholders by adjusting the weights, enabling the style generator to generate more style-corresponding index sequences and improve style quality and diversity. This disclosure does not limit the high-frequency placeholder suppression method. For example, exponential suppression can be performed, that is, the first weight is set to the exponential function value of the corresponding statistical probability, and the second weight can be set to 1 to achieve significant suppression of index values ​​under high-frequency placeholder conditions. In addition, linear suppression can also be used, by setting a fixed ratio between the first weight and the second weight to gradually reduce the impact of high-frequency placeholders. Regardless of the specific method used, the core objective is to optimize the distribution density and expressive power of style information in the index sequence through dynamic adjustment of weights.

[0061] This disclosure does not limit the method for determining statistical probabilities. In an exemplary embodiment, a preset image set may first be obtained, which includes images of various styles; specifically, the preset image set may include as many images of different styles as possible.

[0062] Then, the index sequence corresponding to each image in the preset image set is determined. Specifically, visual features can be extracted from these images, and the extracted visual features are input into an encoder, which encodes the corresponding index sequence based on the style codebook. This encoder is also used to train the aforementioned decoder. Finally, the probability of each index value appearing in the index set formed by the index sequences is calculated, and the probability of appearance is used as the statistical probability corresponding to the index value.

[0063] Specifically, this involves determining the index sequence of a large number of images, thereby determining the probability of each index value appearing. Using this probability as a statistical probability, the statistical probability of each index value can be obtained scientifically, improving the reliability of high-frequency suppression.

[0064] A2. Based on the predicted probability corresponding to the index value and the reference weight corresponding to the index value, the recommendation probability corresponding to the index value is obtained.

[0065] The prediction probability indicates the likelihood of the corresponding index value predicted by the image style generator, while the reference weight indicates high-frequency suppression information. The product of the prediction probability corresponding to the index value and the reference weight corresponding to the index value can be used as the recommendation probability corresponding to the index value, so that the recommendation probability comprehensively balances the prediction results of the image style generator and the need for high-frequency suppression.

[0066] A3. The index value with the highest recommendation probability is determined as the index value corresponding to the next index number.

[0067] The index value with the highest recommendation probability is used as the index value for the next index number. This method effectively balances the importance of different index values ​​in the generation process, preventing certain high-frequency index values ​​from excessively dominating the generated results, while ensuring that low-frequency but important index values ​​receive appropriate attention. This approach not only improves the diversity of generated images but also enhances the consistency between the generated results and the target style, providing a more scientific and reliable basis for subsequent image generation tasks.

[0068] S250. Update the current index number with the next index number and continue to predict the index value until the prediction is completed.

[0069] By iteratively predicting the index values ​​corresponding to each index number, it is ensured that a unique index sequence can be generated based on the initial index value as a seed. If the user provides the same style code each time, the same index sequence can be obtained, ultimately generating images with the same style.

[0070] In one exemplary embodiment, after obtaining the target image, the method further includes: determining style description information, the style description information being used to describe the image style of the target image; recording the correspondence between the style code and the style description information; and displaying the correspondence between the style code and the style description information.

[0071] After obtaining the target image, its image style can be described by analyzing the image, and this style is recorded in text form, forming style description information. In this disclosure, once the style code is determined, the image style of the image generated based on the style code can also be uniquely determined, so the correspondence between the style code and the style description information can be recorded. This disclosure can also display the correspondence between the style code and the style description information. For example, when a user inputs a style code, the corresponding style description information can be displayed. Or, when a user wants to generate an image of a specific style, they can quickly locate the appropriate style code by querying existing style description information, thereby improving operational efficiency. In addition, this mechanism also supports the expansion and maintenance of the existing style library. When a new style is introduced, it is only necessary to generate the corresponding style description information and associate it with the new style code. By recording and displaying the correspondence between style codes and style description information, this disclosure not only helps users quickly understand and select appropriate style codes, but also provides a more intuitive reference for subsequent image generation. This method effectively reduces the time cost for users in style selection and improves the overall convenience of operation. Furthermore, by associating style description information with style codes, users can flexibly select style codes in different scenarios, thereby meeting diverse application needs. This design also has good scalability, allowing for the determination of more style description information and continuous expansion of the style library as the input style codes become more diverse. Next, this disclosure describes the training process of the style generator, decoder, and image generation model used for image generation, which is divided into three stages. In the first stage, the decoder is trained; please refer to... Figure 3 This diagram illustrates the decoder generation method provided in this disclosure. The decoder is trained using the following method: S310. Obtain multiple first sample images and the style label corresponding to each first sample image.

[0072] Each first sample image in this disclosure has a corresponding style tag. This disclosure does not limit the style tag; it indicates the style of the corresponding first sample image. For example, a style tag may indicate that the first sample image belongs to a certain art movement, such as Impressionism, Abstract Expressionism, or Realism. Furthermore, a style tag may also represent the tonal characteristics of the image, such as cool tones, warm tones, or a black and white style. Different first sample images have different content. Two first sample images with the same style tag can form a positive sample image pair, and two first sample images with different style tags can form a negative sample image pair.

[0073] S320. Obtain the encoder, the style codebook, and the decoder.

[0074] This disclosure does not limit the encoder and decoder; both basic encoder-decoder structures and derived variant decoder structures can be used. Examples include FSQ, IBQ, VQGAN, UniTok, and TokenFlow. FSQ is an efficient quantization model that reduces computational resource consumption while maintaining high-quality generation. IBQ focuses on improving information fidelity during encoding and decoding, making it particularly suitable for scenarios with high detail requirements. VQGAN combines the advantages of vector quantization and generative adversarial networks, generating high-resolution images with rich textures. UniTok simplifies the processing of multimodal data through a unified tokenization method, making it excellent in cross-domain tasks. TokenFlow, with its dynamically adjusted token stream at its core, can flexibly adapt to image generation needs of different styles and content, demonstrating strong versatility. These models each have their own characteristics, and the appropriate solution can be selected for application based on actual needs.

[0075] S330. For each of the first sample images, extract the first visual features corresponding to the first sample image, input the first visual features into the encoder, so that the encoder outputs the corresponding first prediction index sequence based on the style codebook; input the first prediction index sequence into the decoder, so that the decoder outputs the corresponding prediction style features.

[0076] This disclosure does not limit the method for extracting the first visual feature corresponding to the first sample image, as long as the visual feature extraction in each step of this disclosure uses the same method. For example, a pre-trained Visual Transformer (ViT) model can be used to extract the visual features of the input image. The pre-trained Visual Transformer (ViT) model is an open-source model trained on a large image dataset, capable of capturing rich semantic information in images. By dividing the input image into multiple small patches and feeding these patches as sequences into the Transformer, the model can effectively learn the relationship between global and local features and generate high-quality visual features. Transformer refers to a deep learning model based on a self-attention mechanism, which can effectively capture long-distance dependencies in sequential data. This model was originally designed for natural language processing tasks, but due to its powerful feature extraction capabilities, it is now widely used in the field of computer vision. By segmenting the image into small patches and treating them as input sequences, Transformer can model image features globally while preserving local detail information.

[0077] The first visual feature is input into the encoder, which outputs a corresponding first prediction index sequence based on the style codebook. The first prediction index sequence is then input into the decoder, so that the decoder outputs the corresponding predicted style feature.

[0078] S340. Construct positive sample pairs based on the predicted style features corresponding to two first sample images with the same style label; construct negative sample pairs based on the predicted style features corresponding to two first sample images with different style labels.

[0079] For example, the predicted style features of the two first sample images in a positive sample image pair are used as the positive sample pair, and the predicted style features of the two first sample images in a negative sample image pair are used as the negative sample pair.

[0080] S350. Based on the positive sample pairs and the negative sample pairs, perform comparative learning and adjust the parameters of the decoder.

[0081] To ensure effective differentiation of different styles, this disclosure uses contrastive loss for training. The purpose of contrastive loss is to constrain the parameters of the decoder so that for positive sample image pairs of the same style, the predicted style features after decoding should be as similar as possible; for negative sample image pairs of different styles, the predicted styles after decoding should be as dissimilar as possible. This disclosure does not limit the contrastive learning loss; the following formula can be used as an example:

[0082]

[0083] Where B is the batch size, yi∈{0,1} represents the label of the i-th sample pair (1 indicates the same style, 0 indicates different styles), and m is the boundary hyperparameter used to define the minimum separation distance required for negative sample pairs. and Let F represent the visual features of the two first sample images in the i-th sample pair, and let F() represent the decoder.

[0084] In some exemplary embodiments, a reconstruction loss may be added to prevent decoder crashes. This reconstruction loss can be used to maintain consistency between the predicted style features and the corresponding first visual features. This disclosure does not limit the reconstruction loss; for example, the following formula can be used, where N in the formula also represents the batch size:

[0085] In some exemplary implementations, quantization loss can also be incorporated. Quantization loss is a common loss in codebook training schemes. It refers to the information distortion or error caused by mapping continuous signals or high-precision data to a finite discrete set. This loss is often unavoidable because quantization is essentially an approximation method. Specific manifestations may include loss of signal details, increased noise, or decreased reconstruction quality. Many factors influence quantization loss, such as codebook size, the choice of quantization method, and the characteristics of the input data. Quantization loss is typically calculated based on error metrics, with the core idea being to quantize the difference between the original and quantized data using a mathematical model. Common calculation methods include mean squared error (MSE) and signal-to-noise ratio (SNR).

[0086] In one exemplary implementation, the training loss L can be determined based on the following formula: ,in, and This represents the weighting coefficient.

[0087] This disclosure allows for adjusting decoder parameters based on training loss. This disclosure does not limit the parameter tuning method; for example, gradient descent, stochastic gradient descent, etc., can be used. This disclosure does not limit the parameter tuning stopping condition; for example, a fixed number of iterations can be set, or the adjustment can be stopped when the change in training loss is less than a preset threshold.

[0088] The decoder training scheme in this disclosure, compared to traditional style extraction methods, chooses to use discrete representations to represent style, and then reconstructs style features from these discrete representations. This discrete process naturally loses some information, making it easier for the decoder to learn to retain style information while discarding content information during training. Furthermore, this discrete process is compatible with the autoregressive style generator described below, facilitating subsequent style creation. In other words, the decoder training process is coordinated with the style generator training process described below; the encoder and style codebook used in the decoder training also play a role in the style generator training process.

[0089] After the decoder training is complete, the second stage of training can begin. This stage can train either the image generation model or the style generator. In other words, the training order of the image generation model or the style generator is not limited, as long as they are trained after the decoder training is complete. Please refer to [link / reference]. Figure 4 This diagram illustrates the training process of the image generation model in this disclosure. After comparing and learning based on the positive and negative sample pairs and adjusting the parameters of the decoder, the image generation model is trained using the following method: S410. Obtain a second sample image and a third sample image, wherein the second sample image and the third sample image have different content but the same style.

[0090] The second and third sample images can belong to the aforementioned multiple first sample images, or they can be members of other image sets independent of the aforementioned multiple first sample images. This flexibility allows the training process to adapt to different data sources, thereby improving the model's generalization ability. Furthermore, when selecting the second and third sample images, it is necessary to ensure that they are consistent in style, but can differ significantly in content.

[0091] S420. Extract the second visual feature of the second sample image, input the second visual feature into the encoder so that the encoder outputs the corresponding sample index sequence based on the style codebook; input the sample index sequence into the decoder so that the decoder outputs the corresponding sample style feature.

[0092] In this model, the encoder and style codebook are used in the first stage. The encoder is trained in the first stage, so the visual feature extraction method used in the first stage is directly used to extract the second visual features of the second sample image. These second visual features are then input into the encoder, causing it to output a corresponding sample index sequence based on the style codebook. This sample index sequence is then input into the decoder, causing it to output the corresponding sample style features. Therefore, the training process of the image generation model requires the training results from the first stage.

[0093] S430. Input the sample style features and the text description corresponding to the third sample image into the image generation model to obtain the predicted image; train the image generation model based on the difference between the predicted image and the third sample image.

[0094] The text description refers to the content information of the third sample image. Since the third and second sample images share the same style, the style features corresponding to the second sample image and the text description of the third sample image are input into the image generation model. In this way, the model learns how to generate predicted images that closely resemble real images based on style features and content information. During training, the difference between the predicted image and the third sample image is calculated, and the parameters of the image generation model are optimized based on this difference, thereby improving the model's ability to jointly generate style and content. This combination ensures the accuracy of style usage and the consistency of content generation, further improving the model's generalization performance.

[0095] This disclosure does not limit the specific process of training the image generation model based on the difference between the predicted image and the third sample image. For example, the model parameters can be optimized by calculating the pixel-level difference between the two, or by combining perceptual loss and style loss. Perceptual loss measures the similarity between the predicted image and the third sample image in the high-level feature space, while style loss focuses on the degree of matching in style features. Furthermore, adversarial training mechanisms can be introduced to further enhance the realism and detail of the generated image using a discriminator. This multi-dimensional loss function design effectively guides the learning process of the image generation model, enabling it to achieve high-quality style generation while maintaining content consistency.

[0096] This disclosure does not limit the specific calculation method of the loss function; for example, mean squared error can be used to measure pixel-level differences. Style loss can be achieved by calculating the difference in the Gram matrix between the predicted image and the third sample image across multiple convolutional layers. In the adversarial training mechanism, the discriminator design can include a multi-scale feature extraction module to enhance the ability to discriminate the authenticity of the generated images.

[0097] This disclosure does not limit the structure of the image generation model. For example, the image generation model may use a multimodal large model as the encoder for text and style images, and a diffusion model as the text-to-image model. Alternatively, the structure of the image generation model may be a hybrid architecture combining an autoregressive generator and an attention mechanism to better capture local details and global consistency in the image.

[0098] This disclosure does not limit the training method of the image generation model. It allows for parameter tuning based on the loss corresponding to the difference between the predicted image and the third sample image. This disclosure does not limit the parameter tuning method or the stopping condition. For example, stochastic gradient descent or its improved version can be used for parameter optimization, and the convergence process can be accelerated by dynamically adjusting the learning rate. The stopping condition can be set to reach a predetermined number of iterations, or to terminate training when the change in the loss function value is less than a certain threshold.

[0099] By training an image generation model, an index sequence can be obtained that outputs based on a style codebook. The style features obtained by decoding this index sequence are then used, combined with textual descriptions, to generate an image generation model with a specific style. The image generation model produces images whose style highly matches the style features, exhibiting a significant advantage in high style accuracy.

[0100] After the second stage of training is completed, the third stage of training can begin. If the second stage trained the image generation model, the third stage can train the style generator. Conversely, if the second stage trained the style generator, the third stage can train the image generation model. In other words, the training order of the image generation model or the style generator is not limited, as long as both are trained after the decoder is complete. Please refer to [reference needed]. Figure 5 This diagram illustrates the training process of the style generator in this disclosure. After performing comparative learning based on the positive and negative sample pairs and adjusting the parameters of the decoder, the style generator is trained using the following method: S510. Obtain the fourth sample image.

[0101] To enhance the diversity of style generation in the style generator and improve its style generation capabilities and style imagination, this disclosure can use a large number of images with different styles as fourth sample images for training.

[0102] S520. Extract the third visual feature of the fourth sample image and input the third visual feature into the encoder so that the encoder outputs the corresponding reference index sequence based on the style codebook.

[0103] The method used to extract the third visual features of the fourth sample image, as well as the encoder, have been described previously and will not be repeated here. This reference index sequence can characterize the true representation of the style of the fourth sample image in the style codebook.

[0104] S530. Using the style generator, a second predicted index sequence is generated autoregressively based on the reference index sequence; the style generator is trained based on the difference between the second predicted index sequence and the reference index sequence.

[0105] The purpose of training the autoregressive style generator in this disclosure is to create styles. That is, only one style code is given to the autoregressive style generator, and an index sequence can be created autoregressively based on the style code. After the index sequence is decoded by the decoder, the style features are obtained. The style features are used to guide the image generator to generate images with the corresponding style.

[0106] This disclosure does not limit the structure of the style generator. For example, it can use the Qwen2 architecture as its basic framework. The Qwen2 architecture refers to an architecture containing multiple layers of neural networks, where each layer is responsible for processing specific feature representations. This structure allows the style generator to progressively learn complex styles during training. Alternatively, an autoregressive Transformer or other sequence generation models can be used as the basic architecture for the style generator. An autoregressive Transformer is a model that uses an autoregressive mechanism to process sequence data, generating a complete sequence by progressively predicting the next element in the sequence. This architecture performs well in handling style features because it can capture the dependencies between elements in the sequence and apply them to the style learning and generation process. Alternatively, the style generator can also be based on a diffusion model or a VAE (Variational Autoencoder), which generates new style features by learning the latent representations of the data.

[0107] The autoregressive style generator in this disclosure can heuristically and automatically create index sequences. Therefore, it essentially performs a heuristic automatic inference process. In an exemplary embodiment, the style generator is used to autoregressively generate a second predicted index sequence based on the reference index sequence; the style generator is trained based on the difference between the second predicted index sequence and the reference index sequence, including: S1. Initialize the current index number in the second predicted index sequence.

[0108] Assuming the index sequence is 20 in length, the autoregressive style generator can predict the corresponding index value for each sequence number from 0 to 19. During the initialization phase, the current index number is 0.

[0109] S2. Using the style generator, predict the predicted index value corresponding to the current index number; based on the difference between the predicted index value and the reference index value obtained by sampling the reference index sequence based on the current index number, determine the loss term corresponding to the current index number.

[0110] In one exemplary implementation, the step of using the style generator to predict the predicted index value corresponding to the current index number includes: when the current index number is not the first index number of the second predicted index sequence, sampling a reference subsequence based on the current index number in the reference index sequence, wherein the reference subsequence is a sequence formed by reference index values ​​whose index numbers are less than the current index number; and using the style generator to predict the predicted index value corresponding to the current index number based on the reference subsequence.

[0111] For example, if the current index number is 0, which is the first index number in the second predicted index sequence, then the information in the reference index sequence cannot be used, and the corresponding predicted index value can only be output directly. If the current index number is j, and j is greater than 0, then the current index number is not the first index number in the second predicted index sequence. In this case, the first j index values ​​can be extracted from the reference index sequence, i.e., the reference values ​​with index numbers from 0 to j-1, to obtain a reference subsequence. Based on this reference subsequence, the index value with index number j in the second predicted index sequence is predicted. This index value is the (j+1)th index value in the second predicted index sequence.

[0112] In the heuristic index prediction process disclosed in this invention, reference index values ​​from the reference index sequence are fully utilized. This reference index sequence reflects the true representation of the style of the fourth sample image in the style codebook. This eliminates the need to rely on heuristically predicted index values ​​during heuristic index prediction, avoiding the accumulation of prediction errors and effectively improving prediction accuracy. Since the reference index sequence originates directly from real data, the information it provides is more reliable, helping to improve the accuracy of the predicted index values ​​output by the style generator. This facilitates the style generator's efficient learning of the real index distribution, especially since the fourth sample image used in training the style generator exhibits diverse styles. This allows it to learn a scientific index distribution across diverse styles, enhancing the robustness of the style generator and enabling it to generate index sequences corresponding to diverse styles. It also explores its ability to generate index sequences corresponding to new styles.

[0113] In this disclosure, the loss term corresponding to the current index number can be determined based on the difference between the predicted index value and the reference index value obtained by sampling the reference index sequence based on the current index number. For example, if the predicted index value is M, the current index number is t, and the index value corresponding to index number t in the reference index sequence is N, then the representation of the difference between M and N is the loss term corresponding to index number t.

[0114] S3. Update the current index number using the next index number of the current index number, and repeatedly predict the predicted index value corresponding to the current index number and the loss term corresponding to the current index number until the current index number is the last index number of the second predicted index sequence.

[0115] S4. Train the style generator based on each of the loss terms.

[0116] This disclosure does not limit the specific method for training the style generator based on the various loss terms. For example, the total loss can be obtained by weighted summation of the various loss terms, and the parameters of the style generator can be adjusted using the total loss. This disclosure does not limit the weights and can be designed according to actual conditions. This disclosure does not limit the parameter tuning method or the stopping condition for parameter tuning. For example, gradient descent, stochastic gradient descent, or other optimization algorithms can be used to adjust the parameters of the style generator. The stopping condition for parameter tuning can be reaching a predetermined number of iterations, the loss value converging below a certain threshold, or the performance on the validation set no longer showing significant improvement.

[0117] The advantage of adjusting the style generator parameters based on the loss term corresponding to each index number in this disclosure is that it allows for fine-grained optimization of the prediction for each index number, thereby improving the style generator's performance at different index positions. This method not only helps reduce local errors but also enhances the model's ability to learn the overall index distribution. By incorporating the loss term corresponding to each index number into the training process, it ensures that the style generator more closely matches the distribution characteristics of the real data when generating index sequences. Furthermore, this approach provides the style generator with greater adaptability, resulting in higher-quality output.

[0118] Please refer to Figure 6 This diagram illustrates the implementation process of an image generation method in an exemplary embodiment of this disclosure. The specific implementation process of the image generation method is described in detail below with reference to this diagram.

[0119] In the first stage indicated by Figure (a), the decoder is trained. This stage can collect a large number of image pairs with style labels, i.e., each data set consists of two images with the same style but different content. During training, a pre-trained Visual Transformer (ViT) model is first used to extract deep features from the input images. Then, the ViT features are input into a discrete style codebook. The encoder of the discrete style codebook outputs a predicted index sequence, and the decoder outputs predicted style features. Positive and negative sample image pairs are constructed using the labels of the image pairs. Based on the positive and negative sample image pairs and the predicted style features corresponding to each image, comparison learning is performed to adjust the parameters of the decoder.

[0120] In the second stage indicated by Figure (a), the decoder parameters are fixed. This stage also collects a large number of image pairs with style labels, i.e., each data set consists of two images with the same style but different content. A multimodal large model is used as the encoder for both text and style images, and a diffusion model is used as the text-generated image model. During training, for each pair of images with the same style... and Using the Visual Transformer (ViT) model from Extracting its ViT features And obtain its corresponding style features through discrete style codebook. With this style characteristic and corresponding to Text prompts As a condition, a text-to-image diffusion model is trained with fixed parameters for the encoders of text and style images.

[0121] In the third stage, as shown in Figure (b), the parameters of the decoder, the encoders for text and images, and the diffusion model are fixed. This stage is trained using image datasets with distinct styles. Specifically, the Qwen2 architecture can be used as the basic framework for the style generator, with parameters randomly initialized from zero. For each training dataset, the encoder in the discrete style codebook obtains a pixel-level sequence of indices corresponding to the image. With the "next index" as the target, the distribution of these index sequences is heuristically learned to obtain the style generator. The indices generated by this style generator can be decoded by the decoder in the discrete style codebook to obtain the corresponding style features.

[0122] (c) The figure illustrates the image generation method implemented after all relevant models have been trained. In this method, the user can freely input a style code, which serves as a seed to guide the style generator to create an index sequence. This index sequence can be decoded into corresponding style features by the decoder. These style features, along with the text prompts provided by the user, are then input into a trained text and image encoder for encoding. The encoded result is then input into a diffusion model to obtain an image with a specific style.

[0123] In one specific implementation, the relevant explanations regarding the aforementioned training process and image generation method are as follows: Style codebook: Codebook size K=8192, sequence length N=192.

[0124] Style generator: Uses a Qwen2-0.5B as the model structure with a hidden layer dimension of 3584.

[0125] Diffusion model: based on Qwen architecture.

[0126] Inference parameters: The style code is a 32-bit integer.

[0127] How to use: Enter any number as the style code, and enter a text description as the prompt word. The system will generate an image that matches both the style code and the text description.

[0128] Figure 7 This is a block diagram of an image generation apparatus according to an exemplary embodiment. The apparatus includes: The content style acquisition module 710 is configured to acquire text prompt information and style code, wherein the text prompt information is used to indicate the image content of the target image and the style code is used to indicate the image style of the target image. The style generation module 720 is configured to perform the following actions: generating target index information based on the style code using a style generator; determining the target style quantization code corresponding to the style code based on the style codebook and the target index information; and decoding the target style quantization code using a decoder corresponding to the style codebook to obtain the target style feature. The image generation module 730 is configured to input the text prompt information and the target style features into the image generation model to obtain the target image.

[0129] In one exemplary embodiment, the style generation module 720 is configured to perform: Based on the style code, the style codebook is sampled to determine an initial index value, which is the first index value of the target index sequence; Using the style generator, the target index sequence is derived autoregressively based on the initial index value.

[0130] In one exemplary embodiment, the style generation module 720 is configured to perform: The initial index value is determined as the index value corresponding to the current index number, where the current index number points to the first index of the target index sequence; Input the index value corresponding to the current index number into the style generator to obtain the prediction distribution corresponding to the next index number; Based on the predicted distribution corresponding to the next index number, determine the index value corresponding to the next index number; Update the current index number using the next index number, and continue to predict the index value until the prediction is complete.

[0131] In one exemplary implementation, the prediction distribution includes multiple index values ​​and a prediction probability corresponding to each index value; the style generation module 720 is configured to execute: For each index value in the predicted distribution, if the statistical probability corresponding to the index value is higher than or equal to a preset threshold, the reference weight corresponding to the index value is determined as the first weight; if the statistical probability corresponding to the index value is lower than the preset threshold, the reference weight corresponding to the index value is determined as the second weight, wherein the first weight is less than the second weight. Based on the predicted probability corresponding to the index value and the reference weight corresponding to the index value, the recommendation probability corresponding to the index value is obtained; The index value with the highest recommendation probability is determined as the index value corresponding to the next index number; The statistical probability is used to indicate the statistical value of the probability that the corresponding index value is sampled.

[0132] In one exemplary embodiment, the style generation module 720 is configured to perform: Obtain a preset image set, which includes images of various styles; Determine the index sequence corresponding to each image in the preset image set; The probability of each index value appearing in the index set formed by the index sequences is calculated, and the probability of appearance is used as the statistical probability corresponding to the index value.

[0133] In one exemplary embodiment, the style generation module 720 is configured to perform: Obtain multiple first sample images and the style label corresponding to each first sample image; Obtain the encoder, the style codebook, and the decoder; For each first sample image, extract the first visual feature corresponding to the first sample image, input the first visual feature into the encoder, so that the encoder outputs the corresponding first prediction index sequence based on the style codebook; input the first prediction index sequence into the decoder, so that the decoder outputs the corresponding prediction style feature; Positive sample pairs are constructed based on the predicted style features corresponding to two first sample images with the same style label; negative sample pairs are constructed based on the predicted style features corresponding to two first sample images with different style labels. The parameters of the decoder are adjusted by comparing and learning from the positive and negative sample pairs.

[0134] In one exemplary embodiment, the style generation module 720 is configured to perform: Acquire a second sample image and a third sample image, wherein the second sample image and the third sample image have different content but the same style; Extract the second visual features from the second sample image, input the second visual features into the encoder, so that the encoder outputs the corresponding sample index sequence based on the style codebook; input the sample index sequence into the decoder, so that the decoder outputs the corresponding sample style features; The sample style features and the text description corresponding to the third sample image are input into the image generation model to obtain the predicted image; The image generation model is trained based on the difference between the predicted image and the third sample image.

[0135] In one exemplary embodiment, the style generation module 720 is configured to perform: Obtain the fourth sample image; The third visual feature of the fourth sample image is extracted and input into the encoder so that the encoder outputs the corresponding reference index sequence based on the style codebook. Using the style generator, a second predicted index sequence is generated autoregressively based on the reference index sequence; the style generator is trained based on the difference between the second predicted index sequence and the reference index sequence.

[0136] In one exemplary embodiment, the style generation module 720 is configured to perform: Initialize the current index number in the second predicted index sequence; Using the style generator, predict the predicted index value corresponding to the current index number; based on the difference between the predicted index value and the reference index value obtained by sampling the reference index sequence based on the current index number, determine the loss term corresponding to the current index number; Update the current index number using the next index number of the current index number, and repeatedly predict the predicted index value corresponding to the current index number and the loss term corresponding to the current index number until the current index number is the last index number of the second predicted index sequence; The style generator is trained based on each of the aforementioned loss terms.

[0137] In one exemplary embodiment, the style generation module 720 is configured to perform: If the current index number is not the first index number of the second predicted index sequence, a reference subsequence is sampled from the reference index sequence based on the current index number. The reference subsequence is a sequence formed by reference index values ​​whose index numbers are less than the current index number. Using the style generator, based on the reference subsequence, predict the predicted index value corresponding to the current index number.

[0138] In one exemplary embodiment, the content style acquisition module 710 is configured to perform: determining style description information, the style description information being used to describe the image style of the target image; recording the correspondence between the style code and the style description information; and displaying the correspondence between the style code and the style description information. Regarding the apparatus in the above embodiments, the specific manner of each step has been described in detail in the embodiments of the foregoing method, and will not be elaborated here.

[0139] Please refer to Figure 8 It illustrates the structural block of a computer device provided in an exemplary embodiment of this disclosure. Figure 1 The computer device may be a terminal. This computer device is used to implement the image generation method provided in the above embodiments. Specifically: Typically, computer device 800 includes a processor 801 and a memory 802.

[0140] Processor 801 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In an exemplary embodiment, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In an exemplary embodiment, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0141] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In an exemplary embodiment, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one instruction, at least one program, code set, or instruction set, configured to be executed by one or more processors to implement the image generation method described above.

[0142] In one exemplary embodiment, the computer device 800 may optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 803 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 804, a touch display screen 805, a camera assembly 806, an audio circuit 807, a positioning assembly 808, and a power supply 809.

[0143] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the computer device 800, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0144] Please refer to Figure 9 It illustrates the structural block of a computer device provided in another exemplary embodiment of this disclosure. Figure 2 The computer device can be a server for executing the image generation method described above. Specifically: Computer device 900 includes a Central Processing Unit (CPU) 901, a system memory 904 including Random Access Memory (RAM) 902 and Read Only Memory (ROM) 903, and a system bus 905 connecting the system memory 904 and the CPU 901. Computer device 900 also includes a basic input / output system (I / O system) 906 that facilitates information transfer between various devices within the computer, and a mass storage device 907 for storing the operating system 913, application programs 914, and other program modules 911.

[0145] The basic input / output system 906 includes a display 908 for displaying information and an input device 909 for user input, such as a mouse or keyboard. Both the display 908 and the input device 909 are connected to the central processing unit 901 via an input / output controller 190 connected to the system bus 905. The basic input / output system 906 may also include the input / output controller 190 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 190 also provides output to a display screen, printer, or other types of output devices.

[0146] Mass storage device 907 is connected to central processing unit 901 via a mass storage controller (not shown) connected to system bus 905. Mass storage device 907 and its associated computer-readable media provide non-volatile storage for computer device 900. That is, mass storage device 907 may include computer-readable media (not shown) such as hard disk or CD-ROM (CompactDisc Read-Only Memory) drive.

[0147] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 904 and mass storage device 907 described above can be collectively referred to as memory.

[0148] According to various embodiments of this disclosure, the computer device 900 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 900 can be connected to a network 912 via a network interface unit 911 connected to a system bus 905, or the network interface unit 911 can be used to connect to other types of networks or remote computer systems (not shown).

[0149] The aforementioned memory also includes a computer program stored in the memory and configured to be executed by one or more processors to implement the aforementioned image generation method.

[0150] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is executed by a processor to implement the image generation method described above.

[0151] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0152] In an exemplary embodiment, a computer-readable storage medium including program code is also provided, such as a memory including program code, which can be executed by a processor to perform the image generation method described above. Optionally, the computer-readable storage medium may be read-only memory (ROM), random access memory (RAM), compact-disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0153] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the image generation method described above.

[0154] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0155] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image generation method, characterized in that, The method includes: Obtain text prompt information and style code, wherein the text prompt information is used to indicate the image content of the target image and the style code is used to indicate the image style of the target image; Using a style generator, target index information is generated based on the style code; and based on the style codebook and the target index information, the target style quantization code corresponding to the style code is determined. The target style quantization encoding is decoded using the decoder corresponding to the style codebook to obtain the target style features; The text prompt information and the target style features are input into the image generation model to obtain the target image.

2. The method according to claim 1, characterized in that, The target index information is represented by a target index sequence, and the generation of target index information based on the style code by the style generator includes: Based on the style code, the style codebook is sampled to determine an initial index value, which is the first index value of the target index sequence; Using the style generator, the target index sequence is derived autoregressively based on the initial index value.

3. The method according to claim 2, characterized in that, The step of using the style generator to derive the target index sequence autoregressively based on the initial index value includes: The initial index value is determined as the index value corresponding to the current index number, where the current index number points to the first index of the target index sequence; Input the index value corresponding to the current index number into the style generator to obtain the prediction distribution corresponding to the next index number; Based on the predicted distribution corresponding to the next index number, determine the index value corresponding to the next index number; Update the current index number using the next index number, and continue to predict the index value until the prediction is complete.

4. The method according to claim 3, characterized in that, The prediction distribution includes multiple index values ​​and a prediction probability corresponding to each index value; determining the index value corresponding to the next index number based on the prediction distribution corresponding to the next index number includes: For each index value in the predicted distribution, if the statistical probability corresponding to the index value is higher than or equal to a preset threshold, the reference weight corresponding to the index value is determined as the first weight; if the statistical probability corresponding to the index value is lower than the preset threshold, the reference weight corresponding to the index value is determined as the second weight, wherein the first weight is less than the second weight. Based on the predicted probability corresponding to the index value and the reference weight corresponding to the index value, the recommendation probability corresponding to the index value is obtained; The index value with the highest recommendation probability is determined as the index value corresponding to the next index number; The statistical probability is used to indicate the statistical value of the probability that the corresponding index value is sampled.

5. The method according to claim 4, characterized in that, The method further includes: Obtain a preset image set, which includes images of various styles; Determine the index sequence corresponding to each image in the preset image set; The probability of each index value appearing in the index set formed by the index sequences is calculated, and the probability of appearance is used as the statistical probability corresponding to the index value.

6. The method according to claim 1, characterized in that, The decoder is trained using the following method: Obtain multiple first sample images and the style label corresponding to each first sample image; Obtain the encoder, the style codebook, and the decoder; For each first sample image, extract the first visual feature corresponding to the first sample image, input the first visual feature into the encoder, so that the encoder outputs the corresponding first prediction index sequence based on the style codebook; input the first prediction index sequence into the decoder, so that the decoder outputs the corresponding prediction style feature; Positive sample pairs are constructed based on the predicted style features corresponding to two first sample images with the same style label; negative sample pairs are constructed based on the predicted style features corresponding to two first sample images with different style labels. The parameters of the decoder are adjusted by comparing and learning from the positive and negative sample pairs.

7. The method according to claim 6, characterized in that, After performing comparative learning based on the positive and negative sample pairs and adjusting the parameters of the decoder, the image generation model is trained using the following method: Acquire a second sample image and a third sample image, wherein the second sample image and the third sample image have different content but the same style; Extract the second visual features from the second sample image, and input the second visual features into the encoder so that the encoder outputs the corresponding sample index sequence based on the style codebook; The sample index sequence is input into the decoder so that the decoder outputs the corresponding sample style features; The sample style features and the text description corresponding to the third sample image are input into the image generation model to obtain the predicted image; The image generation model is trained based on the difference between the predicted image and the third sample image.

8. The method according to claim 6 or 7, characterized in that, After performing comparative learning based on the positive and negative sample pairs and adjusting the parameters of the decoder, the style generator is trained using the following method: Obtain the fourth sample image; The third visual feature of the fourth sample image is extracted and input into the encoder so that the encoder outputs the corresponding reference index sequence based on the style codebook. Using the style generator, a second predicted index sequence is generated autoregressively based on the reference index sequence; The style generator is trained based on the difference between the second predicted index sequence and the reference index sequence.

9. The method according to claim 8, characterized in that, The style generator is used to generate a second predictive index sequence based on the reference index sequence via autoregression. Training the style generator based on the difference between the second predicted index sequence and the reference index sequence includes: Initialize the current index number in the second predicted index sequence; Using the style generator, predict the predicted index value corresponding to the current index number; based on the difference between the predicted index value and the reference index value obtained by sampling the reference index sequence based on the current index number, determine the loss term corresponding to the current index number; Update the current index number using the next index number of the current index number, and repeatedly predict the predicted index value corresponding to the current index number and the loss term corresponding to the current index number until the current index number is the last index number of the second predicted index sequence; The style generator is trained based on each of the aforementioned loss terms.

10. The method according to claim 9, characterized in that, The step of using the style generator to predict the predicted index value corresponding to the current index number includes: If the current index number is not the first index number of the second predicted index sequence, a reference subsequence is sampled from the reference index sequence based on the current index number. The reference subsequence is a sequence formed by reference index values ​​whose index numbers are less than the current index number. Using the style generator, based on the reference subsequence, predict the predicted index value corresponding to the current index number.

11. The method according to claim 1, characterized in that, After obtaining the target image, the method further includes: Determine style description information, which is used to describe the image style of the target image; Record the correspondence between the style code and the style description information; This demonstrates the correspondence between the style code and the style description information.

12. An image generation apparatus, characterized in that, The device includes: The content style acquisition module is configured to acquire text prompt information and style code, wherein the text prompt information is used to indicate the image content of the target image, and the style code is used to indicate the image style of the target image. The style generation module is configured to execute the following steps: generating target index information based on the style code using a style generator; determining the target style quantization code corresponding to the style code based on the style codebook and the target index information; and decoding the target style quantization code using a decoder corresponding to the style codebook to obtain the target style feature. The image generation module is configured to input the text prompt information and the target style features into the image generation model to obtain the target image.

13. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the image generation method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device performs the image generation method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes a computer program stored in a readable storage medium, wherein at least one processor of a computer device reads from and executes the computer program, causing the device to perform the image generation method as described in any one of claims 1 to 11.

Images