Image generation methods, electronic devices, storage media and program products

Abstract

This application discloses an image generation method, electronic device, storage medium, and program product, relating to the field of image generation technology. For any time step in the process of generating a target image using a preset image generation model, the method involves: acquiring historical tokens generated before that time step, and the target token corresponding to that time step; filtering key tokens from the historical tokens based on their attention weights; combining the key tokens and the target tokens corresponding to the time steps to form a token set; determining the target token for the next time step using the token set; and generating the target token for each time step, then generating the target image based on the target tokens for each time step. This application reduces caching and redundant computation by filtering key tokens, thus accelerating the inference process of image generation.

Description

Technical Field

[0001] This application relates to the field of image generation technology, and in particular to image generation methods, electronic devices, storage media, and program products. Background Technology

[0002] Currently, autoregressive models have become one of the important pathways for high-quality image generation due to their powerful sequence modeling capabilities.

[0003] However, its core mechanism of step-by-step decoding also introduces inherent inference efficiency defects. Specifically, each time the autoregressive model generates a new token, it needs to traverse and calculate its correlation with all historical tokens, resulting in a linear growth relationship in the generation of key-value caches. This further leads to the high complexity of generating images through computation, which in turn slows down the inference process of image generation.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide an image generation method, electronic device, storage medium, and program product, aiming to solve the technical problem of how to accelerate the inference process of image generation.

[0006] To achieve the above objectives, this application proposes an image generation method, the method comprising:

[0007] For any time step in the process of generating a target image using a preset image generation model, obtain the historical tokens generated before the time step and the target token corresponding to the time step;

[0008] Based on the attention weight of the historical tokens, key tokens are selected from the historical tokens, wherein the higher the attention weight of the historical token, the greater the probability that it will be selected as the key token;

[0009] Combine the key token and the target token corresponding to the time step to form a token set;

[0010] The target token for the next time step is determined using the set of tokens.

[0011] After generating the target token for each time step, the target image is generated based on the target token for each time step.

[0012] In one feasible embodiment, the step of filtering key tokens from the historical tokens based on the attention weights of the historical tokens includes:

[0013] If the total cache space of the historical tokens is greater than or equal to a preset cache space threshold, calculate the attention weight of the historical tokens.

[0014] Based on the attention weight, a key token is determined from the historical tokens, wherein the overall cache space of the key token is less than the preset cache space threshold, and the attention weight of the key token is greater than the attention weight of the non-key tokens in the historical tokens.

[0015] In one feasible embodiment, the step of determining the target token for the next time step using the token set includes:

[0016] Based on the historical tokens in the token set and the target token corresponding to the time step, the probability of each token in the preset library is determined by the preset image generation model;

[0017] Select the token with the highest probability from the preset library as the target token for the next time step.

[0018] In one feasible embodiment, the preset image generation model includes multiple transformer blocks, and before the step of filtering key tokens from the historical tokens based on the attention weights of the historical tokens, it further includes:

[0019] By using multiple transformer blocks in both conditional and unconditional paths, the hidden state corresponding to the historical token and the target token at the time step is updated. The conditional path refers to using preset text as input to the multiple transformer blocks.

[0020] In a feasible embodiment, the step of updating the hidden state corresponding to the historical token and the target token at the time step through multiple transformer blocks in conditional and unconditional paths includes:

[0021] The conditional distance is obtained by using the hidden state of each transformer block in the conditional path;

[0022] The unconditional distance is obtained by using the hidden state of each transformer block in the unconditional path;

[0023] The target computation strategy for each transformer block is determined by comparing the conditional distance and unconditional distance with a preset distance threshold.

[0024] The hidden states corresponding to the historical token and the target token at the time step are updated using the preset image generation model and the target calculation strategy.

[0025] In a feasible embodiment, the step of determining the target computation strategy for each transformer block based on a comparison of the conditional distance and the unconditional distance with a preset distance threshold includes:

[0026] When both the conditional distance and the unconditional distance are greater than the preset distance threshold, a first target calculation strategy is executed, wherein the first target calculation strategy is the transformer block performing the calculation process of the conditional path and the unconditional path;

[0027] When the conditional distance is greater than or equal to the preset distance threshold, and the unconditional distance is less than or equal to the preset distance threshold, a second target calculation strategy is executed, wherein the second target calculation strategy is the process of the transformer block executing the conditional path calculation.

[0028] If both the conditional distance and the unconditional distance are less than the preset distance threshold, a third target calculation strategy is executed, wherein the third target calculation strategy is that the transformer block does not execute the calculation process of the conditional path and the unconditional path.

[0029] In a feasible embodiment, after the step of updating the hidden state corresponding to the historical token using the preset image generation model and the target calculation strategy, the method further includes:

[0030] Based on the generation quality assessment index at the current time step, the preset distance threshold is dynamically adjusted, wherein the generation quality assessment index is obtained by calculating the coherence score of the generated token sequence.

[0031] When the generation quality evaluation index of a consecutive preset number of time steps is detected to be lower than a preset quality threshold, the preset distance threshold is reduced.

[0032] Furthermore, to achieve the above objectives, this application also proposes an image generation apparatus, the image generation apparatus comprising:

[0033] The token acquisition module is used to acquire, at any time step in the process of generating a target image by a preset image generation model, the historical tokens generated before the time step and the target token corresponding to the time step.

[0034] The token filtering module is used to filter key tokens from the historical tokens based on the attention weight of the historical tokens, wherein the higher the attention weight of the historical token, the greater the probability that the historical token will be filtered as the key token;

[0035] The token combination module is used to combine the key token and the target token corresponding to the time step to form a token set;

[0036] The target token generation module is used to determine the target token for the next time step using the token set.

[0037] An image generation module is used to generate the target image based on the target token at each time step after generating the target token at each time step.

[0038] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image generation method as described above.

[0039] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the image generation method described above.

[0040] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the image generation method described above.

[0041] The one or more technical solutions proposed in this application have at least the following technical effects: First, for any time step in the image generation process of the preset image generation model, the historical tokens generated before the time step and the target token corresponding to the time step are obtained for subsequent filtering and image generation; then, based on the attention weight of the historical tokens, key tokens are filtered from the historical tokens, wherein the higher the attention weight of the historical token, the greater the probability of it being filtered as a key token; the key tokens and the current token are combined to form a token set, avoiding the use of all historical tokens to generate images, but instead filtering the set of key tokens that are highly related to image generation through attention calculation, reducing the linear growth of the key-value cache corresponding to the tokens; then, the target token for the next time step is determined through the token set; after generating the target token for each time step, the target image is generated based on the target token for each time step, reducing the tokens involved in the calculation and image generation from all historical tokens to the token set, greatly reducing the amount of computation. This application ensures that the preset image generation model can maintain high-quality image output while accelerating the inference process by screening key tokens and actively selecting the tokens that have the greatest impact on subsequent generation and have the highest relevance. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating an embodiment of the image generation method of this application.

[0045] Figure 2 An illustration of attention map symbolization provided for an embodiment of this application. Figure 1 ;

[0046] Figure 3 An illustration of attention map symbolization provided for an embodiment of this application. Figure 2 ;

[0047] Figure 4 A visualization of attention weights provided in one embodiment of this application;

[0048] Figure 5 This is a schematic diagram of the module structure of the image generation device according to an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the image generation method in the embodiments of this application.

[0050] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0052] It should be noted that in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0053] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The current autoregressive model's core mechanism of stepwise decoding introduces inherent inference efficiency defects. Specifically, each time an autoregressive model generates a new token, it needs to traverse and calculate its correlation with all historical tokens, resulting in a linear growth relationship in the generation of key-value caches. This further increases the complexity of generating images, leading to a slow inference process for image generation.

[0055] This application provides a solution. First, for any time step in the image generation process of a preset image generation model, it acquires historical tokens generated before that time step and the target token corresponding to that time step, for subsequent filtering and image generation. Then, based on the attention weights of the historical tokens, it filters key tokens from the historical tokens, where a higher attention weight increases the probability of a historical token being selected as a key token. The key tokens are combined with the current token to form a token set. This avoids using all historical tokens to generate images; instead, it uses attention calculation to filter a set of key tokens highly relevant to image generation, reducing the linear growth of the key-value cache corresponding to the tokens. Then, the target token for the next time step is determined using the token set. After generating the target token for each time step, the target image is generated based on the target token for each time step. This reduces the number of tokens involved in the calculation and image generation from all historical tokens to the token set, significantly reducing the computational load. By filtering key tokens and actively selecting the tokens with the greatest impact and highest relevance on subsequent generation, this application ensures that the preset image generation model can maintain high-quality image output while accelerating the inference process.

[0056] It should be noted that the executing entity in this embodiment can be an electronic device with data processing, network communication and program execution functions, such as a tablet computer, personal computer, mobile phone, etc.

[0057] Based on this, embodiments of this application provide an image generation method, referring to... Figure 1 , Figure 1 This is a schematic flowchart of the first embodiment of the image generation method of this application.

[0058] In this embodiment, the image generation method includes steps S10 to S50:

[0059] Step S10: For any time step in the process of generating the target image by the preset image generation model, obtain the historical tokens generated before the time step and the target token corresponding to the time step.

[0060] A pre-trained image generation model is a model that generates images in advance. For example, an autoregressive model is trained on a large amount of image data to learn the features and patterns of images and can generate corresponding images based on the input information.

[0061] The target image refers to the image that is expected to be generated by the preset image generation model based on the output text.

[0062] A time step refers to the stage in the image generation process where a pre-defined image generation model gradually generates each image block.

[0063] Historical tokens refer to the sequence of tokens generated before the current time step.

[0064] A token is a basic unit of an image, which can be a pixel block, a latent vector fragment, or a discrete code.

[0065] Optionally, during the image generation process, the image is processed by an encoder to obtain a series of image blocks. After processing the vectors corresponding to each image block, the index of the corresponding vector is found in a preset library and used as the token of that image block.

[0066] Optionally, in VQ-VAE (Vector Quantized Variational Autoencoder), a token corresponds to the feature representation of a 32×32 pixel region.

[0067] At any given time step, we can obtain the historical tokens generated by the preset image generation module before that time step and the target token corresponding to that time step. The target token is also obtained based on the tokens from the previous time steps.

[0068] Step S20: Based on the attention weight of historical tokens, filter key tokens from historical tokens, where the higher the attention weight of a historical token, the greater the probability that it will be selected as a key token.

[0069] Attention weights are an important part of the attention mechanism in deep learning, helping the model focus on the input sequence. Each token has a corresponding attention weight, reflecting its importance in the current task. The higher the weight, the more important the token is to the target token in the next time step.

[0070] A key token is a token selected from historical tokens that is more important to the generation process of the target token in the next time step.

[0071] The number of historical input tokens can be very large, containing a lot of information that is not important to the task. By filtering out key tokens, data redundancy can be reduced, and the processing efficiency of the preset image generation model can be improved.

[0072] In one feasible implementation, step S20, which involves filtering key tokens from historical tokens based on their attention weights, includes:

[0073] Step S21: If the total cache space of historical tokens is greater than or equal to a preset cache space threshold, calculate the attention weight of historical tokens.

[0074] In scenarios involving historical token processing, historical tokens are typically stored in a cache. A preset cache space threshold is a pre-defined value used to limit the space occupied by historical tokens in the cache. When the total cache space for historical tokens is greater than or equal to the preset cache space threshold, it means that the number of historical tokens in the current cache is too large, which may affect system performance or processing efficiency. In this case, it is necessary to filter these historical tokens. Therefore, in this situation, it is necessary to calculate the attention weight for each historical token.

[0075] Step S22: Based on the attention weight, determine the key token from the historical tokens, wherein the overall cache space of the key token is less than the preset cache space threshold, and the attention weight of the key token is greater than the attention weight of the non-key tokens in the historical tokens.

[0076] After obtaining the attention weights of historical tokens, a subset needs to be selected as key tokens. Key tokens must meet two conditions: the overall cache space of key tokens must be less than a preset cache space threshold. This ensures that the selected token set does not exceed the cache space limit, avoiding a recurrence of insufficient cache space. Additionally, the attention weight of key tokens must be greater than the attention weight of non-key tokens among the historical tokens, prioritizing tokens that are more important in the current task as key tokens.

[0077] For example, the hidden state, maximum length, and historical key-value pairs are input into the preset image generation model. The hidden state of the current step is multiplied by three learnable weight matrices to obtain the query vector, key vector, and value vector of the current step. Here, the hidden state represents the intermediate computation result of the preset image generation model at the current time step, containing sequence information up to the current time step; the maximum length is a parameter set by the preset image generation model, representing a preset cache space threshold, i.e., the upper limit of the number of key tokens that can be obtained; the historical keys and values ​​refer to the key vectors and value vectors calculated in all previous steps during the generation process to save computation. A filtering process is only performed when the accumulated number of historical key-value pairs exceeds the maximum length. If the maximum length is not exceeded, it means that there is not much historical information, and all historical information is used directly.

[0078] In attention mechanisms, tokens are typically transformed into corresponding keys and values ​​for processing. Specifically, each token is mapped to a key vector and a value vector. The key is used to calculate the attention weight. By comparing the key vectors corresponding to different tokens, the relevance between them and other tokens can be determined. For example, in the Transformer architecture, the key vector of each token is multiplied by the query vector, and the result, after a series of processing steps, becomes the attention weight, reflecting the importance of the token in the current context. The value contains the actual information of the token. After calculating the attention weight, the value vectors are weighted and summed according to the weight to obtain the context representation of the current token.

[0079] Record the attention weight of each historical token; the higher the weight, the more important the historical token. Using a function, select the index of the historical token with the highest score among its attention weights, minus the maximum number of tokens. The maximum number is the number of tokens that can be stored within a preset space threshold, and also the maximum length of the key vector corresponding to that token. Based on the calculated index, extract only the keys and values ​​corresponding to that index from the complete set of historical keys and values. This completes information compression, retaining only the most crucial information corresponding to the maximum number of tokens minus one historical token.

[0080] Optionally, the key-value pairs corresponding to the filtered historical tokens (minus one) are concatenated with the key and value of the target token in the current step to form a new key-value set with the maximum number of pairs, which is then retained. The preset image generation model can output the result calculated using approximate attention weights and the merged value vector, as well as the updated key-value cache and score, for calculation in the next time step.

[0081] This embodiment calculates attention weights and filters tokens when the total cache space of historical tokens exceeds a preset threshold, thus preventing an excessive number of historical tokens from impacting system performance and processing efficiency. The total cache space of the filtered key tokens is less than the threshold, preventing insufficient cache space. Key tokens have higher attention weights, prioritizing the retention of tokens more important to the current task, achieving information compression. For example, in a preset image generation model, by subtracting one historical token index from the highest-scoring maximum quantity, the corresponding key and value are extracted, completing information compression. This information can also be concatenated with newly generated keys and values ​​in the current step, providing efficient and accurate data for subsequent calculations and improving overall processing performance.

[0082] For example, refer to Figure 2 , Figure 2Each cell represents the attention weight of a query token to a key token; the darker the color, the higher the weight. Figure 2 This represents a symbolic representation of the attention graph in the dense attention mechanism, with the range corresponding to the triangular region in the lower left corner. At any time step, all generated tokens (the colored cells in the triangular region in the lower left corner, all cells are colored) need to participate in the attention calculation process, which involves a large amount of computation.

[0083] Reference Figure 3 , Figure 3 This diagram illustrates the attention graph symbolization for this embodiment, with the area corresponding to the triangular region in the lower left corner. Only a portion of the region (i.e., cells with displayed color) participates in the calculation, indicating that only the selected key tokens and the token corresponding to the current time step participate in the attention calculation; the remaining tokens are skipped or discarded. This reduces the number of tokens participating in the attention calculation, significantly reducing computational load and GPU memory usage while preserving core information.

[0084] For example, refer to Figure 4 Attention weight visualization is a graphical representation method used to intuitively show the attention distribution among different elements in the attention mechanism. Each cell represents an attention weight, and the darker the color, the higher the weight. Figure 4 Figure A in the middle is a visualization of the attention weights in the first layer of the LlamaGen-l256 model (before softmax). Figure 4 Figure B in the middle is a visualization of the attention weights in layer 19 of the LlamaGen-l256 model (before softmax). Figure 4 This indicates that tokens at adjacent time steps are highly correlated in terms of attention weights. Some tokens contribute very highly to the output, while others contribute less, suggesting that the autoregressive generation process of repeatedly using all previous tokens at each step involves redundant computation. Tokens with darker colors in the graph have a higher probability of being selected as key tokens.

[0085] Step S30: Combine the key token and the target token corresponding to the time step to form a token set;

[0086] A token set is a new sequence of tokens formed by combining selected key tokens with the target token for the current time step. The token set contains information that significantly impacts the generation of the next time step.

[0087] In one feasible implementation, the key token and the target token are arranged sequentially to form a token set.

[0088] Step S40: Determine the target token for the next time step using the token set;

[0089] Each token in the token set is a highly relevant token that has a significant impact on the target token for the next time step. By determining the target token for the next time step using tokens, the impact on the quality of the generated image can be controlled within a manageable range, accelerating the inference process of image generation without compromising the quality of the generated image.

[0090] In one feasible implementation, step S40, which determines the target token for the next time step using the token set, includes:

[0091] Step S41: Based on the historical tokens and the target tokens corresponding to the time steps in the token set, determine the probability of each token in the preset library through a preset image generation model;

[0092] Historical tokens record information generated at previous time steps, including contextual information during image generation; the target token at the current time step reflects the portion of content already generated at that moment. Based on these, the preset image generation model analyzes and calculates the probability of each token in the preset library. The preset library is a predefined set containing all possible tokens. The model uses its own algorithms and structures (such as forward propagation in a neural network) to calculate the probability of each token in the preset library being selected as the next token in the current situation.

[0093] Step S42: Select the token with the highest probability in the preset library as the target token for the next time step.

[0094] The preset image generation model selects the token with the highest probability from the preset library and determines it as the target token for the next time step. This is achieved through a greedy strategy. By default, the token with the highest probability is the most reasonable and likely token to appear in the current context. By continuously repeating this process, the preset image generation model can gradually generate a series of target tokens, which are ultimately used to generate the target image.

[0095] Step S50: After generating the target token for each time step, generate the target image based on the target token for each time step.

[0096] By combining the target tokens from all time steps and performing certain decoding and transformation operations, the final target image is obtained.

[0097] This embodiment first obtains historical tokens generated before this time step and the target token corresponding to this time step for any time step in the image generation process of the preset image generation model, which are used for subsequent filtering and image generation. Then, based on the attention weight of the historical tokens, key tokens are filtered from the historical tokens, where the higher the attention weight of the historical token, the greater the probability of it being filtered as a key token. The key tokens are combined with the current token to form a token set, avoiding the use of all historical tokens to generate images. Instead, attention calculation is used to filter out a set of key tokens that are highly relevant to image generation, reducing the linear growth of the key-value cache corresponding to the tokens. Then, the target token for the next time step is determined through the token set. After generating the target token for each time step, the target image is generated based on the target token for each time step. This reduces the number of tokens involved in the calculation and image generation from all historical tokens to the token set, significantly reducing the amount of computation. By filtering key tokens and actively selecting the tokens with the greatest impact and highest relevance on subsequent generation, this application ensures that the preset image generation model can maintain high-quality image output while accelerating the inference process.

[0098] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, the preset image generation model includes multiple transformer blocks. Before step S20, which filters key tokens from historical tokens according to the attention weights of historical tokens, the following method is also included:

[0099] Step A10 involves updating the hidden state corresponding to the historical token through multiple transformer blocks in both conditional and unconditional paths. The conditional path refers to using preset text as input to guide multiple transformer blocks, while the hidden state is used to calculate the attention weight of the historical token.

[0100] Transformer blocks have powerful feature extraction and information processing capabilities, and are used to handle global dependencies in sequence data.

[0101] The conditional path takes pre-defined text as input to multiple transformer blocks to guide the model's processing. This pre-defined text contains semantic information related to the target image. By inputting it into the transformer blocks, the model can combine this textual information to update the hidden states corresponding to historical tokens, allowing the hidden states to better reflect the semantic requirements of the image generation task. The pre-defined text needs to be encoded as an embedding vector as a guiding signal.

[0102] The hidden state records intermediate information about historical tokens after they have been processed by the model. This hidden state will be used to calculate the attention weights for subsequent historical tokens.

[0103] Conditional path refers to receiving text embeddings and the hidden state of historical tokens as input, and fusing text and image information through cross-attention.

[0104] An unconditional path refers to performing self-attention computation with the hidden state of the historical token as input.

[0105] In a feasible embodiment, step A10, which updates the hidden state corresponding to the historical token through multiple transformer blocks in both conditional and unconditional paths, includes:

[0106] Step A11: Obtain the conditional distance by examining the hidden state of each transformer block in the conditional path;

[0107] Step A12: Obtain the unconditional distance by using the hidden state of each transformer block in the unconditional path;

[0108] Step A13: Determine the target computation strategy for each transformer block by comparing the conditional distance and unconditional distance with the preset distance threshold;

[0109] Step A14: Update the hidden state corresponding to the historical token by using a preset image generation model and target calculation strategy.

[0110] Each time step in the image generation process requires processing the hidden state through multiple modules. This processing involves calculations along two paths: conditional and unconditional. In the conditional path, guided by the embedding vector corresponding to the preset text, the hidden state is calculated to obtain the conditional distance. For example, the conditional distance can be Euclidean distance, measuring the norm or magnitude of change of the hidden state vector within the conditional path. A larger conditional distance indicates a more drastic change in the current hidden state value, potentially containing more information, thus increasing the importance of the conditional calculation in this module. In the unconditional path, when the embedding vector is empty, the unconditional distance is calculated based on the hidden state within the unconditional path. For example, the unconditional distance can be Euclidean distance, measuring the norm or magnitude of change of the hidden state vector within the unconditional path. Similarly, a more drastic change in the current hidden state value, potentially containing more information, also increases the importance of the unconditional calculation in this module.

[0111] The preset distance threshold is a pre-set value used to determine whether conditional paths and unconditional paths need to be skipped.

[0112] The target calculation strategy refers to determining the time steps and modules of the conditional and unconditional paths to be executed based on a preset distance threshold.

[0113] Optionally, after determining the target computation strategy, the preset image generation model will dynamically generate a strategy list to indicate the time steps and modules that can be skipped in the computation process.

[0114] The pre-defined image generation model performs the computation process according to the target computation strategy, updating the hidden state of historical tokens. After multiple modules, the hidden state of historical tokens is updated, incorporating new contextual information and higher-level features for subsequent image generation.

[0115] In this embodiment, the hidden states under conditional and unconditional paths are calculated separately for each transformer block, yielding conditional and unconditional distances. These distances quantify the magnitude of change in the hidden state vector; a larger distance indicates a more drastic state change, richer information content, and higher importance for calculating that path. Next, the conditional and unconditional distances are compared with preset distance thresholds. Based on the comparison results, a target computation strategy for each transformer block is determined, thus avoiding redundant processing. Then, the preset image generation model dynamically generates a strategy list based on the target computation strategy, indicating which time steps and modules can skip computation and only perform necessary path updates to the hidden states. This embodiment achieves intelligent allocation of computational resources through strategy selection based on preset distance thresholds, making it suitable for large-scale image generation tasks and accelerating the inference speed of image generation.

[0116] In one feasible implementation, step A13, which determines the target computation strategy for each transformer block based on a comparison of conditional and unconditional distances with a preset distance threshold, includes:

[0117] Step A131: When both the conditional distance and the unconditional distance are greater than a preset distance threshold, execute the first target calculation strategy, wherein the first target calculation strategy is the process of the transformer block executing the calculation of the conditional path and the unconditional path;

[0118] When both the conditional and unconditional distances exceed a preset distance threshold, the first objective calculation strategy is executed. This means that within this transformer block, both the conditional and unconditional paths are calculated simultaneously. This is because, in this scenario, the current hidden state is in a critical, rapidly changing, and information-rich phase, regardless of whether it's the conditional or unconditional path. The large distances indicate that this module is crucial for fusing textual information and capturing the data's own distribution. Therefore, the preset image generation model does not skip any steps and executes the calculations for both paths completely to ensure the accuracy and richness of the generated content.

[0119] Step A132: When the conditional distance is greater than or equal to a preset distance threshold and the unconditional distance is less than or equal to a preset distance threshold, execute the second target calculation strategy, wherein the second target calculation strategy is the process of the transformer block executing the conditional path calculation.

[0120] When the conditional distance is greater than or equal to a preset distance threshold, while the unconditional distance is less than or equal to the preset distance threshold, a second objective calculation strategy is executed. That is, within this Transformer Block, only the calculation of the conditional path is performed, and the calculation of the unconditional path is skipped. In the current module, the conditional distance is large, but the unconditional distance is small. Conditional guidance plays a dominant role in the final image's direction, eliminating the need to repeatedly calculate an unconditional path with limited information. Skipping the calculation of the unconditional path saves significant computational resources, and because it closely follows the guidance of the conditional path, the generation quality is not significantly affected.

[0121] Step A133: If both the conditional distance and the unconditional distance are less than the preset distance threshold, execute the third target calculation strategy. The third target calculation strategy is that the transformer block does not execute the calculation process of the conditional path and the unconditional path.

[0122] When both the conditional and unconditional distances are less than a preset distance threshold, a third objective calculation strategy is executed. This means that within this Transformer Block, all calculations for both conditional and unconditional paths are skipped. In this specific Transformer Block processing stage, the changes in the hidden states are very gradual for both conditional and unconditional paths, carrying insufficient new information, and their computation contributes little to the final result. Therefore, the preset image generation model skips all calculations in this module and directly outputs the hidden states from the previous layer to the next layer, greatly accelerating the inference process and avoiding repetitive calculations in numerous intermediate layers.

[0123] This embodiment determines the target computation strategy of each transformer block by comparing conditional distance and unconditional distance with a preset distance threshold. This can significantly improve computational efficiency, avoid unnecessary computation, save computational resources, and thus effectively improve the utilization rate of computational resources while ensuring that the image quality generated by the preset image generation model is controllable.

[0124] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Following the step of updating the hidden state corresponding to the historical token by using a preset image generation model and target calculation strategy, the method further includes:

[0125] Step B10: Based on the generation quality assessment index of the current time step, dynamically adjust the preset distance threshold. The generation quality assessment index is obtained by calculating the coherence score of the generated token sequence.

[0126] The quality assessment metric is obtained by calculating the coherence score of the generated token sequence. In the image generation scenario, the generated token sequence corresponds to a series of generated image patches. The coherence score measures the consistency and coherence among these image patch information in terms of content, style, structure, etc.

[0127] Step B20: When the generation quality evaluation index of a consecutive preset number of time steps is detected to be lower than the preset quality threshold, the preset distance threshold is reduced.

[0128] The preset distance threshold is dynamically adjusted based on the generation quality evaluation index at the current time step. When the generation quality evaluation index is high, it indicates that the currently generated image patch sequence has good coherence, and the preset distance threshold can be appropriately increased to make the transformer block more inclined to use simple computational engineering, thereby accelerating the image generation speed. When the generation quality evaluation index is low, it indicates that the currently generated image patch sequence has poor coherence, and the preset distance threshold needs to be decreased to make the transformer block use more computational strategies to ensure that the generated image patches are more stable and coherent.

[0129] The preset quality threshold is a pre-defined standard value used to determine whether the generated image quality is acceptable. When the generated quality evaluation index is lower than this threshold for a consecutive preset number of time steps, it indicates a problem with the consistency of the image generation process, which may lead to poor quality of the final generated image.

[0130] This embodiment adjusts the threshold based on the generated quality evaluation index. When the index is low, the threshold is reduced. A complex calculation strategy is used to ensure stable and coherent image patches, avoiding poor image quality due to poor coherence. When the index is lower than the preset quality threshold for multiple consecutive time steps, the threshold is reduced to promptly correct coherence issues in the generation process and improve the overall quality of the final generated image.

[0131] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the image generation method of this application. Any simple transformations based on this technical concept are all within the protection scope of this application.

[0132] This application also provides an image generation apparatus, please refer to... Figure 5 The image generation apparatus includes:

[0133] The token acquisition module 10 is used to acquire historical tokens generated before the time step and the target token corresponding to the time step at any time step in the process of generating the target image by the preset image generation model.

[0134] The token filtering module 20 is used to filter key tokens from historical tokens based on the attention weight of historical tokens. The higher the attention weight of a historical token, the greater the probability that it will be filtered as a key token.

[0135] The token combination module 30 is used to combine key tokens and target tokens corresponding to time steps to form a token set.

[0136] Target token generation module 40 is used to determine the target token for the next time step using a set of tokens;

[0137] The image generation module 50 is used to generate a target image based on the target token of each time step after generating the target token of each time step.

[0138] The image generation apparatus provided in this application addresses the technical problem of how to accelerate the inference process of image generation using the image generation method described in the above embodiments. Compared with the prior art, the beneficial effects of the image generation apparatus provided in this application are the same as those of the image generation method described in the above embodiments, and other technical features in the image generation apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0139] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the image generation method in the first embodiment described above.

[0140] The following is for reference. Figure 6 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0141] like Figure 6As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0142] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0143] The electronic device provided in this application, employing the image generation method described above, can solve the technical problem of how to accelerate the inference process of image generation. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the image generation method described above, and other technical features of this electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0144] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0145] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0146] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the image generation method described above.

[0147] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0148] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0149] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device causes the following: for any time step in the process of generating a target image using a preset image generation model, the electronic device acquires historical tokens generated before the time step and the target token corresponding to the time step; based on the attention weights of the historical tokens, it filters key tokens from the historical tokens, wherein historical tokens with higher attention weights have a greater probability of being selected as key tokens; it combines the key tokens and the target tokens corresponding to the time step to form a token set; it determines the target token for the next time step using the token set; and after generating the target token for each time step, it generates the target image based on the target tokens for each time step.

[0150] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0153] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described image generation method, thereby solving the technical problem of how to accelerate the inference process of image generation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the image generation method provided in the above embodiments, and will not be repeated here.

[0154] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image generation method described above.

[0155] The computer program product provided in this application can solve the technical problem of how to accelerate the inference process of image generation. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the image generation method provided in the above embodiments, and will not be repeated here.

[0156] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An image generation method characterized by, The image generation method comprises: For any one time step in the process of generating a target image for a preset image generation model, obtaining a historical token generated before the time step and a target token corresponding to the time step; In the case where the overall cache space of the historical token is greater than or equal to a preset cache space threshold, calculating the attention weight of the historical token; According to the attention weight, selecting a key token from the historical token, wherein the overall cache space of the key token is less than the preset cache space threshold, the attention weight of the key token is greater than the attention weight of a non-key token in the historical token, and the higher the attention weight of the historical token, the greater the probability of being selected as the key token; Combining the key token and the target token corresponding to the time step to form a token set; Determining the target token of the next time step through the token set; After generating the target token of each time step, generating the target image according to the target token of each time step.

2. The image generation method of claim 1, wherein, The step of determining the target token of the next time step through the token set comprises: According to the historical token and the target token corresponding to the time step in the token set, determining the probability of each token in a preset library through the preset image generation model; Selecting the token with the maximum probability in the preset library as the target token of the next time step.

3. The image generation method of claim 1, wherein, The preset image generation model comprises a plurality of transformer blocks, and before the step of selecting a key token from the historical token according to the attention weight of the historical token, the method further comprises: Updating the hidden state corresponding to the historical token through a plurality of transformer blocks in a conditional path and a non-conditional path, wherein the conditional path refers to guiding a preset text as the input of a plurality of transformer blocks, and the non-conditional path refers to performing self-attention calculation with the hidden state of the historical token as the input, and the hidden state is used for calculating the attention weight of the historical token.

4. The image generation method of claim 3, wherein, The step of updating the hidden state corresponding to the historical token through a plurality of transformer blocks in a conditional path and a non-conditional path comprises: Obtaining a conditional distance through the hidden state of each transformer block in the conditional path; Obtaining a non-conditional distance through the hidden state of each transformer block in the non-conditional path; According to the comparison of the conditional distance and the non-conditional distance with a preset distance threshold, determining a target calculation strategy of each transformer block; Updating the hidden state corresponding to the historical token through the preset image generation model and the target calculation strategy.

5. The image generation method of claim 4, wherein, The step of determining the target calculation strategy of each transformer block according to the comparison of the conditional distance and the unconditional distance with the preset distance threshold comprises: In the case that both the conditional distance and the unconditional distance are greater than the preset distance threshold, a first target calculation strategy is executed, wherein the first target calculation strategy executes the calculation process of the conditional path and the unconditional path for the transformer block; In the case that the conditional distance is greater than or equal to the preset distance threshold and the unconditional distance is less than or equal to the preset distance threshold, a second target calculation strategy is executed, wherein the second target calculation strategy executes the calculation process of the conditional path for the transformer block; In the case that both the conditional distance and the unconditional distance are less than the preset distance threshold, a third target calculation strategy is executed, wherein the third target calculation strategy does not execute the calculation process of the conditional path and the unconditional path for the transformer block.

6. The image generation method of claim 5, wherein, After the step of updating the hidden state corresponding to the history token through the preset image generation model and the target calculation strategy, the method further comprises: dynamically adjusting the preset distance threshold based on a generation quality evaluation indicator of the current time step, wherein the generation quality evaluation indicator is obtained by calculating a coherence score of the generated token sequence; when it is detected that the generation quality evaluation indicators of a continuous preset number of time steps are lower than a preset quality threshold, reducing the preset distance threshold.

7. An electronic device, comprising: The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the image generation method according to any one of claims 1 to 6.

8. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the image generation method according to any one of claims 1 to 6.

9. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by the processor to implement the steps of the image generation method according to any one of claims 1 to 6.

Images