Image generation method and device, electronic equipment and non-volatile storage medium

Abstract

The application discloses an image generation method and device, electronic equipment and a nonvolatile storage medium. The method comprises: obtaining original input condition information; determining condition encoding corresponding to a multi-scale level in an image generation model according to the original input condition information; in each processing layer, performing implicit balanced solving according to the output result of the previous processing layer in the image generation model and the condition encoding corresponding to the current processing layer, obtaining the output result of the current processing layer, and outputting the output result to the next processing layer; repeating the solving step until the target result output by the processing layer with the finest image resolution scale in the image generation model is obtained; and performing visual decoding on the target result to obtain a target image corresponding to the original input condition information. The application solves the technical problems of high consumption of computing resources and low inference efficiency caused by fixed depth explicit stacking of a visual self-regressive image generation model in the related art.

Description

Technical Field

[0001] This application relates to the field of image generation technology, and more specifically, to an image generation method, apparatus, electronic device, and non-volatile storage medium. Background Technology

[0002] Modern image generation techniques primarily rely on two paradigms: diffusion models and autoregressive models. Among these, autoregressive models have made significant progress in visual generation in recent years due to their clear structure and strong scalability. Visual Autoregressive Modeling (VAR), as one of the mainstream methods, encodes images into multi-scale discrete label sequences and predicts labels in parallel at each scale from coarse to fine. This effectively preserves the two-dimensional spatial structure of the image and overcomes the problems of local correlation destruction and low sampling efficiency caused by flattening images in traditional one-dimensional sequence autoregressive models (such as PixelCNN). In the VAR framework, the generation at each scale depends on the output of the previous scale, and feature transformation is performed through a deep stacked explicit Transformer network to progressively refine image details. This method exhibits excellent performance in terms of generation quality, zero-shot generalization ability, and computational parallelism, and has become an important technical path for high-resolution image generation.

[0003] However, existing VAR methods still have significant drawbacks: they employ a fixed-depth explicit neural network stack within each scale, leading to an exponential increase in model parameters and intermediate activation memory with network depth and resolution. This is particularly problematic in high-resolution image generation tasks, where the key-value (KV) cache expands dramatically, severely limiting hardware resource utilization and inference efficiency. Furthermore, because the network depth is fixed before training, the model cannot dynamically adjust computational load during inference based on input complexity, device computing power, or latency requirements, making it difficult to adapt to diverse deployment scenarios from cloud servers to mobile terminals. Even on low-power devices, full-depth forward computation is still required, resulting in resource waste; while in high-power environments, increasing computational depth cannot further improve generation quality. This "fixed depth, fixed computation" paradigm prevents the model from achieving "on-demand computation," severely limiting its practical application in edge computing, real-time interaction, and low-power devices.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides an image generation method, apparatus, electronic device, and non-volatile storage medium to at least solve the technical problems of high computational resource consumption and low inference efficiency caused by the fixed-depth explicit stacking of visual autoregressive image generation models in related technologies.

[0006] According to one aspect of the embodiments of this application, an image generation method is provided, comprising: acquiring original input condition information, wherein the original input condition information is used to represent the user's requirements for the planned generated image; determining condition codes corresponding to multi-scale levels in an image generation model based on the original input condition information, wherein each processing layer in the multi-scale level is arranged sequentially in order from coarse to fine according to the corresponding image resolution scale, and each processing layer corresponds to a condition code that matches the scale of the current processing layer; in each processing layer, performing implicit equilibrium solution based on the output result of the previous processing layer in the image generation model and the condition code corresponding to the current processing layer to obtain the output result of the current processing layer, and outputting the output result to the next processing layer, repeating the above solution steps until the target result output by the processing layer with the finest image resolution scale in the image generation model is obtained, wherein the implicit equilibrium solution is used to iteratively solve the fixed-point equation through a single-layer neural network with weight sharing; and visually decoding the target result to obtain a target image corresponding to the original input condition information.

[0007] Optionally, determining the conditional encoding corresponding to the multi-scale levels in the image generation model based on the original input conditional information includes: determining the type of the original input conditional information, wherein the type of the original input conditional information includes at least one of the following: text description, reference image, category identifier, random seed; encoding the original input conditional information using an encoder corresponding to the type of the original input conditional information to obtain a global feature encoding; and fusing the global feature encoding with the positional encoding corresponding to the processing layers at different scale levels in the image generation model to obtain a conditional encoding that matches the spatial shape of each processing layer.

[0008] Optionally, each processing layer includes a deep equalization implicit generation layer. Based on the output of the previous processing layer in the image generation model and the conditional encoding corresponding to the current processing layer, implicit equalization is solved to obtain the output of the current processing layer. This includes: using a first shallow projection network in the processing layer to fuse the output of the previous processing layer and the conditional encoding corresponding to the current processing layer to obtain the initial feature representation of the current processing layer; injecting the initial feature representation into the deep equalization implicit generation layer for iterative solving to obtain equalized features, wherein the deep equalization implicit generation layer is a single-layer neural network with shared weights, and the single-layer neural network converges the initial feature representation to an equalized state by iteratively solving fixed-point equations; and projecting the equalized features through a second shallow projection network to obtain the output of the current processing layer.

[0009] Optionally, injecting the initial feature representation into a deep equalization implicit generation layer for iterative solution to obtain equalized features includes: using a deep equalization implicit generation layer to transform the initial feature representation to obtain implicit state features, and then re-injecting the implicit state features as new feature representations into the deep equalization implicit generation layer for transformation. This process is repeated iteratively until the number of iterations reaches the maximum iteration threshold, or the difference parameter between the implicit state features output in the current iteration and those output in the previous iteration is less than a preset parameter threshold. The difference parameter is used to characterize the degree of difference between the implicit state features output in the current iteration and those output in the previous iteration. The implicit state features obtained in the last iteration are determined as equalized features.

[0010] Optionally, the method further includes: determining the current deployment environment state information of the image generation model, wherein the deployment environment state information is at least used to characterize the state of available computing resources in the device currently running the image generation model; determining the total computing resource budget based on the deployment environment state information, wherein the total computing resource budget is used to characterize the maximum amount of computation that the image generation model can consume when completing a single image generation task; and determining the maximum number of iterations thresholds corresponding to processing layers of different scales based on the target allocation scheduling strategy and the total computing resource budget, wherein the target allocation scheduling strategy is used to allocate the number of iterations for each processing layer under the constraint of the total computing resource budget.

[0011] Optionally, determining the maximum iteration thresholds for processing layers at different scales based on the target allocation scheduling strategy and the total computing resource budget includes: determining the task type of the current image generation task, where the task type characterizes the type of image to be generated by the image generation task; determining the importance weights for processing layers at different scales based on the task type, where the importance weights characterize the importance of the image resolution scale corresponding to the processing layer to the image to be generated by the image generation task; determining the allocation ratio between processing layers at different scales based on the importance weights; and determining the maximum iteration thresholds for processing layers at different scales according to the allocation ratio and the total computing resource budget.

[0012] Optionally, the training steps of the image generation model include: for each processing layer at each scale, performing a first number of forward iterations using a depth-equalized implicit generation layer without saving the computational graph, to approximate the implicit equilibrium point, wherein the first number of forward iterations are performed in a context where gradient computation is disabled, and the intermediate states of each iteration do not retain gradient dependencies; after completing the first number of iterations, continuing to perform a second number of forward iterations, and saving the computational graph and intermediate activation values ​​of each iteration, wherein the second number is less than the first number; performing backpropagation based on the computational graph and intermediate activation values ​​saved in the second number of iterations, calculating the gradient of the loss function with respect to the model parameters of the image generation model through the gradient propagation chain, in order to update the model parameters.

[0013] According to another aspect of the embodiments of this application, an image generation apparatus is also provided, comprising: a condition information acquisition module, configured to acquire original input condition information, wherein the original input condition information represents the user's requirements for the planned generated image; a condition encoding determination module, configured to determine the condition encoding corresponding to the multi-scale levels in the image generation model based on the original input condition information, wherein each processing layer in the multi-scale level is arranged sequentially in order from coarse to fine according to the corresponding image resolution scale, and each processing layer corresponds to a condition encoding that matches the scale of the current processing layer; an implicit equalization solution module, configured to perform implicit equalization solution in each processing layer based on the output result of the previous processing layer in the image generation model and the condition encoding corresponding to the current processing layer, obtain the output result of the current processing layer, and output the output result to the next processing layer, repeating the above solution steps until the target result output by the processing layer with the finest image resolution scale in the image generation model is obtained, wherein the implicit equalization solution is used to iteratively solve the fixed-point equation through a single-layer neural network with shared weights; and a target image output module, configured to perform visual decoding on the target result to obtain the target image corresponding to the original input condition information.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes an image generation method during runtime.

[0015] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes an image generation method by running the computer program.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of an image generation method.

[0017] In this embodiment, the following steps are employed: First, original input condition information is obtained, representing the user's requirements for the planned generated image. Second, conditional codes corresponding to multiple scale levels in the image generation model are determined based on the original input condition information. Each processing layer in the multiple scale levels is arranged sequentially from coarse to fine according to its corresponding image resolution scale, and each processing layer corresponds to a conditional code matching the scale of the current processing layer. Third, implicit equalization is performed in each processing layer based on the output of the previous processing layer and the conditional code corresponding to the current processing layer. The output of the current processing layer is then passed to the next processing layer, and this process is repeated until the target result output by the processing layer with the finest image resolution scale in the image generation model is obtained. The implicit equalization solution is used to iteratively solve the fixed-point equation using a single-layer neural network with shared weights. Finally, the target result is visually decoded to obtain the target image corresponding to the original input condition information. This is achieved by integrating the Deep Equilibrium Models (DEQ) implicit layer into the Visual Autoregressive Model. In the "next-scale prediction" paradigm of visual autoregressive image generation (VAR) modeling, a novel image generation architecture with dynamically adjustable computational depth is implemented, thereby solving the technical problems of high computational resource consumption and low inference efficiency caused by the explicit stacking of fixed depths in related technologies. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal (or electronic device) for implementing an image generation method according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of an image generation method flow according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the overall framework of an improved image generation model provided according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram comparing the effects of image generation results according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram comparing the effects of image editing results according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the structure of an image generation device provided according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] In related technologies, the field of image generation is mainly dominated by two architectural paradigms: diffusion models and autoregressive models. Diffusion models (such as DDPM, SGM, DiT, LDM, etc.) have set a high benchmark in terms of generated image quality. In recent years, autoregressive models based on the Transformer architecture and more powerful visual word segmenters (such as VQGAN, ViT-VQGAN, VQ-VAE-2, RQ-AR, Parti, etc.) have made continuous progress, significantly narrowing the gap with diffusion models in terms of generated image quality.

[0028] However, traditional "next-label" image autoregressive methods have a fundamental flaw: their one-dimensional causal order does not match the inherent two-dimensional spatial structure of images. The process of flattening a two-dimensional image into a one-dimensional sequence destroys local correlations, introduces unwanted bidirectional dependencies in practice, and leads to low sampling efficiency (as is evident in models such as PixelCNN).

[0029] To overcome the aforementioned problems, Visual Autoregressive modeling (VAR) redefines autoregression as "next-scale prediction": the model generates multi-scale labeled maps in a coarse-to-fine order, with generation fully parallelized within each scale. This hierarchical sorting method preserves spatial locality, reduces the computational complexity of traditional raster scanning AR models, and exhibits good scalability and zero-shot generalization ability.

[0030] Nevertheless, the VAR method still relies on a deeply stacked explicit backbone network for feature transformation in each subsequent scale prediction step. As model width and image resolution increase, the contradiction between memory consumption, inference latency, sampling depth, and parallelism becomes increasingly prominent. At the same time, the fixed-depth computational paradigm for each scale also limits the on-demand computing power required for model deployment on mobile or edge devices and in low-latency online service scenarios.

[0031] To address the aforementioned issues, this application provides a solution by introducing the implicit layer paradigm of Deep Equilibrium Models (DEQ) into the next-scale prediction framework of VAR, thus constructing a novel Visual Implicit Autoregressive Modeling (VIAR) method. Specifically, the core idea of ​​Deep Equilibrium Models (DEQ) provides important insights for this approach. DEQ utilizes implicit fixed-point layers instead of explicit depth stacking and is trained using methods such as implicit differentiation or Jacobian-free backpropagation, thereby achieving backpropagation with constant memory consumption, controllable convergence, and flexibly adjustable computational load during the testing phase. Embedding such implicit layers into the iterative generation process introduces a new computational control mechanism, supports dynamic allocation of computational resources across steps, and allows reuse of existing solutions among related subproblems, thus achieving a better balance between generation quality and efficiency under a fixed computational budget. These characteristics indicate that integrating deep equilibrium implicit layers into the VAR architecture can overcome the high parallel computational complexity caused by fixed-depth-scale prediction while retaining the original advantages of VAR. A detailed explanation follows.

[0032] According to an embodiment of this application, an embodiment of an image generation method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0033] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or electronic device) for implementing an image generation method is shown. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0034] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or electronic device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image generation method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned image generation method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0037] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or electronic device).

[0038] Under the above operating environment, this application provides an image generation method. Figure 2 This is a schematic diagram of an image generation method flow according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0039] Step S202: Obtain the original input condition information, wherein the original input condition information is used to represent the user's requirements for the planned generated image;

[0040] Among them, the original input condition information serves as the initial input to the generation process and directly determines the user intent boundary that the subsequent image generation task should respond to. Its role is to provide a unique and unchangeable initial constraint for the entire generation process, ensuring that subsequent processing revolves around the original requirements explicitly put forward by the user.

[0041] Step S204: Based on the original input condition information, determine the condition codes corresponding to the multi-scale levels in the image generation model. The processing layers in the multi-scale levels are arranged in order from coarse to fine according to the corresponding image resolution scale, and each processing layer corresponds to a condition code that matches the scale of the current processing layer.

[0042] In the process of gradually generating from coarse to fine scales, the conditional information of each layer is precisely anchored to its own resolution level, ensuring that the injected conditional content is consistent with the image structure granularity responsible for that level, thereby providing semantic guidance for subsequent intra-scale generation that directly corresponds to the current resolution.

[0043] Step S206: In each processing layer, implicit equalization is performed based on the output of the previous processing layer in the image generation model and the conditional encoding corresponding to the current processing layer to obtain the output of the current processing layer. The output is then output to the next processing layer. The above solution steps are repeated until the target result output by the processing layer with the finest image resolution scale in the image generation model is obtained. The implicit equalization solution is used to iteratively solve the fixed-point equation through a single-layer neural network with weight sharing.

[0044] The feature output generated by the previous processing layer and the conditional information associated with the current processing layer are used as input to drive a computational unit consisting of only a single neural network layer to run repeatedly. This unit corrects the features in each iteration through a weight sharing mechanism until the fixed-point convergence condition is met, thereby obtaining the final output of the current processing layer. This output is passed to the subsequent processing layer as its input to participate in a new round of implicit equilibrium solution. This process advances scale by scale until the finest scale in the image resolution is generated. The whole process does not rely on a multi-layer stacked explicit network structure, but achieves feature refinement and information transfer through the continuous iteration of a single-layer network.

[0045] Step S208: Visually decode the target result to obtain the target image corresponding to the original input condition information.

[0046] In this embodiment, visual decoding is performed on the target result, that is, the intermediate result representing the image content generated after model processing is converted into a visual image form that can be recognized by humans or systems. This process is directly based on the visual information structure carried by the target result, and through the decoding mechanism corresponding to the visual word segmenter, the discrete tag sequence or feature representation is restored to the pixel-level image output. This step does not involve modifying or enhancing the content of the target result, but only performs a mapping from the encoding space to the visual space. Its role is to make the abstract representation generated by the model ultimately present an image with clear visual content that matches the original input condition information, thereby completing the closed loop from the calculation result to the perceptible output.

[0047] Through the above steps, by integrating the implicit computational idea of ​​the depth equalization model into the multi-scale visual autoregressive generation framework, the goal of significantly reducing the number of model parameters and dynamically adjusting the computational depth is achieved. This solves the technical problems of high computational resource consumption and low inference efficiency caused by the explicit stacking of fixed depths in visual autoregressive image generation models in related technologies.

[0048] The image generation method in steps S202 to S208 of the embodiments of this application will be further described below.

[0049] This application proposes a Visual Implicit Autoregressive Modeling (VIAR) method. By replacing the deeply stacked explicit intermediate layers in traditional VAR with a single implicit fixed-point layer, it fundamentally reduces the number of model parameters and the memory footprint of intermediate activations. Simultaneously, this design effectively suppresses key-value (KV) caching during large-scale parallel generation, thereby alleviating hardware (especially memory and bandwidth) pressure and creating conditions for deploying larger-scale, higher-resolution models. Figure 3 As shown, the core architectural differences between traditional VAR and VIAR in the embodiments of this application are compared, and the progressive generation process of VIAR is demonstrated. Specifically, as... Figure 3 As shown in the upper part, VAR uses a fixed-depth explicit layer stack (L layers are passed sequentially, each layer has independent parameters), while VIAR uses a single implicit layer to iterate N times (the same set of weights is repeatedly applied until convergence), achieving constant memory usage and dynamic depth adjustment. Figure 3 The lower part presents a coarse-to-fine generation strategy for VIAR: coarse-scale (e.g., 8×8) allocates a high number of iterations to determine the global structure, while fine-scale (e.g., 64×64) reduces iterations and utilizes the fast convergence characteristics of implicit layers to supplement details, ultimately obtaining high-quality images with low computational cost. The visual implicit autoregressive modeling method in the embodiments of this application will be described in detail below.

[0050] During image generation, the original input condition information can be obtained first, and conditional codes that match the resolution stages of the multi-scale hierarchy can be generated accordingly, as follows.

[0051] In some embodiments of this application, determining the conditional encoding corresponding to the multi-scale levels in the image generation model based on the original input conditional information includes: determining the type of the original input conditional information, wherein the type of the original input conditional information includes at least one of the following: text description, reference image, category identifier, random seed; encoding the original input conditional information using an encoder corresponding to the type of the original input conditional information to obtain a global feature encoding; and fusing the global feature encoding with the positional encoding corresponding to the processing layers at different scale levels in the image generation model to obtain a conditional encoding that matches the spatial shape of each processing layer.

[0052] Specifically, we can first determine the type of the original input condition information, which refers to identifying whether the user-provided generation instruction belongs to one or more types such as text description, reference image, category label, or random seed (for example, in the text condition generation scenario, the original input condition information is a text description; in the image editing scenario, the original input condition information may contain both image and text). This provides semantic guidance for subsequent encoding strategies. Then, we can use an encoder corresponding to this type to encode the original input condition information. That is, we can select the most suitable neural network structure for feature extraction based on the characteristics of the input modality to generate a global feature code with high semantic expressiveness, thereby ensuring that condition information from different sources has comparability and compatibility in a unified semantic space.

[0053] For example, when the input is "a Shiba Inu wearing glasses is running on the grass", the system recognizes it as a text description and then calls a specially trained text encoder (such as CLIP text encoder) to extract semantic vectors. When the input is a reference image, a pre-trained visual word segmenter can be used to encode the input image, avoiding semantic distortion caused by using a uniform encoding for all inputs.

[0054] Subsequently, the global feature encoding is fused with the positional encoding corresponding to the processing layers at different scale levels. For example, the global feature encoding can be quantized into discrete token sequences at multiple resolutions from coarse to fine. Specifically, in each resolution level of image generation (such as 8×8, 16×16, 32×32 up to 512×512), the global semantic information can be linearly transformed by adding or concatenating the inherent spatial positional priors of that level (such as two-dimensional sinusoidal positional encoding or learnable coordinate embedding) element by element. This allows the coarse-scale layer to obtain a general guide to the global semantics (such as "the overall composition is a landscape"), while the fine-scale layer inherits the semantic details and accurately locates the local structure (such as "the leaf texture should be distributed in the upper right corner"). This achieves dynamic alignment of semantic content and spatial structure during the implicit equilibrium solution process.

[0055] Through the above mechanism, the model can adaptively handle diverse user input formats and inject conditions that are both globally consistent and conform to local spatial distribution into each level in the multi-scale generation process, significantly improving semantic accuracy and spatial consistency in the implicit solution process.

[0056] In subsequent image generation, the image generation model follows a coarse-to-fine order, predicting all labels in parallel at each scale. Generation at each scale depends only on information from all coarser scales. This hierarchical prediction framework reduces computational complexity by transforming traditional "next label" prediction into "next scale" prediction, while preserving spatial structure. Significantly reduced to It also achieves fully parallelized generation within the scale.

[0057] Meanwhile, this application introduces a depth-equalized implicit generation layer to replace the fixed-depth explicit Transformer stack in traditional VAR. The specific processing flow for each scale generation step is as follows.

[0058] In some embodiments of this application, each processing layer includes a deep equalization implicit generation layer; based on the output of the previous processing layer in the image generation model and the conditional encoding corresponding to the current processing layer, implicit equalization is solved to obtain the output of the current processing layer, including: using a first shallow projection network in the processing layer to fuse the output of the previous processing layer and the conditional encoding corresponding to the current processing layer to obtain the initial feature representation of the current processing layer; injecting the initial feature representation into the deep equalization implicit generation layer for iterative solution to obtain equalized features, wherein the deep equalization implicit generation layer is a single-layer neural network with shared weights, and the single-layer neural network converges the initial feature representation to an equalized state by iteratively solving the fixed-point equation; and projecting the equalized features through a second shallow projection network to obtain the output of the current processing layer.

[0059] Specifically, the deep, balanced implicit generative layer embedded in each processing layer is a single-layer neural network with shared weights (such as an implicit Transformer layer). Its core principle is to iteratively solve fixed-point equations, gradually converging the initial feature representation of the input to a stable, balanced state, rather than relying on a fixed number of forward propagations through a multi-layered, explicitly stacked structure. This significantly reduces parameter redundancy and computational overhead. Mathematically, this process is equivalent to an infinitely deep network, and its actual computational cost is determined by the number of iterations required to achieve convergence.

[0060] In this embodiment, the input to the depth-balanced implicit generation layer is an initial feature representation generated by dynamically fusing the output features of the processing layer at the previous scale with the conditional encoding corresponding to the current scale through the first shallow projection network. This fusion process compresses multi-source information (such as coarse-grained semantics and fine-grained texture guidance) into a unified representation through linear transformation and nonlinear activation function. For example, when generating a high-resolution face, the low-resolution facial contours output by the previous layer and the conditional encoding of the "eye details" of the current layer are projected and fused to form an initial feature representation with contextual relevance.

[0061] Subsequently, the deep equalization implicit generation layer uses this initial feature as a starting point and repeatedly applies a nonlinear mapping function through a single-layer network with fixed weights, allowing the feature to evolve self-consistently during iteration until the fixed-point convergence condition is met, simulating a physical process of "implicit equilibrium." Finally, the second shallow projection network adaptively samples the converged equalization feature, mapping it to output features that conform to the current scale resolution, and outputting all predicted labels for the current scale. This ensures compatibility with the input format of subsequent higher-resolution layers.

[0062] The specific steps for iteratively solving the deep equalization implicit generation layer are as follows.

[0063] In some embodiments of this application, injecting the initial feature representation into a deep equalization implicit generation layer for iterative solution to obtain equalized features includes: using a deep equalization implicit generation layer to transform the initial feature representation to obtain implicit state features, and then re-injecting the implicit state features as new feature representations into the deep equalization implicit generation layer for transformation. This process is repeated iteratively until the number of iterations reaches a maximum iteration threshold, or the difference parameter between the implicit state features output in the current iteration and the implicit state features output in the previous iteration is less than a preset parameter threshold. The difference parameter is used to characterize the degree of difference between the implicit state features output in the current iteration and the implicit state features output in the previous iteration. The implicit state features obtained in the last iteration are determined as equalized features.

[0064] In this embodiment, the aforementioned implicit state features refer to the intermediate feature representations updated by the neural network in each iteration. Their role is to gradually approximate the stable semantic expression required for image generation, rather than relying on explicit stacking of a fixed number of layers. By repeatedly injecting the implicit state features output by the current iteration into the same network layer as the input for the next iteration, dynamic adaptation of computational depth is achieved. The aforementioned difference parameter is a quantitative indicator that measures the magnitude of change between the implicit state features of two consecutive iterations. For example, the Euclidean distance of the feature vector is calculated using the L2 norm. When this value is lower than a preset threshold (e.g., 0.01), it indicates that the features have become stable and there is no need to continue iterating, thereby achieving intelligent pruning of computational resources while ensuring the quality of generation. The aforementioned maximum iteration threshold serves as a safety boundary to prevent infinite loops caused by slow convergence or noise interference. For example, in complex texture generation scenarios, even if the difference parameter does not meet the standard, the system will be forcibly terminated after 50 rounds to ensure inference efficiency. Finally, only the implicit state features output by the last iteration are output as "balanced features," ensuring the convergence consistency of the feature space and the stability of the generation results.

[0065] The aforementioned mechanism achieves a seamless integration of implicit solution and multi-scale generation processes. This design transforms the computational depth of the model from a fixed, discrete number of network layers to a continuous, dynamically adjustable number of implicit fixed-point iterations, thus decoupling computational load from model architecture. This allows the processing of each layer to rely on both preceding semantic evolution and precisely inject conditional guidance at the current scale, completely eliminating the computational rigidity and resource waste caused by the fixed number of explicitly stacked layers in traditional autoregressive models. This enables dynamic adjustment of computational depth, improved inference efficiency, and enhanced system stability while maintaining generation quality.

[0066] Based on the above mechanism, in this embodiment, the computational depth of the model is transformed from a pre-set hyperparameter before training to an iteration number that can be flexibly adjusted during the inference phase according to input complexity, available computing resources, or the required generation quality. A single trained VIAR model can simulate network behavior at different depths during inference, achieving flexible computation from "quick draft" to "fine rendering," without the need to train and store multiple independent models. In the model inference (image generation) phase, this embodiment can utilize the adjustable iteration number of implicit layers to implement a flexible computing resource allocation strategy, as detailed below.

[0067] In some embodiments of this application, the method further includes: determining the current deployment environment state information of the image generation model, wherein the deployment environment state information is at least used to characterize the state of available computing resources in the device currently running the image generation model; determining the total computing resource budget based on the deployment environment state information, wherein the total computing resource budget is used to characterize the maximum amount of computing power that the image generation model can consume when completing a single image generation task; and determining the maximum iteration number thresholds corresponding to processing layers of different scales based on the target allocation scheduling strategy and the total computing resource budget, wherein the target allocation scheduling strategy is used to allocate the iteration number of each processing layer under the constraint of the total computing resource budget.

[0068] Specifically, by acquiring the deployment environment status information of the device currently running the image generation model, key resource indicators such as GPU memory usage, CPU load, memory bandwidth, or power consumption limits can be perceived in real time, thereby dynamically assessing the actual carrying capacity of the device. Based on this, the total computing resource budget can be defined as the maximum amount of computing power allowed to be consumed in a single image generation task. For example, it may be limited to 500 MFLOPs on mobile devices, while it can be relaxed to 5 GFLOPs on cloud servers, ensuring that the budget is strictly aligned with the device's capabilities.

[0069] Under the constraint of the total computational budget, a scheduling strategy can be implemented according to the target, intelligently allocating the maximum number of iterations for each processing layer based on the resolution weights and computational complexity distribution across multiple scale levels. This allows for the allocation of appropriate iteration counts to different scales. For example, more iterations can be allocated to coarse-scale processes responsible for the global structure, while fewer iterations can be allocated to high-resolution, fine-scale processes responsible for details, thereby optimizing the balance between overall generation quality and efficiency. Simultaneously, a convergence threshold can be set to terminate computation at the current scale early when the iteration update is sufficiently small, further conserving computational resources.

[0070] In this embodiment, the above-mentioned target allocation scheduling strategy may include, but is not limited to: 1) Budget-aware scheduling strategy: the total budget is fixed and allocated according to the importance of the scale; 2) High-quality fixed configuration strategy: sufficient iterations at each scale to pursue the best quality; 3) Ultra-fast fixed configuration: minimum iterations at each scale to pursue the fastest speed, which can be used for real-time preview of the scene.

[0071] Among them, the budget-aware scheduling strategy can automatically optimize the iteration depth of each scale according to the semantic characteristics of the generated content while keeping the total computational cost constant, significantly improving the generation quality of key regions, while avoiding redundant computation of non-key scales, and achieving synergistic optimization of generation efficiency and visual accuracy, as detailed below.

[0072] In some embodiments of this application, determining the maximum iteration threshold for processing layers of different scales based on the target allocation scheduling strategy and the total computing resource budget includes: determining the task type of the current image generation task, wherein the task type is used to characterize the type characteristics of the image to be generated by the image generation task; determining the importance weights corresponding to processing layers of different scales based on the task type, wherein the importance weights are used to characterize the importance of the image resolution scale corresponding to the processing layer to the image to be generated by the image generation task; determining the allocation ratio between processing layers of different scales based on the importance weights, and determining the maximum iteration threshold for processing layers of different scales according to the allocation ratio and the total computing resource budget.

[0073] Specifically, the semantic category of the generated image can be identified, such as face, landscape, or medical image. Different task types have fundamentally different sensitivities to image details. For example, face generation requires highly preserving the fine texture of facial features, while landscape generation focuses more on global composition and color transitions. Then, the importance weights corresponding to processing layers at different scales can be determined according to the task type. This means assigning a quantized priority to each resolution level (such as a low-resolution coarse outline layer, a medium-resolution structure layer, and a high-resolution detail layer). The allocation ratio between processing layers at different scales is determined based on the importance weights. This means proportionally splitting the total computational budget according to the relative size of the weights of each layer. For example, if the weight ratio is 0.6:0.3:0.1, the corresponding iteration allocation ratio is 6:3:1. This ensures that resources are tilted towards semantically critical layers, avoiding excessive consumption of computational power at low-value scales, and ensuring that each layer obtains a computational depth that matches its semantic importance before convergence. For example, since implicit layers converge rapidly at high-resolution scales, significantly reducing their iteration count has little impact on the final image quality. This allows a single, well-trained VIAR model to dynamically adapt to various deployment scenarios, from cloud servers to edge devices, meeting the needs of different latency and computing power.

[0074] This mechanism achieves dynamic and intelligent allocation of computing resources through task-type-driven semantic-aware scheduling. While maintaining a constant total computational cost, the model automatically optimizes the iteration depth at each scale based on the semantic characteristics of the generated content, significantly improving the generation quality of key regions while avoiding redundant computation at non-key scales, thus achieving synergistic optimization of generation efficiency and visual accuracy. This prioritizes the stable convergence of low-resolution structures in resource-constrained scenarios, avoiding memory overflow or timeout failures due to excessive computation at fine-grained layers. The synergistic effect of these steps ensures that the iteration termination in the implicit equilibrium solution process no longer depends on a fixed-depth structure but adapts adaptively to device capabilities. This prevents inference interruptions caused by overload and maximizes generation quality when resources are abundant, achieving efficient, stable, and adaptive execution of image generation tasks in heterogeneous device environments.

[0075] In addition, to avoid the huge memory overhead when training the image generation model containing the deep balanced implicit generation layer, this application also proposes an efficient training mechanism based on random Jacobi-less backpropagation, as follows.

[0076] In some embodiments of this application, the training steps of the image generation model include: for each scale processing layer, performing a first number of forward iterations using a depth-equalized implicit generation layer without saving the computation graph, to approximate the implicit equilibrium point, wherein the first number of forward iterations are performed through a context that disables gradient calculation, and the intermediate states of each iteration do not retain gradient dependencies; after completing the first number of iterations, continuing to perform a second number of forward iterations, and saving the computation graph and intermediate activation values ​​of each iteration, wherein the second number is less than the first number; performing backpropagation based on the computation graph and intermediate activation values ​​saved in the second number of iterations, calculating the gradient of the loss function with respect to the model parameters of the image generation model through the gradient propagation chain, in order to update the model parameters.

[0077] This training method randomly executes two phases of iteration in each training iteration: first, several forward iterations are performed without saving the computation graph to approach the equilibrium point; then, a small number of iterations are performed with the computation graph saved. Gradient calculation is approximated only through backpropagation in these last few iterations. This method achieves near-constant memory usage while ensuring training stability. Specifically, for the training process of each scale processing layer of the image generation model, the first number of forward iterations can be performed in a context where gradient calculation is disabled. That is, the system repeatedly calls the same parameter module to solve for implicit equilibrium through a single-layer neural network with shared weights without recording any intermediate activation values ​​or computational dependencies, thereby quickly approximating the convergence state of the fixed-point equation. For example, in 50 iterations, only the first 30 coarsely stable feature representations are used, and all intermediate tensors do not participate in the graph construction during backpropagation, thus significantly reducing memory overhead.

[0078] Subsequently, only the complete computation graph and activation values ​​of the last 20 iterations (the second number) are retained. That is, the system only enables gradient tracking after confirming that it is close to the equilibrium point. By saving a small number of iteration states that are close to the convergence interval, it is ensured that the gradient propagation chain can still effectively transmit error signals, while avoiding redundant storage of all 50 iteration states. Finally, backpropagation is performed only based on the computation graph saved in the last 20 iterations, so that the model parameter update depends on the local but accurate gradient path, rather than the entire iteration chain. This is similar to "only recording the footprints of the last few steps" in the optimization process to infer the influence of the initial input, rather than recording every step of the entire path. This strategy divides the solution process into two stages: memoryless approximation and memory fine-tuning. It achieves the goal of maintaining the convergence accuracy of the implicit layer while compressing the training memory usage from a linear increase proportional to the number of iterations to a constant level related to the second number. This completely solves the memory explosion problem caused by saving the computation graph for all iterations in deep equalization models, and enables high-resolution multi-scale image generation models to be trained stably with limited hardware resources.

[0079] Furthermore, embodiments of this application verify the effectiveness of the proposed VIAR method through system experiments and compare it with mainstream visual autoregressive (VAR) models, such as... Figure 4 and Figure 5 The figures show a comparison of image generation (editing) effects in image generation and image editing scenarios, respectively. The results demonstrate that VIAR significantly improves computational and storage efficiency while maintaining high-quality generation.

[0080] Specifically, the VIAR method in this application generates higher quality images with a significantly simplified model: In the ImageNet 256×256 generation task, VIAR outperforms the deep VAR model on several key metrics. Its Inception Score reaches 330.7, and its spatial fidelity (sFID of 7.92) is also significantly better. Furthermore, when CFG equals 1.5, the FID of VIAR and VAR is very close. It is worth emphasizing that VIAR achieves this performance using only 770 million parameters, a 61.6% reduction in parameters compared to VAR, demonstrating that implicit layer design can capture better visual distributions with a smaller model.

[0081] Meanwhile, the VIAR method offers faster inference speeds and significantly reduced memory usage: deployment tests on an RTX 4090 show that VIAR can flexibly balance efficiency and quality by adjusting the number of iterations. Under optimized configuration, VIAR achieves a generation speed of 32.08 images per second with a peak memory requirement of only 8.53 GB; in comparison, VAR achieves 15.16 images per second with a memory usage of 19.24 GB. This means that VIAR achieves a 2.1x speedup and 2.26x memory savings, while maintaining highly competitive generation quality.

[0082] Furthermore, in image editing tasks that do not require fine-tuning (such as interior drawing and in-frame generation), VIAR-generated regions blend more naturally with the context, exhibit clearer details, and smoother boundary transitions. This is thanks to the implicit layers' ability to effectively integrate global information, outputting stable, high-quality results even with fewer iterations.

[0083] In summary, VIAR significantly outperforms existing explicit stacking architectures in terms of model size, inference speed, and resource efficiency, providing an efficient solution for the practical application of visual generative models on edge devices and in low-latency scenarios.

[0084] This application aims to seamlessly inherit the inherent multi-scale generation advantages of VAR (such as spatial locality preservation, intra-scale parallelization, good scalability, and zero-shot generalization ability), while absorbing the core advantages of DEQ (such as constant memory training, controllable convergence, and variable computation during testing). Ultimately, it achieves a better balance between generation quality, computational efficiency, and deployment flexibility.

[0085] Specifically, a novel generative model architecture is proposed. While retaining the coarse-to-fine "next-scale prediction" process, this architecture makes a fundamental improvement: replacing the fixed-depth explicit neural network stacks within each scale in traditional methods with a core module consisting of an implicit equalization layer and lightweight projection networks before and after it. This implicit layer refines features through iterative solving until convergence, and its computational depth is determined by a dynamic parameter—the number of iterations—rather than a fixed number of network layers. During the model usage (inference) phase, computational resources at each scale can be dynamically allocated according to actual needs. For example, more iterations can be allocated to the coarse scale responsible for the global structure, and fewer iterations to the fine scale responsible for details; or a convergence threshold can be set to achieve early stopping.

[0086] The aforementioned characteristics make this application's solution particularly suitable for edge and mobile deployment environments that are latency-sensitive and have limited computing resources. For example, when performing real-time image generation, editing, and enhancement on smartphones, embedded devices, or smart glasses, VIAR can adaptively allocate the number of computational iterations at each scale based on the device's current computing power and battery status, maximizing energy efficiency while ensuring user experience. This provides a feasible technical path for bringing high-quality visual generation models from the cloud to terminal devices.

[0087] Specifically, this technical solution integrates a depth-balanced implicit layer into a multi-scale autoregressive framework, achieving a significant reduction (over 60%) in model parameters and dynamically adjustable computational depth. This allows high-quality image generation technology to move beyond its reliance on high-computing cloud resources and be efficiently deployed on mobile devices such as smartphones and tablets, empowering users to create and edit images in real-time and at low cost, greatly lowering the barrier to entry for professional-grade visual content production. Simultaneously, its adaptive computational characteristics provide a flexible quality-efficiency balancing tool for professional fields such as film and television, design, and e-commerce, improving workflow efficiency. These fields have a high demand for visual content generation and editing. This technical solution not only provides high-quality visual output, but its single model supports multiple quality levels (from rapid sketches to refined finished products), providing professionals with a flexible workflow tool. Designers can quickly generate multiple concept sketches and then refine selected designs, significantly improving the efficiency of creative exploration and content production. Furthermore, this technical solution is applicable to cutting-edge scenarios such as augmented reality (AR) devices and smart glasses that have stringent requirements for real-time performance and lightweight design. It can achieve real-time restoration and enhancement of environmental images, promoting the upgrade of immersive experiences. Finally, this technology aligns with the trends of edge computing and green low-carbon development, providing a key technical foundation for the widespread application of visual AI in a wide range of fields such as the Internet of Things and smart cities.

[0088] According to an embodiment of this application, an embodiment of an image generation apparatus is also provided. Figure 6 This is a schematic diagram of the structure of an image generation apparatus according to an embodiment of this application. Figure 6 As shown, the device includes:

[0089] The condition information acquisition module 60 is used to acquire raw input condition information, wherein the raw input condition information is used to represent the user's requirements for the planned generated image;

[0090] The condition coding determination module 62 is used to determine the condition coding corresponding to the multi-scale level in the image generation model based on the original input condition information. The processing layers in the multi-scale level are arranged in order from coarse to fine according to the corresponding image resolution scale, and each processing layer corresponds to a condition coding that matches the scale of the current processing layer.

[0091] The implicit equalization solution module 64 is used to perform implicit equalization solution in each processing layer based on the output result of the previous processing layer in the image generation model and the conditional encoding corresponding to the current processing layer, to obtain the output result of the current processing layer, and output the output result to the next processing layer. The above solution steps are repeated until the target result output by the processing layer with the finest image resolution scale in the image generation model is obtained. The implicit equalization solution is used to iteratively solve the fixed point equation through a single-layer neural network with weight sharing.

[0092] The target image output module 66 is used to perform visual decoding on the target result to obtain the target image corresponding to the original input condition information.

[0093] Optionally, determining the conditional encoding corresponding to the multi-scale levels in the image generation model based on the original input conditional information includes: determining the type of the original input conditional information, wherein the type of the original input conditional information includes at least one of the following: text description, reference image, category identifier, random seed; encoding the original input conditional information using an encoder corresponding to the type of the original input conditional information to obtain a global feature encoding; and fusing the global feature encoding with the positional encoding corresponding to the processing layers at different scale levels in the image generation model to obtain a conditional encoding that matches the spatial shape of each processing layer.

[0094] Optionally, each processing layer includes a deep equalization implicit generation layer. Based on the output of the previous processing layer in the image generation model and the conditional encoding corresponding to the current processing layer, implicit equalization is solved to obtain the output of the current processing layer. This includes: using a first shallow projection network in the processing layer to fuse the output of the previous processing layer and the conditional encoding corresponding to the current processing layer to obtain the initial feature representation of the current processing layer; injecting the initial feature representation into the deep equalization implicit generation layer for iterative solving to obtain equalized features, wherein the deep equalization implicit generation layer is a single-layer neural network with shared weights, and the single-layer neural network converges the initial feature representation to an equalized state by iteratively solving fixed-point equations; and projecting the equalized features through a second shallow projection network to obtain the output of the current processing layer.

[0095] Optionally, injecting the initial feature representation into a deep equalization implicit generation layer for iterative solution to obtain equalized features includes: using a deep equalization implicit generation layer to transform the initial feature representation to obtain implicit state features, and then re-injecting the implicit state features as new feature representations into the deep equalization implicit generation layer for transformation. This process is repeated iteratively until the number of iterations reaches the maximum iteration threshold, or the difference parameter between the implicit state features output in the current iteration and those output in the previous iteration is less than a preset parameter threshold. The difference parameter is used to characterize the degree of difference between the implicit state features output in the current iteration and those output in the previous iteration. The implicit state features obtained in the last iteration are determined as equalized features.

[0096] Optionally, the image generation apparatus is further configured to: determine the current deployment environment state information of the image generation model, wherein the deployment environment state information is at least used to characterize the state of available computing resources in the device currently running the image generation model; determine the total computing resource budget based on the deployment environment state information, wherein the total computing resource budget is used to characterize the maximum amount of computation that the image generation model can consume when completing a single image generation task; and determine the maximum iteration number thresholds corresponding to processing layers of different scales based on the target allocation scheduling strategy and the total computing resource budget, wherein the target allocation scheduling strategy is used to allocate the iteration number of each processing layer under the constraint of the total computing resource budget.

[0097] Optionally, determining the maximum iteration thresholds for processing layers at different scales based on the target allocation scheduling strategy and the total computing resource budget includes: determining the task type of the current image generation task, where the task type characterizes the type of image to be generated by the image generation task; determining the importance weights for processing layers at different scales based on the task type, where the importance weights characterize the importance of the image resolution scale corresponding to the processing layer to the image to be generated by the image generation task; determining the allocation ratio between processing layers at different scales based on the importance weights; and determining the maximum iteration thresholds for processing layers at different scales according to the allocation ratio and the total computing resource budget.

[0098] Optionally, the training steps of the image generation model include: for each processing layer at each scale, performing a first number of forward iterations using a depth-equalized implicit generation layer without saving the computational graph, to approximate the implicit equilibrium point, wherein the first number of forward iterations are performed in a context where gradient computation is disabled, and the intermediate states of each iteration do not retain gradient dependencies; after completing the first number of iterations, continuing to perform a second number of forward iterations, and saving the computational graph and intermediate activation values ​​of each iteration, wherein the second number is less than the first number; performing backpropagation based on the computational graph and intermediate activation values ​​saved in the second number of iterations, calculating the gradient of the loss function with respect to the model parameters of the image generation model through the gradient propagation chain, in order to update the model parameters.

[0099] It should be noted that each module in the above-mentioned image generation device can be a program module (for example, a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0100] It should be noted that the image generation apparatus provided in this embodiment can be used to perform... Figure 2 The image generation method shown above is also applicable to the embodiments of this application, and will not be repeated here.

[0101] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following image generation method by running the computer program: acquiring original input condition information, wherein the original input condition information represents the user's requirements for the planned generated image; determining the condition codes corresponding to the multi-scale levels in the image generation model based on the original input condition information, wherein each processing layer in the multi-scale level is arranged sequentially from coarse to fine according to the corresponding image resolution scale, and each processing layer corresponds to a condition code matching the scale of the current processing layer; in each processing layer, performing implicit equalization solution based on the output result of the previous processing layer in the image generation model and the condition code corresponding to the current processing layer to obtain the output result of the current processing layer, and outputting the output result to the next processing layer, repeating the above solution steps until the target result output by the processing layer with the finest image resolution scale in the image generation model is obtained, wherein the implicit equalization solution is used to iteratively solve the fixed-point equation through a single-layer neural network with shared weights; and visually decoding the target result to obtain the target image corresponding to the original input condition information.

[0102] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image generation method described in various embodiments of this application: obtaining original input condition information, wherein the original input condition information represents the user's requirements for the planned generated image; determining the condition codes corresponding to the multi-scale levels in the image generation model based on the original input condition information, wherein each processing layer in the multi-scale level is arranged sequentially in order from coarse to fine according to the corresponding image resolution scale, and each processing layer corresponds to a condition code that matches the scale of the current processing layer; in each processing layer, performing implicit equalization solution based on the output result of the previous processing layer in the image generation model and the condition code corresponding to the current processing layer to obtain the output result of the current processing layer, and outputting the output result to the next processing layer, repeating the above solution steps until the target result output by the processing layer with the finest image resolution scale in the image generation model is obtained, wherein the implicit equalization solution is used to iteratively solve the fixed-point equation through a single-layer neural network with shared weights; and visually decoding the target result to obtain the target image corresponding to the original input condition information.

[0103] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0104] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0106] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0107] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0108] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0109] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image generation method, characterized in that, include: Obtain the original input condition information, wherein the original input condition information is used to represent the user's requirements for the planned generated image; Based on the original input condition information, the condition codes corresponding to the multi-scale levels in the image generation model are determined. The processing layers in the multi-scale levels are arranged in order from coarse to fine according to the corresponding image resolution scale, and each processing layer corresponds to a condition code that matches the scale of the current processing layer. In each processing layer, implicit equalization is performed based on the output of the previous processing layer in the image generation model and the conditional encoding corresponding to the current processing layer to obtain the output of the current processing layer. The output is then passed to the next processing layer to continue solving until the target result of the processing layer with the finest image resolution scale in the image generation model is obtained. The implicit equalization is used to iteratively solve the fixed-point equation through a single-layer neural network with weight sharing. Each processing layer includes a deep equalization implicit generation layer. Based on the output of the previous processing layer in the image generation model and the conditional encoding corresponding to the current processing layer, implicit equalization is performed to obtain the output of the current processing layer. This includes: using a first shallow projection network in the processing layer to fuse the output of the previous processing layer and the conditional encoding corresponding to the current processing layer to obtain an initial feature representation of the current processing layer; injecting the initial feature representation into the deep equalization implicit generation layer for iterative solving to obtain equalized features, wherein the deep equalization implicit generation layer is a single-layer neural network with shared weights, and the single-layer neural network converges the initial feature representation to an equalized state by iteratively solving fixed-point equations; and projecting and sampling the equalized features through a second shallow projection network to obtain the output of the current processing layer. The target result is visually decoded to obtain the target image corresponding to the original input condition information.

2. The image generation method according to claim 1, characterized in that, Based on the original input condition information, the conditional encoding corresponding to the multi-scale levels in the image generation model is determined as follows: The type of the original input condition information is determined, wherein the type of the original input condition information includes at least one of the following: text description, reference image, category identifier, and random seed; The original input condition information is encoded using an encoder corresponding to the type of the original input condition information to obtain global feature encoding; The global feature encoding is fused with the position encoding corresponding to the processing layers at different scales in the image generation model to obtain the conditional encoding that matches the spatial shape of each processing layer.

3. The image generation method according to claim 1, characterized in that, The initial feature representation is injected into the deep equalization implicit generation layer for iterative solution to obtain the equalization features, including: The initial feature representation is transformed using the deep equalization implicit generation layer to obtain implicit state features. These implicit state features are then re-injected into the deep equalization implicit generation layer as new feature representations for further transformation. This process of injecting and transforming feature representations into the deep equalization implicit generation layer is repeated iteratively until the number of iterations reaches a maximum iteration threshold, or the difference parameter between the implicit state features output in the current iteration and those output in the previous iteration is less than a preset parameter threshold. The difference parameter is used to characterize the degree of difference between the implicit state features output in the current iteration and those output in the previous iteration. The implicit state feature obtained in the last iteration is determined as the equilibrium feature.

4. The image generation method according to claim 3, characterized in that, The method further includes: Determine the current deployment environment status information of the image generation model, wherein the deployment environment status information is at least used to characterize the status of available computing resources in the device currently running the image generation model; Based on the deployment environment status information, the total computing resource budget is determined, wherein the total computing resource budget is used to characterize the maximum amount of computing power that the image generation model can consume when completing a single image generation task; Based on the target allocation scheduling strategy and the total computing resource budget, the maximum iteration number thresholds corresponding to the processing layers at different scales are determined, wherein the target allocation scheduling strategy is used to allocate the iteration number of each processing layer under the constraint of the total computing resource budget.

5. The image generation method according to claim 4, characterized in that, Based on the target allocation scheduling strategy and the total computing resource budget, the maximum iteration thresholds corresponding to the processing layers at different scales are determined as follows: Determine the task type of the current image generation task, wherein the task type is used to characterize the type characteristics of the image that the image generation task plans to generate; Based on the task type, importance weights corresponding to the processing layers at different scales are determined, wherein the importance weights are used to characterize the degree of importance of the image resolution scale corresponding to the processing layer to the image to be generated by the image generation task plan; Based on the importance weights, the allocation ratio among the processing layers of different scales is determined, and according to the allocation ratios and the total computing resource budget, the maximum number of iterations thresholds corresponding to the processing layers of different scales are determined respectively.

6. The image generation method according to claim 1, characterized in that, The training steps of the image generation model include: For each scale of the processing layer, without saving the computation graph, the depth-equalized implicit generation layer is used to perform a first number of forward iterations to approximate the implicit equilibrium point, wherein the first number of forward iterations are performed through a context in which gradient computation is disabled, and the intermediate states of each iteration do not preserve gradient dependencies. After completing the first number of iterations, continue to execute the second number of forward iterations, and save the computation graph and intermediate activation values ​​for each iteration, wherein the second number is less than the first number; Backpropagation is performed based on the computation graph and intermediate activation values ​​saved in the second number of iterations. The gradient of the loss function with respect to the model parameters of the image generation model is calculated through the gradient propagation chain to update the model parameters.

7. An image generation apparatus, characterized in that, include: The condition information acquisition module is used to acquire raw input condition information, wherein the raw input condition information is used to represent the user's requirements for the planned generated image; The condition coding determination module is used to determine the condition coding corresponding to the multi-scale level in the image generation model based on the original input condition information. The processing layers in the multi-scale level are arranged in order from coarse to fine according to the corresponding image resolution scale, and each processing layer corresponds to a condition coding that matches the scale of the current processing layer. An implicit equalization solution module is used to perform implicit equalization solution in each processing layer based on the output result of the previous processing layer in the image generation model and the conditional encoding corresponding to the current processing layer, to obtain the output result of the current processing layer, and output the output result to the next processing layer to continue solving, until the target result output by the processing layer with the finest image resolution scale in the image generation model is obtained. The implicit equalization solution is used to iteratively solve the fixed-point equation through a single-layer neural network with weight sharing. Each processing layer includes a deep equalization implicit generation layer. Based on the output of the previous processing layer in the image generation model and the conditional encoding corresponding to the current processing layer, implicit equalization is performed to obtain the output of the current processing layer. This includes: using a first shallow projection network in the processing layer to fuse the output of the previous processing layer and the conditional encoding corresponding to the current processing layer to obtain an initial feature representation of the current processing layer; injecting the initial feature representation into the deep equalization implicit generation layer for iterative solving to obtain equalized features, wherein the deep equalization implicit generation layer is a single-layer neural network with shared weights, and the single-layer neural network converges the initial feature representation to an equalized state by iteratively solving fixed-point equations; and projecting and sampling the equalized features through a second shallow projection network to obtain the output of the current processing layer. The target image output module is used to perform visual decoding on the target result to obtain the target image corresponding to the original input condition information.

8. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the image generation method according to any one of claims 1 to 6.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the image generation method according to any one of claims 1 to 6 by running the computer program.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the image generation method according to any one of claims 1 to 6.

Images