Image rendering methods, apparatuses, electronic devices, storage media and software products

Abstract

This application provides an image rendering method, apparatus, electronic device, storage medium, and program product, belonging to the field of image processing technology. The method includes: receiving input data; inputting the input data into an image rendering model on a neural network processor, and obtaining the next frame image of the input image output by the image rendering model; the image rendering model includes a diffusion model module and a diffusion forced control module; the diffusion forced control module is used to determine the noise level and the number of denoising steps after compression according to the rendering instructions; the diffusion model module is used to combine the input image with noise generated according to the noise level to form an intermediate state, and under the control of the diffusion forced control module, perform autoregressive denoising processing on the intermediate state based on the number of denoising steps after compression to obtain the next frame image. This application utilizes a neural network processor and a diffusion forced control module to improve the image generation speed and solves the problem of difficulty in balancing rendering performance, power consumption, and image quality.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image rendering method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development of fields such as game development, industrial design, architectural visualization, and virtual reality, the market has an increasingly urgent need for technologies that can generate high-quality dynamic images with photorealistic qualities in real time in a low-power and low-cost manner.

[0003] To meet these requirements, existing technologies primarily employ physically based rendering pipelines using Graphics Processing Units (GPUs). This approach generates images by simulating the physical laws governing light propagation and material interactions in the real world. Specifically, it first performs geometric modeling of the 3D scene, and then performs extensive mathematical calculations on a high-performance GPU using techniques such as rasterization or ray tracing to determine the color of each pixel. In recent years, there have also been approaches that utilize artificial intelligence models (such as Deep Learning Super Sampling (DLSS)), Generative Adversarial Networks (GANs), and Neural Radiance Fields (NeRF) to super-upscale low-resolution images generated by physically rendered images, aiming to improve rendering frame rates while maintaining image quality. These approaches form the foundation of current mainstream real-time rendering technologies.

[0004] However, in practical applications, the performance of existing GPU-based physical rendering pipelines is tightly coupled with the geometric complexity of the scene. When the number of facets, objects, or lighting complexity in a scene increases dramatically, the required physical computation grows exponentially, leading to a significant drop in rendering frame rate and making it difficult to maintain the stability required for real-time interaction. Secondly, high-performance GPUs consume enormous amounts of power and are expensive. Their hardware architecture is primarily optimized for graphics computing and is not energy-efficient when performing general-purpose computations that do not involve graphics features, such as generative AI models. This limits the application of high-quality real-time rendering on power-sensitive platforms such as mobile devices and edge computing. Finally, existing rendering workflows rely on complex physical parameter settings and manual adjustments, resulting in a high creative threshold. Furthermore, the entire rendering process is based on forward calculations using preset physical rules, making it difficult to directly learn from data and generate subtle and rich visual effects that are difficult for physical models to accurately describe. In summary, existing GPU-based physical rendering pipelines struggle to balance rendering performance, power consumption, and image quality. Summary of the Invention

[0005] This application provides an image rendering method, apparatus, electronic device, storage medium, and program product, aiming to solve the problem that existing physical rendering pipelines based on graphics processing units (GPUs) cannot balance rendering performance, power consumption, and image quality.

[0006] In a first aspect, this application provides an image rendering method, including: Receive input data, which includes an input image and rendering instructions; The input data is input to the image rendering model on the neural network processor to obtain the next frame image of the input image output by the image rendering model; The image rendering model includes a diffusion model module and a diffusion forced control module; The diffusion forced control module is used to determine the noise level and the number of denoising steps for generating the next frame image according to the rendering instructions; The diffusion model module is used to combine the input image with noise generated according to the noise level to form an intermediate state, and under the control of the diffusion forced control module, to perform autoregressive denoising processing on the intermediate state based on the compressed denoising steps to obtain the next frame image.

[0007] In one embodiment, the diffusion model module includes a diffusion transformer model that runs on the tensor core of the neural network processor. Furthermore, the diffusion forced control module runs on the vector processor of the neural network processor.

[0008] In one embodiment, the on-chip memory of the neural network processor includes a state memory module, which is used to store the hidden state of the previous frame of the input image; Accordingly, before the diffusion model module performs autoregressive denoising processing on the intermediate state based on the compressed denoising steps, the diffusion model module is further configured to: The hidden state of the previous frame image is fused with the rendering command to generate the current context vector; During the process of performing autoregressive denoising on the intermediate state based on the number of denoising steps after compression, the current context vector is used as a global condition constraint of the diffusion model module to guide the generation of the next frame image.

[0009] In one embodiment, the diffusion model module fuses the hidden state of the previous frame image with the rendering instructions to generate a current context vector, including: Using the hidden state of the previous frame image as a query and the rendering instruction as a key and value, a cross-attention operation is performed on the query, the key, and the value to obtain the first operation result; The result of the first operation and the hidden state of the previous frame are subjected to residual connection and layer normalization processing, and then the processing result is subjected to feedforward network processing to obtain the current context vector.

[0010] In one embodiment, the diffusion model module determines the noise level and the number of compressed denoising steps for generating the next frame image based on the rendering instructions, including: The parameters of the preset noise scheduling function are adjusted based on the rendering instructions; The noise level and the number of denoising steps after compression are determined by a noise scheduling function based on the parameters.

[0011] In one embodiment, the image rendering model further includes a codec running on a convolutional acceleration unit of the neural network processor; Accordingly, before the diffusion model module combines the input image with noise generated according to the noise level, it further includes: The encoder of the codec is used to compress the input image to obtain a first latent representation; Furthermore, the intermediate state is formed by combining the first latent representation with the noise; The next frame image is obtained by decoding the second latent representation obtained by performing autoregressive denoising processing on the diffusion model module using the decoder of the codec.

[0012] In one embodiment, the neural network processor runs a frame prediction loopback module; Accordingly, after the diffusion model module performs autoregressive denoising processing to obtain the second latent representation, the method further includes: The frame prediction loopback module is used to fuse the second latent representation with the hidden state of the previous frame image to obtain a fused latent representation; The decoder of the codec is used to decode the fused latent representation as the input image for the next round of image rendering.

[0013] In one embodiment, the frame prediction loopback module fuses the second latent representation with the hidden state of the previous frame image to obtain a fused latent representation, specifically including: Perform linear interpolation on the second latent representation and the hidden state of the previous frame image to obtain an intermediate latent representation; Using the intermediate latent representation as a query and the hidden state of the previous frame image as a key and value, a cross-attention operation is performed to obtain the second operation result; Residual connections and layer normalization are performed on the second operation result and the intermediate latent representation to obtain the fused latent representation.

[0014] Secondly, this application also provides an image rendering apparatus, comprising: A receiving module is used to receive input data, which includes an input image and rendering instructions; An image rendering module is used to input the input data into an image rendering model on a neural network processor and obtain the next frame image of the input image output by the image rendering model. The image rendering model includes a diffusion model module and a diffusion forced control module; The diffusion forced control module is used to determine the noise level and the number of denoising steps for generating the next frame image according to the rendering instructions; The diffusion model module combines the input image with noise generated according to the noise level to form an intermediate state, and under the control of the diffusion forced control module, performs autoregressive denoising processing on the intermediate state based on the compressed denoising steps to obtain the next frame image. Thirdly, this application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described image rendering methods.

[0015] Fourthly, this application also provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described image rendering methods.

[0016] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements any of the above-described image rendering methods. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 It is the GPU physical rendering technology architecture used in existing technologies; Figure 2 This is a schematic diagram of existing technology that combines GPU physical rendering with AI-assisted rendering. Figure 3 This is a schematic diagram of the NeRF rendering technology used in existing technologies; Figure 4 This is one of the flowcharts illustrating the image rendering method provided in this application; Figure 5 This is the second flowchart illustrating the image rendering method provided in this application; Figure 6 This is a schematic diagram of the structure of the computing system provided in this application; Figure 7 This is a schematic diagram of the image rendering apparatus provided in this application; Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0021] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0022] Existing image rendering technologies mainly include GPU physically-based rendering, GPU physically-based rendering combined with AI-assisted rendering, and other rendering methods.

[0023] In GPU-based physical rendering, both traditional 3D rendering pipelines and real-time ray tracing rely on high-performance GPUs. For example... Figure 1 As shown, current mainstream games and visualization engines (such as Unreal and Unity) use GPU rasterization and RTX ray tracing technology to achieve semi-real-time global illumination with hardware acceleration, but some details or frame rate still need to be sacrificed in ultra-complex scenes.

[0024] In GPU physical rendering combined with AI-assisted rendering, such as Figure 2 As shown, the DLSS series utilizes convolutional networks for super-resolution, improving image clarity and increasing frame rate; it also includes technologies such as neural texture compression and neural materials, using AI to compress textures and accelerate material processing. These technologies are used as post-processing or acceleration modules in existing rendering pipelines, but a rough image still needs to be rendered first or path tracing needs to be performed before optimization.

[0025] Currently, GANs and NeRF (such as...) are also used. Figure 3 While deep neural networks (as shown) can be used for specific rendering tasks, such as view compositing or stylized rendering of static scenes, these methods are typically only effective for limited scenes or static viewpoints and do not form a complete solution for real-time dynamic rendering. Currently, there is no mature dedicated hardware solution to fully replace traditional pipelines with neural rendering technology in industrial applications.

[0026] In practical applications, the performance of existing GPU-based physical rendering pipelines is tightly coupled with the geometric complexity of the scene. When the number of facets, objects, or lighting complexity in a scene increases dramatically, the required physical computation grows exponentially, leading to a significant drop in rendering frame rate and making it difficult to maintain the stability required for real-time interaction. Secondly, high-performance GPUs consume enormous amounts of power and are expensive. Their hardware architecture is primarily optimized for graphics computing and is not energy-efficient when performing general-purpose computations that do not involve graphics characteristics, such as generative AI models. This limits the application of high-quality real-time rendering on power-sensitive platforms such as mobile devices and edge computing. Finally, existing rendering workflows rely on complex physical parameter settings and manual adjustments, resulting in a high creative threshold. Furthermore, the entire rendering process is based on forward calculations using preset physical rules, making it difficult to directly learn from data and generate the subtle and rich visual effects that are difficult for physical models to accurately describe.

[0027] Based on the above, this application provides an image rendering method, apparatus, electronic device, storage medium, and program product. The image rendering method of this application can be executed by a computing system integrating a Neural Processing Unit (NPU). Figure 6An example of a computing system is given, which is a heterogeneous computing system with a central processing unit (CPU) and a neural network processor (NPU).

[0028] Neural Processing Units (NPUs) are hardware-optimized for deep learning operations such as matrix multiplication and addition, supporting massively parallel computing. Furthermore, NPUs possess neural network inference capabilities, achieving frame rates far exceeding those of ordinary GPUs. Therefore, image rendering based on NPUs can significantly improve rendering speed and real-time performance. NPUs use dedicated NPU chips instead of traditional GPUs, significantly reducing power consumption. Moreover, under the same power consumption conditions, NPUs accelerate neural networks by hundreds of times compared to GPUs, enabling edge devices and mobile terminals to operate efficiently without the need for external cooling or expensive power supplies.

[0029] In this application, an image rendering model runs on a neural network processor. This image rendering model includes at least a diffusion model module and a diffusion enforcement control module. The diffusion model module is a neural network component that performs the core image rendering task. It is a learning-based deep generative model that, through training on a large amount of image data, learns the inherent rules for recovering clear, realistic images from random noise or noisy data. The trained diffusion model module is used to perform noise denoising based on the input image and rendering instructions, thereby generating a new image and achieving image rendering. The diffusion enforcement control module is a logic control component that controls and schedules the real-time operation of the diffusion model module. It is responsible for dynamically adjusting the generation process according to the input data to ensure the real-time performance and interactivity of the rendering.

[0030] The image rendering method, apparatus, electronic device, storage medium, and program product provided in this application are described below with reference to the accompanying drawings.

[0031] It should be noted that the image rendering method provided in this application embodiment is implemented based on an image rendering device. On the basis of the high-performance computing power of the neural network processor, the image rendering method uses a diffusion forced control module to improve the image generation speed of the diffusion model module, so as to meet the real-time and interactive image rendering scenarios. It solves the problem that rendering performance, power consumption and image quality are difficult to balance in the prior art. The embodiment of this application can update the image in real time according to the user's real-time operation, ensuring the smoothness and continuity in game and design interaction. In addition, the embodiment of this application is applicable to a variety of fields such as industrial design, game development, and architectural visualization. It can be transferred between different scenarios by only training or fine-tuning for new content.

[0032] This application describes the image rendering method using an image rendering device as the execution subject as an example.

[0033] Figure 4 This is one of the flowcharts illustrating the image rendering method provided in this application. For example... Figure 4 As shown, the image rendering method provided in this application includes: S410: Receive input data, which includes an input image and rendering instructions.

[0034] The input image is the source image used to generate the next frame. In a continuous video rendering pipeline, the input image is the image of the previous frame from which the rendered image will be obtained. This input image is typically a digital image format, such as a pixel matrix in RGB format.

[0035] Rendering commands are control signals used to instruct changes in the scene or viewpoint. These control signals can come from various input devices, such as mouse movement or click events to change the viewing angle; keyboard key events to control the movement of objects in the scene; or posture data from other sensors (such as VR controllers).

[0036] Please combine Figure 6 The image rendering model includes an input preprocessing module, which connects to the CPU and external input devices and is responsible for capturing raw control signals and image data. After receiving the input image and rendering instructions, the input preprocessing module performs standardization processing on the input image, including color space conversion and resolution verification; and encodes the rendering instructions, converting mouse / keyboard events into embedding vectors. The input preprocessing module then transmits the processed data to the input buffer module.

[0037] The input buffer module is a first-in, first-out (FIFO) buffer queue used to store multiple frames of input images (e.g., 5-10 frames) to prevent data loss during real-time interaction. The input buffer module employs a circular buffer structure and supports dynamic resizing. Access speed to the input buffer module can be optimized using the memory management unit of the Neural Processing Unit (NPU) to ensure low-latency input. The buffered data from the input buffer module is then fed into the diffusion model module for processing.

[0038] S420: Input the input data into the image rendering model on the neural network processor, and obtain the next frame image of the input image output by the image rendering model.

[0039] The image rendering model includes a diffusion model module and a diffusion forced control module; The diffusion forced control module is used to determine the noise level and the number of denoising steps for generating the next frame image according to the rendering instructions; The diffusion model module is used to combine the input image with noise generated according to the noise level to form an intermediate state, and under the control of the diffusion forced control module, to perform autoregressive denoising processing on the intermediate state based on the compressed denoising steps to obtain the next frame image.

[0040] Please refer to Figure 6 The input buffer module inputs preprocessed data of the input image and rendering instructions into the rendering core. The rendering core includes a diffusion enforcement control module and a diffusion model module.

[0041] The Diffusion Forcing control module aims to replace the slow, multi-step denoising process of traditional diffusion models. The diffusion forcing mechanism assists the diffusion model module in achieving autoregressive generation by configuring an independent noise level for each frame. During the inference phase, as the diffusion model module progressively denoises, the diffusion forcing mechanism adjusts the noise level of each frame, forcibly reducing the number of denoising steps to achieve rapid convergence, rather than the complete diffusion process, thus ensuring real-time performance.

[0042] Noise level is the intensity of noise introduced at the beginning of the image rendering or image generation process. Compressed denoising steps significantly reduce the hundreds or even thousands of denoising steps typically required in traditional diffusion models to an order of magnitude suitable for real-time frame rate requirements, such as the number of steps that can be completed within 20 milliseconds.

[0043] The diffusion enforcement control module selects the noise level and denoising steps based on the type or magnitude of the rendering instruction. For example, when the rendering instruction indicates a small change in viewpoint, the diffusion enforcement control module selects a lower noise level and fewer denoising steps (e.g., 5 steps) because the difference between the next frame and the input image is small. Conversely, when the rendering instruction indicates a drastic scene change, the diffusion enforcement control module selects a relatively higher noise level and more denoising steps (e.g., 20 steps) to give the model more generative space.

[0044] By combining diffusion coercion with DiT (Differentiation-Driven Transformer) and parallelizing denoising iterations on the NPU, the NPU is leveraged to accelerate Transformer computation and noise injection, achieving real-time rendering at 48 FPS. Compared to the slow bidirectional processing of traditional diffusion models, diffusion coercion can improve rendering efficiency by more than 50% while maintaining high fidelity, resulting in artifact-free and more natural lighting effects.

[0045] Combining the input image with noise generated according to the noise level is a process of adding noise to the input image information, resulting in an intermediate state. The diffusion model module then performs denoising starting from this intermediate state. The entire autoregressive denoising process is unidirectional and sequential, using information from frame N-1 (the input image) to generate frame N (the next frame). Under the control of the diffusion forced control module, the denoising process is completed within a preset, compressed number of denoising steps. In each denoising step, the diffusion model module predicts and removes a portion of the noise in the intermediate state, gradually bringing it closer to the clear target image. After completing all the compressed denoising steps, the data output by the diffusion model module is processed to finally generate the next frame.

[0046] The diffusion model module runs on the neural network processor (NPU), which fully utilizes the parallel computing capabilities of the neural network processor to ensure that each step of the calculation can be completed quickly within the compressed number of steps.

[0047] Furthermore, during the training process of the diffusion model, it learns and integrates lighting, material, and texture responses. The trained diffusion model can automatically generate richly detailed and realistic images, replacing physically based rendering. This eliminates the need for manually setting parameters for each light source or shader, reducing reliance on specialized knowledge, thus simplifying the rendering process, lowering the barrier to entry, reducing hardware resource consumption, and decreasing manual production costs, while simultaneously improving creative efficiency. Moreover, the neural network processor eliminates some of the deviations in physical simulation during image rendering, generating images with physically accurate shadows, reflections, refractions, and delicate material textures, while easily achieving various artistic styles or enhancement effects.

[0048] Please combine Figure 6 After obtaining the next frame image, it is output through the output interface module of the image rendering model. It can be directly rendered into the frame buffer of the display device (such as a monitor) to form a continuous dynamic picture; or the next frame image can be encoded into the video stream, or provided as image data to other applications (such as industrial software, game engines or construction tools) through a standardized API.

[0049] In one possible implementation, please combine Figure 6 Before outputting the next frame image, the generated RGB frame is also enhanced by the post-processing module of the image rendering model, including anti-aliasing (such as using multi-sample anti-aliasing technology), color correction, and lightweight noise removal.

[0050] Based on the high-performance computing capabilities of neural network processors, this application embodiment utilizes a diffusion forced control module to improve the image generation speed of the diffusion model module, thereby meeting the needs of real-time, interactive image rendering scenarios. This solves the problem of balancing rendering performance, power consumption, and image quality in existing technologies. This application embodiment can update images instantly based on real-time user operations, ensuring smoothness and continuity in game and design interactions. Furthermore, this application embodiment is applicable to various fields such as industrial design, game development, and architectural visualization, and can be transferred between different scenarios simply by training or fine-tuning for new content.

[0051] In one possible implementation, the diffusion model module includes a diffusion transformer (DiT) model that runs on the tensor core of the neural network processor. Furthermore, the diffusion enforcement control module runs on the vector processor of the neural network processor.

[0052] The DiT model is a diffusion model that uses a transformer architecture as its backbone network. It differs from traditional diffusion models that employ convolutional neural network architectures such as U-Net. In this application, the diffusion transformer model can better handle global dependencies in images, thereby generating higher-quality images with more reasonable structures.

[0053] The specific structure and principle of the DiT model are as follows: The DiT model mainly consists of multiple cascaded transformer blocks. When processing image data, it first divides the noisy intermediate state of the input into a sequence containing multiple image tokens in the spatial dimension. For example, a 160×90 latent map is split into 160×90 image tokens, with each image token corresponding to a local region in the latent space.

[0054] Subsequently, this sequence of image tokens is fed into cascaded transformer blocks for processing. Each transformer block includes components such as self-attention, multi-head attention, a feed-forward network (FFN), and a normalization (LayerNorm) layer. The self-attention mechanism extracts features from each image token. The multi-head self-attention mechanism calculates the association weights between each token and all other tokens in the image token sequence, thereby capturing the global structural information of the image, such as the lighting relationships between objects and light sources in the scene, and the spatial occlusion relationships between different objects. The feed-forward network performs nonlinear transformations and refines the features extracted by the self-attention mechanism, enhancing the model's expressive power. Through layer-by-layer processing by multiple transformer blocks, the model can gradually recover image features with complex structures and rich details from pure noise.

[0055] The cascaded transformer blocks can be divided into at least one shallow block, at least one medium block, and at least one deep block. The shallow block performs operations on the image token sequence to capture local detail features between tokens within the sequence; the medium block performs operations on the local detail features to establish structural relationships between local and global elements; and the deep block performs operations on the structural relationships to incorporate global semantic information and control the denoising result. The output of each transformer block serves as the input to the next transformer block, progressively reducing the noise level and ultimately generating a realistic image frame from noise.

[0056] The diffusion model module (i.e., the diffusion transformer model) runs on the Tensor Core of the neural network processor. Neural network processors (NPUs) are typically heterogeneous architectures, containing various dedicated computing units optimized for different AI computing tasks. Among these, the Tensor Core is a hardware unit specifically designed to perform large-scale matrix and tensor operations (especially matrix multiplication and addition).

[0057] The core computations of the diffusion transformer model—namely, the generation and multiplication of the query (Q), key (K), and value (V) matrices in the multi-head self-attention mechanism, and the linear transformations in the feedforward network—are essentially large-scale matrix multiplication operations. Therefore, scheduling the computational tasks of the diffusion transformer model onto the tensor cores of the NPU and mapping matrix operations to hardware instructions for the tensor cores enables parallel acceleration. This significantly leverages the parallel computing advantages of the hardware, efficiently processing these operations in a pipelined manner. Consequently, the originally computationally intensive transformer model inference process is accelerated to a level sufficient to meet the latency requirements of real-time rendering.

[0058] Corresponding to the tensor core, the vector processor is another important type of computing unit in the NPU. It excels at performing vector or scalar-level operations, such as addition, subtraction, multiplication, and division of the entire vector, as well as performing activation functions and data scheduling operations. The core tasks of the diffusion enforcement control module include determining and adjusting the noise level of each frame according to rendering instructions (e.g., calculating the noise variance), managing the number of denoising steps after compression, and injecting noise into intermediate states during the denoising process. Most of these tasks involve direct mathematical operations on feature vectors or parameter scalars. Assigning these control and scheduling logic tasks to the vector processor enables low-latency, high-precision control of the rendering process, ensuring real-time adjustment of noise levels and precise management of the number of denoising steps.

[0059] This application embodiment effectively separates the computationally intensive tasks undertaken by the diffusion converter model and the control-intensive tasks undertaken by the diffusion forced control module, and maps them respectively to the hardware units in the neural network processor that are best suited to their computational characteristics, realizing deep software and hardware co-design. Compared with mixed execution on a general-purpose processor, this application embodiment achieves higher computational efficiency and lower power consumption.

[0060] In one possible implementation, the diffusion-forced control module determines the noise level and the number of compressed denoising steps for generating the next frame image based on the rendering instructions, including: P1: Adjust the parameters of the preset noise scheduling function based on the rendering instructions.

[0061] P2: The noise level and the number of denoising steps after compression are determined based on the noise scheduling function determined by the parameters.

[0062] A noise scheduling function is a predefined mathematical function that describes how noise intensity changes with time steps (or iterations) throughout the denoising process. This function typically accepts several parameters to control its behavior.

[0063] In one embodiment, the noise scheduling function can be a linear scheduling function. For example, the function ,in It is the current time step. This is the number of denoising steps after compression. and These are the initial and final noise intensities, respectively. , and It can be used as an adjustable parameter.

[0064] In another embodiment, a cosine schedule function can be used. This function has a smoother curve, which helps to retain more image information in the early stages of denoising, thereby improving the quality of the generated image.

[0065] After receiving the rendering command, the diffusion forced control module first parses the command type (view rotation, object movement, etc.) and magnitude (e.g., rotation degree). Then, based on this parsed information, it dynamically adjusts the key parameters of the aforementioned noise scheduling function. For example: If the rendering command is a smooth move with a small amplitude, then the number of denoising steps will be... Adjust to a smaller value (e.g., 10 steps) and reduce the initial noise. The settings are set too low.

[0066] If the rendering command involves a rapid scene switch, then the number of denoising steps will be increased. Adjust to a relatively large value (e.g., 20 steps) and use a high starting noise. This allows the model more room to generate entirely new content.

[0067] At each step of the denoising process, the diffusion forced control module calls the noise scheduling function to calculate the precise noise level that should be used in the current step. This noise level is then used to generate the noise corresponding to the current step, combined with intermediate states, and fed into the diffusion model module for processing.

[0068] This application embodiment analyzes rendering commands and uses a dynamic adjustment mechanism based on the noise scheduling function to control the noise level and the number of denoising steps. This enables the diffusion forced control module to flexibly balance rendering speed and generation quality according to different interaction requirements, thus providing a solid technical guarantee for achieving stable and high-quality real-time neural rendering.

[0069] In one possible implementation, the on-chip memory of the neural network processor includes a state memory module, which is used to store the hidden state of the previous frame of the input image. Accordingly, before the diffusion model module performs autoregressive denoising processing on the intermediate state based on the compressed denoising steps, the diffusion model module is further configured to: The hidden state of the previous frame image is fused with the rendering command to generate the current context vector; Furthermore, during the process of performing autoregressive denoising on the intermediate state based on the number of denoising steps after compression, the current context vector is used as a global condition constraint of the diffusion model module to guide the generation of the next frame image.

[0070] The state memory module is an implicit memory component used to remember the rendering state. It employs a Transformer-based key-value caching mechanism to store the hidden state of the previous frame, typically with a dimension of 512-1024. The state memory module does not rely on external storage; it maintains the state vector solely in the NPU's on-chip memory. By connecting with the attention mechanism of the diffusion model module, it enables state injection, helping to generate coherent sequences of frames, such as maintaining consistent object motion in games.

[0071] The hidden state is a low-dimensional feature vector that has been refined by a neural network and condenses key information of an image. For example, it may contain high-level semantic information such as the position, pose, and lighting direction of objects in the scene. It can be a 512-dimensional or 1024-dimensional vector. The hidden state of the previous frame is updated by the system based on the generated image information and stored in the state memory module during the previous rendering cycle; the hidden state of the previous frame is the historical hidden state.

[0072] In this embodiment, the hidden state of the previous frame image is fused with the rendering instruction. The generated current context vector combines the historical hidden state and the rendering instruction to form a complete description of the current rendering task. The current context vector can be understood as a high-level instruction for the diffusion model module, containing a composite semantic representation for the current rendering task. For example, moving an object previously located at point A to a new position to its right according to a rightward movement instruction.

[0073] The current context vector is injected as a global condition into each iteration of the diffusion model module. Thus, the diffusion model module performs each step of denoising under the guidance of global condition constraints, thereby ensuring that the next frame image it generates is not only reasonable in content and structure, but also that its dynamic changes (such as the movement of objects and the rotation of viewpoints) strictly follow the user instructions and historical motion trajectories.

[0074] This application embodiment fuses historical hidden states provided by the state memory module with rendering instructions to generate a context vector as a global constraint, providing key temporal memory capabilities for the entire rendering process. It transforms the isolated single-frame generation task into a coherent, memory-based sequence generation task, thereby effectively solving the problems of screen flickering and discontinuous motion, and significantly improving the quality and realism of the generated dynamic video.

[0075] In one possible implementation, the diffusion model module fuses the hidden state of the previous frame image with the rendering instructions to generate a current context vector, including: Q1: Using the hidden state of the previous frame image as a query and the rendering instruction as a key and value, perform a cross-attention operation on the query, the key, and the value to obtain the first operation result.

[0076] The query performs operations such as dot product with all keys to calculate an attention score, which represents the degree of attention the historical state pays to various aspects of the rendering command. This attention score is then used to perform a weighted summation of the values ​​to obtain the final output, the first operation result. The first operation result represents the part most relevant to the historical state extracted from the rendering command. For example, if the historical state is a static scene and the rendering command is to rotate 15 degrees to the right, then the result of the cross-attention operation will emphasize the viewpoint rotation information.

[0077] Q2: Perform residual connection and layer normalization processing on the result of the first operation and the hidden state of the previous frame, and then perform feedforward network processing on the processing result to obtain the current context vector.

[0078] The residual connection directly adds the hidden state of the previous frame to the result of the first operation, ensuring a smooth gradient flow during backpropagation. This effectively avoids the vanishing gradient problem common in deep networks, making model training more stable. It also guarantees that even if the cross-attention operation doesn't learn useful information, the original historical state information is not lost. After the residual connection, the result is normalized to stabilize the data distribution of each layer's output, making its mean 0 and variance 1, thereby accelerating model convergence and improving its generalization ability. Finally, a feedforward network is executed. The feedforward network performs further nonlinear transformations and refinements on the fused features, enhancing the expressive power of the entire fusion module and enabling it to learn more complex mapping relationships between history and instructions. The final output of the feedforward network is the current context vector.

[0079] This application embodiment, through the processing flow of cross-attention-residual connection & layer normalization-feedforward network, not only achieves the fusion of historical state and rendering instructions, but also performs it in a highly nonlinear, dynamically weighted manner. Compared with simple linear fusion, this method can more deeply understand the intent of the instructions and their impact on the historical scene, thereby generating a current context vector with richer information and stronger guidance, laying a solid foundation for the subsequent generation of high-quality, highly coherent video frames.

[0080] In one possible implementation, the image rendering model further includes a codec running on the convolutional acceleration unit of the neural network processor. Specifically, the codec is part of the rendering core.

[0081] Accordingly, before the diffusion model module combines the input image with noise generated according to the noise level, it further includes: The encoder of the codec is used to compress the input image to obtain a first latent representation. After obtaining the first latent representation, the first latent representation is input into the diffusion model module.

[0082] Furthermore, the intermediate state is formed by combining the first latent representation with the noise; the next frame image is obtained by decoding the second latent representation obtained by performing autoregressive denoising processing on the diffusion model module using the decoder of the codec.

[0083] An encoder-decoder is used to shift core computations to a lower-dimensional, more computationally efficient latent space. The encoder-decoder can be a variational autoencoder (VAE) or other autoencoder architectures with similar functionality. The encoder consists of an encoder and a decoder. The encoder compresses high-dimensional input data (such as an RGB image) into a low-dimensional feature representation containing the core information of the original data, i.e., the latent representation. The decoder restores (or decodes) the low-dimensional latent representation back into the high-dimensional original data format (such as an RGB image).

[0084] The computational tasks of the codec are run on the convolution acceleration unit of the NPU, which is a hardware module in the NPU specifically designed for efficient execution of convolution operations. By accelerating this unit, the image compression and decoding process can be significantly accelerated, ensuring that this preprocessing and postprocessing step does not become the performance bottleneck of the entire real-time rendering pipeline.

[0085] In this embodiment, the input image is first compressed using the encoder of the codec to obtain a first latent representation. For example, a 720P RGB frame is compressed into a low-dimensional latent vector (160×90×4, compression ratio 8:1).

[0086] After the first latent representation is input into the diffusion model module, it is combined with the noise, allowing the noise addition operation to be performed in a low-dimensional latent space, thus forming a new noisy intermediate state in the latent space. Since the dimension of the latent representation is much lower than that of the original image, the amount of data that the subsequent diffusion model module needs to process is significantly reduced. After obtaining the second latent representation through autoregressive denoising, it is decoded using the decoder of the encoder-decoder to reconstruct a high-resolution RGB image, which serves as the next frame image.

[0087] This application's embodiments successfully shift the core, computationally intensive diffusion denoising process from the high-dimensional pixel space to the low-dimensional latent space by introducing a codec that runs efficiently on a convolution acceleration unit. This significantly reduces the amount of data that the diffusion model module needs to process, thereby significantly reducing computational overhead and memory usage. Without sacrificing too much image quality, it improves the overall speed and efficiency of the rendering method, making it possible to achieve high-quality real-time rendering on resource-constrained devices.

[0088] In one possible implementation, the neural network processor runs a frame prediction loopback module; Accordingly, after the diffusion model module performs autoregressive denoising processing to obtain the second latent representation, the method further includes: The frame prediction loopback module is used to fuse the second latent representation with the hidden state of the previous frame image to obtain a fused latent representation; The codec is used to decode the fused latent representation as the input image for the next round of image rendering.

[0089] In one possible implementation, the frame prediction loopback module runs on the vector processor of the neural network processor, supporting closed-loop rendering such as real-time game interactions and ensuring sequence consistency.

[0090] The frame prediction loopback module is responsible for receiving the output of the current rendering cycle and preparing the input image for the next rendering cycle. The second latent representation is a latent vector representing the original content of the current Nth frame image, which is just obtained from the diffusion model module. The hidden state of the previous frame image is a feature vector that condenses historical temporal information, obtained from the state memory module.

[0091] The frame prediction loopback module combines the second latent representation with the hidden state representing historical trends to correct the current generation result using historical information, thereby obtaining a fused latent representation that includes the content of the current frame and better conforms to the overall motion logic. Subsequently, the decoder restores the fused latent representation from the low-dimensional latent space to the high-dimensional pixel space, generating a fused RGB image. The fused RGB image is looped back to the beginning of the entire rendering pipeline as the input image in the next round (N+1 round) of image rendering cycle.

[0092] In this embodiment, the original, unfused second latent representation is decoded and displayed to the user to ensure the lowest possible interaction latency. On the other hand, the second latent representation and the hidden state of the previous frame image are fused through the frame prediction loop closure module, and the decoded image is used as the starting point for the next round of calculation. This effectively suppresses the frame-by-frame accumulation of errors during the generation process and greatly improves the stability of long-term continuous rendering and the smoothness of the video.

[0093] In one possible implementation, the frame prediction loopback module fuses the second latent representation with the hidden state of the previous frame image to obtain a fused latent representation, specifically including: R1: Perform linear interpolation on the second latent representation and the hidden state of the previous frame image to obtain an intermediate latent representation.

[0094] R2: Using the intermediate latent representation as the query and the hidden state of the previous frame image as the key and value, perform a cross-attention operation to obtain the second operation result.

[0095] R3: Perform residual connection and layer normalization on the result of the second operation and the intermediate latent representation to obtain the fused latent representation.

[0096] Linear interpolation is used for an initial, smooth blending of information from the two sources. Then, the intermediate latent representation is used as the query, and the hidden state of the previous frame is used as the key and value. A cross-attention operation is performed to dynamically calculate how much information each feature should draw from the historical state. This makes the fusion process no longer a globally uniform linear weighting, but rather allows for targeted and selective strengthening or weakening of certain features, thereby more accurately correcting inconsistencies between the current generated result and the historical data, resulting in the second operation result. Residual connections between the intermediate latent representation and the second operation result ensure that the attention mechanism incrementally corrects the intermediate latent representation, guaranteeing effective information transfer and training stability. Layer normalization standardizes the result after the residual connections to stabilize the data distribution and ensure that the output feature vector has good mathematical properties. This results in a high-quality fused latent representation that contains rich details of the current frame and has been smoothly corrected by historical information.

[0097] The embodiments of this application optimize the generated results in the latent space through a fusion mechanism of linear interpolation, cross-attention, and stabilization processing, ensuring that the input data looping back to the next round of calculation has high temporal coherence, and providing a strong guarantee for the long-term stable operation of the entire system.

[0098] Based on the above, the following combination Figure 5 and Figure 6 Here is a specific example of an image rendering process: like Figure 6 As shown, the image rendering model includes an input preprocessing module, an input buffer module, a state memory module, a rendering core, a frame prediction loopback module, a post-processing module, and an output interface module. These modules are connected via data streams and control signals to form a closed-loop rendering pipeline.

[0099] The rendering core includes the VAE encoder, DiT model, and diffusion forced control module.

[0100] like Figure 5 As shown, the image rendering process includes: S510: The input preprocessing module receives the current image frame (RGB frame, for example, provided by the frame prediction loopback module) and the rendering instructions issued by the control interaction side, and normalizes the current image frame and encodes the rendering instructions.

[0101] S520: The input preprocessing module transmits the processed data to the input buffer module for buffering, and then injects it into the state memory module to update the hidden state of the previous frame.

[0102] S530: The input buffer module inputs the cached data of the current image frame and rendering instructions into the rendering core.

[0103] S540: The VAE encoder compresses the input RGB frame into the latent vector space (compression ratio 8:1) to obtain the first latent representation, with dimensions of approximately 160x90x4. Then, the first latent representation is input into the DiT model. The DiT model, combined with the diffusion forced control module, processes the first latent representation to generate the second latent representation. Finally, the VAE decoder decodes the second latent representation back into an RGB frame to generate the next frame image. The frame prediction loopback module processes the second latent representation and, after decoding by the VAE decoder, feeds it back to the input preprocessing module as the input image for the next round of image rendering, forming a closed loop.

[0104] S550: The post-processing module enhances the next frame image and outputs the enhanced next frame image to the display interaction side through the output interface module.

[0105] This application's embodiments differ from physical renderers with explicit mathematical rules. They achieve better rendering effects through generative neural networks and combine NPU chips to replace traditional GPU physical rendering, forming a hardware and software rendering architecture that enables real-time interactive rendering and supports general applications such as industrial design, game development, and architectural visualization.

[0106] Based on the above, this application also provides an image rendering apparatus. The image rendering apparatus and the image rendering method described above can be referred to in correspondence with each other.

[0107] As an example, such as Figure 7 As shown, the image rendering apparatus provided in this application includes: The receiving module 710 is used to receive input data, which includes an input image and rendering instructions; The image rendering module 720 is used to input the input data into the image rendering model on the neural network processor and obtain the next frame image of the input image output by the image rendering model; The image rendering model includes a diffusion model module and a diffusion forced control module; The diffusion forced control module is used to determine the noise level and the number of denoising steps for generating the next frame image according to the rendering instructions; The diffusion model module combines the input image with noise generated according to the noise level to form an intermediate state. Under the control of the diffusion forced control module, autoregressive denoising processing is performed on the intermediate state based on the compressed denoising steps to obtain the next frame image. This embodiment, based on the high-performance computing capabilities of the neural network processor, utilizes the diffusion forced control module to improve the image generation speed of the diffusion model module, thus meeting the needs of real-time, interactive image rendering scenarios. It solves the problem of balancing rendering performance, power consumption, and image quality in existing technologies. This embodiment can update images instantly based on real-time user operations, ensuring smoothness and continuity in game and design interactions. Furthermore, this embodiment is applicable to various fields such as industrial design, game development, and architectural visualization; it can be transferred between different scenarios simply by training or fine-tuning for new content.

[0108] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute an image rendering method, which includes: Receive input data, which includes an input image and rendering instructions; Using the diffusion forced control module, the noise level and the number of denoising steps after compression are determined according to the rendering instructions; Using the diffusion model module, the input image is combined with noise generated according to the noise level to form an intermediate state. Under the control of the diffusion forced control module, autoregressive denoising processing is performed on the intermediate state based on the compressed denoising steps to obtain the next frame image. Output the next frame image.

[0109] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the image rendering method provided in the above embodiments, the method including: Receive input data, which includes an input image and rendering instructions; Using the diffusion forced control module, the noise level and the number of denoising steps after compression are determined according to the rendering instructions; Using the diffusion model module, the input image is combined with noise generated according to the noise level to form an intermediate state. Under the control of the diffusion forced control module, autoregressive denoising processing is performed on the intermediate state based on the compressed denoising steps to obtain the next frame image. Output the next frame image.

[0111] In another aspect, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the image rendering methods provided in the above embodiments, the method comprising: Receive input data, which includes an input image and rendering instructions; Using the diffusion forced control module, the noise level and the number of denoising steps after compression are determined according to the rendering instructions; Using the diffusion model module, the input image is combined with noise generated according to the noise level to form an intermediate state. Under the control of the diffusion forced control module, autoregressive denoising processing is performed on the intermediate state based on the compressed denoising steps to obtain the next frame image. Output the next frame image.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image rendering method, characterized in that, include: Receive input data, which includes an input image and rendering instructions; The input data is input to the image rendering model on the neural network processor to obtain the next frame image of the input image output by the image rendering model; The image rendering model includes a diffusion model module and a diffusion forced control module; The diffusion forced control module is used to determine the noise level and the number of denoising steps for generating the next frame image according to the rendering instructions; The diffusion model module is used to combine the input image with noise generated according to the noise level to form an intermediate state, and under the control of the diffusion forced control module, to perform autoregressive denoising processing on the intermediate state based on the compressed denoising steps to obtain the next frame image.

2. The image rendering method according to claim 1, characterized in that, The diffusion model module includes a diffusion transformer model, which runs on the tensor core of the neural network processor. Furthermore, the diffusion forced control module runs on the vector processor of the neural network processor.

3. The image rendering method according to claim 1, characterized in that, The on-chip memory of the neural network processor is provided with a state memory module, which is used to store the hidden state of the previous frame of the input image; Accordingly, before the diffusion model module performs autoregressive denoising processing on the intermediate state based on the compressed denoising steps, the diffusion model module is further configured to: The hidden state of the previous frame image is fused with the rendering command to generate the current context vector; During the process of performing autoregressive denoising on the intermediate state based on the number of denoising steps after compression, the current context vector is used as a global condition constraint of the diffusion model module to guide the generation of the next frame image.

4. The image rendering method according to claim 3, characterized in that, The diffusion model module fuses the hidden state of the previous frame image with the rendering instructions to generate the current context vector, including: Using the hidden state of the previous frame image as a query and the rendering instruction as a key and value, a cross-attention operation is performed on the query, the key, and the value to obtain the first operation result; The result of the first operation and the hidden state of the previous frame are subjected to residual connection and layer normalization processing, and then the processing result is subjected to feedforward network processing to obtain the current context vector.

5. The image rendering method according to claim 1, characterized in that, The diffusion forced control module determines the noise level and the number of denoising steps for generating the next frame image based on the rendering instructions, including: The parameters of the preset noise scheduling function are adjusted based on the rendering instructions; The noise level and the number of denoising steps after compression are determined by a noise scheduling function based on the parameters.

6. The image rendering method according to claim 3, characterized in that, The image rendering model also includes a codec running on the convolution acceleration unit of the neural network processor; Accordingly, before the diffusion model module combines the input image with noise generated according to the noise level, it further includes: The encoder of the codec is used to compress the input image to obtain a first latent representation; Furthermore, the intermediate state is formed by combining the first latent representation with the noise; The next frame image is obtained by decoding the second latent representation obtained by performing autoregressive denoising processing on the diffusion model module using the decoder of the codec.

7. The image rendering method according to claim 6, characterized in that, The neural network processor runs a frame prediction loopback module. Accordingly, after the diffusion model module performs autoregressive denoising processing to obtain the second latent representation, the method further includes: The frame prediction loopback module is used to fuse the second latent representation with the hidden state of the previous frame image to obtain a fused latent representation; The decoder of the codec is used to decode the fused latent representation as the input image for the next round of image rendering.

8. The image rendering method according to claim 7, characterized in that, The frame prediction loopback module fuses the second latent representation with the hidden state of the previous frame image to obtain a fused latent representation, specifically including: Perform linear interpolation on the second latent representation and the hidden state of the previous frame image to obtain an intermediate latent representation; Using the intermediate latent representation as a query and the hidden state of the previous frame image as a key and value, a cross-attention operation is performed to obtain the second operation result; Residual connections and layer normalization are performed on the second operation result and the intermediate latent representation to obtain the fused latent representation.

9. An image rendering apparatus, characterized in that... include: A receiving module is used to receive input data, which includes an input image and rendering instructions; An image rendering module is used to input the input data into an image rendering model on a neural network processor and obtain the next frame image of the input image output by the image rendering model. The image rendering model includes a diffusion model module and a diffusion forced control module; The diffusion forced control module is used to determine the noise level and the number of denoising steps for generating the next frame image according to the rendering instructions; The diffusion model module is used to combine the input image with noise generated according to the noise level to form an intermediate state, and under the control of the diffusion forced control module, to perform autoregressive denoising processing on the intermediate state based on the compressed denoising steps to obtain the next frame image.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image rendering method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium, wherein a computer program is stored on the non-transitory computer-readable storage medium, characterized in that, When the computer program is executed by a processor, it implements the image rendering method as described in any one of claims 1 to 8.

12. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image rendering method as described in any one of claims 1 to 8.

Images