An in-situ adaptive diffusion sampling architecture for diffusion model hardware acceleration and a working method thereof

Abstract

The application provides an in-situ adaptive diffusion sampling architecture for diffusion model hardware acceleration and a working method thereof, and belongs to the technical field of integrated circuits, diffusion models and artificial intelligence hardware acceleration. The architecture comprises a time step scheduling module, a noise intensity mapping module, an in-situ Gaussian noise generation module, a feature cache module, a sampling calculation module, a state updating module and a control management module. The control management module is in communication connection with the other modules, and through the close cooperation of the time step scheduling module, the noise intensity mapping module and the in-situ Gaussian noise generation module, a native hardware scaling strategy is realized at the chip bottom layer, and Gaussian noise samples matching the statistical intensity of the diffusion evolution stage are output. On the working method, the algorithm logic of the diffusion model is mapped to the closed-loop workflow of the architecture, and the complete adaptive diffusion training and image generation process is realized in-situ on the chip. The application has the characteristics of low memory access and high energy efficiency, and can improve the execution efficiency of diffusion model hardware implementation.

Description

Technical Field

[0001] This invention relates to the fields of integrated circuits, diffusion models, and artificial intelligence hardware acceleration technologies, specifically to an in-situ adaptive diffusion sampling architecture (iADS) and its working method for hardware-accelerated diffusion models. More specifically, this invention relates to a hardware system co-designed around the diffusion model sampling closed loop. This system unifies the organization of time step scheduling, noise intensity control, in-situ Gaussian noise generation, on-chip cache organization, sampling calculation, and state updates to reduce external random number dependence, storage bandwidth pressure, and cross-module data transfer overhead during diffusion model execution, thereby improving the throughput, energy efficiency, and scalability of the hardware implementation of the diffusion model. Background Technology

[0002] Diffusion models, as one of the important technical approaches in generative artificial intelligence in recent years, have been widely applied in scenarios such as image generation, speech synthesis, video generation, molecular design, and multimodal content generation. These models typically achieve complex distribution modeling and high-quality sample generation by progressively adding noise to the data and learning a reverse denoising process. Compared with traditional generative models, diffusion models have significant advantages in sample quality, training stability, and modeling ability, thus gradually becoming an important target for generative model hardware deployment.

[0003] Specifically, the complete lifecycle of the diffusion model mainly consists of two core stages: training mode and image generation mode. In training mode, the system undergoes a forward diffusion process, which involves injecting Gaussian noise of a specific standard deviation into the original clean data (such as an image) according to a preset time step to construct noisy samples. Subsequently, a neural network is used to fit and predict the added noise to update the network parameters. In image generation mode, the system starts from a purely random Gaussian noise state, iteratively calls the pre-trained denoising network at multiple time steps, and combines the random noise scheduling strategy of the current time step to gradually reduce noise, ultimately recovering a high-quality target image. It can be seen that, whether in training mode or image generation mode, the algorithm loop is highly dependent on the generation of a large amount of Gaussian noise across multiple time steps, precise dynamic amplitude scaling, and frequent tensor state updates.

[0004] However, diffusion models typically face high computational complexity and bandwidth pressure during hardware implementation in both of the aforementioned modes. Firstly, in both image generation and training modes, diffusion models require repeated calls to the denoising network across multiple time steps, incorporating the current state, time information, and noise injection process at each time step, resulting in long computational chains and numerous iterations. Secondly, diffusion models often require injecting Gaussian noise of varying intensities at different time steps, or updating statistics based on a pre-defined noise scheduling curve. This means the system must frequently generate, scale, and transmit noise samples over time. Thirdly, in existing hardware deployments, Gaussian noise is often pre-generated by an external digital random number module or software, then transported to the computing unit via off-chip memory or on-chip buffers, leading to high bandwidth consumption, increased latency, and low energy efficiency. Fourthly, the time step control, noise figure scheduling, feature caching, and computing array in diffusion models are often loosely coupled, lacking overall collaborative optimization for the diffusion model's execution process, resulting in significant redundant data movement and control overhead within the system.

[0005] Existing hardware acceleration research for diffusion models mainly focuses on accelerating operators such as convolution, attention, matrix multiplication, or the denoising network itself. However, research on dedicated architectures for the closed-loop process of "noise generation—time step scheduling—state update—data write-back" in the diffusion model sampling process is relatively insufficient. In particular, when the system needs to generate Gaussian noise with different standard deviations quickly, continuously, and statistically consistently across multiple time steps, using traditional external random source schemes will significantly increase architectural complexity and diminish the benefits of hardware acceleration.

[0006] Furthermore, in many diffusion model applications, the execution process of the diffusion model does not need to be completely independent of the random number subsystem outside the computing array. Instead, it is more suitable to adopt on-chip in-situ generation and local injection methods to reduce the repeated handling of noisy data in the bus and buffer. However, in the existing technology, there is still a lack of a complete solution for the architecture design of deep integration of in-situ Gaussian noise generation and diffusion model network computing, especially a unified hardware architecture that can simultaneously cover time step scheduling, noise intensity adaptive control, feature path organization, local cache reuse, and iterative control.

[0007] Therefore, there is an urgent need to propose a new hardware acceleration architecture for diffusion models to coordinate Gaussian noise generation, noise standard deviation scheduling, diffusion time step control, and denoising network computation at the system level, thereby improving the hardware execution efficiency of diffusion models. Summary of the Invention

[0008] To address the problems of random noise dependence on external generation, fragmented time-step control and noise scheduling, high data transfer costs, insufficient on-chip resource utilization, and low system energy efficiency in existing hardware acceleration technologies for diffusion models, this invention provides an in-situ adaptive diffusion sampling (iADS) architecture and its working method for hardware acceleration of diffusion models. This architecture integrates a time-step scheduling module, a noise intensity mapping module, an in-situ Gaussian noise generation module, a feature caching module, a sampling calculation module, a state update module, and a control management module at the hardware level, enabling noise generation, noise injection, denoising calculation, and state write-back to be completed collaboratively within the same hardware system.

[0009] One objective of this invention is to provide a dedicated hardware architecture suitable for diffusion model training and image generation, enabling it to reduce off-chip Gaussian noise sample transmission and external random number dependence. Another objective is to achieve in-situ adjustment of noise statistics under different operating modes and time steps by mapping the target time step to Gaussian noise sample intensity as a local control parameter. A further objective is to enhance the versatility of this architecture, making it adaptable to both single-step sampling acceleration scenarios and multi-time-step iterative sampling scenarios.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] An in-situ adaptive diffusion sampling (iADS) architecture for hardware acceleration of diffusion models includes a time step scheduling module, a noise intensity mapping module, an in-situ Gaussian noise generation module, a feature caching module, a sampling calculation module, a state update module, and a control management module.

[0012] The control and management module is communicatively connected to the other modules to coordinate the working timing, data paths, parameter loading processes, and pipeline scheduling control of each module. The time-step scheduling module generates the current sampling time-step information of the diffusion model and sends the scheduling parameters corresponding to this time-step, which characterize the target noise intensity, to the noise intensity mapping module. Simultaneously, it sends the time-step information to the sampling calculation module. The noise intensity mapping module receives the scheduling parameters output by the time-step scheduling module and converts them according to a preset control table, calibration mapping relationship, or segmented control relationship to generate in-situ noise control parameters corresponding to the target noise intensity of the current time-step. These in-situ noise control parameters characterize the statistical intensity of the target Gaussian noise at the current time-step and control the in-situ Gaussian noise generation module to output Gaussian noise samples that meet the statistical intensity requirements.

[0013] The in-situ Gaussian noise generation module is used to generate Gaussian noise samples that match the current time step in-situ inside the chip according to the in-situ noise control parameters. The Gaussian noise samples are directly injected into the sampling calculation module or temporarily stored in the feature cache module.

[0014] The feature cache module is used to store and retrieve the current diffusion state, intermediate features of the diffusion network, or the output state of the previous time step within the chip.

[0015] The sampling calculation module serves as the physical execution carrier of the core denoising network in the diffusion model. It receives the current diffusion state from the feature caching module, the time step information from the time step scheduling module, and the Gaussian noise samples from the in-situ Gaussian noise generation module. It then performs denoising prediction, residual estimation, and feature transformation calculations of the diffusion network at the hardware level and outputs the calculation results to the state update module for recursive updating of the diffusion state.

[0016] The state update module is used to recursively update the current diffusion state based on the denoising prediction results output by the sampling calculation module, the corresponding in-situ noise control parameters and Gaussian noise samples, and the adaptive diffusion sampling formula of the corresponding diffusion model, and write the generated next time step diffusion state back to the feature cache module for subsequent time step iterations.

[0017] Furthermore, the in-situ Gaussian noise generation module employs an on-chip probability bit array, random number array, or adjustable random circuit array, along with corresponding driving circuits, reading circuits, statistical transformation circuits, or calibration circuits. Utilizing the adjustable output statistical characteristics of the underlying on-chip noise generation unit, the module changes the operating bias, driving state, or statistical output distribution of the on-chip noise generation unit through the in-situ noise control parameters. This allows the unit to directly output Gaussian noise samples that meet the target noise intensity requirements of the current time step during the noise generation stage, thereby achieving the integration of noise generation and noise intensity adjustment at the hardware level.

[0018] Furthermore, the sampling calculation module includes underlying computing hardware for performing neural network acceleration: a graphics processing unit (GPU), a neural network processor (NPU), or a customized AI application-specific integrated circuit (ASIC); the underlying computing hardware performs denoising prediction, feature transformation, and residual estimation operations, thereby physically completing the denoising prediction and backsampling steps at a specific time step in the diffusion model.

[0019] Furthermore, the control and management module is used to perform hardware timing coordination of in-situ noise generation, denoising prediction calculation and diffusion state write-back during multi-time-step sampling, so that noise generation of subsequent data to be processed, denoising prediction calculation of current data and state write-back of previous data are carried out overlappingly inside the chip.

[0020] Furthermore, the in-situ adaptive diffusion sampling architecture, through the close collaboration of the time-step scheduling module, the noise intensity mapping module, and the in-situ Gaussian noise generation module, implements a native hardware scaling strategy at the chip level. The scaling refers to adjusting the statistical intensity of the Gaussian noise sample, where the statistical intensity is one of the following: standard deviation, variance, amplitude range, or equivalent noise intensity. The time-step scheduling module advances according to the beat and outputs scheduling parameters. The noise intensity mapping module converts these parameters into corresponding in-situ noise control parameters. Upon receiving the physical adjustment of these in-situ noise control parameters, the in-situ Gaussian noise generation module utilizes the intrinsic physical response law of its underlying on-chip noise generation unit to change its operating bias or driving state, enabling it to naturally output Gaussian noise samples matching the statistical intensity of the diffusion evolution stage at the moment of excitation.

[0021] This invention also provides a working method for the in-situ adaptive diffusion sampling (iADS) architecture for hardware acceleration of diffusion models. This method maps the algorithmic logic of the diffusion model to the closed-loop workflow of the in-situ adaptive diffusion sampling architecture, thereby realizing the complete adaptive diffusion training and image generation process on-chip in-situ. It is universally adaptable to the training and image generation modes of diffusion models, and specifically includes the following steps:

[0022] S1. Time step scheduling and parameter reading: Under the unified timing coordination of the control management module, the time step scheduling module generates the sampling time step information of the current diffusion process and determines the scheduling parameters corresponding to the time step.

[0023] S2. Noise intensity adaptive mapping: The noise intensity mapping module receives the scheduling parameters and adaptively maps them in the hardware to the in-situ noise control parameters required for the current time step, thereby avoiding the dependence on external software for noise scaling calculation.

[0024] S3. In-situ Gaussian noise generation and injection: The in-situ Gaussian noise generation module generates Gaussian noise samples that match the target noise intensity of the current time step directly inside the chip according to the in-situ noise control parameters. The samples can be directly injected into the sampling calculation module in a streaming manner, or they can be temporarily stored in the feature cache module and sent in batches as needed, thereby greatly reducing the bandwidth consumption of off-chip random number transmission.

[0025] S4. State and Feature Reading: Read the diffusion state of the sample to be processed in the diffusion model at the current time step and the intermediate features of the diffusion network from the feature cache module to realize the local reuse of data; wherein, the diffusion state includes the initial input state pre-written into the feature cache module and the intermediate diffusion state written back to the feature cache module after the state is updated in the previous time step.

[0026] S5. Diffusion network collaborative calculation: The current diffusion state, current time step information and Gaussian noise samples generated in situ are synchronously sent to the sampling calculation module to perform denoising prediction calculation of the denoising network in the diffusion model at the hardware level.

[0027] S6. Adaptive Diffusion Sampling State Update: The state update module updates the diffusion state at the current time step by recursively updating the diffusion state at the current time step based on the prediction results output by the sampling calculation module and the adaptive diffusion sampling formula of the corresponding diffusion model, generating the diffusion state of the next adjacent time step, and continuing to participate in the calculation as the new current diffusion state in subsequent iterations.

[0028] S7. Feature Writeback and Iterative Control: Write the updated diffusion state of the next time step back to the feature cache module; in the multi-time step iterative sampling scenario, the control management module continues to coordinate the in-situ noise generation, denoising prediction calculation and diffusion state update of subsequent time steps, so that the diffusion state of the next time step enters the subsequent processing flow as the new current diffusion state, and executes steps S1 to S7 in a loop until the entire diffusion model training or image generation process is completed; in the single-step processing scenario, output the updated state corresponding to the time step and end the process.

[0029] Furthermore, to demonstrate the high reusability and versatility of this architecture, the working method for hardware acceleration of the diffusion model can be flexibly configured as either a training mode or an image generation mode:

[0030] When configured in training mode, the architecture performs forward noise addition and network parameter learning. In step S4, the current diffusion state is read as a training sample or a noisy state constructed from the training sample. In step S5, the sampling calculation module performs denoising prediction calculation of the diffusion network in combination with the Gaussian noise sample generated in situ. In step S6, the training error is calculated based on the denoising prediction result and the supervision target, and the network parameter learning is completed accordingly. In step S7, the relevant intermediate features or intermediate results formed during the training process are written back to the feature cache module.

[0031] When configured for image generation mode, the architecture performs a reverse denoising inference process. In the initial time step, the Gaussian noise sample generated in step S3 is written to the feature cache module as the initial current diffusion state. In step S4, the diffusion state at the current time step is initially a purely random Gaussian noise initial state, and in subsequent stages it is an intermediate diffusion state that is recursively updated by the previous time step and written back to the feature cache module. In step S6, the state update module gradually removes the noise components in the current diffusion state according to the adaptive diffusion sampling formula of different diffusion models and the denoising prediction results, calculates and generates the diffusion state of the next time step, and finally outputs a high-quality generated image after multiple time step iterations.

[0032] Compared with the prior art, the present invention has at least the following beneficial effects:

[0033] First, by setting a control signal (in-situ noise control parameters) evolution mechanism over time, and applying the control signal directly to the on-chip noise generation unit; relying on the adjustable output statistical characteristics of the underlying physical device, the intensity of the generated Gaussian noise sample evolves over time, naturally determined by the physical characteristics of the device itself. Thus, at the moment of physical excitation, the architecture of this invention directly forms a hardware scheduling path completely determined by the underlying physical mechanism, thereby completely bypassing the multiplier array used for digital amplitude modulation in the data path and greatly reducing the control overhead introduced to fit complex mathematical functions. Second, by integrating an in-situ Gaussian noise generation module within the hardware, the need for external random number sample handling and off-chip storage bandwidth consumption can be significantly reduced. Third, the collaborative design of diffusion time step scheduling and noise intensity mapping enables rapid and continuous adjustment of noise injection parameters at different time steps. Fourth, feature caching and local data flow organization reduce repetitive data handling across modules and improve data reuse during sampling. Fifth, pipelined control and parallel module operation reduce waiting overhead between processing stages and improve the sampling throughput of the diffusion model. Sixth, this architecture is suitable for both diffusion model training and image generation, exhibiting strong scalability and compatibility. Seventh, in practical deployments, the low memory access and high energy efficiency of this architecture significantly reduce power consumption and heat generation on edge devices running generative AI, supporting local real-time generation; simultaneously, in cloud scenarios, it effectively shortens image generation latency and improves the chip's concurrent processing capabilities and overall energy efficiency. Attached Figure Description

[0034] Figure 1 This is a block diagram of the overall structure of the in-situ adaptive diffusion sampling architecture according to an embodiment of the present invention.

[0035] Figure 2 This is a flowchart illustrating the use of the iADS architecture for diffusion model training in an embodiment of the present invention.

[0036] Figure 3 This is a flowchart illustrating image generation using the iADS architecture in an embodiment of the present invention.

[0037] Figure 4 This is a schematic diagram illustrating the changes of the hardware scaling strategy and the traditional scaling strategy with time step in an embodiment of the present invention. Detailed Implementation

[0038] The present invention will be further illustrated below with examples. It should be noted that the purpose of disclosing the embodiments is to aid in further understanding the present invention; however, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

[0039] Example 1: Hardware composition of the iADS in-situ adaptive diffusion sampling architecture

[0040] This example illustrates an in-situ adaptive diffusion sampling (iADS) architecture for hardware acceleration of diffusion models, the overall structure of which is as follows: Figure 1 As shown, this architecture is used to complete random noise generation, noise amplitude adjustment, diffusion sampling calculation, and state iteration update in a closed loop within the chip during diffusion model training and image generation modes, thereby significantly reducing external random number handling and cross-module communication overhead.

[0041] like Figure 1 As shown, the iADS architecture comprises seven core collaborative modules: a control and management module, a time-step scheduling module, a noise intensity mapping module, an in-situ Gaussian noise generation module, a feature caching module, a sampling calculation module, and a state update module. These seven modules together constitute an in-situ adaptive hardware architecture operating around a diffusion sampling closed loop.

[0042] Specifically, the control and management module (in hardware implementation, such as a microcontroller CPU or a field-programmable gate array FPGA) serves as the global coordination unit, communicating with each of the other modules. It is responsible for coordinating the working timing, data path configuration, parameter loading process, and pipeline scheduling control of each module, enabling in-situ noise generation, denoising network calculation, and diffusion state write-back to proceed efficiently and in parallel, thereby reducing waiting overhead between processing stages and improving system throughput.

[0043] During the data and control flow, the time step scheduling module (which may be composed of a hardware counter or an on-chip lookup table LUT) updates the current sampling time step index t in a preset order during the diffusion process, and sends the scheduling parameters corresponding to the time step to the noise intensity mapping module. At the same time, the time step information (specifically the current time step index t, or the time step encoded feature vector converted from it, which serves as the conditional input of the denoising network to indicate the current diffusion stage) is synchronously sent to the sampling calculation module.

[0044] In the in-situ noise generation and injection link, the noise intensity mapping module (e.g., may include a digital-to-analog converter (DAC), a voltage divider resistor network, or a dedicated row / column drive circuit) receives the scheduling parameters output by the time step scheduling module, and converts the scheduling parameters according to a preset control table, calibration mapping relationship, or segmented control relationship to generate in-situ noise control parameters corresponding to the target noise intensity of the current time step. The in-situ noise control parameters are used to characterize the statistical intensity of the target Gaussian noise at the current time step, and are used to control the in-situ Gaussian noise generation module to output Gaussian noise samples that meet the statistical intensity requirements. The in-situ noise control parameters include at least one or more of the following: control voltage, bias parameters, scaling factor, group control information, normalization factor, and calibration compensation parameters. The in-situ noise control parameters can be in digital or analog form and are output to the in-situ Gaussian noise generation module via the DAC, voltage divider resistor network, or dedicated drive circuit.

[0045] Subsequently, the in-situ Gaussian noise generation module (in this example, an in-situ noise generator composed of voltage-adjustable probability bit units) directly adjusts the statistical randomness of the p-bit array output on demand within the chip according to the in-situ noise control parameters (such as control voltage), thereby generating Gaussian noise samples that strictly match the requirements of the current time step. The in-situ Gaussian noise generation module can also be implemented using a random number array, an adjustable random circuit array, or other hardware structures that support local random output, along with corresponding drive circuits, readout circuits, statistical transformation circuits, or calibration circuits. Its output Gaussian noise samples can be directly injected into the sampling calculation module without first being cached in external memory.

[0046] In the state calculation and closed-loop iteration chain, the feature caching module uses on-chip storage media, such as on-chip static random access memory (SRAM), embedded dynamic random access memory (eDRAM), or global buffer (GLB). This is used to cache the current diffusion state, intermediate features of the diffusion network, and the output state of the previous time step locally within the chip, thereby significantly reducing the number of off-chip memory accesses during cross-time step iterations. Specifically, the current diffusion state refers to the state tensor to be input to the sampling calculation module for denoising prediction at the current time step; the intermediate features of the diffusion network refer to the intermediate layer feature tensors generated during the diffusion network calculation by the sampling calculation module and reused in the current or subsequent calculations; and the output state of the previous time step refers to the diffusion state calculated by the state update module in the previous time step and written back to the feature caching module. This state can be read in the current time step and used as the current diffusion state to continue participating in the calculation.

[0047] Subsequently, the sampling calculation module can be implemented using a graphics processing unit (GPU), a neural network processing unit (NPU), or a custom AI application-specific integrated circuit. Internally, it may include low-level computing hardware such as a neural network computing array, tensor operation unit, multiply-accumulate array, and attention calculation unit. It receives the current diffusion state from the feature caching module, time step information from the time step scheduling module, and Gaussian noise samples from the in-situ Gaussian noise generation module in parallel. It then performs denoising prediction, residual estimation, and feature transformation calculations at the current time step, thereby physically completing the denoising prediction and backsampling steps for a specific time step in the diffusion model and outputting the calculation results. These results include the predicted noise tensor, residual estimation results, intermediate denoising results, and intermediate calculation results used by the state update module to generate the diffusion state for the next time step.

[0048] After the calculation is completed, the state update module can be implemented using a dedicated state recursion logic unit, a programmable control logic unit, or a dedicated update calculation unit that works in conjunction with the sampling calculation module. It is used to recursively update the current diffusion state based on the denoising prediction results output by the sampling calculation module and the adaptive diffusion sampling formula of the corresponding diffusion model, and write the generated next time step diffusion state back to the feature cache module.

[0049] Through the close collaboration of the above seven modules, the iADS architecture not only realizes in-situ noise generation and time-step adaptive noise scaling at the input end, but also completely opens up the closed loop of state update and feature cache write-back at the output end, completely getting rid of the dependence of traditional architectures on off-chip random number generators and digital post-stage multipliers, thus forming a complete hardware acceleration technology solution for diffusion models.

[0050] This invention also provides a working method for an in-situ adaptive diffusion sampling (iADS) architecture for hardware acceleration of diffusion models. This method maps the algorithm logic of the diffusion model to the closed-loop workflow of the above-mentioned in-situ adaptive diffusion sampling architecture, thereby realizing the complete adaptive diffusion training and image generation process on-chip in-situ; it is universally compatible with the training mode and image generation mode of the diffusion model.

[0051] Example 2: Diffusion Model Training Mode Based on iADS Architecture

[0052] This example illustrates the working method and process of configuring the iADS architecture in diffusion model training mode, and its corresponding appendix. Figure 2The training mode comprises two processes: forward noise addition and network parameter learning. Forward noise addition involves injecting in-situ generated Gaussian noise samples into the training samples based on the target noise intensity corresponding to the current training time step, thus constructing a noisy training input for the current time step. Network parameter learning involves sending the noisy training input to the sampling calculation module to perform denoising prediction calculations for the diffusion network, and updating the diffusion network parameters based on the error between the prediction result and the supervised target. The core of this training process lies in directly generating and adjusting the noise required for training within the chip for different noise intensities needed at different time steps during the training phase, and then using this noise to construct the noisy input for the training samples to complete the diffusion model parameter training.

[0053] Specifically, the training mode can be executed according to the following steps:

[0054] S1. Time step scheduling and parameter reading:

[0055] During training, the original training samples are first input, and the current training time step is determined under the coordination of the control and management module. Subsequently, the time step scheduling module outputs the scheduling parameters corresponding to the current training time step; the scheduling parameters are used to characterize the target noise intensity at the current time step.

[0056] S2, Noise Intensity Adaptive Mapping:

[0057] The noise intensity mapping module receives the scheduling parameters and adaptively maps them within the hardware to the in-situ noise control parameters required for the current time step. The in-situ noise control parameters may be, for example, group control voltage, bias parameters, or other control quantities used to control the output statistical characteristics of the in-situ Gaussian noise generation module.

[0058] S3. In-situ generation and injection of Gaussian noise:

[0059] The in-situ Gaussian noise generation module adjusts the output probability of the underlying array directly within the chip as needed, based on the in-situ noise control parameters, thereby generating Gaussian noise samples that match the target noise intensity requirements of the current time step. These samples can be directly injected into the sampling calculation module in a streaming manner, or temporarily stored in the feature cache module and then sent in batches as needed, thus significantly reducing the bandwidth consumption of off-chip random number transmission.

[0060] S4. Status and Feature Reading:

[0061] The diffusion state of the sample to be processed in the diffusion model at the current time step, as well as the intermediate features of the diffusion network, are read from the feature cache module. In this training mode, the diffusion state at the current time step is the original training sample, or a noisy training input obtained by combining the original training sample with Gaussian noise samples generated in situ.

[0062] S5, Diffusion Network Collaborative Computation:

[0063] The current diffusion state, current time step information, and in-situ generated Gaussian noise samples are synchronously fed into the sampling calculation module. The sampling calculation module uses its internal convolution array or tensor operation unit to perform feature transformation and residual estimation on the high-dimensional image data, executes the denoising prediction calculation of the diffusion network, and outputs the denoising prediction result corresponding to the current time step.

[0064] S6. Adaptive Diffusion Sampling State Update and Network Parameter Learning:

[0065] In training mode, loss is calculated based on the error between the denoised prediction result output by the sampling calculation module and the supervised target, and network parameter learning is completed. The supervised target is the in-situ Gaussian noise sample actually added at the current time step or its corresponding equivalent supervised quantity. For scenarios requiring cross-time step training state transfer, the state update module can also generate the diffusion state for the next adjacent time step based on the calculation result at the current time step combined with the adaptive diffusion sampling formula of the corresponding diffusion model.

[0066] S7. Feature Writeback and Iteration Control:

[0067] The updated diffusion state, relevant intermediate features, or intermediate results generated during training are written back to the feature cache module. In multi-time-step training scenarios, the control management module continues to coordinate the in-situ noise generation, denoising prediction calculation, and training state update of subsequent time steps, so that the updated diffusion state becomes the new current diffusion state and enters the subsequent processing flow. The above steps are executed repeatedly until the entire training process is completed.

[0068] Furthermore, in batch training scenarios, the in-situ Gaussian noise generation module can generate random outputs in parallel for multiple samples in the current batch, and the noise intensity mapping module performs corresponding hardware scaling according to the unified time step setting of the current batch, thereby providing parallel noisy input to the training network; in sample-by-sample training scenarios, the control and management module can select training samples sequentially and update the time step parameters sequentially to complete sequential training. Regardless of the implementation method, the core lies in relying on the attached... Figure 2 The process shown completes the main training chain of "time step selection - forward noise addition - in-situ noise generation - hardware scaling - noisy training - network parameter learning".

[0069] Example 3: Image generation mode based on iADS architecture

[0070] This example illustrates the working method and process of configuring the iADS architecture in image generation mode, and its corresponding appendix... Figure 3The image generation mode executes the inverse denoising inference process of the diffusion model. Starting from an initial random state, it iteratively performs in-situ Gaussian noise generation, denoising prediction calculation, and diffusion state recursive update at multiple time steps, gradually completing the image state update and finally outputting the generated image. The core of the image generation process lies in: for different target noise intensities corresponding to different time steps, directly generating and adjusting Gaussian noise samples matching the current time step within the chip, enabling noise generation, noise intensity adjustment, diffusion network denoising prediction, and diffusion state update to be completed efficiently in a closed loop within the same hardware architecture. Specifically, the image generation mode can be executed according to the following steps:

[0071] S1. Time step scheduling and parameter reading:

[0072] At the start of image generation, the control management module coordinates with the time step scheduling module to provide the current time step information according to a preset reverse sampling order, and outputs the scheduling parameters corresponding to that time step; the scheduling parameters are used to characterize the target noise intensity of the current time step. For the initial time step, the current diffusion state is a purely random Gaussian noise initial state.

[0073] S2, Noise Intensity Adaptive Mapping:

[0074] The noise intensity mapping module receives the scheduling parameters and adaptively maps them to the corresponding in-situ noise control parameters within the hardware. The in-situ noise control parameters may be, for example, group control voltage, bias parameters, or other control quantities used to control the output statistical characteristics of the in-situ Gaussian noise generation module.

[0075] S3. In-situ generation and injection of Gaussian noise:

[0076] The in-situ Gaussian noise generation module generates Gaussian noise samples that match the target noise intensity at the current time step directly within the chip, based on the in-situ noise control parameters. At the initial time step, the in-situ generated Gaussian noise samples can be written into the feature cache module as the initial current diffusion state. In subsequent time steps, the Gaussian noise samples can be directly injected into the sampling calculation module, or temporarily stored in the feature cache module and then sent in as needed, thereby reducing the bandwidth consumption of off-chip random number transmission.

[0077] S4. Status and Feature Reading:

[0078] The diffusion state of the sample to be processed in the diffusion model at the current time step, as well as the intermediate features of the diffusion network, are read from the feature cache module. In the image generation mode, the diffusion state at the current time step is initially a purely random Gaussian noise initial state, and in subsequent stages it is an intermediate diffusion state that is recursively updated from the previous time step and written back to the feature cache module.

[0079] S5, Diffusion Network Collaborative Computation:

[0080] The current diffusion state, current time step information, and in-situ generated Gaussian noise samples are synchronously fed into the sampling calculation module. The sampling calculation module uses its internal neural network computing array, tensor operation unit, or attention calculation unit and other underlying computing hardware to perform denoising prediction calculations for the diffusion network, thereby outputting the predicted noise tensor, residual estimation results, denoising intermediate results, and intermediate calculation results for the state update module to generate the diffusion state for the next time step at the current time step.

[0081] S6. Adaptive diffusion sampling state update:

[0082] The state update module recursively updates the diffusion state at the current time step based on the prediction results output by the sampling calculation module, combined with the Gaussian noise sample corresponding to the current time step and the adaptive diffusion sampling formula of the diffusion model, to generate the diffusion state of the next adjacent time step. The diffusion state of the next adjacent time step is calculated from the current diffusion state and continues to participate in the calculation as the new current diffusion state in subsequent iterations.

[0083] Furthermore, the adaptive diffusion sampling formula can be configured according to the type of diffusion model used. For example, when using the DDPM model, the state update module recursively updates the current diffusion state based on the DDPM random backsampling formula; when using the DDIM model, the state update module recursively updates the current diffusion state based on the DDIM deterministic or weakly random sampling formula. By configuring the sampling recursion formulas corresponding to different diffusion models, the iADS architecture can adapt to the image generation process of different diffusion models.

[0084] S7. Feature Writeback and Iteration Control:

[0085] The updated diffusion state of the next time step is written back to the feature cache module. In the multi-time-step iterative image generation scenario, the control management module continues to coordinate the in-situ noise generation, denoising prediction calculation and diffusion state update of the subsequent time steps, so that the diffusion state of the next time step is used as the new current diffusion state to enter the subsequent processing flow, and the above steps are executed repeatedly until the entire image generation process is completed and the final generated image is output.

[0086] In the image generation process, the iADS architecture of this invention does not limit itself to generating random noise alone. Instead, it dynamically adjusts the in-situ Gaussian noise output according to the target noise intensity at different time steps, ensuring that image generation at each time step achieves noise control results that match the theoretical diffusion schedule. Compared to pre-generating and storing noise for all time steps offline, this approach reduces external storage dependencies and noise transport overhead during image generation, and enables noise generation, hardware scaling, denoising prediction, and diffusion state updates to be executed continuously within a unified hardware path. Diffusion state updates are completed continuously within the same hardware architecture.

[0087] In the iADS architecture of this invention, the close collaboration between the time step scheduling module, the noise intensity mapping module, and the in-situ Gaussian noise generation module implements a native hardware scaling strategy at the chip level. A schematic diagram illustrating the variation of this hardware scaling strategy with the time step size compared to traditional scaling strategies (such as linear scheduling, cosine scheduling, and S-shaped scheduling) is shown below. Figure 4 As shown, the diffusion model corresponds to different noise intensity requirements at different time steps. Traditional strategies typically rely on complex mathematical formulas to calculate the theoretical target scaling value in software or external digital processing units, and then call a digital multiplier to apply this scaling value to the standard random noise sample. The traditional architecture involves a "digital path where the generator outputs standard noise, and then an independent digital multiplier is called to multiply it with the algorithm scheduling coefficient." However, this invention abandons this mathematical calculation path and traditional architecture, directly relying on the underlying physical mechanism within the iADS architecture to naturally achieve the time-evolving scheduling output. This invention only needs to set a mechanism for the evolution of a control signal (such as control voltage or drive current) over time steps and apply this control signal directly to the on-chip noise generation unit. Relying on the adjustable output statistical characteristics of the underlying physical devices (such as the device's natural nonlinear physical response curve), the evolution of the generated Gaussian noise sample intensity over time steps is entirely determined by the device's own physical characteristics. Thus, at the moment of physical excitation, the iADS architecture directly forms a hardware scheduling path that is entirely determined by the underlying physical mechanism, thereby completely bypassing the multiplier array used for digital amplitude modulation in the data path and greatly reducing the control overhead introduced to fit complex mathematical functions.

[0088] In this invention, the time-step scheduling module outputs corresponding scheduling parameters at each time step, which are then fed into the noise intensity mapping module. The noise intensity mapping module directly converts these parameters into in-situ noise control parameters for driving the underlying devices. The hardware scaling strategy is implemented based on the intrinsic statistical adjustability of voltage-adjustable p-bit devices. By changing physical control methods such as the control voltage, without any subsequent digital proportional multiplication coefficients, the in-situ Gaussian noise generation module, governed by its own nonlinear physical laws, naturally exhibits different standard deviation characteristics at different time steps. Thus, time-step information is no longer a mathematical label fed into the digital multiplier, but directly participates in and defines the evolution process of the underlying physical excitation.

[0089] Appendix Figure 4 This aims to illustrate the fundamental differences in generation mechanism and macroscopic performance between traditional mathematical scheduling and the native hardware scheduling of this invention. The linear, cosine, and sigmoid scheduling shown in the figure represent ideal theoretical mathematical curves, which highly rely on external digital processing units for precise calculations and subsequent multiplication scaling. In contrast, the hardware scaling strategy curve (thick solid line) in the figure represents an evolutionary trajectory naturally formed by the physical response characteristics (such as voltage-probabilistic nonlinear curves) of the underlying semiconductor device itself under the influence of simple control parameters (such as a linearly monotonically changing bias voltage). Further observation... Figure 4 As can be seen from the curve shape, compared to the linear scheduling that declines too rapidly in the early stages, the hardware scaling strategy curve of this invention exhibits a very smooth downward trend. Influenced by the physical characteristics of the underlying devices, its naturally formed curve trend is very similar to the cosine scheduling and S-shaped scheduling, which produce better actual generation results; both exhibit the characteristics of gradual initial changes and a smooth overall transition. This nonlinear smooth evolution, spontaneously formed by a purely physical mechanism, perfectly matches the stringent statistical requirements of high-order diffusion model algorithms for high-quality noise scheduling without introducing any complex mathematical compensation or digital multiplication fitting overhead. This results in excellent image generation effects and sampling convergence in actual hardware operation. Thus, the overhead of digital conversion, data transmission, and multi-level caching introduced to fit complex mathematical functions is completely eliminated.

[0090] In practical implementation, the hardware scaling strategy can be achieved by setting a simple set of control signal evolution relationships (such as a preset voltage lookup table or segmented control lines). Specifically, the time-step scheduling module advances according to the beat and outputs scheduling parameters, which are then converted into corresponding in-situ noise control parameters by the noise intensity mapping module. Upon receiving this physical control, the in-situ Gaussian noise generation module utilizes the intrinsic physical response law of its underlying on-chip noise generation unit to change its operating bias or driving state, so that it naturally outputs Gaussian noise samples matching the statistical intensity of the diffusion evolution stage at the moment of excitation. Thus, the iADS architecture perfectly reduces the pure mathematical scheduling of the algorithm layer to pure physical actions at the underlying level during training and image generation, eliminating the need to rely on traditional external digital scaling processes.

[0091] In summary, the specific embodiments of the present invention are illustrated in the appendix. Figure 1 To be continued Figure 4 The content shown is the basis for further development, including the appendix. Figure 1 Corresponding to the overall structure of the iADS in-situ adaptive diffusion sampling architecture, attached Figure 2 The workflow for training diffusion models using the iADS architecture is attached. Figure 3 The corresponding workflow for image generation using the iADS architecture is attached. Figure 4 The relationship between the hardware scaling strategy and the traditional scaling strategy as a function of time step is shown. As can be seen from the above examples, the key point of this invention is to unify the in-situ Gaussian noise generation, time step control, and hardware scaling with the diffusion model training and image generation processes within the same hardware system, thereby improving the execution efficiency of the hardware implementation of the diffusion model.

[0092] Although the present invention has been described through preferred embodiments, those skilled in the art should understand that various changes or equivalent substitutions can be made to its structural form, connection method, control logic and parameter settings without departing from the concept of the present invention, and such changes or substitutions should all fall within the protection scope defined by the claims of the present invention.

Claims

1. An in-situ adaptive diffusion sampling architecture for hardware acceleration of diffusion models, characterized in that, It includes a time step scheduling module, a noise intensity mapping module, an in-situ Gaussian noise generation module, a feature caching module, a sampling calculation module, a state update module, and a control and management module; The control and management module is communicatively connected to the other modules to coordinate the working timing, data paths, parameter loading processes, and pipeline scheduling control of each module. The time step scheduling module generates the current sampling time step information of the diffusion model and sends the scheduling parameters corresponding to this time step, which characterize the target noise intensity, to the noise intensity mapping module. Simultaneously, it sends the time step information to the sampling calculation module. The noise intensity mapping module receives the scheduling parameters output by the time step scheduling module and converts them according to a preset control table, calibration mapping relationship, or segmented control relationship to generate in-situ noise control parameters corresponding to the target noise intensity of the current time step. The in-situ noise control parameters characterize the statistical intensity of the target Gaussian noise at the current time step and control the in-situ Gaussian noise generation module to output Gaussian noise samples that meet the statistical intensity requirements. The in-situ Gaussian noise generation module is used to generate Gaussian noise samples that match the current time step in-situ inside the chip according to the in-situ noise control parameters. The Gaussian noise samples are directly injected into the sampling calculation module or temporarily stored in the feature cache module. The feature cache module is used to store and retrieve the current diffusion state, intermediate features of the diffusion network, or the output state of the previous time step within the chip. The sampling calculation module serves as the physical execution carrier of the denoising network in the diffusion model. It is used to receive the current diffusion state from the feature cache module, the time step information from the time step scheduling module, and the Gaussian noise samples from the in-situ Gaussian noise generation module. It performs denoising prediction, residual estimation, and feature transformation calculation of the diffusion network at the hardware level and outputs the calculation results to the state update module for diffusion state recursive update. The state update module is used to recursively update the current diffusion state based on the denoising prediction results output by the sampling calculation module, the corresponding in-situ noise control parameters and Gaussian noise samples, and the adaptive diffusion sampling formula of the corresponding diffusion model, and write the generated next time step diffusion state back to the feature cache module for subsequent time step iterations.

2. The in-situ adaptive diffusion sampling architecture as described in claim 1, characterized in that, The in-situ Gaussian noise generation module employs an on-chip probability bit array, random number array, or adjustable random circuit array, along with corresponding drive circuits, readout circuits, statistical transformation circuits, or calibration circuits. Utilizing the adjustable output statistical characteristics of the underlying on-chip noise generation unit, the module changes the operating bias, drive state, or statistical output distribution of the on-chip noise generation unit through the in-situ noise control parameters. This allows the unit to directly output Gaussian noise samples that meet the target noise intensity requirements of the current time step during the noise generation stage, thereby achieving the integration of noise generation and noise intensity adjustment at the hardware level.

3. The in-situ adaptive diffusion sampling architecture as described in claim 1, characterized in that, The sampling calculation module includes underlying computing hardware for performing neural network acceleration: a graphics processing unit (GPU), a neural network processor (NPU), or a custom AI application-specific integrated circuit (ASIC). The underlying computing hardware performs denoising prediction, feature transformation, and residual estimation operations, thereby physically completing the denoising prediction and backsampling steps at a specific time step in the diffusion model.

4. The in-situ adaptive diffusion sampling architecture as described in claim 1, characterized in that, The control and management module is used to perform hardware timing coordination of in-situ noise generation, denoising prediction calculation and diffusion state write-back during multi-time step sampling, so that noise generation of subsequent data to be processed, denoising prediction calculation of current data and state write-back of previous data are carried out overlappingly inside the chip.

5. The in-situ adaptive diffusion sampling architecture as described in claim 1, characterized in that, Through the close collaboration of the time-step scheduling module, the noise intensity mapping module, and the in-situ Gaussian noise generation module, a native hardware scaling strategy is implemented at the chip level. The scaling refers to adjusting the statistical intensity of the Gaussian noise sample, where the statistical intensity is one of the following: standard deviation, variance, amplitude range, or equivalent noise intensity. The time-step scheduling module advances according to the beat and outputs scheduling parameters. The noise intensity mapping module converts these parameters into corresponding in-situ noise control parameters. After receiving the physical adjustment of the in-situ noise control parameters, the in-situ Gaussian noise generation module utilizes the intrinsic physical response law of its on-chip noise generation unit to change its operating bias or driving state, so that it naturally outputs a Gaussian noise sample with a statistical intensity matching the diffusion evolution stage at the moment of excitation.

6. A method for operating the in-situ adaptive diffusion sampling architecture as described in claim 1, characterized in that, The algorithmic logic of the diffusion model is mapped to the closed-loop workflow of the in-situ adaptive diffusion sampling architecture, thereby realizing the complete adaptive diffusion training and image generation process on-chip in-situ; specifically, it includes the following steps: S1. Time step scheduling and parameter reading: Under the unified timing coordination of the control management module, the time step scheduling module generates the sampling time step information of the current diffusion process and determines the scheduling parameters corresponding to the time step. S2. Noise intensity adaptive mapping: The noise intensity mapping module receives the scheduling parameters and adaptively maps them in the hardware to the in-situ noise control parameters required for the current time step. S3. In-situ Gaussian noise generation and injection: The in-situ Gaussian noise generation module generates Gaussian noise samples that match the target noise intensity of the current time step directly inside the chip according to the in-situ noise control parameters; the samples are injected directly into the sampling calculation module in a streaming manner, or temporarily stored in the feature cache module and then sent in batches as needed. S4. State and Feature Reading: Read the diffusion state of the sample to be processed in the diffusion model at the current time step and the intermediate features of the diffusion network from the feature cache module to realize the local reuse of data; wherein, the diffusion state includes the initial input state pre-written into the feature cache module and the intermediate diffusion state written back to the feature cache module after the state is updated in the previous time step. S5. Diffusion network collaborative calculation: The current diffusion state, current time step information and Gaussian noise samples generated in situ are synchronously sent to the sampling calculation module to perform denoising prediction calculation of the denoising network in the diffusion model at the hardware level. S6. Adaptive Diffusion Sampling State Update: The state update module updates the diffusion state at the current time step by recursively updating the diffusion state at the current time step based on the prediction results output by the sampling calculation module and the adaptive diffusion sampling formula of the corresponding diffusion model, generating the diffusion state of the next adjacent time step, and continuing to participate in the calculation as the new current diffusion state in subsequent iterations. S7. Feature Writeback and Iterative Control: Write the updated diffusion state of the next time step back to the feature cache module; in the multi-time step iterative sampling scenario, the control management module continues to coordinate the in-situ noise generation, denoising prediction calculation and diffusion state update of subsequent time steps, so that the diffusion state of the next time step enters the subsequent processing flow as the new current diffusion state, and executes steps S1 to S7 in a loop until the entire diffusion model training or image generation process is completed; in the single-step processing scenario, output the updated state corresponding to the time step and end the process.

7. The working method as described in claim 6, characterized in that, Configure as training mode or image generation mode: When configured in training mode, the architecture performs forward noise addition and network parameter learning. In step S4, the current diffusion state is read as a training sample or a noisy state constructed from the training sample. In step S5, the sampling calculation module performs denoising prediction calculation of the diffusion network in combination with the Gaussian noise samples generated in situ. In step S6, the training error is calculated based on the denoising prediction result and the supervision target, and the network parameter learning is completed accordingly. In step S7, the relevant intermediate features or intermediate results formed during the training process are written back to the feature cache module. When configured in image generation mode, the architecture performs a reverse denoising inference process. In the initial time step, the Gaussian noise sample generated in step S3 is written into the feature cache module as the initial current diffusion state. In step S4, the diffusion state at the current time step is initially a purely random Gaussian noise initial state, and in subsequent stages it is an intermediate diffusion state that is recursively updated by the previous time step and written back to the feature cache module. In step S6, the state update module gradually removes the noise components in the current diffusion state according to the adaptive diffusion sampling formula of different diffusion models and the denoising prediction results, calculates and generates the diffusion state of the next time step, and outputs the final generated image after multiple time step iterations.

Images