Heterogeneous nerve processing system of generative AI model

Abstract

A heterogeneous neural processing system includes a first processor configured to perform encoding and decoding operations of an autoencoder, and a second processor configured to perform neural network operations of a particular task, the operations having iterative processing. The processors perform compute tasks through synchronous data exchange to implement a generative AI model. The first processor processes a feature map divided into data rows, caches the data into active memory using row-based depth-first scheduling, and selects deeper operations in the network hierarchy when processing branch inputs, outputs, and residual connections. And H reuses boundary pixels between the cache storage space segments, so that convolution and element-level operation can be executed simultaneously. A neural network adjusts the device analysis model to identify layer dependencies, applies search space constraints, performs an iterative search to generate a fusion plan, and selects an optimal plan based on external memory access and execution delays.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 689,015, filed August 30, 2024. The contents of that application are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to technical implementations of supporting generative artificial intelligence on edge devices. BACKGROUND

[0004] Generative artificial intelligence (AI) exhibits tremendous potential to revolutionize user experiences through its ability to generate images with superior perceptual quality. The ability of generative models to process diverse input modalities enables a variety of image synthesis and editing applications, such as text-to-image generation, high-resolution image inpainting, and image restoration. These applications often employ sophisticated neural network architectures, including latent diffusion models, autoencoder-based inpainting systems, and multi-modal conditioning networks.

[0005] Supporting generative AI applications on edge devices presents a series of important technical challenges that require novel solutions to address multiple key constraints simultaneously. These constraints include maintaining low latency, minimizing power consumption, reducing bandwidth usage, and optimizing silicon area. Edge devices such as smartphones, tablets, autonomous vehicles, and Internet of Things (IoT) systems face resource limitations that are fundamentally different from the abundant resources available in cloud computing environments. For example, edge devices must provide real-time performance to support user-interactive applications while operating within strict thermal budgets, limited battery capacity, and stringent cost-sensitive manufacturing targets. These limitations make it impractical to deploy high-power, large-area processing solutions that are typically used in data center or cloud systems.

[0006] The area limitations of edge devices further complicate the situation. Due to limited on-chip memory and processing capabilities, designing a neural processing unit (NPU) capable of efficiently processing complex AI models becomes challenging without consuming excessive power or occupying excessive silicon area. The silicon area allocated to neural processing is typically only a small fraction of the total area of a system-on-chip (SoC), which must also accommodate other essential components such as central processing units (CPUs), graphics processors, communication interfaces, and power management units. Therefore, the area available for the NPU limits the number of processing elements, the capacity of on-chip memory, and the complexity of control logic that can be integrated.

[0007] In addition to these hardware challenges, edge devices must also remain compatible with existing software ecosystems while providing sufficient compute capability to support the growing complexity of modern neural network architectures. This trend toward increasingly complex models enhances the need for efficient processing architectures capable of handling complex neural network topologies. These architectures must not only minimize external memory access and reduce execution latency, but also ensure high hardware utilization within the compact area and power budgets typical of edge devices, thereby enabling real-time AI performance without sacrificing battery life or overall device functionality. SUMMARY

[0008] A heterogeneous neural processing system is provided, including a first processor for performing encoding operations and decoding operations of an autoencoder, and a second processor for performing neural network operations of a specific task that perform iterative processing. The first processor and the second processor perform a computational task through synchronous data exchange to implement a generative AI model.

[0009] In certain aspects, the encoding operations convert input data to a latent representation, and the decoding operations reconstruct the latent representation back to output data. One embodiment provides concurrent execution, where the first processor performs the encoding operations and the decoding operations, while the second processor simultaneously performs the neural network operations of the specific task on the same data of the generative AI model.

[0010] In certain aspects, a row-based depth-first processing capability is provided, where the first processor is further to process feature maps of the neural network, partitioned into a plurality of data rows, cache the plurality of data rows to an activation memory, and determine whether a required portion of the plurality of data rows is cached to the activation memory and select a deeper operation in a network hierarchy. In certain aspects, the neural network includes branch inputs, branch outputs, and residual connections processed within a fusion layer stack.

[0011] In certain aspects, the first processor includes an H-reuse cache to store boundary pixels between adjacent spatial segments. In certain aspects, the first processor is further to concurrently perform a convolution and an element-level operation, the element-level operation including an addition and / or concatenation. One embodiment provides a ping-pong buffer to fetch a next data segment while processing a current segment.

[0012] In certain aspects, a neural network tuning device is to analyze a neural network model to identify layer dependencies and fusion boundaries, apply constraints to a search space according to memory capacity and processing capability to define a fusion configuration, perform an iterative search within the search space to generate a plurality of fusion plans, and select a fusion plan from the plurality of fusion plans according to external memory access and execution latency.

[0013] In certain aspects, the neural network operation for a particular task includes a denoising operation performed by a U-Net architecture. In certain aspects, the neural network operation for a particular task includes a conditioning operation that encodes semantic vectors or text token embeddings into latent space representations, or an attention mechanism with query, key, and value components.

[0014] To the accomplishment of the foregoing and related ends, certain embodiments comprise the features hereinafter fully described and particularly pointed out in the claims. The following description and the annexed drawings set forth in detail certain illustrative aspects of the embodiments. These aspects are indicative, however, of but a few of the various ways in which principles of the present disclosure can be employed and the present disclosure is intended to include all such aspects and their equivalents. These and other objects, along with the advantages thereof over prior devices of a similar nature, are apparent upon reading the detailed description of the preferred embodiment and upon referring to the associated drawings. BRIEF DESCRIPTION OF DRAWINGS

[0015] The following drawings form part of the present specification and are included to further demonstrate certain aspects of the present disclosure. The application can be better understood by reference to one or more of these drawings in combination with the detailed description of specific embodiments presented herein.

[0016] Figure 1A A generative AI model architecture running in pixel space and latent space is shown.

[0017] Figure 1B A heterogeneous neural processing system for executing a generative artificial intelligence model according to one embodiment is shown.

[0018] Figure 2A An overall architecture of a MAE (Mini AutoEncoder) chip according to one embodiment is shown.

[0019] Figure 2B A MAE data flow in a computing process according to one embodiment is shown.

[0020] Figure 3A One layer fusion and branching example is shown, representing a typical neural network stack with complex connection patterns.

[0021] Figure 3B A row-based depth-first execution method according to one embodiment is shown.

[0022] Figure 3C A scheduling order and memory management according to one embodiment is shown.

[0023] Figure 4A Performance comparison results showing reduced external memory access relative to L1 size using depth-first fusion and branching processing are shown.

[0024] Figure 4B The performance comparison results show the latency reduction relative to bandwidth achieved by using depth-first fusion and branching processing.

[0025] Figure 5A The data flow architecture within the convolutional and addition / connection units according to one embodiment is shown.

[0026] Figure 5B A simplified architecture overview of direct links between convolutional units and A / C units according to one embodiment is shown.

[0027] Figure 6 A convolution row processing workflow with local reuse is illustrated according to one embodiment.

[0028] Figure 7A The effectiveness of the H reuse cache compared to rereading data from L1 memory is shown.

[0029] Figure 7B The performance comparison results are presented, showing the reduction in latency when using H reuse caching and direct links.

[0030] Figure 8A An example of the optimization challenge of scheduling a DNN model with multiple branches is shown.

[0031] Figure 8B A global scheduling optimization based on one embodiment is illustrated.

[0032] Figure 9A The performance analysis results are presented, showing the feasible region analysis for EMA-delay.

[0033] Figure 9B The performance comparison results are presented, showing improvements in EMA and delay.

Detailed Implementation Methods

[0034] Heterogeneous neural processing systems represent a paradigm shift from traditional homogeneous architectures by integrating multiple specialized processing units, each optimized for a different computational workload. Unlike traditional single-processor approaches that attempt to handle all neural network operations with a unified architecture, heterogeneous systems strategically distribute computational tasks across processing units with complementary capabilities. This architectural approach recognizes that different neural network operations have drastically different computational characteristics, memory access patterns, and performance requirements that cannot be optimally served by a single processing unit design.

[0035] In a heterogeneous neural processing system, general-purpose neural processing units (NPUs) typically provide high throughput and low latency for compute-intensive operations, while specialized small NPUs handle specific workloads with high efficiency and low power consumption. General-purpose NPUs are designed with large-scale parallel processing capabilities, considerable on-chip memory, and flexible instruction sets to accommodate diverse neural network architectures and rapidly changing algorithmic requirements. In contrast, specialized small NPUs are optimized for specific operation types, with dedicated hardware modules, streamlined data paths, and minimal control overhead to maximize efficiency within strict area and power budgets. Each task is assigned to the most efficient processing core, leveraging the complementary strengths of different processing units through intelligent workload distribution algorithms. This task distribution process takes into account factors such as computational complexity, memory bandwidth requirements, data dependencies, and real-time constraints to determine the optimal processor allocation. The heterogeneous approach enables different neural network components to execute concurrently, allowing for overlapped computations and improved resource utilization compared to sequential processing methods.

[0036] For example, when deploying a generative diffusion model, processing cores can work in tandem, with specialized small NPUs handling autoencoder operations and general-purpose NPUs executing latency-dominated denoising networks and conditioning modules. Autoencoder operations involve relatively regular computational patterns and can benefit from specialized optimization, making them well-suited for the focused architecture of small NPUs. At the same time, complex iterative denoising processes and multi-modal conditioning operations require the flexibility and computational power provided by general-purpose NPUs. This collaborative processing approach enables efficient resource utilization while maintaining the performance characteristics required for real-time generative AI applications.

[0037] However, designing specialized small NPUs presents unique challenges due to very limited area budgets, which inherently restrict the number of multiply-accumulate (MAC) units and the amount of on-chip memory available. This limitation gives rise to two primary technical issues. First, the reduced on-chip memory necessitates more energy-consuming external memory accesses, with DRAM accesses potentially accounting for a significant portion of the total energy consumption of a tiered scheduling approach, impacting low-power objectives. Second, temporal underutilization can occur when computations are limited by input / output bandwidth or non-MAC operations, constraining overall performance and hardware efficiency.

[0038] Traditional neural network processing methods typically employ a tiered execution schedule, where the complete feature map of each layer is computed before proceeding to subsequent layers. While this approach is conceptually simple, it requires substantial intermediate storage and leads to frequent external memory accesses, which dominate power consumption in resource-constrained implementations. Tiered scheduling also fails to exploit opportunities for overlapped computations and memory accesses, which can improve overall efficiency.

[0039] Depth-first scheduling concepts have been proposed as an alternative to the hierarchical approach, where feature maps are segmented into chunks and processed in a depth-first order to reduce on-chip memory usage by consuming generated data immediately. Prior depth-first scheduling implementations have proven to be more efficient than traditional hierarchical approaches in reducing external memory accesses, as it prioritizes computation of deep layers in the fusion layer stack as long as input data is available.

[0040] Existing depth-first scheduling techniques face limitations when applied to modern neural network architectures. As neural network topologies become more complex, with branching inputs, branching outputs, and residual connections, prior depth-first scheduling methods typically revert to less efficient hierarchical scheduling when encountering such branching patterns. This limitation limits the potential fusion space and prevents these methods from achieving optimal memory access reduction in modern neural network architectures commonly used in generative AI applications.

[0041] Furthermore, traditional solutions lack mechanisms to enable concurrent execution of different types of operations, such as convolutions and element-wise operations, leading to underutilization of processing resources and increased latency. Traditional scheduling methods also do not provide a comprehensive optimization framework to handle the exponential search space complexity that arises when optimizing fusion decisions for networks with multiple branching points.

[0042] To address the aforementioned challenges, the following disclosure provides detailed descriptions of various embodiments. While specific implementation details are provided herein to facilitate a thorough understanding of the disclosure, it will be apparent to those skilled in the art that the present invention can be implemented without necessarily adhering to all of these details. In some cases, detailed descriptions of well-established methods, procedures, components, and circuits have been omitted to avoid obscuring the present disclosure. It should be understood that technical features described separately in individual figures can be implemented individually or in combination with other features, as described in this specification.

[0043] Figure 1A A heterogeneous neural processing system for executing generative artificial intelligence models is demonstrated in accordance with one embodiment. The system shows the mapping between models and corresponding hardware.

[0044] Figure 1A The model architecture in FIG. 1 represents a generative AI system that operates in pixel space and latent space. The model includes an encoder (ε) located in pixel space that converts input data from a pixel representation to a compressed latent representation (z). This encoding operation reduces the dimensionality of the input data while preserving essential features. The model also includes a decoder that reconstructs the latent representation back into a pixel space output The decoder performs the inverse operation of the encoder, expanding the latent representation into full-resolution output data.

[0045] The model operates in latent space, including a denoising U-Net that performs iterative denoising operations on the latent representation (z). The U-Net processes data through multiple stages with skip connections, cross-attention mechanisms, and various neural network operations, including switches and concatenations. The parameter τ θ represents a time-regulated denoising U-Net model. The denoising process repeats N iterations to progressively refine the latent representation, with each iteration controlled by a noise parameter τ θ guidance, which controls the denoising schedule. In the U-Net architecture, the query (Q), key (K), and value (V) components of the cross-attention mechanism enable text-conditioned generation. The Q vector is derived from the latent image representation, and the K and V vectors are generated from the conditioning information, such as text embeddings, enabling the model to focus on relevant semantic information during the denoising process. Additionally, the model contains a conditioning module as a task-specific component for integrating external regulatory information, such as semantic maps, text representations, or other control signals. The conditioning module influences the denoising process to guide the generation towards the desired output.

[0046] A hardware solution for the above model is illustrated in Figure 1B , which implements a heterogeneous processing system that optimally allocates different model components to specialized processing units. The system includes specialized hardware, including a specialized neural processing unit (NPU) designed specifically for performing encoder and decoder operations. The specialized hardware can be optimized for task-agnostic autoencoders, including encoders and decoders, to provide high efficiency and low power consumption. The system also contains a general-purpose NPU, which serves as a general-purpose neural processing unit to handle the denoising U-Net and the conditioning module. The general-purpose NPU provides higher throughput and lower latency for compute-intensive and task-specific operations.

[0047] Heterogeneous systems leverage the complementary strengths of each processing unit, where task-agnostic components benefit from specialized hardware optimizations, while task-specific components utilize general-purpose processing for flexibility and performance. Workload distribution optimizes resource utilization across the entire generative AI pipeline, and concurrent execution allows for parallel processing of different model components. The system processes data by letting input data enter an encoder on specialized hardware, the encoded latent representation flows to a denoising U-Net on general-purpose NPU, integrating conditional information during the denoising process. The refined latent representation then returns to a decoder on specialized hardware, with the final output generated in pixel space. This architecture balances specialized optimization with general-purpose flexibility, enabling efficient deployment of complex generative AI models on edge devices, ensuring high performance and energy efficiency.

[0048] Figure 2A and 2B An architecture of a MAE (Mini AutoEncoder) chip is shown, which includes on-chip memory for efficient neural network processing on edge devices.

[0049] Figure 2A An architecture of a MAE chip according to one embodiment is shown. The central processing core includes a tensor core, which contains a convolution unit, an add and concatenation unit (A / C unit), a vector unit for element-wise operations, and a tuner. The convolution unit contains a 576 8-bit MAC (multiply-accumulate) array, which is capable of processing convolutions with various dilation rates, strides, and kernel sizes. The MAC array generates quantized and activated outputs, which are sent to an output alignment buffer that formats the data as 4 pixels per word to match the L1 data format. The convolution unit includes an H-reuse local buffer, which stores boundary pixels while processing each row, reducing the need to access the main activated L1 memory and improving bandwidth utilization for stride and depth convolutions.

[0050] The outputs of the convolution unit are directly linked to the A / C unit through a dedicated connection path, allowing concurrent execution of convolution and element-wise operations to minimize core idle time during residual addition or concatenation operations. This direct link reduces latency by eliminating pipeline stalls between sequential operations. The tensor core components are coordinated by a local L1 arbiter, which manages memory access conflicts and ensures efficient utilization of shared L1 bandwidth between different processing units.

[0051] The control infrastructure includes a command engine (CMDE) that contains a layer configuration module, a command decoder, and an instruction L1 memory. The command engine determines the appropriate layer configuration, including parameters such as input / output dimensions, quantization parameters, and weight addresses. The command system also specifies the temporal execution order of each row and its corresponding active L1 address. To prevent pauses between command switches, the CMDE prefetches the next set of commands while the ongoing computation is in progress, ensuring continuous operation without interruption.

[0052] The compiler and scheduling system implements a multi-objective scheduler with an optimized iterative search. The system includes a fusion scheduler that determines the optimal layer fusion layout, a memory allocator that manages address allocation for various operation heights and active ranges, and a power / latency evaluator that evaluates the performance trade-off. The compiler generates a binary codec that produces a binary file (BinFile) containing all pre-compiled weights, commands, and configurations, which are stored in external DRAM for runtime access.

[0053] The memory subsystem includes a main active L1 memory and a comprehensive DMA (Direct Memory Access) system with format converters, pad / crop operations, command buffers, and data buffers to efficiently move data between on-chip and external memories. The entire system is connected through an interconnect network that facilitates communication between all components and maintains high bandwidth and low latency data transfer.

[0054] Figure 2B The MAE dataflow during computation is demonstrated, showing a row-based depth-first layer fusion with branch processing capability. The dataflow demonstrates the interaction between the weight L1 memory, the Conv Core, and the input feature map (IFM) and output feature map (OFM), as well as the active control system. The system processes operation parameters, including shape, kernel, and stride information, through the layer configuration module, while the command decoder manages command execution based on the instructions from the instruction L1 memory. The Conv Core receives weight data and operation parameters (OP Params), including shape, kernel, and stride specifications, through the weight address path of the layer configuration module.

[0055] The activation control (Act. Ctrl) module coordinates the data flow by managing the activation addresses (Act Addresses) that determine the locations of intermediate data storage and retrieval from the activation LI memory. This coordination ensures that the row-based processing approach can efficiently handle branching input, output, and residual connections in the fusion layer stack. The data flow architecture represents an advancement over traditional layer-wise scheduling approaches by enabling more fine-grained control over data movement and processing order, allowing deeper parts of the network to start processing before the entire intermediate layer is completed, while maintaining computational efficiency and reducing memory footprint requirements.

[0056] Figure 3A A layer fusion with branching examples is shown, representing a typical neural network stack with complex connection patterns. The network starts with a remote direct memory access (RDMA) operation (0) that loads a 224x224x32 input tensor, followed by a convolution 3x3 (Conv3x3) operation (1) with a stride of 2, reducing the spatial dimension to 112x112x48. The network then includes a convolution 1x1 (Conv1x1) operation (5) and a depthwise convolution 3x3 (DW Conv3x3) operation (6) that process data with dimensions 112x112x144. A key aspect of this topology is the presence of residual paths, indicated by arrows connecting different layers, and branching input points where data from multiple sources converge. The network also demonstrates branching output scenarios where the output of a single layer enters multiple subsequent operations, including convolution 1x1 operations (7, 9, 4) and addition operations (8, 10). The final output is handled by write direct memory access (WDMA) operations (2, 11) that write the results back to memory. The complex network topology with multiple branching points and residual connections represents a challenging architecture that previous depth-first scheduling approaches could not efficiently handle, often requiring a fallback to the less ideal layer-wise scheduling approach.

[0057] Figure 3B A row-based depth-first execution approach and scheduling system is demonstrated that can efficiently handle the complex topology shown in Figure 3A . The approach divides the feature maps into data rows, with each row serving as a computational unit. Operations are labeled as OP0 to OP11, corresponding to the network layers shown in Figure 3A , and are processed based on the availability of rows rather than strict layer-wise execution. The diagram shows how rows are tracked with specific row numbers (Ln 69 / 70 / 71, Ln 35, Ln 33, etc.) and demonstrates the temporal execution, with some operations shown as "pending" and others as "ready." This approach allows deeper operations in the network hierarchy to be executed as soon as their required input rows are available, rather than waiting for the entire feature map to be computed.

[0058] This row-based approach allows deeper operations in the network hierarchy to be executed as soon as their required input rows are available, rather than waiting for the entire feature map to be computed. The scheduling system evaluates at each time step which operations have all the required input rows in the active memory and selects the deeper operations in the network hierarchy while maintaining balance across parallel branches. This depth-first priority with branch balancing prevents any single branch from dominating the computational resources and ensures efficient utilization of available processing power. The graph shows complex data dependencies where operations such as OP5, OP6, OP7+8, and OP9+10 are coordinated according to the availability of rows. Operations OP7+8 and OP9+10 are shown as combined operations, indicating that these represent fused operations that can be executed together when their input dependencies are satisfied. The row-based processing approach represents a fundamental shift from traditional tile or hierarchical approaches, enabling more fine-grained scheduling decisions that can accommodate complex data dependencies in branched network architectures. This approach is particularly effective for neural networks with residual connections and branching patterns, as it allows deeper parts to start processing before the entire intermediate layer is completed, reducing the memory footprint required for intermediate activations while maintaining computational efficiency.

[0059] Figure 3C A scheduling order and memory management strategy that coordinates depth-first execution is demonstrated. The scheduling order is managed in the instruction LI memory, which maintains an operation queue (OP0→OP1→OP2→OP6→OP7+8→OP9+10→OP11) that balances between depth-first priority and parallel branch execution to prevent any single branch from dominating the computation. The architecture includes complex memory management, including: a weight LI memory that stores convolution weights and parameters, a convolution and addition / connection (Conv+A / C) unit that performs convolution and has addition / connection functionality, an activation LI memory that holds intermediate activation data and feature maps, an instruction LI memory that manages operation scheduling and control flow, a layer configuration module that manages operation parameters, an activation control block that coordinates data flow, a resizer that handles feature map resizing, and a vector unit for element-level operations. These components coordinate to minimize on-chip activation footprint while maintaining efficient data flow throughout the processing pipeline.

[0060] The required buffer over time part shows how memory allocation and deallocation is dynamically managed to adapt to different operations' memory requirements over time. This illustration demonstrates the temporal memory usage pattern where different operations (op8_33, op5_34, op5_33, op4_33, op4_34, etc.) require different amounts of buffer space at different execution stages. The system allocates memory blocks when an operation starts processing its data and automatically deallocates these blocks when the intermediate results are no longer needed by the subsequent operations in the fusion stack. This dynamic memory management approach allows multiple operations with overlapping execution windows to efficiently share the limited on-chip memory resources, with the allocator tracking the active ranges of different data buffers to minimize the peak memory consumption while ensuring that all required intermediate data remains available when needed.

[0061] The active memory allocation demonstrates how the active LI memory manages the memory of cache lines with different heights and overlapping active ranges. The memory allocator allocates addresses to minimize the required memory footprint, with different operations (op0_33, op1_34, op3_34, etc.) being allocated to specific memory regions that are effectively reused after the operations are completed. The timeline shows the evolution of memory allocation, with some regions being deallocated while others are allocated to achieve optimal memory utilization throughout the execution. Compared to traditional approaches, the demonstrated row-based depth-first approach with branch processing expands the potential fusion space with branch processing, enabling the system to reduce external memory accesses and maintain efficient utilization of processing resources, showing progress over previous depth-first scheduling techniques.

[0062] Figure 4A The EMA reduction using depth-first fusion with branch processing is demonstrated, with results shown for LI sizes from a pool of active dominant and weight dominant neural network models. The horizontal axis represents the normalized EMA, ranging from 0.4 to 1.0, while the vertical axis displays the total LI size in kilobytes, ranging from 160KB to 320KB. Three different scheduling methods are compared: depth-first with branch processing, depth-first without branch processing, and traditional layer-by-layer scheduling. The depth-first method with branch processing demonstrates an EMA reduction, improving by 28% compared to depth-first without branch processing and by 29% compared to layer-by-layer scheduling. The results show that the proposed branch-aware depth-first fusion outperforms both traditional methods across different LI memory sizes, with the proposed method's performance ranging from 0.46 to 0.54, while depth-first without branch ranges from 0.60 to 0.73, and layer-by-layer scheduling ranges from 0.94 to 1.00.

[0063] Figure 4BThe latency reduction using the depth-first fusion with branch handling is illustrated, and the latency improvement is shown against bandwidth. The horizontal axis represents the normalized latency, from 0.4 to 1.0, and the vertical axis shows the bandwidth limit, in GB / s, from 0 to 4. The proposed depth-first method with branch handling outperforms the depth-first method without branch handling by 19% and the layer-wise scheduling by 15%. Figure 4A Similarly, the three scheduling methods are compared using the same marking convention. The proposed depth-first method with branch handling achieves a significant latency reduction, 19% compared to the depth-first method without branch handling and 15% compared to the layer-wise scheduling. The performance data show that the proposed method keeps the latency value between 0.48 and 0.69 under different bandwidth limits, while the depth-first method without branch handling ranges from 0.50 to 0.85 and the layer-wise scheduling is close to the baseline value of 1.00. The results show that bandwidth-limited scenarios benefit particularly from the proposed method, and more improvement is observed under lower bandwidth limits.

[0064] The ability to handle branch input, output, and residual connections within the fusion layer stack enables deeper fusion than previous depth-first methods, which typically fall back to the less efficient layer-wise scheduling when encountering complex network topologies. The average results across activation-dominant and weight-dominant models demonstrate the universal applicability and robustness of the proposed method in different neural network architectures commonly used in generative AI applications.

[0065] Figure 5A A dataflow architecture within the convolution and A / C units is illustrated according to one embodiment. The figure demonstrates the H-reuse buffer mechanism and direct link paths, enabling convolution and element-level operations to be executed simultaneously to reduce latency and improve bandwidth utilization.

[0066] The upper part shows the convolution unit dataflow, which receives input from the activation L1 arbiter. The convolution unit contains an input local buffer that employs a ping-pong mechanism, one buffer set fetches the next data segment from the activation L1 while the other set is connected to the MAC array, ensuring continuous processing without pipeline stalls. The input H-reuse buffer stores the boundary pixels between adjacent spatial segments to avoid re-reading overlapping data from the main L1 memory.

[0067] A data sequencer coordinates the input data flow to the MAC array, which processes spatial segments of 8 pixels in a parallel fashion, while different segments are mapped to temporal execution. Weight data is provided to the MAC array from the weight L1 memory through weight registers. The MAC array generates accumulations, which are processed through quantization and activation functions (annotated as PReLU / ReLU / Logistic in the figure) before being sent to the output alignment buffer. The output alignment buffer formats the processed data to match the L1 data format requirements. The output is then packed by the output packer and sent to the WDMA (Weight Direct Memory Access) and / or the activation L1 arbiter for storage or further processing.

[0068] The lower portion illustrates the A / C unit and the direct link path, enabling simultaneous execution between the convolution unit and the A / C unit. Data can flow directly from the convolution unit to the A / C unit through the direct link path, bypassing intermediate storage, enabling simultaneous execution of convolution and element-wise operations. Alternatively, data can arrive at the A / C unit from the RDMA (Remote Direct Memory Access) and / or the activation L1 arbiter. The processing results can be formatted by the output packer and sent to the WDMA and / or the activation L1 arbiter.

[0069] Figure 5B A simplified architecture overview of the direct link between the convolution unit and the A / C unit is illustrated according to one embodiment. The figure shows the connectivity and data flow paths. The activation L1 memory serves as the central data repository for the system. Two L1 arbiters (L1 ARB) manage access to the shared activation L1 memory, providing conflict-free operation when multiple units require simultaneous memory access. The convolution unit and the A / C unit are positioned through the direct link path, showing the direct connection enabling data to flow directly from the convolution unit output to the A / C unit input without intermediate storage in the activation L1 memory.

[0070] There is also an optional path that represents the data flow when direct linking is not used by activating the LI memory. The architecture overview demonstrates the flexibility of the architecture to operate in a direct link mode for maximum efficiency or in a traditional memory mediated data flow operation when required for a specific sequence of operations. The LI arbiter coordinates the access mode to prevent memory conflicts and ensure efficient bandwidth utilization for all processing units. Furthermore, the architecture minimizes core idle time during residual addition or connection operations by allowing the convolution and A / C units to operate simultaneously. The LI arbiter resolves memory contention at runtime, ensuring conflict-free and efficient use of the shared LI bandwidth between different processing units. The combination of the H-reuse cache and direct link connections reduces the average latency compared to traditional methods that sequentially process the convolution and element-level operations. The ping-pong buffering mechanism ensures that the row processing remains continuous and there is no pipeline stall, while the H-reuse cache specifically targets the redundant memory accesses that occur when processing overlapping spatial regions during convolution operations.

[0071] Figure 6 A 3x3 convolution row processing workflow is demonstrated with local reuse according to one embodiment. The illustration demonstrates how the h-reuse cache mechanism reduces memory accesses by storing boundary pixels between adjacent spatial segments during row-based processing. The workflow demonstrates the time progression of processing a row of data through three main stages: input (In), calculation (Calc), and output (Out). The horizontal axis represents the time progression, demonstrating how different pixel segments are processed sequentially while maintaining data reuse efficiency. The input stage demonstrates the flow of data from the LI memory to the input buffer, with specific pixel segments labeled p0-3, p4-7, p5-6, p8-11, p12-15, p14-15, p16-19, and p20-23. These segments represent overlapping windows of pixels processed by the 3x3 convolution operation.

[0072] The h-reuse cache mechanism demonstrates how boundary pixels are stored locally to avoid redundant memory accesses. For example, when processing segment p4-7, the h-reuse cache retains the boundary pixels that will be needed for the subsequent segment p8-11. This caching approach is particularly effective because the 3x3 convolution requires overlapping input data from adjacent spatial locations. The cache stores overlapping regions (e.g., pixels p5-6 that span segments p0-3 and p4-7) locally, eliminating the need to re-read these pixels from the LI memory.

[0073] The calculation stage demonstrates how the MAC array processes these segments, with operation labels p0-6, p7-14, and p15-22. These calculation windows demonstrate that the processing segments are slightly larger than the input segments due to the convolution kernel requirements. The overlapping nature of these calculation segments provides the necessary boundary pixels without additional memory accesses through the h-reuse cache.

[0074] The output stage shows the processing results formatted and sent back to memory through a packing mechanism. Output stages p0-3, p4-6, p7, p8-11, p12-14, p15, p15-19, and p20-21 show the final processed pixels written back to the L1 memory. The workflow includes a "flush if no more pixels" indication, showing the system processing the end of a row to ensure all remaining data is correctly output.

[0075] This row processing workflow with local reuse shows the reduction in memory accesses compared to traditional methods. By caching boundary pixels locally, the system avoids the redundant L1 memory reads that would otherwise be required for the overlapping regions in a 3x3 convolution. The temporal organization shows how the continuous processing is maintained through careful coordination of the input buffering, local caching, computation, and output packing stages.

[0076] Figure 7A The effectiveness of the H-reuse cache compared to re-reading data from the L1 memory is shown, demonstrating the quantitative performance improvement for different convolution operation configurations. The left side of the figure shows the spatial data organization for the convolution processing, showing the input data structure for each unrolled spatial stage. The illustration shows a data segment of width d*(kh-1) with 8*s spatial elements, where the boundaries represent overlapping regions that can be either re-read from the L1 memory or retrieved from the H-reuse cache. The parameters d, s, k h , and k v represent the dilation, stride, and kernel size, respectively. The overlapping boundary regions can be efficiently cached locally instead of repeatedly accessed from the main L1 memory.

[0077] The right side shows the quantitative performance results comparing the total L1 access amount for the traditional re-reading method and the H-reuse cache method. Three different convolution configurations are evaluated: (k h , k v , d, s) = (3, 3, 1, 1), (3, 3, 1, 2), and (3, 3, 2, 1), representing different combinations of kernel size, dilation rate, and stride values. For the (3, 3, 1, 1) configuration, the H-reuse cache achieves a 20% reduction in total L1 accesses compared to the re-reading method. The (3, 3, 1, 2) configuration shows an 11% improvement, while the (3, 3, 2, 1) configuration shows the advantage of a 33% reduction in L1 accesses.

[0078] The results show that the H-reuse cache is particularly effective for operations with different stride and dilation patterns.

[0079] The 33% improvement for the (3, 3, 2, 1) configuration indicates that the dilated convolution benefits from the boundary pixel cache due to its sparse access pattern. The different improvement percentages for different configurations reveal the adaptive nature of the H-reuse cache performance across different sets of convolution parameters, with more benefits observed for operations with higher spatial overlap or more complex access patterns.

[0080] The area overhead of implementing the H-reuse buffer is approximately 2.3% of the total chip area, showing the possibility of achieving significant performance improvement with low hardware cost. The mechanism also brings two additional benefits. First, since L1 memory contention is reduced, the latency is no longer limited by on-chip bandwidth. Second, more on-chip bandwidth is available for the direct link path between processing units. This released bandwidth enables convolution and element-wise operations to be performed simultaneously, resulting in an overall 22% latency reduction achieved by combining H-reuse and direct link architecture. The H-reuse buffer mechanism thus becomes an efficient trade-off between moderate area overhead and memory access optimization, particularly suitable for bandwidth-limited scenarios common in edge device implementations.

[0081] Figure 7B The latency reduction achieved by H-reuse buffer and direct link is demonstrated, embodying the performance advantage brought by the combination of H-reuse buffer and direct link architecture. The bar chart compares four different system configurations: Re-read + No DL (without direct link), Reuse + No DL (without direct link), Re-read + DL (with direct link), and Reuse + DL (with direct link). The normalized latency values decrease from approximately 1.0 for the baseline configuration to approximately 0.78 for the optimized configuration.

[0082] The results show that the latency improves gradually with optimization. The use of H-reuse buffer alone (without direct link) provides a 6% latency reduction compared to the baseline re-read method. The use of direct link alone (without H-reuse) achieves a 15% latency improvement compared to the baseline. The combination of implementing H-reuse buffer and direct link brings the greatest benefit, achieving a total of 22% latency reduction.

[0083] Figure 8A The scheduling of a DNN model with multiple branches is demonstrated, embodying the optimization challenges faced in determining efficient execution strategies for complex neural network topologies. The figure shows a representative deep neural network (DNN) architecture, containing local and global branch structures. The local branch and merge blocks on the left represent a common pattern in modern neural networks, where data is converged through multiple parallel paths. This local branch pattern includes several processing blocks connected through direct sequential paths and bypass connections, demonstrating the residual or skip connection type common in ResNet or similar designs. The blocks within this section show various interconnection patterns, including parallel processing paths that merge at specific points.

[0084] The right-hand global branching and fusion block represents longer-range branching behavior, where data paths diverge early in the network and re-converge after passing through multiple intermediate processing stages. The global branching structure reveals that network topology can become more complex when multiple branching points are distributed throughout the architecture. The optimization challenge is to find a network partition that minimizes memory accesses and latency, representing a fundamental problem that scheduling systems must solve when dealing with complex network topologies. This challenge is exacerbated by two factors. First, as the number of branching points increases, the computational complexity of finding an optimal scheduling solution grows exponentially, making brute-force optimization approaches infeasible for practical neural networks. Second, the optimization problem involves balancing two potentially competing objectives: minimizing external memory accesses (EMA) and reducing latency.

[0085] Figure 8B A scheduling optimization method according to one embodiment is demonstrated. This method addresses the complex branching challenge shown in Figure 8A The greedy scheduling with local fusion represents a traditional layer fusion optimization approach. The graph shows a series of processing blocks with local fusion boundaries, where operations are grouped into fusion segments (N / 2, N) according to local optimization decisions. The greedy scheduling follows a "fuse as deep as possible" strategy, attempting to maximize the number of layers that can be fused within each segment in a local region. However, this approach results in a total access volume of = 2N for memory operations.

[0086] In contrast, the optimized fusion scheduling demonstrates an improved scheduling approach that creates a fusion pattern with segments of varying sizes, resulting in a total access volume of = 1.5N. This represents a 25% reduction in memory accesses compared to the greedy scheduling. The optimized fusion scheduling employs a strategy that considers the minimum access breakpoint globally, indicating that fusion decisions are made based on a global analysis of the entire network rather than local optimization. This global consideration enables the identification of optimal breakpoints, thereby minimizing the overall memory access across the entire execution sequence.

[0087] The compilation flow achieved through the optimized scheduling begins with NN model input, followed by DAG computation graph analysis to understand network structure and dependencies. Search space limitations through iterative search and result-guided adjustments are implemented to manage the exponential complexity of the optimization problem. This iterative approach optimizes the search space step-by-step based on intermediate results, making optimization of complex networks computationally feasible.

[0088] Within the restricted search space, a fusion scheduler running on subgraphs is used to evaluate different fusion possibilities. An evaluation component evaluates the effectiveness / EMA / latency of each potential fusion configuration, ensuring that the proposed method meets hardware constraints while optimizing external memory accesses and execution latency. The evaluation process considers multiple objectives simultaneously, balancing the trade-off between memory efficiency and performance.

[0089] The compilation flow ends with fusion stack generation, followed by memory allocation optimization and binary generation (BinaryGen) to produce a binary file (BinFile) for execution on the target hardware. The memory allocation component ensures that the fusion operations can be performed within the available on-chip memory constraints, while the binary generation creates executable instructions that implement the optimization scheduling decisions.

[0090] This comprehensive optimization framework addresses the exponential search space problem identified in Figure 8A by using a constrained search technique and iterative optimization. The result is a practical solution that can handle complex neural network topologies with multiple branches while achieving memory access efficiency improvements compared to traditional greedy scheduling.

[0091] Figure 9A and Figure 9B demonstrate the effectiveness of the optimized fusion scheduling, including the feasible operating region and the quantified performance improvements achieved through the optimized fusion scheduling.

[0092] Figure 9A demonstrate the EMA-delay feasible region analysis using YOLO v7 as the target model. The graph shows the delay (ms) on the vertical axis, ranging from 34 to 38, and the memory access (MB) on the horizontal axis, ranging from 32 to 44. The region within the curve represents the feasible region where an efficient scheduling solution can be implemented within the system hardware constraints. The curve boundary line defines the upper limit of this feasible region, indicating the trade-off relationship between memory access and delay among different scheduling methods.

[0093] Three different operating points are marked on the graph to illustrate different scheduling strategies. The greedy scheduling point lies within the feasible region but represents a suboptimal solution in terms of memory access and delay performance. The minimum EMA point represents the scheduling configuration that minimizes external memory access, located at the leftmost boundary of the feasible region with the lowest memory access requirement but possibly with higher delay. The minimum time point represents the scheduling configuration optimized for minimum delay, located at the position that achieves the fastest execution time while maintaining an acceptable level of memory access.

[0094] The Pareto front (optimized scheduling) curve connects the best operating points and demonstrates the range of efficient scheduling solutions that can be obtained through the optimization methods described above. The Pareto front represents a set of non-dominated solutions where improvements in one objective (EMA or delay) cannot be achieved without degrading the other objective. Depending on the specific requirements and priorities of the target application, the optimization algorithm can choose any point along this Pareto front.

[0095] Figure 9BQuantized EMA and latency improvements are provided, comparing three scheduling methods: greedy scheduling, minimum time, and minimum EMA. Bar graphs show normalized performance metrics, with a baseline value of 1.0. Each method presents two sets of bars: normalized EMA and normalized time.

[0096] The greedy scheduling method serves as a baseline reference point, with both normalized EMA and normalized time approximately 1.0. The minimum time scheduling configuration achieves an 8% improvement in latency (normalized time) and a 4% improvement in EMA compared to the greedy method. The minimum EMA scheduling configuration provides more significant improvements, with a 16% reduction in external memory accesses and a 2% improvement in latency performance.

[0097] The results demonstrate that the optimized scheduling methods provide advantages over the traditional greedy scheduling. The minimum EMA configuration is particularly effective for memory-constrained scenarios common in edge device implementations, significantly reducing memory accesses while maintaining competitive latency performance. The minimum time configuration provides balanced improvements in both metrics, making it suitable for applications where execution speed is the primary concern but memory efficiency is still important. Additionally, the ability to select different points on the Pareto frontier provides flexibility to system designers to optimize for specific application needs and hardware constraints. This multi-objective optimization capability represents an advancement over traditional scheduling methods, which typically optimize for a single objective without considering the trade-off between memory accesses and execution latency.

[0098] The terminology used in the description of the various embodiments herein is intended to be interpreted in a descriptive sense, and not a limiting sense. In the context of this description and the additional claims, the singular forms "a," "an," and "one" are intended to include plural forms unless the context clearly dictates otherwise.

[0099] It will be understood that the term "and / or," as used herein, is intended to encompass all possible combinations of one or more of the associated listed items. Moreover, it should be noted that the terms "comprise" and / or "comprising," when used in this specification, are intended to mean that the features, integers, steps, operations, elements, and / or components that follow the word are included, but not to the exclusion of the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0100] In the context of the present disclosure, the terms "connected," "coupled," and similar terms are used in broad sense to encompass electrically or electronically connected states. Also, when an entity is in "communication" with another entity (or entities), it is understood that the entity is electrically transmitting and / or receiving information signals to / from the other entity, whether or not the signals include image / speech information or data / control information, and whether the signals are of analog or digital type. It is important to note that such communication can be carried out through wired or wireless means. The use of these terms is intended to encompass all forms of electrically or electronically connected states relevant to the described embodiments.

[0101] Ordinal designations, such as "first," "second," and the like, used in the specification and the claims are used to distinguish between multiple instances of like-named elements. These designations do not imply any inherent order, priority, or temporal sequence in the manufacturing process or functional relationship between elements. Rather, they are used solely to identify and distinguish different instances of elements having common names or descriptions.

[0102] Directional terms used in the embodiments, such as up, down, left, right, top, bottom, front, or rear, merely refer to the orientation in the drawings. The directional terms used in the present disclosure are used for explanation and are not intended to limit the scope of the present disclosure. It is noted that the specifically described or labeled elements can exist in various forms to those skilled in the art.

[0103] Approximating language such as "substantially," "approximately," "generally," "about," "near," and the like, used in the specification and the appended claims, is intended to convey that the approximating terms and conditions do not preclude the exact elements or conditions being present. Such approximating language is used in the context of the specification and the claims to describe features that can be outside the perfect numerical range or outside the perfect structural or functional relationship. These variations are to be expected and are within the scope of the disclosure. The approximating language is not to be construed as literally reciting only the perfect or ideal condition. Rather, such approximating language is used to describe and claim features coming within the scope of the disclosure. In the context of the present disclosure, the terms "connected," "coupled," and similar terms are used in broad sense to encompass electrically or electronically connected states. Also, when an entity is in "communication" with another entity (or entities), it is understood that the entity is electrically transmitting and / or receiving information signals to / from the other entity, whether or not the signals include image / speech information or data / control information, and whether the signals are of analog or digital type. It is important to note that such communication can be carried out through wired or wireless means. The use of these terms is intended to encompass all forms of electrically or electronically connected states relevant to the described embodiments.

[0104] The various illustrative components, logic, logical blocks, modules, circuits, operations and algorithm processes associated with the embodiments disclosed herein can be implemented as electronic hardware, firmware, software, or combinations of both hardware, firmware and software including the structural means disclosed in the present specification and structural equivalents thereof. The interchangeability of hardware, firmware and software has been described generally, and in terms of the various illustrative components, blocks, modules, circuits and processes described above. Such functionality can be implemented in the hardware, firmware or software, depending on the particular application and design constraints imposed on the overall system.

[0105] The hardware and data processing apparatus used to implement the various illustrative components, logics, blocks, modules and circuits described herein can include, but are not limited to one or more general-purpose or special-purpose microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field- programmable gate arrays (FPGAs), other programmable logic devices (PLDs), discrete gates or transistor logic, discrete hardware components, or any suitable combination thereof. This hardware and apparatus should be configured to perform the functions described herein.

[0106] The general-purpose processor can include, but is not limited to, a microprocessor, or alternatively, any conventional processor, controller, microcontroller, or state machine. In some implementations, the processor can be implemented as a combination of computing devices. These combinations can include, for example, a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other suitable configuration for the intended application.

[0107] It should be understood that in certain embodiments, a particular function, operation, or method can be performed by circuitry that is specifically designed for that purpose. Such specific function circuitry can be optimized to improve performance, efficiency, or other relevant metrics to accomplish a particular task. The choice of specific hardware implementation should be determined in accordance with the specific requirements of the application, which can include performance specifications, power consumption limitations, cost considerations, and size constraints, among others.

[0108] In certain aspects, the subject matter described herein can be implemented as software. Specifically, various functions of the disclosed components, or steps of the methods, operations, processes, or algorithms described herein, can be implemented in one or more computer programs that are executable on one or more computers or computing devices. These computer programs can include non-transitory processor-executable or computer-executable instructions that are encoded on one or more non-transitory processor-readable or computer-readable storage media. These instructions configure a data processing apparatus to perform or control the operations described herein, including the components of the devices described herein. The above storage media can include, but are not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), compact disc read only memory (CD-ROM) or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other non-transitory medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. It should be understood that a combination of one or more of the above storage media can also fall within the scope of computer-readable storage media as intended herein.

[0109] Various modifications to the implementations described in this disclosure can be readily apparent to those with ordinary skill in the art, and the generic principles defined herein can be applied to other implementations without departing from the spirit or scope of the disclosure. Thus, the claims should not be limited to the implementations shown herein but should be given the broadest interpretation consistent with the specification, the principles and the novel features disclosed.

[0110] In certain implementations, embodiments can include disclosed features and can optionally include additional features not explicitly described herein. Conversely, alternative implementations can be characterized by substantially or completely lacking undisclosed elements. For the avoidance of doubt, it is understood that in certain embodiments, undisclosed elements can be intentionally omitted, either partially or entirely, without departing from the scope of the invention. Such omissions of undisclosed elements are not to be construed as limiting the breadth of the claimed subject matter so long as the explicitly disclosed features remain present in the embodiments.

[0111] Moreover, various features that are described in this specification as separate embodiments can be implemented in a single implementation. Conversely, various features that are described in a single implementation can be implemented in multiple embodiments. Accordingly, although features can be described above as acting in particular combinations, one or more features from a described combination can in some cases be deleted or released from a combination, and the claimed combination can be directed to a sub-combination or a variation of a sub-combination.

[0112] Operations depicted in the figures should not be viewed as requiring a strict order in practice, and that the illustrated operations can be performed in any order, not necessarily the order in which they are depicted. Additional operations can also be added, or some operations can be omitted, in various implementations.

[0113] Furthermore, it should be understood that the various diagrams and component graphs provided and discussed herein are merely illustrative of the embodiments and should not be construed as precise technical diagrams or limiting the scope of the invention to the particular arrangements depicted.

[0114] In certain implementations, multitasking and parallel processing can prove advantageous. Furthermore, although various system components are described as separate entities in certain embodiments, such separation should not be taken as mandatory in all embodiments. The program components and system described herein can be integrated into a single software package or distributed over multiple software packages according to the particular implementation requirements.

[0115] It should be noted that other embodiments beyond those explicitly described are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order than presented within the claims, while still achieving the desired results. Such flexibility in executing the order of actions is inherent in the claimed processes and should be considered within the scope of the invention.

[0116] While the present application has been described in connection with certain embodiments, those skilled in the art will understand that various modifications and adaptations can be made without departing from the scope of the present application. The specific embodiments provided are meant to be illustrative only and not limiting of the application's application or construction. Those skilled in the art will readily observe many modifications and changes that can be made to the devices and methods without departing from the scope of the present application. Accordingly, the disclosure should only be limited by the scope of the appended claims.

Claims

1. A heterogeneous neural processing system, comprising: a first processor configured to perform encoding and decoding operations of an autoencoder; and a second processor configured to perform a neural network operation of a specific task, the neural network operation performing an iterative process; wherein the first processor and the second processor perform a computing task by exchanging data synchronously to implement a generative AI model.

2. The heterogeneous neural processing system of claim 1, wherein the encoding operation converts input data into a latent representation, and the decoding operation reconstructs the latent representation back into output data.

3. The heterogeneous neural processing system of claim 1, wherein the first processor performs the encoding and decoding operations, and the second processor simultaneously performs the neural network operation of the specific task on the same data of the generative AI model.

4. The heterogeneous neural processing system of claim 1, wherein the first processor is further configured to: process a feature map of a neural network into a plurality of data rows; cache the plurality of data rows into an activation memory; and determine if a required portion of the plurality of data rows is cached into the activation memory and select a deeper operation in a network hierarchy.

5. The heterogeneous neural processing system of claim 4, wherein the neural network includes branch inputs, branch outputs, and residual connections processed within a fusion layer stack.

6. The heterogeneous neural processing system of claim 4, wherein the first processor is further configured to allocate memory addresses for the cached data rows having different heights and overlapping active ranges.

7. The heterogeneous neural processing system of claim 1, wherein the first processor includes an H-reuse cache configured to store boundary pixels between adjacent spatial segments.

8. The heterogeneous neural processing system of claim 7, wherein the first processor is further configured to simultaneously perform a convolution and an element-level operation, the element-level operation including an addition and / or concatenation.

9. The heterogeneous neural processing system of claim 7, wherein the first processor further includes a ping-pong buffer configured to fetch a next data segment while processing a current segment.

10. The heterogeneous neural processing system of claim 7, wherein the H-reuse cache stores boundary pixels between spatial segments having a kernel size, a dilation rate, and a stride.

11. The heterogeneous neural processing system of claim 1, further comprising a neural network conditioning device configured to: analyze a neural network model to identify layer dependencies and fusion boundaries; apply constraints to a search space based on memory capacity and processing capability to define a fusion configuration; perform an iterative search within the search space to generate a plurality of fusion plans; and select a fusion plan from the plurality of fusion plans based on external memory access and execution latency.

12. The heterogeneous neural processing system of claim 11, wherein the neural network conditioning device is further configured to limit a branch factor and a fusion depth.

13. The heterogeneous neural processing system of claim 11, wherein the neural network conditioning device is further configured to generate a plurality of fusion configurations and select a configuration from the plurality of fusion configurations based on external memory access and execution latency.

14. The heterogeneous neural processing system of claim 11, wherein the neural network conditioning device is further configured to allocate activation memory addresses for operations having different data heights and non-overlapping time execution windows.

15. The heterogeneous neural processing system of claim 11, wherein the neural network conditioning device is further configured to process neural network topologies having skip connections and residual connections.

16. The heterogeneous neural processing system of claim 1, wherein the task-specific neural network operation comprises a denoising operation performed by a U-Net architecture.

17. The heterogeneous neural processing system of claim 1, wherein the task-specific neural network operation comprises a conditioning operation that encodes semantic vectors or text token embeddings into latent space representations.

18. The heterogeneous neural processing system of claim 1, wherein the task-specific neural network operation comprises an attention mechanism having query, key, and value components.

19. The heterogeneous neural processing system of claim 1, wherein the generative AI model comprises a latent diffusion model for text-to-image generation.

20. The heterogeneous neural processing system of claim 1, wherein the generative AI model comprises an image inpainting model for super-resolution or face restoration.

Images