Optimization processing method, system, device and medium of ai accelerator

Abstract

The application provides an AI accelerator processing optimization method, system, device and medium, and the method is applied to an AI accelerator in communication connection with a main memory. Through the collaborative architecture of the AI accelerator and the 3D DRAM, the application realizes reduction of transmission overhead and power consumption in the data carrying process, improvement of the computing throughput and the energy efficiency ratio, and simultaneously supports large-scale parallel computing tasks relying on the high bandwidth characteristics of the 3D DRAM.

Description

Technical Field

[0001] This invention relates to the field of AI accelerator technology, and in particular to an optimization processing method, system, device and medium for AI accelerators. Background Technology

[0002] Current AI accelerators are primarily designed for computationally intensive tasks, with a particular focus on optimizing CNN convolution acceleration. They integrate L1, L2, and L3 caches, forming an inverted pyramid-shaped bandwidth hierarchy. This means that caches closer to the computation unit (such as L1) have higher bandwidth, precisely matching the high bandwidth requirements of the computation unit. Data is sequentially transferred from main memory through the L3, L2, and L1 caches before finally being processed by the computation unit. The core objective of this design is to reduce the frequency of main memory access and improve data access efficiency through multi-level caching. This architecture is highly adaptable to tasks with data locality, such as CNN convolution computation, where data can be reused multiple times within the cache, fully leveraging the efficiency of cache resources.

[0003] However, when faced with novel tasks such as Large Language Models (LLM), due to the massive number of parameters and the low locality of data access patterns (read / write ratio close to 1:1), the multi-level caching architecture of current accelerators struggles to perform as expected, resulting in significant resource waste. In summary, current AI accelerators have obvious limitations in handling novel AI tasks such as LLM. Therefore, an effective optimization method for AI accelerators is urgently needed to address these issues. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide an optimized processing method, system, device and medium for an AI accelerator that overcomes or at least partially solves the above problems.

[0005] To achieve the above and other related objectives, the present invention provides an optimization processing method for an AI accelerator, applied to an AI accelerator communicatively connected to 3D DRAM, the method comprising:

[0006] Upon receiving a configuration instruction sent by the main CPU via the configuration bus, the configuration instruction is parsed to generate a corresponding serialized memory access instruction stream; wherein, the serialized memory access instruction stream includes the physical access address, access type, data length, and access triggering conditions;

[0007] According to the memory access instruction stream, the target data stream is read from the 3D DRAM via the main bandwidth, and the target data stream is parsed and separated into a weight data stream and a feature map data stream;

[0008] The GEMV operation corresponding to the configuration instruction is performed on the weighted data stream and the feature map data stream to generate calculation results, and the calculation results are written back to the 3D DRAM through the main bandwidth.

[0009] Optionally, parsing the configuration instructions to generate a corresponding serialized memory access instruction stream includes:

[0010] The configuration instructions are parsed to identify the access type, operation parameters, and preset access trigger conditions of the GEMV operation;

[0011] Based on the computational parameters and the storage layout of the 3D DRAM, the dependencies between each data block are derived, and their physical access addresses and corresponding data lengths are calculated.

[0012] The physical access address, the data length, the access type, and the access triggering condition are encapsulated into a standardized serialized memory access instruction package to form a serialized memory access instruction stream that can be executed by hardware.

[0013] Optionally, the AI ​​accelerator includes a demultiplexing unit, a weight storage channel, a feature map storage channel, and a computation unit. The step of parsing and splitting the target data stream into a weight data stream and a feature map data stream includes:

[0014] The target data stream is separated according to data attributes by the demultiplexing unit and then input into the weight storage channel and the feature map storage channel respectively.

[0015] Within the weight storage channel, block reassembly processing is performed on the received weight data stream to generate a weight data stream in a matrix block format adapted to GEMV operations;

[0016] Within the feature map storage channel, the received feature map data stream is subjected to dimension alignment and parallel splitting processing to generate a feature map data stream in vector slice format adapted for GEMV parallel operation;

[0017] The weight data stream and the feature map data stream are aligned and synchronously delivered to the computing unit in real time in a streaming manner to ensure that the data dimensions of the two are matched and the pipeline timing is synchronized, so that the GEMV operation can be performed.

[0018] Optionally, performing the GEMV operation corresponding to the configuration instruction on the weighted data stream and the feature map data stream to generate the calculation result includes:

[0019] According to the access triggering conditions defined by the serialized memory access instruction stream, the weight data stream and the feature map data stream are time-aligned, and the GEMV operation is performed under the condition that the dimensions of the two are matched to generate intermediate vector results divided into blocks according to the output dimension.

[0020] The intermediate vector results are concatenated and organized according to their logical index positions in the output vector to finally generate the complete calculation result.

[0021] Optionally, writing the calculation result back to the 3D DRAM via the main bandwidth includes:

[0022] The calculation results are formatted to generate write data adapted to the main bandwidth transmission protocol, and then written back to the storage address corresponding to the 3D DRAM via the main bandwidth.

[0023] Optionally, the AI ​​accelerator further includes a computing unit, and after the step of writing the computing result back to the 3D DRAM via the main bandwidth, it further includes:

[0024] Real-time monitoring of the main bandwidth utilization and the computation completion status of the computing units;

[0025] When the bandwidth utilization meets the prefetching condition and the computing unit completes the current cycle GEMV operation, a pipeline ready signal is generated.

[0026] In response to the pipeline ready signal, the serialized memory access instructions required for the next stage of the pipeline are prefetched in parallel from the instruction cache, and the next batch of target data streams to be processed is prefetched through the main bandwidth, thereby achieving parallel overlap of computation and memory access.

[0027] Only when the next batch of target data stream to be processed is prefetched and the calculation results of the current batch are completely written back to 3DDRAM, the current pipeline state is locked and a stage switch is triggered, and the next batch of target data stream to be processed is injected into the computing unit to execute the GEMV operation of the next cycle.

[0028] Secondly, the present invention also provides an optimized processing system for an AI accelerator, applied to an AI accelerator communicatively connected to 3D DRAM, the system comprising:

[0029] The parsing module is used to parse the configuration instruction sent by the main CPU through the configuration bus after receiving the configuration instruction, and generate a corresponding serialized memory access instruction stream; wherein, the serialized memory access instruction stream includes physical access address, access type, data length and access triggering conditions;

[0030] The read module is used to read the target data stream from the 3D DRAM via the main bandwidth according to the memory access instruction stream, and to parse and separate the target data stream into a weight data stream and a feature map data stream;

[0031] The calculation module is used to perform GEMV operations corresponding to the configuration instructions on the weight data stream and the feature map data stream, generate calculation results, and write the calculation results back to the 3D DRAM through the main bandwidth.

[0032] Thirdly, the present invention provides an electronic device comprising: a memory and a processor; the memory for storing a computer program; and the processor for executing the computer program stored in the memory to cause the electronic device to perform the steps of the AI ​​accelerator optimization processing method as described above.

[0033] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the steps of the AI ​​accelerator optimization processing method as described above.

[0034] Fifthly, the present invention provides a computer program product including computer program code, which, when run on a computer, causes the computer to implement the method described above.

[0035] The above-described one or more technical solutions provided by this invention can have the following advantages or at least achieve the following technical effects:

[0036] This invention achieves reduced transmission overhead and power consumption during data transfer, improved computing throughput and energy efficiency through a collaborative architecture of AI accelerator and 3D DRAM, while leveraging the high bandwidth characteristics of 3D DRAM to support large-scale parallel computing tasks. Attached Figure Description

[0037] Figure 1 The diagram shows a flowchart of an optimization method for an AI accelerator in one embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram illustrating the interaction between an AI accelerator and 3D DRAM in one embodiment of the present invention;

[0039] Figure 3 The diagram shows an architecture schematic of an AI accelerator in one embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of the functional modules of the optimization processing system of an AI accelerator in one embodiment of the present invention;

[0041] Figure 5 The diagram shown is a schematic representation of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0042] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0043] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0044] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0045] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0046] Unless otherwise stated, the term "multiple" means two or more.

[0047] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0048] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0049] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0050] Please see Figure 1An embodiment of the present invention provides an optimization processing method for an AI accelerator, applied to an AI accelerator communicating with 3D DRAM. The method may include the following steps S10-S30:

[0051] Step S10: After receiving the configuration instruction sent by the main CPU through the configuration bus, the configuration instruction is parsed to generate a corresponding serialized memory access instruction stream; wherein, the serialized memory access instruction stream includes physical access address, access type, data length and access triggering conditions.

[0052] The configuration instructions are used to configure the set of control parameters for the computing units (CUs) in the AI ​​accelerator. The control parameters include at least: control words that define the operation mode, quantization parameters for data precision, address mapping information for memory layout, and synchronization signals for pipeline dependencies.

[0053] A serialized memory access instruction stream is used to represent a control instruction stream generated based on configuration instruction parsing and arranged in chronological order, used to drive the transfer of data between the computing unit and 3D DRAM; each instruction includes at least: physical access address, access type (read / write), data length, and access triggering condition (e.g., timing trigger / state trigger).

[0054] Access trigger conditions are core fields in serialized memory access instructions that control the timing of instruction execution, ensuring precise synchronization between memory access operations and the computational process and hardware status. These include:

[0055] Timing-triggered mechanism: Memory access operations are triggered based on a preset time sequence or timing nodes. For example, data reading / writing is initiated at fixed clock cycle intervals, or instruction execution is triggered at a specified pipeline timing stage (such as a preset time slot before the start of the next GEMV operation). This mechanism is suitable for pipelined scenarios with fixed computation and memory access timings.

[0056] State-triggered mechanism: Memory access operations are triggered based on real-time state feedback signals from hardware modules. For example, the corresponding memory access instruction is executed only after specific state signals such as the computing unit completing the current GEMV operation, 3D DRAM bandwidth being idle, or data cache being ready are detected. This is suitable for scenarios that require dynamic adaptation to changes in hardware load and processing irregular data streams.

[0057] Please see Figure 2This diagram illustrates the data interaction between the AI ​​accelerator and 3D DRAM. Specifically, data flows from 3D DRAM to the computing units of the AI ​​accelerator via the main bandwidth for processing. The AI ​​accelerator contains computing units that are primarily responsible for performing core computational tasks such as matrix-vector multiplication (GEMV). As a high-bandwidth storage unit, 3D DRAM is responsible for providing the AI ​​accelerator with the necessary weights and feature maps (mp).

[0058] Please see Figure 3 This diagram illustrates the functional units and data flow paths within the AI ​​accelerator's internal architecture. Data enters from the "read data" input port on the left, first passing through the "demux" demultiplexing unit, which distributes the input data to two independent data channels: "weight" and "fm". Subsequently, the data from the "weight" and "fm" units are synchronously fed into the computation unit for fusion operations (such as convolution or matrix multiplication). After computation, the results are processed by the "output" unit and output as "write data", ultimately written back to external storage or the next-level processing unit. In addition, the AI ​​accelerator includes a "cmddecode" instruction parsing unit, used to receive and parse external requests (req), coordinating the scheduling and control of the entire data path.

[0059] In its implementation, when the AI ​​accelerator receives a configuration instruction from the main CPU via the configuration bus, it can parse the instruction and generate a corresponding serialized memory access instruction stream. This serialized instruction stream includes the physical access address, access type, data length, and access trigger conditions. This allows the AI ​​accelerator to automatically, efficiently, and synchronously stream the target data stream from 3D DRAM using the main bandwidth, providing data support for subsequent GEMV operations.

[0060] Step S20: According to the memory access instruction stream, the target data stream is read from the 3D DRAM via the main bandwidth, and the target data stream is parsed and separated into a weight data stream and a feature map data stream.

[0061] The target data stream represents the core data set that the AI ​​accelerator reads from 3D DRAM using the main bandwidth to perform GEMV operations. Specifically, it includes two key data types:

[0062] The weighted data stream corresponds to the operand matrix of the GEMV operation, which is usually an M×K dimension matrix.

[0063] The feature map data stream corresponds to the operand vector of the GEMV operation, which is usually a K×1 dimension vector.

[0064] The target data stream needs to be preprocessed (data splitting, weight reorganization, feature map processing) before it can be adapted to the GEMV operation requirements of the computing unit.

[0065] The calculation result represents the output data obtained after performing the GEMV operation on the preprocessed weight data and feature map data. Its dimension is M×1 (matching the row dimension M of the weight matrix), and it is the final product of the GEMV operation. It needs to be written back to 3D DRAM through the main bandwidth.

[0066] In its implementation, the AI ​​accelerator can read the target data stream from 3D DRAM via the main bandwidth according to the control of the memory access instruction stream. Then, it performs real-time parsing and splitting of the target data stream, including data splitting, weight reorganization, and feature map processing, to generate a weight data stream and a feature map data stream adapted for GEMV operation. Subsequently, the weight data stream and feature map data stream can be provided to the computing unit of the AI ​​accelerator to perform GEMV operation. This achieves seamless data flow from 3D DRAM to the computing unit, effectively improving the energy efficiency ratio of the AI ​​accelerator.

[0067] Step S30: Perform the GEMV operation corresponding to the configuration instruction on the weight data stream and the feature map data stream to generate the calculation result, and write the calculation result back to the 3D DRAM through the main bandwidth.

[0068] The calculation result represents the output vector data generated by the calculation unit after performing the GEMV operation; it can be used as input for subsequent calculation stages or stored in a hierarchical storage system.

[0069] In the specific implementation, after the computing unit receives the weight data stream and the feature map data stream, it can perform the GEMV operation corresponding to the configuration instructions on the time-aligned weight data stream and feature map data stream to generate the calculation result. Subsequently, the calculation result is written back to 3D DRAM through the main bandwidth. Thus, based on the efficient in-memory computing closed loop of the pipelined architecture, the coordinated operation of computing resources and storage bandwidth is realized, which significantly reduces the inference latency of AI models and improves the energy efficiency ratio.

[0070] Furthermore, in one embodiment, step S10 may include the following sub-steps S101~S103:

[0071] Sub-step S101: Parse the configuration instructions to identify the access type, operation parameters and preset access trigger conditions of the GEMV operation;

[0072] Sub-step S102: Based on the operation parameters and the storage layout of the 3D DRAM, deduce the dependency relationship between each data block and calculate its physical access address and corresponding data length.

[0073] Sub-step S103 encapsulates the physical access address, the data length, the access type, and the access triggering condition into a standardized serialized memory access instruction package to form a serialized memory access instruction stream that can be executed by hardware.

[0074] The serialized memory access instruction packet represents the smallest encapsulated unit containing the physical access address, access type, data length, and access triggering conditions; this instruction packet serves as the basic control unit for hardware to perform memory access operations.

[0075] In its implementation, the AI ​​accelerator includes an instruction parsing unit. This unit parses configuration instructions to generate and identify the access type, computational parameters, and preset access triggering conditions of the target operator. Then, based on the computational parameters and the 3D DRAM storage layout, it determines the dependencies between data blocks and calculates the corresponding physical access addresses and data lengths. Subsequently, the physical access addresses, data lengths, access types, and access triggering conditions are encapsulated into standardized serialized memory access instruction packages, forming a serialized memory access instruction stream executable by hardware. This achieves automated memory access scheduling and hardware-level parallel processing, laying the foundation for overlapping computation and memory access execution.

[0076] In this embodiment, the collaborative architecture of AI accelerator and 3D DRAM reduces transmission overhead and power consumption during data transfer, improves computing throughput and energy efficiency, and supports large-scale parallel computing tasks by leveraging the high bandwidth characteristics of 3D DRAM.

[0077] Based on the foregoing embodiments, a second embodiment of the processing optimization method for the AI ​​accelerator of the present invention is proposed. In this embodiment, step S20 may include the following sub-steps S201~S204:

[0078] In sub-step S201, the target data stream is separated according to data attributes by the demultiplexing unit and input into the weight storage channel and feature map storage channel respectively.

[0079] The AI ​​accelerator also includes a demultiplexing unit, a weight storage channel, a feature map storage channel, and a computing unit.

[0080] The demultiplexing unit is configured to perform real-time parsing and splitting of the target data stream to generate a weighted data stream and a feature map data stream.

[0081] Weight storage channel, configured to deliver weight data streams to the computing unit.

[0082] Feature map storage channel, configured to deliver feature map data streams to the computing unit.

[0083] Data path closed loop: instruction parsing → demultiplexing and splitting → dual-channel supply (weight storage channel, feature map storage channel) → calculation and write-back.

[0084] In practical implementation, the target data stream can be parsed and split by a demultiplexing unit to generate a weight data stream and a feature map data stream. The weight data stream and feature map data stream are then input into the weight storage channel and feature map storage channel, respectively. This allows for independent prefetching, caching, and scheduling optimization based on the different access characteristics of the weights and feature maps, avoiding resource conflicts between different types of data on the transmission path.

[0085] Sub-step S202: Within the weight storage channel, block reorganization processing is performed on the received weight data stream to generate a weight data stream in a matrix block format adapted to GEMV operations.

[0086] In the specific implementation, the received weight data stream can be reorganized into blocks within the weight storage channel to generate a weight data stream with a matrix block format to meet the computational requirements of GEMV operation; thereby avoiding the computational unit from processing unaligned or non-blocked raw data and effectively improving computational efficiency.

[0087] Sub-step S203: Within the feature map storage channel, perform dimension alignment and parallel splitting processing on the received feature map data stream to generate a feature map data stream in vector slice format adapted for GEMV parallel computation.

[0088] In the specific implementation, within the feature map storage channel, the received feature map data stream can be dimensionally aligned and parallelized split to generate a feature map data stream with a vector slice format, which is adapted to the computational granularity of GEMV parallel operation; thereby avoiding the computational unit from processing unaligned or non-blocked raw data and effectively improving computational efficiency.

[0089] Sub-step S204: The weight data stream and the feature map data stream are aligned and synchronously delivered to the computing unit in real time in a streaming manner to ensure that the data dimensions of the two are matched and the pipeline timing is synchronized, so that the GEMV operation can be performed.

[0090] In the specific implementation, the weight data stream and feature map data stream can be aligned and synchronously delivered to the computing unit in real time in a streaming manner, ensuring that the data dimensions of the two are matched and the pipeline timing is synchronized, so that the computing unit can perform GEMV operation; thus, deep coupling between data stream supply and GEMV computing paradigm is achieved on the storage channel side, and an efficient streaming computing path is constructed.

[0091] In this embodiment, by achieving seamless integration between the data stream and the GEMV computing paradigm on the storage channel side, a high-throughput, low-latency streaming computing data path is constructed, which effectively alleviates the storage bandwidth bottleneck problem in AI model computing and significantly reduces the redundant overhead of data transportation.

[0092] Based on the foregoing embodiments, a third embodiment of the processing optimization method for the AI ​​accelerator of the present invention is proposed. In this embodiment, step S30 may include the following sub-steps S301 to S303:

[0093] Sub-step S301: Based on the access triggering conditions defined by the serialized memory access instruction stream, the weight data stream and the feature map data stream are time-aligned, and the GEMV operation is performed under the condition that the dimensions of the two are matched to generate intermediate vector results divided into blocks according to the output dimension.

[0094] The intermediate vector result represents a local output fragment generated by the computation unit after performing the GEMV operation on the current input weight data block and feature map data block. This result reflects the transient computation state of the current data stream window and requires subsequent accumulation, fusion, or activation processing to form the final effective output vector.

[0095] In the specific implementation, within the computation unit, the weight data stream and the feature map data stream can be time-aligned according to the access trigger conditions defined by the serialized memory access instruction stream. Then, under the condition that the dimensions of the two are confirmed to match, the GEMV operation is triggered to generate intermediate vector results divided into blocks according to the output dimension. This achieves a tightly coupled pipeline of computation and memory access, eliminating idle cycles of the computation unit caused by data waiting or dimension verification.

[0096] Sub-step S302 involves concatenating and organizing the intermediate vector results according to their logical index positions in the output vector to finally generate a complete calculation result.

[0097] The logical index position represents the starting offset or array index of the intermediate vector in the logical address space of the final output vector. It defines the target address where the intermediate vector result should be written to the on-chip cache or aggregation unit in the AI ​​accelerator, and is a key basis for realizing data concatenation, accumulation and fusion, and maintaining the integrity of the tensor structure.

[0098] In the specific implementation, the computing unit can concatenate and organize the results of each intermediate vector according to their logical index positions in the output vector, thereby realizing the logical reorganization and numerical accumulation of the computing results and generating a complete output vector with standardized format.

[0099] Sub-step S303: The calculation result is formatted to generate write data adapted to the main bandwidth transmission protocol, and then written back to the storage address corresponding to the 3D DRAM via the main bandwidth.

[0100] The AI ​​accelerator may also include an output unit configured to format the computation results, generate write data adapted to the main interface bandwidth transmission protocol, and write it back to the storage address corresponding to the 3D DRAM through the main interface.

[0101] In the specific implementation, after the output unit receives the calculation result transmitted by the computing unit, the output unit can format the calculation result to generate write data adapted to the main bandwidth transmission protocol, and write the write data back to the corresponding storage address of the 3D DRAM via the main bandwidth. By packaging the data into write data that conforms to the characteristics of the main bandwidth, the burst transmission efficiency of off-chip memory access is effectively improved, bandwidth waste caused by data format mismatch or uneven access granularity is reduced, and the end-to-end latency of the entire AI inference task is optimized.

[0102] In this embodiment, a data-driven streaming computing mode is achieved by performing GEMV operations corresponding to configuration instructions on the weight data stream and feature map data stream and writing them back to 3D DRAM. This process utilizes the high throughput of the main bandwidth to efficiently write back the computation results, effectively alleviating the data transport bottleneck. This not only improves the utilization rate of the computing unit but also enhances the hardware architecture's adaptability to diverse AI models through the instruction-driven configuration method.

[0103] Based on the foregoing embodiments, a fourth embodiment of the processing optimization method for the AI ​​accelerator of the present invention is proposed. In this embodiment, after step S30, the method may include the following steps A10~A40:

[0104] Step A10: Monitor the bandwidth utilization of the main bandwidth and the computation completion status of the computing unit in real time.

[0105] Among them, bandwidth utilization rate is used to represent the ratio of the amount of data actually transmitted through the bus in the main bandwidth of the AI ​​accelerator within a specific time period to the theoretical maximum amount of data that the bus can transmit.

[0106] Operation completion status is a status flag or register value used to indicate whether the current instruction or micro-operation of the computing unit has been completed.

[0107] In the specific implementation, after the calculation results are written back, the control logic hardware inside the AI ​​accelerator can monitor the bandwidth utilization of the main bandwidth and the completion status of the computing unit in real time. Based on the bandwidth utilization and the completion status of the computing unit, the hardware logic can determine whether the current pipeline stage meets the prefetching conditions. This enables efficient pipelined parallelism of computation and memory access without waiting for external CPU intervention, reducing the system's scheduling latency.

[0108] Step A20: When the bandwidth utilization meets the prefetched bandwidth threshold and the computing unit confirms the completion of the current cycle GEMV operation, a pipeline ready signal is generated.

[0109] The prefetch bandwidth threshold is used to represent the upper limit of main bandwidth utilization that is pre-set to allow the start of data prefetching operations.

[0110] The pipeline ready signal is a control signal used to indicate that the calculation and write-back of the current cycle has been completed and the prefetching conditions for the next batch of data have been met.

[0111] In the specific implementation, when the bandwidth utilization meets the prefetch bandwidth threshold and the computing unit confirms the completion of the current cycle's GEMV operation, a pipeline ready signal is generated. This triggers a parallel prefetch mechanism, enabling parallel overlap between computation and memory access, effectively masking memory access latency, and maximizing pipeline throughput efficiency.

[0112] Step A30: In response to the pipeline ready signal, the serialized memory access instructions required by the next stage pipeline are prefetched in parallel from the instruction cache, and the next batch of target data streams to be processed is prefetched through the main bandwidth, thereby realizing the parallel overlap of computation and memory access.

[0113] In a practical implementation, in response to the pipeline ready signal, the serialized memory access instructions required by the next stage of the pipeline can be prefetched in parallel from the instruction cache; then, the next batch of target data streams to be processed can be prefetched from the 3D DRAM through the main bandwidth, thereby achieving parallel overlap of computation and memory access.

[0114] Step A40: Only when the prefetch of the next batch of target data stream to be processed is completed and the calculation results of the current batch are completely written back to 3D DRAM, lock the current pipeline state and trigger a stage switch, inject the next batch of target data stream to be processed into the computing unit to execute the GEMV operation of the next cycle.

[0115] In the specific implementation, the current pipeline state can be locked, the stage switch can be triggered, and the next batch of target data streams can be injected into the computing unit to execute the GEMV operation of the next cycle, only when the prefetching of the next batch of target data streams is completed and the calculation results of the current batch are completely written back to 3D DRAM; thus realizing the conflict-free closed-loop scheduling of the pipeline.

[0116] In this embodiment, memory access latency is hidden through a data prefetching mechanism, and the utilization of computing units is maximized through pipeline parallel overlap.

[0117] Based on the same inventive concept, the fifth embodiment of this invention also provides an AI accelerator processing optimization system corresponding to the AI ​​accelerator processing optimization method of the foregoing embodiments. Since the principle of the system in the fifth embodiment of this invention for solving the problem is similar to the AI ​​accelerator processing optimization method of the foregoing embodiments, the implementation of the system can refer to the implementation of the method; repeated details will not be elaborated further. Please refer to... Figure 4 The present invention provides a processing optimization system for an AI accelerator, applied to an AI accelerator communicating with 3D DRAM. The system may include:

[0118] The parsing module 10 is used to parse the configuration instruction sent by the main CPU through the configuration bus after receiving the configuration instruction, and generate a corresponding serialized memory access instruction stream; wherein, the serialized memory access instruction stream includes physical access address, access type, data length and access triggering conditions;

[0119] The reading module 20 is used to read the target data stream from the 3D DRAM through the main bandwidth according to the memory access instruction stream, and to parse and separate the target data stream into a weight data stream and a feature map data stream;

[0120] The calculation module 30 is used to perform GEMV operation corresponding to the configuration instruction on the weight data stream and the feature map data stream, generate calculation results, and write the calculation results back to the 3D DRAM through the main bandwidth.

[0121] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described processing optimization method for an AI accelerator.

[0122] Figure 5 This is a schematic block diagram of the electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 5 The general will label all buses as bus systems.

[0123] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0124] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0125] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic device 400. Examples of this data include: any executable program for operation on the electronic device 400, such as the operating system 4021 and application programs 4022; the operating system 4021 includes various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application program 4022 may include various applications, such as a media player, browser, etc., for implementing various application services. The processing optimization method for implementing the AI ​​accelerator provided in this embodiment of the invention can be included in the application program 4022.

[0126] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the processing optimization method for AI accelerators provided in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0127] In an exemplary embodiment, the electronic device 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0128] In summary, this invention achieves reduced transmission overhead and power consumption during data transfer, improved computing throughput and energy efficiency through a collaborative architecture of AI accelerator and 3D DRAM, while also supporting large-scale parallel computing tasks by leveraging the high bandwidth characteristics of 3D DRAM.

[0129] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A processing optimization method for an AI accelerator, characterized in that, The method, applied to an AI accelerator communicating with 3D DRAM, includes: Upon receiving a configuration instruction sent by the main CPU via the configuration bus, the configuration instruction is parsed to generate a corresponding serialized memory access instruction stream. This serialized memory access instruction stream includes a physical access address, access type, data length, and access triggering conditions. The parsing of the configuration instruction to generate the corresponding serialized memory access instruction stream includes: parsing the configuration instruction to identify the access type, operation parameters, and preset access triggering conditions of the GEMV operation; deriving the dependencies between data blocks based on the operation parameters and the 3D DRAM storage layout, and calculating their physical access addresses and corresponding data lengths; and encapsulating the physical access address, data length, access type, and access triggering conditions into a standardized serialized memory access instruction package to form a serialized memory access instruction stream executable by hardware. According to the memory access instruction stream, the target data stream is streamed from 3D DRAM via the main bandwidth, and the target data stream is parsed and separated into a weight data stream and a feature map data stream. The AI ​​accelerator includes a demultiplexing unit, a weight storage channel, a feature map storage channel, and a computation unit. Parsing and splitting the target data stream into a weight data stream and a feature map data stream includes: separating the target data stream according to data attributes using the demultiplexing unit and inputting them into the weight storage channel and the feature map storage channel respectively; performing block reassembly processing on the received weight data stream in the weight storage channel to generate a matrix block format weight data stream adapted for GEMV operations; performing dimension alignment and parallel splitting processing on the received feature map data stream in the feature map storage channel to generate a vector slice format feature map data stream adapted for GEMV parallel operations; and aligning and synchronously transmitting the weight data stream and the feature map data stream to the computation unit in real time in a streaming manner to ensure that their data dimensions match and their pipeline timing is synchronized for GEMV operations. Perform the GEMV operation corresponding to the configuration instruction on the weight data stream and the feature map data stream to generate the calculation result, and write the calculation result back to the 3D DRAM through the main bandwidth; wherein, performing the GEMV operation corresponding to the configuration instruction on the weight data stream and the feature map data stream to generate the calculation result includes: aligning the weight data stream and the feature map data stream in time according to the access triggering conditions defined by the serialized memory access instruction stream, and performing the GEMV operation under the condition that the dimensions of the two are confirmed to match, generating intermediate vector results divided into blocks according to the output dimension; concatenating and regularizing each intermediate vector result according to its logical index position in the output vector, and finally generating the complete calculation result.

2. The method according to claim 1, characterized in that, The step of writing the calculation result back to the 3D DRAM via the main bandwidth includes: The calculation results are formatted to generate write data adapted to the main bandwidth transmission protocol, and then written back to the storage address corresponding to the 3D DRAM via the main bandwidth.

3. The method according to claim 1, characterized in that, The AI ​​accelerator also includes a computing unit, and after the step of writing the computing result back to the 3D DRAM via the main bandwidth, it further includes: Real-time monitoring of the main bandwidth utilization and the computation completion status of the computing units; When the bandwidth utilization meets the prefetched bandwidth threshold and the computing unit confirms the completion of the current cycle GEMV operation, a pipeline ready signal is generated. In response to the pipeline ready signal, the serialized memory access instructions required for the next stage of the pipeline are prefetched in parallel from the instruction cache, and the next batch of target data streams to be processed is prefetched through the main bandwidth, thereby achieving parallel overlap of computation and memory access. Only when the next batch of target data stream to be processed is prefetched and the calculation results of the current batch are completely written back to 3DDRAM, the current pipeline state is locked and a stage switch is triggered, and the next batch of target data stream to be processed is injected into the computing unit to execute the GEMV operation of the next cycle.

4. A processing optimization system for an AI accelerator, characterized in that, An AI accelerator applied to a communication connection with 3D DRAM, the system comprising: The parsing module is used to parse the configuration instruction sent by the main CPU through the configuration bus after receiving the configuration instruction, and generate a corresponding serialized memory access instruction stream; wherein, the serialized memory access instruction stream includes data access address, access type, data length and access timing information; The parsing module is further configured to parse the configuration instructions, identify the access type, operation parameters, and preset access trigger conditions of the GEMV operation; deduce the dependencies between data blocks based on the operation parameters and the storage layout of the 3D DRAM, and calculate their physical access addresses and corresponding data lengths; and encapsulate the physical access addresses, data lengths, access types, and access trigger conditions into standardized serialized memory access instruction packages to form a serialized memory access instruction stream that can be executed by hardware. The read module is used to read the target data stream from the 3D DRAM via the main bandwidth according to the memory access instruction stream, and to parse and separate the target data stream into a weight data stream and a feature map data stream; The reading module is further configured to separate the target data stream according to data attributes through the demultiplexing unit, and input them into the weight storage channel and the feature map storage channel respectively; in the weight storage channel, the received weight data stream is subjected to block reorganization processing to generate a weight data stream in a matrix block format adapted to GEMV operation; in the feature map storage channel, the received feature map data stream is subjected to dimension alignment and parallel splitting processing to generate a feature map data stream in a vector slice format adapted to GEMV parallel operation; the weight data stream and the feature map data stream are aligned and synchronously delivered to the computing unit in real time in a streaming manner to ensure that the data dimensions of the two are matched and the pipeline timing is synchronized, so that it can perform GEMV operation; The calculation module is used to perform GEMV operation corresponding to the configuration instruction on the weight data stream and the feature map data stream, generate calculation results, and write the calculation results back to the 3D DRAM through the main bandwidth; The calculation module is further configured to perform time-series alignment of the weight data stream and the feature map data stream according to the access triggering conditions defined by the serialized memory access instruction stream, and perform GEMV operation under the condition that the dimensions of the two are matched to generate intermediate vector results divided into blocks according to the output dimension; and concatenate and regularize each intermediate vector result according to its logical index position in the output vector to finally generate a complete calculation result.

5. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the processor to perform the steps of the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed, performs the steps of the method according to any one of claims 1 to 3.

7. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the steps of the method according to any one of claims 1 to 3.

Images