Artificial intelligence chip and operating method thereof

Abstract

An artificial intelligence chip and an operating method thereof are provided. The artificial intelligence chip includes a shared memory, execution units, and a load store circuit. The load store circuit is coupled to the shared memory and the execution units. The load store circuit extracts a plurality of raw pixel data required for a plurality of iterations as a plurality of buffered pixel data from the shared memory at one time. In a first iteration, a sliding window covers a plurality of first respective pixel data in the plurality of buffered pixel data, and the load store circuit distributes the plurality of first respective pixel data to a corresponding execution unit. In a second iteration, the sliding window covers a plurality of second respective pixel data in the plurality of buffered pixel data, and the load store circuit distributes the plurality of second respective pixel data to the corresponding execution unit.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence (AI) chip technology, and more particularly to an artificial intelligence chip and its operating method. Background Technology

[0002] Depthwise Separable Convolution (DWC) is a convolutional operation widely used in lightweight neural networks (such as MobileNet and EfficientNet). The DWC operation consists of two stages: channel-wise convolution and pointwise convolution. DWC effectively reduces computational cost and parameter count, improving model inference speed and runtime efficiency. Unlike conventional convolution, where each convolution kernel (or filter) operates on all input channels (ICH), in DWC, each kernel processes only one input channel; that is, each input channel is convolved by only one kernel.

[0003] Although depthwise separable convolutions have fewer parameters and lower computational cost than regular convolutions, the computational intensity (the ratio of computational cost to memory access) is significantly reduced because the execution unit needs to repeatedly read a large amount of intermediate data from shared memory during computation. This can easily cause depthwise separable convolution operations to fall into the memory bound region of the roofline model. For example, suppose the input image size is 1. 34 (a pixel sequence including 34 pixels: p0, p1, p2, ..., p33), with a convolution kernel size of 1. 3 (coefficients f0, f1, f2), and the number of threads is 32 (including thread0 to thread31). In the first iteration, the loading storage unit loads coefficient f0 and pixel data p0 to p31 from shared memory, while the execution unit executes threads thread0 to thread31 with coefficient f0. p0, f0 p1、...、f0 p31. In the second iteration, the loading storage unit loads coefficient f1 and pixel data p1~p32 from shared memory, while the execution unit executes threads thread0~thread31 for f1. p1, f1 p2、...、f1 p32. In the third iteration, the loading storage unit loads coefficient f2 and pixel data p2~p33 from shared memory, while the execution unit executes threads thread0~thread31 for f2. p2, f2 p3、...、f2 p33. This execution method causes pixel data p2 to p31 to be repeatedly read from shared memory three times, resulting in a large number of redundant memory accesses. Summary of the Invention

[0004] This invention relates to an artificial intelligence chip and its operating method, in order to avoid multiple reads of the same data from shared memory.

[0005] In an embodiment of the present invention, the artificial intelligence chip includes shared memory, at least one execution unit, and a load-store circuit. The load-store circuit is coupled to the shared memory and the at least one execution unit. The load-store circuit continuously extracts multiple original pixel data required for multiple iterations from the shared memory as multiple buffered pixel data. In a first iteration, a sliding window covers multiple first corresponding pixel data among the multiple buffered pixel data. In the first iteration, the load-store circuit distributes the multiple first corresponding pixel data to the corresponding execution units of the at least one execution unit. In a second iteration, the sliding window covers multiple second corresponding pixel data among the multiple buffered pixel data. In the second iteration, the load-store circuit distributes the multiple second corresponding pixel data to the corresponding execution units of the at least one execution unit.

[0006] In an embodiment of the present invention, the operation method includes: a loading and storage circuit of an artificial intelligence chip continuously extracting multiple original pixel data required for multiple iterations from the shared memory of the artificial intelligence chip as multiple buffered pixel data; in a first iteration, a sliding window encompassing multiple first corresponding pixel data among the multiple buffered pixel data; the loading and storage circuit distributing the multiple first corresponding pixel data to corresponding execution units in the at least one execution unit in the first iteration; in a second iteration, a sliding window encompassing multiple second corresponding pixel data among the multiple buffered pixel data; and the loading and storage circuit distributing the multiple second corresponding pixel data to corresponding execution units in the at least one execution unit in the second iteration.

[0007] Based on the above, and considering the characteristic of data reuse within a sliding window in depthwise separable convolution operations, the loading and storage circuit reads sufficient data at once to cover the entire sliding window's range across multiple iterations, avoiding the need for the loading and storage circuit to read the same data from shared memory multiple times. This aggregates previously scattered, small-scale shared memory accesses into a single large-scale, continuous access, improving the efficiency and bandwidth utilization of shared memory. During the execution of memory loading instructions, a dedicated hardware module automatically reassembles the sliding window of data, requiring no software intervention and reducing programming complexity.

[0008] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of a circuit block of an artificial intelligence chip according to an embodiment of the present invention;

[0010] Figure 2 This is a flowchart illustrating an operation method of an artificial intelligence chip according to an embodiment of the present invention;

[0011] Figure 3 This is a schematic diagram illustrating multiple original pixel data according to an embodiment of the present invention;

[0012] Figure 4 This is a schematic diagram of a loading and storage circuit module according to an embodiment of the present invention;

[0013] Figure 5 This is a schematic diagram of a circuit module of a sliding window loading unit according to an embodiment of the present invention.

[0014] Explanation of icon numbers

[0015] 100: Artificial intelligence chips

[0016] 110: Instruction decoder

[0017] 120: Shared memory,

[0018] 130: Loading storage circuit,

[0019] 131: Load memory unit,

[0020] 132: Sliding window loading unit,

[0021] 140_1: Execution Unit 1

[0022] 140_2: Execution Unit 2

[0023] 140_N: The Nth execution unit,

[0024] 310: Convolution kernel,

[0025] 320: Input image,

[0026] 331: First iteration,

[0027] 332: Second iteration,

[0028] 333: Third iteration,

[0029] 510: Address generation unit,

[0030] 520: Data caching unit

[0031] 530: Sliding window unit. Detailed Implementation

[0032] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same element symbols are used in the drawings and description to denote the same or similar parts.

[0033] The term "coupled (or connected)" as used throughout this specification (including the claims) may refer to any direct or indirect means of connection. For example, if the text describes a first device coupled (or connected) to a second device, it should be interpreted as the first device being directly connected to the second device, or the first device being indirectly connected to the second device through other devices or some means of connection. The terms "first," "second," etc., used throughout this specification (including the claims) are used to name components or distinguish different embodiments or scopes, and are not intended to limit the upper or lower limit of the number of components, nor to limit the order of components. Furthermore, wherever possible, components / components / steps using the same reference numerals in the drawings and embodiments represent the same or similar parts. Components / components / steps using the same reference numerals or the same terms in different embodiments may be referred to mutually in the relevant descriptions. It should be understood that the features of the following embodiments can be combined with each other. For example, the features of the second embodiment can be combined with the features of the first embodiment. Those skilled in the art can select appropriate combinations of features according to actual design requirements.

[0034] Computing devices such as artificial intelligence (AI) chips can provide enormous computing power. This immense computing power stems from the large number of internal hardware cores. An AI chip typically contains multiple programmable processors, such as a Stream Processor Cluster (SPC). Each programmable processor typically contains multiple Compute Units (CUs). By organizing these CUs through programming, AI chips can support general-purpose computing, scientific computing, and neural network computing. Each CU typically contains multiple Execution Units (EUs) and shared memory. Execution Units may include at least one of the following: tensor cores (Tcores), integer cores (INTs), floating-point cores (FPs), and vector cores (Vcores). Different execution units within the same CU can exchange data with each other through local shared memory.

[0035] Figure 1 This is a schematic diagram of a circuit module of an artificial intelligence chip according to an embodiment of the present invention. Figure 1 The AI ​​chip 100 shown includes an instruction decoder 110, shared memory 120, load-memory circuitry 130, and one or more execution units (e.g., Figure 1 The execution units shown are 1st execution unit 140_1, 2nd execution unit 140_2, ..., Nth execution unit 140_N. The number N of execution units 1st execution unit 140_1 to Nth execution unit 140_N can be determined according to the actual design and application. The load-store circuit 130 is coupled to the instruction decoder 110, the shared memory 120, and the 1st execution unit 140_1 to Nth execution unit 140_N. The instruction decoder 110 routes the Load with Sliding Window (LDSW) instruction to the load-store circuit 130. The load-store circuit 130 executes the LDSW instruction to continuously extract multiple original pixel data required for multiple iterations from the shared memory 120 at once. Depending on the design, in some embodiments, at least one of the instruction decoder 110, the load-store circuit 130, and the 1st execution unit 140_1 to Nth execution unit 140_N can be implemented as hardware circuitry. In other embodiments, at least one of the instruction decoder 110, the load storage circuit 130, and the first execution unit 140_1 to the Nth execution unit 140_N can be implemented as a combination of hardware, firmware, and software (i.e., programs).

[0036] In terms of hardware, at least one of the instruction decoder 110, the load-store circuit 130, and the first execution unit 140_1 to the Nth execution unit 140_N can be implemented as logic circuitry on an integrated circuit. For example, the relevant functions of the instruction decoder 110, the load-store circuit 130, and at least one of the first execution unit 140_1 to the Nth execution unit 140_N can be implemented in various logic blocks, modules, and circuits within one or more hardware controllers, microcontrollers, hardware processors, microprocessors, application-specific integrated circuits (ASICs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), central processing units (CPUs), or other processing units. The related functions of at least one of the instruction decoder 110, the load storage circuit 130, and the first execution unit 140_1 to the Nth execution unit 140_N can be implemented as hardware circuits, such as various logic blocks, modules, and circuits in an integrated circuit, using hardware description languages ​​(such as Verilog HDL or VHDL) or other suitable programming languages.

[0037] In software or firmware form, the functions of the instruction decoder 110, the load-store circuit 130, and at least one of the first execution unit 140_1 to the Nth execution unit 140_N can be implemented as programming codes. For example, the instruction decoder 110, the load-store circuit 130, and at least one of the first execution unit 140_1 to the Nth execution unit 140_N can be implemented using general programming languages ​​(such as C, C++, or assembly language) or other suitable programming languages. The programming codes can be recorded / stored in a non-transitory machine-readable storage medium. In some embodiments, the non-transitory machine-readable storage medium includes, for example, semiconductor memory and / or a storage device. Electronic devices (such as CPUs, hardware controllers, microcontrollers, hardware processors, or microprocessors) can read and execute programming code from a non-transitory machine-readable storage medium to realize the related functions of instruction decoder 110, load storage circuit 130, and at least one of the first execution unit 140_1 to the Nth execution unit 140_N.

[0038] The size of shared memory 120 can be determined according to the actual design and application. Shared memory 120 can consist of X banks, each with a depth of Z and a size of Y bytes. For example, shared memory 120 can consist of 32 banks, each with a depth of 1024 and a size of 4 bytes. The smallest granularity of access to shared memory 120 is at the bank level. Load storage circuit 130 can read M bytes of data from shared memory 120 simultaneously per cycle, where M = X bytes. Y. For example, the load memory circuit 130 can simultaneously read 32 bytes from shared memory 120 per cycle. 4 = 128 bytes of data.

[0039] Figure 2 This is a flowchart illustrating an operation method of an artificial intelligence chip according to an embodiment of the present invention. Please refer to... Figure 1 and Figure 2 In step S210, the loading storage circuit 130 continuously extracts multiple original pixel data required for multiple iterations from the shared memory 120 at once as multiple buffered pixel data. For example, in some applications, the multiple original pixel data are all the pixel data of an image. In other applications, the multiple original pixel data include at least (T + f - 1) consecutive pixel data in an image, where T is the number of execution unit threads and f is the convolution kernel size.

[0040] For example, Figure 3 This is a schematic diagram illustrating multiple original pixel data according to an embodiment of the present invention. Figure 3 In the illustrated embodiment, it is assumed that the size f of the convolution kernel 310 is 1. 3 (coefficients are f0, f1, f2), input image size 320 is 1 34 (including a pixel sequence of 34 pixel data, p0, p1, p2, ..., p31, p32, p33), with an execution unit thread count T of 32. In step S210, the loading storage circuit 130 continuously extracts multiple original pixel data of p0 to p33 required for multiple iterations (such as the first iteration 331, the second iteration 332, and the third iteration 333) from the shared memory 120 at one time as multiple buffered pixel data.

[0041] The addressing method of the load-store circuit 130 for the shared memory 120 can be determined according to the actual design and application. For example, the load-store circuit 130 can calculate the shared memory address based on the input parameters of the sliding window load instruction and generate at least one load request for the shared memory 120. Depending on the actual design and application, the input parameters of the sliding window load instruction may include the shared memory base address base_addr (in bytes), the kernel size filter_size (ranging from 1 to F, where F represents the maximum supported kernel size), the number of input channels ich (requiring Y-byte alignment, where Y is the size of each library in the shared memory 120), the pixel pitch parameter step (in bytes) between different pixels in the shared memory 120, and the data element bit width dem_size (in bytes).

[0042] Based on the input parameters provided by the sliding window loading instruction: the kernel size `filter_size` and the number of input channels `ich`, the load-store circuit 130 calculates the number of requests for at least one load request. For example (but not limited to), the load-store circuit 130 calculates the following formula 1 to generate the number of requests for at least one load request. In formula 1, `req_number` is the number of requests, `ceil()` is the ceiling function, `T` is the number of execution unit threads, `f` is the kernel size, `C` is the number of input channels, `dem_size` is the data element bit width (in bytes), and `X` and `M` are the output bandwidth of the shared memory 120 (`X` is in pixels and `M` is in bytes).

[0043] req_number = ceil((T + f – 1 + X – 1) / X X C dem_size / M) (Calculation formula 1)

[0044] For each load request, the load storage circuit 130 calculates the shared memory address of the shared memory 120 based on the request sequence number and the input parameter (shared memory base address base_addr) provided by the sliding window load instruction. For example (but not limited to), for each request of at least one load request, the load storage circuit calculates the following formula 2 to generate the shared memory address. In formula 2, addr is the shared memory address, base_addr is the shared memory base address (the input parameter provided by the sliding window load instruction), req_id is the request sequence number in the request number req_number, step is the pixel pitch parameter (the input parameter provided by the sliding window load instruction, in bytes), ich_id is the input channel sequence number, and dem_size is the data element bit width (in bytes).

[0045] addr = base_addr + req_id X step + ich_id dem_size (Calculation formula 2)

[0046] Based on at least one load request generated by the load storage circuit 130, the shared memory 120 continuously feeds back multiple original pixel data to the load storage circuit 130 at once. For example (but not limited to), the shared memory 120 can execute the following pseudocode to process each M-byte request, and then return the Y-byte data of X pixels (a total of M bytes) as a whole to the load storage circuit 130.

[0047] for i = 0 to (X-1): / / Process X consecutive pixels

[0048] p_i_addr = addr + i × step

[0049] Read Y bytes of data continuously starting from p_i_addr (reading one library-sized amount of input channel data at a time).

[0050] Based on the convolution kernel position, the loading and storage circuit 130 automatically moves the sliding window to output the required data. The loading and storage circuit 130 calculates the starting and ending pixel data positions of the sliding window in different iterations based on the iteration number and the number of execution unit threads T. In some applications, the starting pixel data position is the iteration number k, and the ending pixel data position is (T + k - 1), where k is an integer from 0 to f-1, and f is the convolution kernel size. For example, the loading and storage circuit 130 outputs pixel data p0 to p(T-1) for the coefficients f0 of the convolution kernel (iteration number 0) to the first execution unit 140_1 to the Nth execution unit 140_N, where T is the number of execution unit threads. The loading and storage circuit 130 outputs pixel data p1 to pT for the coefficients f1 of the convolution kernel (iteration number 1) to the first execution unit 140_1 to the Nth execution unit 140_N, and so on. For the kth coefficient fk of the convolution kernel (kth iteration, iteration number k), the loading and storage circuit 130 outputs pixel data pk to p(T+k-1) to the first execution unit 140_1 to the Nth execution unit 140_N. The loading and storage circuit 130 divides the Y bytes of channel data into G groups, each group containing K = Y / G bytes, and sends them to the G corresponding execution units in the first execution unit 140_1 to the Nth execution unit 140_N.

[0051] as Figure 3 In the example shown, in the first iteration 331, the sliding window covers multiple first corresponding pixel data p0 to p31 from the buffered pixel data p0 to p33 of the loading storage circuit 130 (step S220). In step S230, the loading storage circuit 130 distributes the first corresponding pixel data p0 to p31 to the corresponding execution units among the first execution unit 140_1 to the Nth execution unit 140_N in the first iteration 331. For example, the threads thread0 to thread31 executed by the first execution unit 140_1 to the Nth execution unit 140_N are f0. p0, f0 p1、…、f0 p31. In the second iteration 332, the sliding window encompasses multiple second corresponding pixel data p1 to p32 from the buffered pixel data p0 to p33 of the loading storage circuit 130 (step S240). In step S250, the loading storage circuit 130 distributes the second corresponding pixel data p1 to p32 to the corresponding execution units among the first execution unit 140_1 to the Nth execution unit 140_N in the second iteration 332. For example, the threads thread0 to thread31 executed by the first execution unit 140_1 to the Nth execution unit 140_N are f1. p1, f1 p2、...、f1 p32. Similarly, in the third iteration 333, the sliding window encompasses multiple third corresponding pixel data p2 to p33 from the buffered pixel data p0 to p33 of the load storage circuit 130. In the third iteration 333, the load storage circuit 130 distributes the third corresponding pixel data p2 to p33 to the corresponding execution units in the first execution units 140_1 to 140_N. For example, threads thread0 to thread31 executed by the first execution unit 140_1 to the Nth execution unit 140_N are f2. p2, f2 p3, ..., f2 p33.

[0052] In summary, considering the characteristic of data reuse within a sliding window in depthwise separable convolution operations, the loading and storage circuit 130 reads sufficient data at once to cover the entire sliding window across multiple iterations (e.g., ...). Figure 3 The sliding window range shown in the first iteration 331, second iteration 332, and third iteration 333 prevents the loading storage circuit 130 from reading the same data from the shared memory 120 multiple times. This aggregates previously scattered, small-scale shared memory accesses into a single large-scale, continuous access, improving the efficiency and bandwidth utilization of the shared memory 120. During the execution of the memory loading instruction, a dedicated hardware module automatically reorganizes the sliding window of data, eliminating the need for software intervention and reducing programming complexity.

[0053] Figure 4 This is a schematic diagram of a loading and storage circuit module according to an embodiment of the present invention. Figure 4 The load storage circuit 130 shown can be used as Figure 1 This is one of many implementation examples of the load storage circuit 130 shown. Figure 4 The instruction decoder 110, shared memory 120, load-memory circuit 130, and execution units 140_1 to 140_N shown can be referenced. Figure 1 The relevant explanations. In Figure 4 In the illustrated embodiment, the load-memory circuit 130 includes a load-memory unit 131 and a sliding window load unit 132. The load-memory unit 131 is coupled to an instruction decoder 110, shared memory 120, and first execution units 140_1 to Nth execution units 140_N. The instruction decoder 110 routes non-sliding window load instructions to the load-memory unit 131. In some practical applications, the load-memory unit 131 may be a general (conventional) load-memory unit or other load-memory units. The load-memory unit 131 executes non-sliding window load instructions to retrieve data from shared memory 120.

[0054] The sliding window loading unit 132 is coupled to the instruction decoder 110, the shared memory 120, and the first execution unit 140_1 to the Nth execution unit 140_N. The instruction decoder 110 routes the sliding window loading instruction to the sliding window loading unit 132. The sliding window loading unit 132 executes the sliding window loading instruction to continuously extract multiple original pixel data required for multiple iterations from the shared memory 120 at once. For example, the sliding window loading unit 132 executes the sliding window loading instruction to calculate the above-mentioned calculation formula 1 and calculation formula 2, and then continuously extracts multiple original pixel data required for multiple iterations from the shared memory 120 at once.

[0055] Figure 5 This is a schematic diagram of a circuit module of a sliding window loading unit according to an embodiment of the present invention. Figure 5 The sliding window loading unit 132 shown can be used as Figure 4 This is one of many implementation examples of the sliding window loading unit 132 shown. Figure 5 The shared memory 120, sliding window loading unit 132, and first execution unit 140_1 to Nth execution unit 140_N shown can be referenced. Figure 4 The relevant explanations. In Figure 5 In the embodiment shown, the sliding window loading unit 132 includes an address generation unit 510, a data caching unit 520, and a sliding window unit 530.

[0056] Address generation unit 510 calculates the shared memory address based on the input parameters provided by the sliding window loading instruction and generates at least one load request for shared memory 120. Address generation unit 510 calculates the number of requests for at least one load request based on the convolution kernel size and the number of input channels. For each load request, address generation unit 510 calculates the shared memory address based on the request number and input parameters. For example, address generation unit 510 calculates the above formulas 1 and 2, calculates the shared memory address, and generates at least one load request for shared memory 120.

[0057] Shared memory 120 continuously feeds back multiple original pixel data to data buffer unit 520 at once based on at least one load request. Data buffer unit 520 is S bytes in size. S can be determined based on the actual design and application, for example, S = (T + F). Y, where T is the number of execution unit threads, F represents the maximum supported convolutional kernel size, and Y is the size of each library in shared memory 120. Data cache unit 520 is used to temporarily store data read from shared memory 120. Data cache unit 520 is sufficient to hold (T + F - 1) pixels of data.

[0058] The sliding window unit 530 automatically slides to generate pixel data streams corresponding to each convolution kernel position based on the convolution kernel size. Specifically, the sliding window unit 530 calculates the starting and ending pixel data positions of the sliding window in different iterations based on the iteration number k and the number of execution unit threads T. In some applications, the starting pixel data position is the iteration number k, and the ending pixel data position is (T + k - 1), where k is an integer from 0 to f-1, and f is the convolution kernel size. For example, the sliding window unit 530 outputs pixel data p0 to p(T-1) for the coefficients f0 of the convolution kernel (iteration number 0) to the first execution unit 140_1 to the Nth execution unit 140_N, where T is the number of execution unit threads. The sliding window unit 530 outputs pixel data p1 to pT for the coefficients f1 of the convolution kernel (iteration number 1) to the first execution unit 140_1 to the Nth execution unit 140_N, and so on. For the kth coefficient fk of the convolution kernel (kth iteration), the sliding window unit 530 outputs pixel data pk to p(T+k-1) to the first execution unit 140_1 to the Nth execution unit 140_N. The sliding window unit 530 divides the Y bytes of channel data into G groups, each group containing K = Y / G bytes, and sends them to the G corresponding execution units in the first execution unit 140_1 to the Nth execution unit 140_N.

[0059] For example, suppose there are 8 libraries (X) in a shared memory pool with a depth of Z of 1024, a size of Y of 16 bytes, and the smallest granularity of memory access is at the library level. Each cycle can read 128 bytes of data simultaneously (M = X). Y = 8 16 = 128). Assume the data cache unit 520 size S is 640 bytes. Assume the number of execution unit threads T is 32, meaning the number of pixels that can be calculated at one time is 32. Assume the maximum supported convolutional kernel size F is 8. The following will describe the process of the sliding window loading unit 132 processing the sliding window loading instruction.

[0060] Assuming the number of input channels (ich) is 128, the data element width (dem_size) is 2 bytes, the kernel size (filter_size) is 5, and the shared memory base address (base_addr) is 0, the load memory circuit 130 calculates the number of requests for at least one load request based on the input parameters provided by the sliding window load instruction: the kernel size (filter_size) and the number of input channels (ich). For example (but not limited to), the address generation unit 510 calculates the above formula 1 to generate the number of requests for the load request. For example, the number of requests (req_number) = ceil((32 + 5 – 1 + 7) / 8 (8 × 128 × 2 / 128) = 80. That is, the sliding window loading unit 132 needs to generate a total of 80 loading requests to the shared memory 120, and each loading request can retrieve 128 bytes of data.

[0061] For each request, address generation unit 510 calculates the above formula 2 to generate the shared memory address 120. For example, the shared memory address addr = 0 + pixel_id × step + ich_id dem_size, where pixel_id = req_id 8 (req_id is the request sequence number, each request processes 8 pixels), data element bit width dem_size is 2, pixel spacing parameter step is 128. 2 = 256 bytes (meaning the interval between two adjacent pixels in shared memory 120 is 256 bytes).

[0062] Based on the load request generated by the address generation unit 510, the shared memory 120 continuously feeds back multiple original pixel data to the data cache unit 520 at once. For example (but not limited to this), the shared memory 120 can execute the following pseudocode to process each 128-byte request, and then return 16 bytes of data for 8 pixels as a whole to the data cache unit 520.

[0063] for i = 0 to 7: / / Process 8 consecutive pixels

[0064] p_i_addr = addr + i × step

[0065] Read 16 bytes of data continuously starting from p_i_addr (reading one library-sized amount of input channel data at a time).

[0066] The sliding window unit 530 automatically outputs the required data based on the convolution kernel position. For example, for the convolution kernel coefficient f0 (iteration number 0), the sliding window unit 530 outputs pixel data p0 to p31 to the first execution unit 140_1 to the Nth execution unit 140_N. For the convolution kernel coefficient f1 (iteration number 1), the sliding window unit 530 outputs pixel data p1 to p32 to the first execution unit 140_1 to the Nth execution unit 140_N. For the convolution kernel coefficient f2 (iteration number 2), the sliding window unit 530 outputs pixel data p2 to p33 to the first execution unit 140_1 to the Nth execution unit 140_N. For the convolution kernel coefficient f3 (iteration number 3), the sliding window unit 530 outputs pixel data p3 to p34 to the first execution unit 140_1 to the Nth execution unit 140_N. The sliding window unit 530 outputs pixel data p4 to p35 for the coefficients f4 of the convolution kernel (iteration number 4) to the first execution unit 140_1 to the Nth execution unit 140_N.

[0067] The sliding window unit 530 divides the 16 bytes of channel data into four groups of four bytes each, and sends them to four execution units from the first execution unit 140_1 to the Nth execution unit 140_N (e.g., the first execution unit 140_1 to the fourth execution unit (not shown in the figure)). The first execution unit 140_1 receives data from channel 0 and channel 1. The second execution unit 140_2 receives data from channel 2 and channel 3. The third execution unit (not shown in the figure) receives data from channel 4 and channel 5. The fourth execution unit (not shown in the figure) receives data from channel 6 and channel 7.

[0068] In summary, this embodiment can significantly reduce the number of accesses to shared memory 120. (Based on 3...) Taking a 3-kernel convolution as an example, traditional methods require reading the pixels in the middle region three times, while this embodiment can meet the needs of all convolution kernel positions with a single read. Actual test data shows that for typical depthwise separable convolution workloads, access to shared memory 120 can be reduced by more than 60%. This embodiment can improve memory bandwidth utilization. By aggregating memory accesses 128 bytes wide, it fully utilizes the 128-byte bandwidth capability of shared memory 120 per cycle, reducing memory access latency. This embodiment can reduce programming complexity. The sliding window loading instruction encapsulates the complex sliding window data organization logic in hardware. Programmers only need to call this instruction in the key parts of the depthwise separable convolution calculation, without manually optimizing the data layout, significantly improving development efficiency.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An artificial intelligence chip, characterized by, The artificial intelligence chip comprises: a shared memory; at least one execution unit; and a load-store circuit coupled to the shared memory and the at least one execution unit, wherein the load-store circuit fetches from the shared memory a plurality of original pixel data required for a plurality of iterations as a plurality of buffered pixel data at one time, the load-store circuit automatically moves a sliding window to output required data, the sliding window covers a plurality of first corresponding pixel data in the plurality of buffered pixel data in a first iteration, the load-store circuit distributes the plurality of first corresponding pixel data to a corresponding execution unit in the at least one execution unit in the first iteration, the sliding window covers a plurality of second corresponding pixel data in the plurality of buffered pixel data in a second iteration, and the load-store circuit distributes the plurality of second corresponding pixel data to a corresponding execution unit in the at least one execution unit in the second iteration.

2. The artificial intelligence chip according to claim 1, wherein, The plurality of original pixel data is all pixel data of an image.

3. The artificial intelligence chip of claim 1, wherein, The plurality of original pixel data is (T + f - 1) pixel data in an image, wherein T is a number of execution unit threads, and f is a convolution kernel size.

4. The artificial intelligence chip of claim 1, wherein: the load-store circuit calculates a shared memory address according to an input parameter carried by a sliding window load instruction and generates at least one load request to the shared memory; and the shared memory feeds back the plurality of original pixel data to the load-store circuit at one time according to the at least one load request.

5. The artificial intelligence chip of claim 4, wherein: the load-store circuit calculates a number of the at least one load request according to a convolution kernel size and a number of input channels; and for each request of the at least one load request, the load-store circuit calculates the shared memory address according to a request serial number and the input parameter.

6. The artificial intelligence chip of claim 5, wherein: The load store circuit calculates req_number = ceil((T + f - 1 + X - 1) / X X C dem_size / M) to generate the request number of the at least one load request, wherein req_number is the request number, ceil() is a ceiling function, T is an execution unit thread number, f is the convolution kernel size, C is the input channel number, dem_size is a data element bit width, the unit of dem_size is byte, X and M are output bandwidths of the shared memory, the unit of X is the number of pixels, and the unit of M is byte. and For each request of the at least one load request, the load storage circuit calculates addr = base_addr + req_id X step + ich_id dem_size to generate the shared memory address, where addr is the shared memory address, base_addr is the shared memory base address of the input parameter, req_id is one of the request sequence numbers in the request number, step is the pixel pitch parameter in the input parameter, the unit of step is byte, and ich_id is the input channel sequence number.

7. The artificial intelligence chip of claim 1, wherein: the load-store circuit calculates a start pixel data position and an end pixel data position of the sliding window in different iterations according to an iteration serial number and a number of execution unit threads.

8. The artificial intelligence chip according to claim 7, characterized in that, the start pixel data position is the iteration serial number, and the end pixel data position is (T + k - 1), wherein T is a number of execution unit threads, k is an integer from 0 to f - 1, and f is a convolution kernel size.

9. The artificial intelligence chip of claim 1, wherein, The artificial intelligence chip further comprises: an instruction decoder coupled to the load-store circuit, wherein the instruction decoder routes a sliding window load instruction to the load-store circuit, and the load-store circuit executes the sliding window load instruction to fetch the plurality of original pixel data required for the plurality of iterations from the shared memory at one time.

10. The artificial intelligence chip according to claim 9, characterized in that, The load-store circuit comprises: a load store unit coupled to the instruction decoder, the shared memory, and the at least one execution unit, wherein the instruction decoder routes the non-sliding window load instruction to the load store unit, and the load store unit executes the non-sliding window load instruction to fetch data from the shared memory; and a sliding window load unit coupled to the instruction decoder, the shared memory, and the at least one execution unit, wherein the instruction decoder routes the sliding window load instruction to the sliding window load unit, and the sliding window load unit executes the sliding window load instruction to fetch the plurality of original pixel data required by the plurality of iterations from the shared memory in one continuous shot.

11. The artificial intelligence chip according to claim 10, characterized in that, The sliding window load unit comprises: an address generation unit to calculate a shared memory address according to input parameters carried by the sliding window load instruction and to generate at least one load request to the shared memory; a data buffer unit, wherein the shared memory feeds back the plurality of original pixel data to the data buffer unit in one continuous shot according to the at least one load request; and a sliding window unit to calculate a start pixel data position and an end pixel data position of the sliding window in different iterations according to an iteration serial number and a number of execution unit threads.

12. The artificial intelligence chip of claim 11, wherein: the address generation unit calculates a number of the at least one load request according to a convolution kernel size and a number of input channels; and for each of the at least one load request, the address generation unit calculates the shared memory address according to a request serial number and the input parameters.

13. The artificial intelligence chip of claim 12, wherein: The address generation unit calculates req_number = ceil((T + f - 1 + X - 1) / X X C dem_size / M) to generate the request number of the at least one loading request, wherein req_number is the request number, ceil() is a ceiling function, T is an execution unit thread number, f is the convolution kernel size, C is the input channel number, dem_size is a data element bit width, the unit of dem_size is byte, X and M are output bandwidths of the shared memory, the unit of X is the number of pixels, and the unit of M is byte. the start pixel data position is the iteration serial number, and the end pixel data position is (T + k - 1), where T is the number of execution unit threads, k is the iteration serial number and k is an integer from 0 to f-1, and f is the convolution kernel size. For each request of the at least one loading request, the address generation unit calculates addr = base_addr + req_id X step + ich_id dem_size to generate the shared memory address, where addr is the shared memory address, base_addr is the shared memory base address of the input parameter, req_id is one of the request serial numbers in the request number, step is the pixel pitch parameter in the input parameter, the unit of step is byte, and ich_id is the input channel serial number.

14. The artificial intelligence chip of claim 11, wherein, The operation method comprises:

15. A method for operating an artificial intelligence chip, characterized in that, fetching, by a load store circuit of the artificial intelligence chip, a plurality of original pixel data required by a plurality of iterations from a shared memory of the artificial intelligence chip as a plurality of buffered pixel data in one continuous shot, wherein the load store circuit is coupled to the shared memory and at least one execution unit of the artificial intelligence chip, the load store circuit automatically moves a sliding window to output required data; in a first iteration, a plurality of first respective pixel data in the plurality of buffered pixel data is covered by the sliding window; in the first iteration, the load store circuit distributes the plurality of first respective pixel data to a respective execution unit in the at least one execution unit; in a second iteration, a plurality of second respective pixel data in the plurality of buffered pixel data is covered by the sliding window; and in the second iteration, the load store circuit distributes the plurality of second respective pixel data to a respective execution unit in the at least one execution unit. The plurality of original pixel data is all pixel data of an image.

16. The method of operation of claim 15, wherein, ​ 17. The method of claim 15, wherein, The plurality of original pixel data is (T + f - 1) pixel data in a continuous image, wherein T is the number of execution unit threads, and f is the size of the convolution kernel.

18. The operating method according to claim 15, characterized in that, The operation method further comprises: The loading and storing circuit calculates the shared memory address according to the input parameters of the sliding window loading instruction, and generates at least one loading request to the shared memory; and The shared memory continuously feeds back the plurality of original pixel data to the loading and storing circuit according to the at least one loading request.

19. The method of operation of claim 18, wherein, The operation method further comprises: The loading and storing circuit calculates the number of loading requests of the at least one loading request according to the size of the convolution kernel and the number of input channels; and For each request of the at least one loading request, the loading and storing circuit calculates the shared memory address according to the request serial number and the input parameters.

20. The method of operation of claim 19, wherein, The operation method further comprises: req_number = ceil((T + f - 1 + X - 1) / X X C dem_size / M) to generate the request number of the at least one load request, wherein req_number is the request number, ceil() is a ceiling function, T is an execution unit thread number, f is the convolution kernel size, C is the input channel number, dem_size is a data element bit width, the unit of dem_size is byte, X and M are output bandwidths of the shared memory, the unit of X is the number of pixels, and the unit of M is byte; and, For each request of the at least one load request, the load storage circuit calculates addr = base_addr + req_id X step + ich_id dem_size to generate the shared memory address, where addr is the shared memory address, base_addr is the shared memory base address of the input parameter, req_id is one of the request serial numbers in the request number, step is the pixel pitch parameter in the input parameter, the unit of step is byte, and ich_id is the input channel serial number.

21. The operating method according to claim 15, characterized in that, The operation method further comprises: The loading and storing circuit calculates the starting pixel data position and the ending pixel data position of the sliding window in different iterations according to the iteration serial number and the number of execution unit threads.

22. The method of operation of claim 21, wherein, The starting pixel data position is the iteration serial number, and the ending pixel data position is (T + k - 1), wherein T is the number of execution unit threads, k is the iteration serial number and k is an integer from 0 to f-1, and f is the size of the convolution kernel.

23. The method of claim 15, wherein, The operation method further comprises: The instruction decoder of the artificial intelligence chip routes the sliding window loading instruction to the loading and storing circuit, wherein the instruction decoder is coupled to the loading and storing circuit; and The loading and storing circuit executes the sliding window loading instruction to continuously extract the plurality of original pixel data required for the plurality of iterations from the shared memory at one time.

24. The method of operation of claim 23, wherein, The operation method further comprises: The instruction decoder routes the non-sliding window loading instruction to the loading and storing unit of the loading and storing circuit, wherein the loading and storing unit is coupled to the instruction decoder, the shared memory and the at least one execution unit; The loading and storing unit executes the non-sliding window loading instruction to extract data from the shared memory; The instruction decoder routes the sliding window loading instruction to the sliding window loading unit of the loading and storing circuit, wherein the sliding window loading unit is coupled to the instruction decoder, the shared memory and the at least one execution unit; and The sliding window loading unit executes the sliding window loading instruction to continuously extract the plurality of original pixel data required for the plurality of iterations from the shared memory at one time.

25. The method of operation of claim 24, wherein, The operation method further comprises: The address generation unit of the sliding window loading unit calculates the shared memory address according to the input parameters of the sliding window loading instruction, and generates at least one loading request to the shared memory; The shared memory continuously feeds back the plurality of original pixel data to the data buffer unit of the sliding window loading unit according to the at least one loading request; and The operation method further comprises: The operation method further comprises: The loading and storing circuit calculates the shared memory address according to the input parameters of the sliding window loading instruction, and generates at least one loading request to the shared memory; and The shared memory continuously feeds back the plurality of original pixel data to the loading and storing circuit according to the at least one loading request. The operation method further comprises: The operation method further comprises: The loading and storing circuit calculates the starting pixel data position and the ending pixel data position of the sliding window in different iterations according to the iteration serial number and the number of execution unit threads. The starting pixel data position is the iteration serial number, and the ending pixel data position is (T + k - 1), wherein T is the number of execution unit threads, k is the iteration serial number and k is an integer from 0 to f-1, and f is the size of the convolution kernel. The operation method further comprises: The instruction decoder of the artificial intelligence chip routes the sliding window loading instruction to the loading and storing circuit, wherein the instruction decoder is coupled to the loading and storing circuit; and The loading and storing circuit executes the sliding window loading instruction to continuously extract the plurality of original pixel data required for the plurality of iterations from the shared memory at one time. The operation method further comprises: The instruction decoder routes the non-sliding window loading instruction to the loading and storing unit of the loading and storing circuit, wherein the loading and storing unit is coupled to the instruction decoder, the shared memory and the at least one execution unit; The loading and storing unit executes the non-sliding window loading instruction to extract data from the shared memory; The instruction decoder routes the sliding window loading instruction to the sliding window loading unit of the loading and storing circuit, wherein the sliding window loading unit is coupled to the instruction decoder, the shared memory and the at least one execution unit; and The sliding window loading unit executes the sliding window loading instruction to continuously extract the plurality of original pixel data required for the plurality of iterations from the shared memory at one time. The operation method further comprises: The address generation unit of the sliding window loading unit calculates the shared memory address according to the input parameters of the sliding window loading instruction, and generates at least one loading request to the shared memory; The shared memory continuously feeds back the plurality of original pixel data to the data buffer unit of the sliding window loading unit according to the at least one loading request; and The operation method further comprises: The operation method further comprises: The loading and storing circuit calculates the shared memory address according to the input parameters of the sliding window loading instruction, and generates at least one loading request to the shared memory; and The shared memory continuously feeds back the plurality of original pixel data to the loading and storing circuit according to the at least one loading request. The sliding window loading unit calculates the start pixel data position and the end pixel data position of the sliding window in different iterations according to the iteration serial number and the number of execution unit threads.

26. The method of operating according to claim 25, wherein, The operation method further comprises: The address generation unit calculates the number of requests of the at least one loading request according to the convolution kernel size and the number of input channels; and For each request of the at least one loading request, the address generation unit calculates the shared memory address according to the request serial number and the input parameter.

27. The method of operation of claim 26, wherein, The operation method further comprises: req_number = ceil((T + f - 1 + X - 1) / X X C dem_size / M) to generate the request number of the at least one loading request, wherein req_number is the request number, ceil() is a ceiling function, T is an execution unit thread number, f is the convolution kernel size, C is the input channel number, dem_size is a data element bit width, the unit of dem_size is byte, X and M are output bandwidths of the shared memory, the unit of X is the number of pixels, and the unit of M is byte; and For each request of the at least one load request, the address generation unit calculates addr = base_addr + req_id X step + ich_id dem_size to generate the shared memory address, where addr is the shared memory address, base_addr is the shared memory base address of the input parameter, req_id is one of the request sequence numbers in the request number, step is the pixel pitch parameter in the input parameter, the unit of step is byte, and ich_id is the input channel sequence number.

28. The method of claim 25, wherein: The start pixel data position is the iteration serial number, and the end pixel data position is (T + k - 1), wherein T is the number of execution unit threads, k is the iteration serial number and k is an integer from 0 to f-1, and f is the convolution kernel size.

Images