Artificial intelligence image processing device and data multiplexing and parallel computing method

Abstract

The invention provides an artificial intelligence image processing device and a data multiplexing and parallel computing method. The device comprises a convolution calculation unit which comprises a multiply-accumulate array; the vector calculation unit is used for executing non-convolution class operation; the data bypass is directly coupled between the output end of the convolution calculation unit and the input end of the vector calculation unit; the scalar engine is used for controlling ordered execution of the convolution calculation unit and the vector calculation unit; the internal storage unit is used for temporarily storing data in the input image; wherein the multiply-accumulate array is configured to load a plurality of slices in an input image from an internal storage unit in a column direction, and then multiply-accumulate results between each pixel value in the plurality of slices and each value in a convolution kernel are calculated in parallel; and parallelly computing the plurality of output channels in the row direction. The device adopts an image width / height / channel three-dimensional parallel mode to realize dimension complementation so as to improve the parallelism of image processing.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and image processing, specifically relating to an artificial intelligence image processing device and a data reuse and parallel computing method. Background Technology

[0002] Typical artificial intelligence image processing devices employ Figure 1 The structure includes a Direct Memory Access (DMA) controller 101, a vector computation unit 102, a convolution computation unit 103, a microcontroller 104, and a storage unit 105. The convolution computation unit 103 typically uses a two-dimensional multiply-accumulate (MAC) array (e.g., 64×64) as its core computing power source. The typical processing flow of this device is as follows: The input image is first stored in external memory 106. The DMA controller 101 reads the input image data from external memory 106 into the storage unit 105. The convolution computation unit 103 sends the data from the storage unit 105 into the multiply-accumulate array, which consists of multiply-accumulate units arranged in a matrix. The multiply-accumulate units perform multiply-accumulate calculations and write the results back to the storage unit 105. Subsequently, the vector computation unit 102 reads the multiply-accumulate result from the storage unit 105 and performs subsequent processing such as activation, normalization, or element-wise operations.

[0003] To fully utilize the multiply-accumulate array, traditional designs typically reuse and parallelize data along the "channel dimension." For example, a multiply-accumulate array might be 64x64 (usually square or rectangular), and the input image has 64 channels (c). Each read operation involves reading data from the same position across all 64 channels and then placing this data into the corresponding positions of the multiply-accumulate array for parallel computation across all 64 channels. This approach performs well in classification networks with a large number of channels. However, in image enhancement networks with fewer channels and higher resolutions (such as super-resolution, noise reduction, and HDR), the limited number of channels often results in the multiply-accumulate array not being fully filled, leading to a significant decrease in array utilization and limitations in performance and energy efficiency.

[0004] Moreover, existing neural network processing devices generally follow a three-stage process of "read-calculate-write" when performing convolution, that is, the convolution calculation unit first reads the input image from the storage unit, completes the convolution calculation, and then writes the result back to the storage unit. The frequent and high-bandwidth data exchange between the convolution calculation unit 103 and the storage unit not only increases the power consumption of memory access, but also becomes a bottleneck for system performance.

[0005] Therefore, there is an urgent need for a parallel computing and data reuse method that can simultaneously improve the utilization of multiply-accumulate arrays, reduce memory access bandwidth, and adapt to the characteristics of image quality networks. Summary of the Invention

[0006] In view of this, the purpose of this disclosure is to provide an artificial intelligence image processing apparatus and a data reuse and parallel computing method to solve the above problems.

[0007] In a first aspect, embodiments of this disclosure provide an artificial intelligence image processing apparatus, comprising:

[0008] The convolution computation unit includes a multiply-accumulate array, which is composed of multiply-accumulate units arranged in a matrix by rows and columns;

[0009] Vector computation unit, used to perform non-convolution operations;

[0010] Data bypass is directly coupled between the output of the convolution calculation unit and the input of the vector calculation unit, so that the output of the convolution calculation unit is used as the input of the vector calculation unit;

[0011] A microcontroller is used to control the orderly execution of various hardware units;

[0012] Internal storage unit, used to temporarily store data from the input image;

[0013] The multiply-accumulate array is configured to: load multiple slices of the input image from the internal storage unit in the column direction, and then calculate the multiply-accumulate result between each pixel value in the multiple slices and each value in the convolution kernel in parallel; and calculate multiple output channels in parallel in the row direction, wherein the same row of each channel data in the multiple channel data of the input image is divided into slices, and the loaded multiple slices are multiple slices at the same position in multiple adjacent rows of the same channel data of the input image.

[0014] In some embodiments, there is at least one single cycle in which the multiply-accumulate array computes multiple slices in two or more channels of data in the input image in parallel.

[0015] In the internal storage unit, multiple slices of data from the same adjacent row at the same location in the same channel are organized together.

[0016] In some embodiments, the artificial intelligence image processing apparatus further includes a storage access controller for moving sub-images containing slices into or out of the artificial intelligence image processing apparatus.

[0017] In some embodiments, the data bypass is turned off when no fusion operation is required.

[0018] In some embodiments, the multiply-accumulate array is divided into multiple groups in the row direction corresponding to the number of the plurality of output channels, each group sharing the same convolution kernel, and each group generating partial sums of the corresponding output channels in parallel.

[0019] In some embodiments, the size of the slice is determined based on the size of the multiply-accumulate array, the width, height, and number of channels of the input image.

[0020] In some embodiments, the size of the slice is equal to the column size of the multiply-accumulate array, and the product of the number of slices per row of the input image and the number of channels of the input image is equal to the number of columns of the multiply-accumulate array.

[0021] Secondly, embodiments of this disclosure provide a chip that integrates the artificial intelligence image processing device described above.

[0022] Thirdly, embodiments of this disclosure provide a data reuse and parallel computing method applied to an artificial intelligence image processing device, the artificial intelligence image processing device comprising a multiply-accumulate array arranged in a two-dimensional array, the method comprising:

[0023] The same row of data from each channel in the input image is divided into slices, and each slice includes multiple consecutive pixel values ​​from the same row of the same channel in the input image.

[0024] Multiple slices at the same position in multiple adjacent rows of the same channel data of the input image are loaded into multiple columns of the multiply-accumulate array so that the multiple columns of the multiply-accumulate array can calculate the multiply-accumulate result between each pixel value in the multiple slices and each value in the convolution kernel in parallel.

[0025] In some embodiments, the data of the input image is organized and transferred to the internal storage unit of the artificial intelligence image processing device according to slices.

[0026] In some embodiments, the size of the slice is determined based on the size of the multiply-accumulate array and the width, height, and number of channels of the input image.

[0027] In summary, the artificial intelligence image processing apparatus of this disclosure fully utilizes the parallelism of the multiply-accumulate array by allocating slices at the same position of multiple adjacent rows in each of multiple channels to multiple columns of the multiply-accumulate array. This allows the multiply-accumulate units of multiple columns to perform multiply-accumulate calculations in parallel. Furthermore, data bypassing between the convolution calculation unit and the vector calculation unit reduces memory access power consumption. Additionally, reusing convolution kernels and inter-row data in the multiply-accumulate array also reduces memory access power consumption.

[0028] It should be noted that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the present invention. Attached Figure Description

[0030] The above and other objects, features, and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0031] Figure 1 This is a schematic diagram of a typical artificial intelligence image processing device;

[0032] Figure 2 This is a schematic diagram of the architecture of an artificial intelligence image processing chip for super-resolution image processing;

[0033] Figure 3 This is a structural diagram of an artificial intelligence image processing device proposed in an embodiment of this disclosure.

[0034] Figure 4 This is an example of a multi-dimensional parallel scheme provided in the embodiments of this disclosure;

[0035] Figure 5 A flowchart of a data reuse and parallel computing method applied to an artificial intelligence image processing device is presented. Detailed Implementation

[0036] The present application will now be described in more detail with reference to the accompanying drawings. In the various drawings, the same elements are indicated by similar reference numerals. For clarity, the various parts in the drawings are not drawn to scale. Furthermore, some well-known parts may not be shown.

[0037] The following terms are used in this article.

[0038] Convolution computation uses a learnable convolutional kernel as a "sliding window" that slides across the input image from left to right and from top to bottom with a fixed stride. At each position, each weight in the convolutional kernel is multiplied by the corresponding pixel value, and all products are summed to obtain the single-channel scalar output of the window.

[0039] Data bypass refers to forwarding intermediate results that have just been calculated but have not yet been written back to the register file or memory directly to the next / next operation unit, thereby saving the round-trip delay of writing back and then reading.

[0040] Figure 2This is a schematic diagram of the architecture of an artificial intelligence image processing chip for super-resolution image processing. As shown in the figure, low-resolution video frame images are processed by the MIPI (Mobile Industry Processor Interface) receivers RX0 and RX1. After initial processing by preprocessor 201, the frame images are fed into the AI ​​(Artificial Intelligence) high-resolution image processing device (AISR, AI-SuperResolution) 202, which outputs high-resolution frame images. These then enter the motion estimation and compensation unit (MEMC, Motion Estimation and Motion Compensation) 203 for frame interpolation to obtain high-frame-rate images. Finally, after passing through the post-processor 206, the images are output to the transmitter TX. The post-processor 206 performs dynamic contrast enhancement, color enhancement, and 3D-LUT (3D Look-Up Table) operations on the input image. The motion vector and depth unit 204 receives motion vector and depth information from receiver RX1, performs preprocessing, and writes it to the frame buffer unit 205 for inter-frame motion estimation. The frame buffer unit 205 is used to buffer each frame image.

[0041] Figure 2 In this chip, various hardware modules can be selectively switched on and off, and the chip can be connected after, for example, a mobile phone's GPU chip. The mobile phone's GPU can render games or videos at a lower resolution / frame rate, and after passing through this chip, a higher resolution / frame rate image is obtained, thereby saving power consumption and increasing battery life.

[0042] Figure 3 This is a structural diagram of an artificial intelligence image processing device 300 according to an embodiment of this disclosure. The artificial intelligence image processing device 300 can be integrated into... Figure 2 AI high-resolution image processing device 202 or completely replace Figure 2 The AI ​​high-resolution image processing device 202 is mentioned. However, the artificial intelligence image processing device 300 is not limited to... Figure 2 It can be used in super-resolution image processing scenarios, as well as other image processing scenarios, such as target recognition in videos and medical image analysis.

[0043] like Figure 3As shown, the artificial intelligence image processing device 300 includes a microcontroller 301, a convolution kernel buffer 302, a storage access controller 303, a convolution calculation unit 304, a vector calculation unit 305, an input line / output line buffer 306, a code buffer 330, a scalar engine 310, and an internal storage unit 311, all connected via an instruction bus 308 and a data bus 309.

[0044] The scalar engine 310, acting as the control unit for executing the artificial intelligence model, reads instructions from the code buffer 330, decodes them, and executes irregular operation instructions such as control, loops, and address calculations. It then sends the remaining instructions to the corresponding hardware units; for example, it sends convolution calculation instructions to the convolution calculation unit 304, vector calculation instructions to the vector calculation unit 305, and data transfer instructions to the memory access controller 303. The scalar engine 310 can also adjust the output rhythm of the convolution calculation unit 304 through a state machine under two-operand instructions to synchronously drive the vector calculation unit 305, achieving the fusion of the two-layer operations.

[0045] The input / output line buffer 301 has an input / output capacity limit, allowing it to temporarily store only sub-images of the input image or the output image. The input / output line buffer 301 can be connected to an external source via an interface circuit, such as a MIPI receiver, to obtain sub-images of the input image from upstream and output sub-images of the output image to downstream.

[0046] The storage access controller 303 is used to move data from the input / output line buffer 301 that is about to be convolved in a sub-image to the internal storage unit 311, and to move data after convolution or vector calculation from the internal storage unit 311 to the input / output line buffer 301. Specifically, the storage access controller 303 performs data movement according to data movement instructions. When the scalar engine 310 detects a new sub-image in the input / output line buffer 307, it initiates the data movement operation of the storage access controller 303. The data movement instructions can specify the starting position and length of the data movement. The convolution kernel buffer 302 is used to buffer convolution kernels. When the convolution operation unit 304 needs a convolution kernel, it reads it from the convolution kernel buffer 302.

[0047] The convolution computation unit 304 integrates a two-dimensional multiply-accumulate (MAC) array as its core computing power source. Each MAC unit consists of a multiplier and an adder cascaded together. The convolution computation unit 304 retrieves multiple pixel values ​​to be convolved from the internal storage unit 311 and distributes these pixel values ​​into the corresponding MAC units of the MAC array so that the corresponding MAC units can perform multiply-accumulate calculations. The vector computation unit 305 may include a configurable activation function module, an element-wise addition module, and a normalization module, enabling post-processing of the convolution results.

[0048] A direct data connection can be established between the convolution calculation unit 304 and the vector calculation unit 305 via a data bypass. This data bypass can be formed, for example, by associating the output register of the convolution calculation unit 304 with the input register of the vector calculation unit 305, so that the convolution result can be processed by the vector calculation unit 305 without passing through the internal storage unit 311. The data bypass can also be turned off when no fusion operation is required, allowing the vector calculation unit 305 to obtain data from other locations (e.g., internal memory 211) for independent calculation.

[0049] It should be understood that, in actual product manufacturing, the artificial intelligence image processing device provided in the above embodiments may include more than Figure 3 More or fewer hardware units, for example, omitting the input / output line buffer 307, having the storage access controller 303 directly move data from external memory to internal storage unit 311, and moving calculation results from internal storage unit 311 to external memory; or, for example, deciding to add certain hardware units or change the number and connection method of buses based on the artificial intelligence model to be executed by the artificial intelligence image processing device.

[0050] based on Figure 4 The provided artificial intelligence image processing device 300, in this embodiment, also proposes a more flexible parallel scheme. Compared with the "channel dimension" parallel method in the prior art, this parallel scheme adopts a three-dimensional parallel approach of the input image's width (w), height (h), and channels (c) to achieve multi-dimensional complementarity, thereby improving the parallelism of the image processing artificial intelligence network model. Furthermore, this scheme can achieve data reuse in the image width (w) and height (h) directions, reducing memory access and thus saving power consumption.

[0051] The following is combined Figure 3 and Figure 4 This multi-dimensional parallel scheme is described in detail.

[0052] exist Figure 4In this process, each channel of the input image is divided into multiple slices along each row of the image width (w). Each slice contains multiple consecutive pixels in the same row. Multiple slices at the same position in adjacent rows within the same channel are input in parallel into the corresponding columns of the multiply-accumulate array. Assuming the input image size is w×h×c=1280×720×16, due to the limited space of the internal storage unit 311, the image is divided into 240x76 sub-images, with one sub-image read into the internal storage unit 311 at a time. Each row of the 240x76 sub-image is divided into 4 slices. Then, 4 vectors (e.g., h0 to h3 in the figure) consisting of 4 slices at the same position in 4 adjacent rows of the first channel data are fed in parallel into the 4 columns of the multiply-accumulate array. The 64 columns of the multiply-accumulate array are grouped into 16 groups of four. The above 4 vectors are repeatedly fed to these 16 groups, with each group calculating one output channel, for a total of 16 output channels.

[0053] Therefore, in conjunction with the above text Figure 3 As described, the storage access controller 303 loads a sub-image of the input image into the internal storage unit 311 each time. The convolution calculation unit 304 reads multiple vectors composed of multiple slices at the same position in adjacent rows from the internal storage unit 311. Each vector contains multiple consecutive pixel values ​​in the same row. These multiple vectors are then loaded onto multiple columns of the multiply-accumulate array of the convolution calculation unit 304. Within a single cycle, the multiply-accumulate array of the convolution calculation unit 304 can perform parallel multiply-accumulate calculations of one or more vectors in the column direction, while dividing the data into multiple groups (multiple rows are divided into one group) corresponding to the number of output channels in the row direction. Each group shares the same convolution kernel, and each group generates partial sums of the corresponding output channels in parallel. Furthermore, preferably, in the internal storage unit 311, multiple vectors composed of multiple slices at the same position in multiple adjacent rows of the same channel data are organized together to facilitate data acquisition by the convolution calculation unit 304.

[0054] Based on the above embodiments, when the number of channels in the input image is insufficient, the parallelism of the multiply-accumulate array in the column direction is fully utilized by inputting multiple slices at the same position in adjacent rows of each of the multiple channels to multiple columns of the multiply-accumulate array. Furthermore, during convolution calculation, the parallelism in width (w) and height (h) allows for the reuse of the convolution kernel, while the parallelism in the height (h) direction allows for the reuse of inter-row data, thereby reducing memory access power consumption.

[0055] As a variation, the individual channel data of the input image can be sliced ​​into multiple slices along each column of the image height (h), with each slice containing multiple consecutive pixel values ​​from the same column. Multiple slices at the same position in adjacent columns of the same channel are then input in parallel into the corresponding rows of a multiply-accumulate array. This example is similar to... Figure 4 The difference is that, Figure 4 In some cases, the insufficient number of channels is compensated by the image width (w), while in this example, the insufficient number of channels is compensated by the image height (h).

[0056] It should be understood that the above scheme is due to insufficient parallelism of the multiply-accumulate array using the channel dimension because the number of channels in the input image is insufficient. Therefore, parallelism in the width direction is used to supplement the parallelism in the channel direction. However, if the number of channels is sufficient, a channel-dimensional parallelism scheme can still be used. In view of this, the compiler can determine which scheme to use based on the size of the multiply-accumulate array and the width (w), height, and channels (c) of the input image, and accordingly drive the memory access controller to move data in the appropriate manner, and configure the convolution calculation unit 304 to execute the multiply-accumulate units on the corresponding rows or columns in parallel in the appropriate manner, thereby improving the parallelism of the multiply-accumulate array. In some optimal cases, the size of the slice can be equal to the column size of the multiply-accumulate array, and the product of the number of slices divided in each row of the input image and the number of channels in the input image equals the number of columns of the multiply-accumulate array. In this way, at a certain moment, the pixel values ​​in the input image will exactly fill the multiply-accumulate array.

[0057] Accordingly, embodiments of this disclosure also provide an application for, for example Figure 3 The data multiplexing and parallel computing methods in the artificial intelligence image processing device shown are illustrated. Figure 5 As shown, the method includes the following steps.

[0058] In step S501, for each channel of the input image data, each row in the image width direction is divided into multiple slices, and each slice includes multiple consecutive pixels in the same row of the same channel.

[0059] In step S502, multiple slices at the same position of multiple adjacent rows of the same channel data of the input image are loaded into multiple columns of the multiply-accumulate array.

[0060] According to this embodiment, for each channel of the input image's multiple channel data, each row in the image width direction is divided into multiple slices, and the slices are laid out on the multiply-accumulate array in a way that combines the image width and the image channel dimension, so as to fill the multiply-accumulate array as much as possible. In this way, the corresponding multiple columns of the multiply-accumulate array can calculate the multiply-accumulate result between each pixel value in the multiple slices and each value in the convolution kernel in parallel. At the same time, the multiply-accumulate array can be divided into multiple groups in the row direction corresponding to the number of multiple output channels. Each group shares the same convolution kernel, and each group calculates the multiply-accumulate result of the corresponding output channel in parallel. Thus, the parallelism of the multiply-accumulate array is improved in both the row and column directions.

[0061] In summary, the methods and artificial intelligence image processing apparatus provided in the embodiments of this disclosure help improve the utilization rate of the multiply-accumulate array, increase data reuse, reduce data memory access, and help reduce chip power consumption and improve chip energy efficiency ratio.

[0062] Accordingly, the data reuse and parallel computing method provided in this disclosure can also be formed as a computer program, which is formed as a computer-readable storage medium that stores one or more computer instructions. When the one or more computer instructions are executed, the steps in the above-mentioned data reuse and parallel computing method are implemented.

[0063] It should be understood that the methods, systems, computer-readable storage media, and image processing chips of the embodiments of this disclosure are all based on the same conceptual idea and therefore can be referenced to each other.

[0064] Although the embodiments of this application are disclosed above with reference to preferred embodiments, they are not intended to limit the claims. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application shall be determined by the scope defined by the claims of this application.

[0065] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An artificial intelligence image processing apparatus, characterized by, The artificial intelligence image processing device comprises: a convolution calculation unit comprising a multiply-accumulate array composed of multiply-accumulate units arranged in a matrix; a vector calculation unit for performing non-convolution type operations; a data bypass directly coupled between the output of the convolution calculation unit and the input of the vector calculation unit, so that the output of the convolution calculation unit is taken as the input of the vector calculation unit; a scalar engine for controlling the ordered execution of the convolution calculation unit and the vector calculation unit; an internal storage unit for temporarily storing data in an input image; wherein the multiply-accumulate array is configured to load multiple slices in the input image from the internal storage unit in the column direction, and then calculate multiply-accumulate results between each pixel value in the multiple slices and each value in a convolution kernel in parallel. In addition, multiple output channels are calculated in parallel in the row direction, and each row of multiple channel data of the input image is split into slices, and the multiple slices are multiple slices at the same position of multiple adjacent rows of the same channel data of the input image.

2. The artificial intelligence image processing device according to claim 1, wherein there is at least one single cycle in which the multiply-accumulate array calculates multiple slices in two or more channel data in the input image in parallel. 3.The artificial intelligence image processing apparatus of claim 1, characterized in that, In the internal storage unit, multiple slices at the same position of multiple adjacent rows of the same channel data are organized together.

4. The artificial intelligence image processing apparatus of claim 3, wherein, The artificial intelligence image processing device further comprises a storage access controller for inputting or outputting a sub-image where the slices are located into or out of the artificial intelligence image processing device. 5.The artificial intelligence image processing apparatus of claim 1, characterized in that, The data bypass is turned off when no fusion operation is needed. 6.The artificial intelligence image processing apparatus of claim 1, wherein, The multiply-accumulate array is divided into multiple groups in the row direction corresponding to the number of the multiple output channels, each group sharing the same convolution kernel, and each group generates a partial sum of the corresponding output channel in parallel. 7.The artificial intelligence image processing apparatus of claim 1, characterized in that, The size of the slice is determined according to the size of the multiply-accumulate array, the width, height and channel number of the input image. 8.The artificial intelligence image processing apparatus of claim 7, wherein, The size of the slice is equal to the column size of the multiply-accumulate array, and the product of the number of slices per row of the input image and the channel number of the input image is equal to the number of columns of the multiply-accumulate array.

9. A chip integrated with the artificial intelligence image processing device according to any one of claims 1 to 8.

10. A data multiplexing and parallel computing method applied to an artificial intelligence image processing device, the artificial intelligence image processing device comprising a multiply-accumulate array composed of multiply-accumulate units arranged in a matrix, characterized in that, The artificial intelligence image processing device comprises: slices are split from each row of each channel data of multiple channel data of an input image, each slice comprising multiple consecutive pixel values of the same row of the same channel in the input image; multiple slices at the same position of multiple adjacent rows of the same channel data of the input image are loaded into multiple columns of the multiply-accumulate array, so that multiple columns of the multiply-accumulate array calculate multiply-accumulate results between each pixel value in the multiple slices and each value in a convolution kernel in parallel.

11. The data multiplexing and parallel computing method of claim 10, wherein, Data of the input image are organized and transported to the internal storage unit of the artificial intelligence image processing device according to slices.

12. The data multiplexing and parallel computing method of claim 10, wherein, The size of the slice is determined according to the size of the multiply-accumulate array and the width, height and channel number of the input image.

Images