A multi-vision task accelerator and a control method of multi-vision task processing

Abstract

The application provides a multi-vision task accelerator and a control method for multi-vision task processing, wherein the vision task refers to a task of processing input images by using a convolutional neural network, each image convolutional neural network comprises at least one convolutional layer, the accelerator comprises: a calculation array for performing convolution operation of the convolutional layer; and a controller for controlling the calculation array to perform convolution operation corresponding to one vision task or dividing the calculation array into at least two areas to simultaneously perform convolution operation corresponding to at least two vision tasks in the multiple vision tasks when there are idle calculation resources in the calculation array performing convolution operation corresponding to a single vision task in response to an acceleration calculation request of one or more vision tasks.

Description

Technical Field

[0001] This invention relates to the field of computer vision task processing technology, specifically to the field of hardware acceleration technology for neural network model computation, and more specifically, to a multi-vision task accelerator and a control method for multi-vision task processing. Background Technology

[0002] Currently, a plethora of neural network algorithms for processing computer vision are emerging. To achieve real-time performance, most neural network algorithms can be deployed on accelerators, such as those based on Field Programmable Gate Arrays (FPGAs), Neural-network Processing Units (NPUs), or custom Application Specific Integrated Circuits (ASICs). While these technologies are relatively mature, some challenges remain: to quickly execute a single vision task, computing units typically allocate all their hardware resources to that single task. This minimizes the computational latency of a single task; however, the size of the computing units required for a vision task does not perfectly match the actual design size. If multiple vision tasks request accelerated computation, they are often queued according to the order of their requests, resulting in typically low hardware utilization and performance. Summary of the Invention

[0003] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-vision task accelerator and a control method for multi-vision task processing.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] According to a first aspect of the present invention, a multi-visual-task accelerator is provided, wherein the visual task refers to the task of processing an input image using a convolutional neural network, each image convolutional neural network including at least one convolutional layer, and the accelerator includes: a computing array for performing convolution operations of the convolutional layers; and a controller for controlling the computing array to perform a convolution operation corresponding to a visual task or, when there are idle computing resources available for the computing array to perform a convolution operation corresponding to a single visual task, dividing the computing array into at least two regions to simultaneously perform convolution operations corresponding to at least two of the multiple visual tasks.

[0006] In some embodiments of the present invention, the computation array includes multiple multipliers and multiple accumulators. Each multiplier is used to perform a multiplication operation between the weight parameters and the corresponding data, and each accumulator corresponds to one or more multipliers to accumulate the results of the corresponding multiplication operations to obtain the convolution sum corresponding to the convolution operation.

[0007] In some embodiments of the present invention, the controller is configured to perform the following operations when there are multiple visual tasks to be accelerated: allocate multipliers and accumulators in a computation array to convolutional layers in multiple visual tasks according to a predetermined priority, wherein each convolutional channel in the convolutional layer is allocated a corresponding multiplier and accumulator, and when the corresponding convolutional layer of the visual task that was allocated earlier has not fully utilized all the multipliers in the computation array, dynamically partition the computation array to utilize the multipliers and accumulators in different regions to process the convolutional operations corresponding to the convolutional layers of different visual tasks.

[0008] In some embodiments of the present invention, the accelerator further includes: a data storage unit for storing feature maps that need to be processed by the convolutional layer, the feature maps being an input image or an intermediate image obtained by processing an image convolutional neural network; a weight storage unit for storing weight parameters of the convolutional layer; a data buffer unit for storing all data that the computing array will process next, the data being data in the feature maps allocated to the next operation in the computing array; and a weight buffer unit for storing all weight parameters that the computing array will utilize next, the weight parameters being weight parameters used for multiplying the data in the feature maps allocated to the next operation in the computing array.

[0009] In some embodiments of the present invention, the predetermined priority is determined based on the initiation time of the accelerated computing request for the visual task and / or the latency requirements of the visual task.

[0010] In some embodiments of the present invention, the accelerator further includes: a first multiplexer for filling data from a data buffer unit into a corresponding region of the computing array, and a second multiplexer for filling weight parameters from a weight buffer unit into a corresponding region of the computing array, wherein the controller is configured to control the first multiplexer to fill data into the corresponding region from the left-right direction of the computing array and the second multiplexer to fill weight parameters into the corresponding region from the top-bottom direction of the computing array, or to control the first multiplexer to fill data into the corresponding region from the top-bottom direction of the computing array and the second multiplexer to fill weight parameters into the corresponding region from the left-right direction of the computing array.

[0011] In some embodiments of the present invention, each multiplier includes: a first gate for sending data input from the left or right side of the computation array to the data terminal of the corresponding multiplication unit via a first multiplexer; a second gate for sending weight parameters input from the top or bottom of the computation array via a second multiplexer to the weight terminal of the corresponding multiplication unit; a multiplication unit for performing multiplication operations on the data obtained at the data terminal and the weight parameters obtained at the weight terminal, and outputting the result of the multiplication operation from the output terminal of the multiplication unit; and a third gate for transmitting the result of the multiplication operation output by the multiplication unit from the left or right side of the computation array to the accumulator corresponding to the multiplier where the multiplication unit is located; the controller is configured to control the first multiplexer, the second multiplexer, the first gate, the second gate, and the third gate based on the divided regions so that the data and weight parameters of one region are transmitted without crossing regions.

[0012] In some embodiments of the present invention, the computation array includes m×m multipliers, where m is an integer greater than or equal to 128. In each computation cycle, when the number of channels of the corresponding convolutional layer of the visual task that was previously allocated is less than the value of m×m, the remaining multipliers in the computation array are allocated to the convolutional channels of the corresponding convolutional layers of other visual tasks.

[0013] In some embodiments of the present invention, the accelerator further includes a data conversion unit for converting images received from sensors, cameras, or other different types of image acquisition devices into data formats that conform to the requirements of the input image convolutional neural network.

[0014] According to a second aspect of the present invention, a control method for multi-visual task processing implemented on the multi-visual task accelerator described in the first aspect is provided, comprising: obtaining an accelerated computing request for a visual task, including the number of visual tasks to be executed simultaneously, the number of convolutional channels in each convolutional layer of each visual task, and the priority of each visual task; responding to the accelerated computing request of one or more visual tasks, controlling a computing array to execute a convolution operation corresponding to one visual task, or dividing the computing array into at least two regions to execute convolution operations corresponding to at least two of the multiple visual tasks simultaneously when there are idle computing resources for the computing array to execute a convolution operation corresponding to a single visual task; filling the corresponding region of the computing array with data and weight parameters that participate in filling the computing array once, and using the computing array to execute a convolution operation corresponding to one visual task or to execute convolution operations corresponding to at least two visual tasks simultaneously in parallel, to obtain the convolution sum corresponding to the convolution operation.

[0015] Compared with the prior art, the advantages of the present invention are as follows:

[0016] This invention provides a multi-vision task accelerator that can allocate the hardware resources of a computing array to multiple vision tasks. This not only makes use of idle hardware resources but also improves performance, including throughput. It can enhance the acceleration efficiency of multi-vision tasks and reduce computation latency. Attached Figure Description

[0017] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0018] Figure 1 This is a schematic diagram of the structure of a multi-vision task accelerator according to an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of another multi-vision task accelerator according to an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the structure of a multiplier according to an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram illustrating the process of allocating computing resources according to an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0023] As mentioned in the background section, the size of the computing units required for visual tasks and the actual design size are not always perfectly matched. If there are multiple accelerated computing requests for visual tasks, the accelerator is often queued according to the order of the requests, resulting in typically low hardware utilization and performance. To address the aforementioned problems, this invention proposes a multi-visual task accelerator that can allocate the hardware resources of the computing array to multiple visual tasks. This utilizes idle hardware resources while improving performance, including throughput; it can enhance the acceleration efficiency for multiple visual tasks and reduce computation latency.

[0024] Before describing the embodiments of the present invention in detail, some of the terms used therein are explained as follows:

[0025] Convolutional Neural Networks (CNNs) are feedforward neural networks that incorporate convolutional computations. Each CNN includes at least one (or more) convolutional layers.

[0026] Visual tasks refer to tasks that utilize convolutional neural networks to process input images. For example, tasks such as classifying or segmenting input images. There are many visual tasks in this field, such as those corresponding to existing convolutional neural networks like AlexNet, VGGNet, ResNet, and LeNet, as well as user-defined convolutional neural networks or future convolutional neural networks. This invention does not impose any limitations on these.

[0027] A feature map is an input image or an intermediate image obtained by processing an image using a convolutional neural network. If a convolutional layer of a convolutional neural network directly processes the image input to the network, then the feature map input to that layer is the input image itself; otherwise, the feature map input to that layer is an intermediate image obtained after processing by other layers of the convolutional neural network (such as pooling layers, convolutional layers, etc.), or an intermediate feature map.

[0028] A convolutional channel refers to the channel used in the convolution operation within a convolutional layer. For example, the red, green, and blue channels in an RGB image, or each channel in a feature map (often a multi-channel feature map).

[0029] According to one embodiment of the present invention, see Figure 1 A multi-vision task accelerator includes: a data storage unit 101, a weight storage unit 102, a data buffer unit 103, a weight buffer unit 104, a computing array 201, and a controller 202, or a combination thereof; wherein the accelerator includes at least the computing array 201 and the controller 202. To explain the principle of the invention in detail, the various components constituting the accelerator are described below. Specifically:

[0030] Data storage unit 101 is used to store feature maps that need to be processed by the convolutional layer. These feature maps are either the input image or intermediate maps obtained by image convolutional neural networks. Preferably, the intermediate maps can be obtained by a computational array, or by pooling and / or activation after processing by the computational array. Preferably, data storage unit 101 can be implemented in the accelerator's memory. In other words, data storage unit 101 stores the input data (data contained in the input image) and corresponding intermediate data (data contained in the intermediate maps, or data contained in the intermediate feature maps) for the visual task to be processed; it is responsible for transferring the data required for the next computation to the data buffer unit 103. If there are multiple visual tasks, the feature maps corresponding to the multiple visual tasks are stored in data storage unit 101; for example, assuming there are four visual tasks, the data D1 of the feature map corresponding to the first visual task, the data D2 of the feature map corresponding to the second visual task, the data D3 of the feature map corresponding to the third visual task, and the data D4 of the feature map corresponding to the fourth visual task are stored in data storage unit 101.

[0031] The weight storage unit 102 is used to store the weight parameters of the convolutional layers. Preferably, the weight storage unit 102 can be implemented in the accelerator's memory. If there are multiple vision tasks, the weight parameters of the convolutional layers of the convolutional neural networks corresponding to the multiple vision tasks are stored in the weight storage unit 102; for example, assuming there are four vision tasks, the weight parameters W1 corresponding to the first vision task, W2 corresponding to the second vision task, W3 corresponding to the third vision task, and W4 corresponding to the fourth vision task are stored in the weight storage unit 102. In other words, the weight storage unit 102 is used to store the weight data corresponding to each layer of the vision task to be processed, and is responsible for transferring the weight parameters required for the next calculation to the weight buffer unit 104.

[0032] Data buffer unit 103 is used to fill all the data that the computing array will process next. This data is the data in the feature map that is allocated to the next operation in the computing array. In other words, data buffer unit 103 can store at least all the data needed to fill the multipliers of the computing array at once (provided there is enough data to be processed), for inputting data from data storage unit 101 and transmitting its buffered data to the data path multiplexer. For example, assuming the computing array includes m×m multipliers, data buffer unit 103 can cache at least m×m of data. This allows all the data needed for the next calculation to be quickly obtained after the computing array completes one calculation. It should be understood that the amount of data stored in the data storage unit 101 is greater than that in the data buffer unit 103. However, since the data buffer unit 103 is positioned closer to the computing array, when the computing array is performing a cycle of operation, the data buffer unit 103 can read from the data storage unit 101 and cache the data required for the next cycle of operation. Thus, when the computing array completes the operation of the current cycle, the data required for the next operation can be quickly filled through the data buffer unit 103, thereby improving the overall computing efficiency.

[0033] The weight buffer unit 104 stores all weight parameters that the computation array will use next. These weight parameters are used to multiply the data in the feature map that are assigned to be processed in the computation array next time. In other words, the weight buffer unit 104 can store at least all the weights needed to fill the computation array at once, and is used to input weights from the weight storage unit 102 and transmit its buffered weights to the weight path multiplexer. For example, assuming that the computation array includes m×m multipliers, the weight buffer unit 104 can cache at least m×m weight parameters. This allows all the weight parameters needed for the next computation to be quickly obtained after the computation array has completed one computation. It should be understood that the number of weight parameters stored in the weight storage unit 102 is greater than that in the weight buffer unit 104. However, since the weight buffer unit 104 is positioned closer to the computing array, when the computing array is performing a cycle of computation, the weight buffer unit 104 can read from the weight storage unit 102 and cache the weight parameters required for the next cycle of computation. Thus, when the computing array completes the computation of the current cycle, the weight parameters required for the next cycle can be quickly filled through the weight buffer unit 104, thereby improving the overall computational efficiency.

[0034] The computation array 201 is used to perform convolution operations on the convolutional layers. Preferably, the computation array includes multiple multipliers and multiple accumulators. Each multiplier performs a multiplication operation between the weight parameters and the corresponding data, and each accumulator or multiple multipliers accumulates the results of the corresponding multiplication operations to obtain the convolution sum corresponding to the convolution operation. The accumulators are pipelined accumulators. The accumulators are connected to the data storage unit 101 to transmit intermediate data and store it in the data storage unit 101. For example, the computation array includes m×m multipliers and at least m×m accumulators, where m is a positive integer greater than 2. Since current convolutional neural networks have a large number of channels, to improve computational efficiency, m can be set to an integer greater than 64, such as preferably 128, 256, 512, or 1024.

[0035] Controller 202 is configured to, in response to acceleration requests from one or more visual tasks, control the computing array to execute convolution operations corresponding to a single visual task, or, when idle computing resources exist for executing convolution operations corresponding to a single visual task, divide the computing array into at least two regions to simultaneously execute convolution operations corresponding to at least two of the multiple visual tasks. Idle computing resources are considered to exist only when at least one idle multiplier and at least one idle accumulator are present. Preferably, controller 202 is configured to perform the following operations when there are multiple visual tasks to be accelerated: allocate multipliers and accumulators in the computing array to convolutional layers in multiple visual tasks according to a predetermined priority, wherein each convolutional channel in the convolutional layer is allocated a corresponding multiplier and accumulator; when the corresponding convolutional layer of a previously allocated visual task does not fully utilize all the multipliers in the computing array, dynamically partition the computing array to utilize the multipliers and accumulators in different regions to process the convolution operations corresponding to the convolutional layers of different visual tasks. For example, when there are two accelerated computing requests for visual tasks, assuming they are a high-priority first visual task and a low-priority second visual task, the computing array contains 128x128 multipliers. If, during a certain computing cycle, a certain convolutional layer of the first visual task only occupies 128x48 multipliers, then according to existing technology, the remaining 128x80 multipliers are idle. According to the present invention, the computing array is partitioned. The first region contains 128x48 multipliers to perform multiplication operations on a certain convolutional layer of the first visual task, and also contains at least 128x48 accumulators to accumulate the corresponding multiplication operations to obtain the convolution sum corresponding to the convolution operation. Correspondingly, the second region contains 128x80 multipliers and corresponding accumulators to perform convolution operations on the second visual task, thereby achieving efficient utilization of the computing array. It should be understood that this is for illustrative purposes only. Depending on the number of visual tasks and / or the size of the computation array, the partitioning can create other numbers of regions, such as 3, 4, 5, 6, or even more. Similarly, after the multiplier is divided into multiple regions, each accumulator corresponds to one or more multipliers (this can be determined by the structure of the convolutional layer, i.e., determining which multiplication results need to be accumulated together, which is determined by the structure of the neural network), and the corresponding accumulator will also be divided into multiple regions. See again. Figure 1PE can refer to multipliers and / or accumulators. Of course, PE can also refer to multipliers, and accumulators define the outer regions of these PEs. Furthermore, preferably, the predetermined priority is determined based on the initiation time of the accelerated computation request for the visual task and / or the latency requirements of the visual task. For example, visual tasks with earlier initiation times have higher priorities; or visual tasks with higher latency requirements have higher priorities; or, the priority of each visual task can be determined jointly based on the initiation time and the latency requirements of the visual task. To further simplify control, the controller can be configured such that if the convolutional layer currently involved in the allocation for a visual task has not completed its corresponding computation, subsequent convolutional layers for that visual task will not participate in the allocation. Of course, it should be understood that if, for the purpose of efficient processing of a visual task, the data required for the computation of subsequent convolutional layers is partially completed, subsequent convolutional layers can also participate in the allocation; this invention does not impose any limitations on this. The technical solution of this embodiment can achieve at least the following beneficial technical effects: The embodiment of the present invention allocates a corresponding multiplier and accumulator to each convolutional channel in the convolutional layer, which can efficiently complete the allocation process of the convolutional layer of each visual task during the allocation process, and can make full use of the idle computing resources after the high-priority visual task is allocated computing resources, and complete the efficient processing of multiple visual tasks in parallel, thereby improving computing efficiency.

[0036] To enable the accelerator to directly process raw images, according to one embodiment of the present invention, the accelerator further includes a data conversion unit for converting images received from sensors, cameras, or other different types of image acquisition devices into images conforming to the data format requirements of image convolutional neural networks. It should be understood that the data conversion unit is not mandatory; the functions corresponding to the data conversion unit can be implemented by an external general-purpose computer, which can then input the images conforming to the data format requirements of image convolutional neural networks into the accelerator.

[0037] To enable the accelerator to perform other functions of convolutional neural networks, according to one embodiment of the present invention, the accelerator further includes: a pooling unit for performing pooling processing on the feature map and / or an activation unit for performing activation processing on the feature map. Pooling and activation are common techniques in the art, and correspondingly, the pooling unit and activation unit can adopt existing structures, which are not limited by the present invention.

[0038] To facilitate data flow control and reduce the complexity of partitioned deployment, while also making it easier to distinguish data, weight parameters, and the results of corresponding multiplication operations for different visual tasks, data and weight parameters can be transmitted from different directions. See also Figure 2According to one embodiment of the present invention, based on the foregoing embodiments, the accelerator further includes: a first multiplexer 105 for filling data from a data buffer unit to a corresponding region of the computing array, and a second multiplexer 106 for filling weight parameters from a weight buffer unit to a corresponding region of the computing array. The controller 202 is configured to control the first multiplexer 105 to fill data from the left-right direction of the computing array 201 to the corresponding region, and the second multiplexer 106 to fill weight parameters from the up-down direction of the computing array 201 to the corresponding region. Preferably, the computing array 201 includes m×m multipliers, see [link to previous section]. Figure 3 Each multiplier includes: a first selector for sending data input from the left (corresponding to the second data input interface) or right (corresponding to the first data input interface) of the first multiplexer 105 to the data terminal of the corresponding multiplication unit; a second selector for sending weight parameters input from the top (corresponding to the first weight input interface) or bottom (corresponding to the second weight input interface) of the second multiplexer 106 to the weight terminal of the corresponding multiplication unit; and a multiplication operation for the data obtained at the data terminal and the weight parameters obtained at the weight terminal. The controller 202 is configured to: control a first multiplexer 105, a second multiplexer 106, a first selector, a second selector, and a third selector, based on the divided regions, to ensure that data and weight parameters of a region are not transmitted across regions. Thus, the computing array can be divided into up to four regions, and data and weight parameters of a region are not transmitted through cross-region transmission lines. For example, if the computing array is divided into upper and lower regions, the weight parameters of the upper region flow into the region from the top, and data flows into or out of the region from the left or right; the weight parameters of the lower region flow into the region from the bottom, and data flows into or out of the region from the left or right.

[0039] For a clearer illustration, see [link to relevant documentation]. Figure 3Taking the division of the computing array into four regions as an example, considering the difficulty of multi-task routing and allocation, the weight parameters are only input from the top and bottom of the computing array, and the data is only input and output from the left and right. Therefore, the input and output of the multiplication unit are modified to include a first weight input interface, a second weight input interface, a first data input interface, a second data input interface, a first data output interface, and a second data output interface. The weight parameters corresponding to the first and second weight input interfaces both come from the weight buffer unit 104; the data corresponding to the first and second data input interfaces both come from the data buffer unit 103; the data output by the first and second data output interfaces is transmitted to the data storage unit 101. When the computing array 201 is divided into four regions, the controller controls the first multiplexer, the second multiplexer, the first gate, the second gate, and the third gate to transmit data and weight parameters in the following manner based on the divided regions:

[0040] The data flow in the upper left area is as follows: the input data is sent to the multiplication unit through the first data input interface and the first selector; the weight parameters are sent to the multiplication unit through the first weight input interface and the second selector; and the calculation result (output data) is output from the first data output interface through the third selector.

[0041] The data flow in the lower left area is as follows: data is sent to the multiplication unit through the first data input interface and the first selector; the weight parameters are sent to the multiplication unit through the second weight input interface and the second selector; and the calculation result is output from the first data output interface through the third selector.

[0042] The data flow in the upper right area is as follows: data is sent to the multiplication unit through the second data input interface and the first selector; the weight parameters are sent to the multiplication unit through the first weight input interface and the second selector; and the calculation result is output from the second data output interface through the third selector.

[0043] The data flow in the lower right area is as follows: data is sent to the multiplication unit through the second data input interface and the first selector; the weight parameters are sent to the multiplier through the second weight input interface and the second selector; and the calculation result is output from the second data output interface through the third selector.

[0044] It should be understood that the above embodiments are merely illustrative, and implementers can make corresponding adjustments. For example, the controller can be configured to control the first multiplexer to fill data into the corresponding area from the vertical direction of the computing array, and the second multiplexer to fill weight parameters into the corresponding area from the horizontal direction of the computing array. Alternatively, when the computing array is divided into three areas, other control methods for the transmission directions of corresponding data and weight parameters can be used to achieve data and weight parameters in the corresponding areas without crossing areas. Furthermore, it should be understood that other data and weight parameter transmission methods are also feasible without considering wiring and control difficulties, such as using existing bus transmission methods. Moreover, in the case of only one accelerated computing request for a visual task, the accelerator of this invention can perform accelerated computing for that visual task without partitioning.

[0045] According to one embodiment of the present invention, the present invention also provides a control method for multi-task processing implemented on the multi-visual task accelerator of the foregoing embodiments, comprising: obtaining an accelerated computing request for a visual task, including the number of visual tasks to be executed simultaneously, the number of convolutional channels in each convolutional layer of each visual task, and the priority of each visual task; responding to the accelerated computing request of one or more visual tasks, controlling the computing array to execute a convolutional operation corresponding to one visual task, or dividing the computing array into at least two regions to execute convolutional operations corresponding to at least two of the multiple visual tasks simultaneously when there are idle computing resources in the computing array executing a convolutional operation corresponding to a single visual task; filling the corresponding region of the computing array with data and weight parameters that participate in filling the computing array once, and using the computing array to execute a convolutional operation corresponding to one visual task or to execute convolutional operations corresponding to at least two visual tasks in parallel, to obtain the convolutional sum corresponding to the convolutional operation. Wherein, the priority of the visual tasks can determine the order in which each visual task participates in the allocation when allocating the computing array each time; the number of convolutional channels in each convolutional layer of each visual task can be used to determine the number of multipliers and accumulators required when a single visual task executes the computation of a certain convolutional layer.

[0046] According to one embodiment of the present invention, for ease of understanding, taking the allocation of computing resources involving up to two visual tasks as an example, it is assumed that visual task 1 has a higher priority than visual task 2, and that each convolutional channel is allocated one multiplier. See [link to relevant documentation]. Figure 4 The control method for this multitasking process includes:

[0047] Step 1: Determine the number of visual tasks to be performed at once. If the number of visual tasks to be performed is 1, proceed to step 2.1. If the number of visual tasks to be performed is 2, proceed to step 2.2.

[0048] Step 2.1: Does the number of convolutional channels in the convolutional layer that the visual task currently needs to compute exceed the maximum computational capacity of the computing array? If yes, allocate computing resources to a portion of the convolutional channels in the currently computed convolutional layer, store the data for the remaining convolutional channels, and complete the computation in batches. If no, allocate computing resources to the currently computed convolutional layer. For example: determine whether the number of convolutional channels A in the convolutional layer that the visual task currently needs to compute exceeds the maximum computational capacity B of the computing array. If A exceeds B, store the data and complete the computation in batches.

[0049] Step 2.2: Prioritize allocating computing resources to vision task 1. If there are remaining computing resources after allocation, then allocate computing resources to vision task 2. For example: Determine if the number of convolution channels A of the convolutional layer currently required for vision task 1 is greater than the maximum computational size B of the computing array. If A is greater than B, the remaining convolutional channels are stored and computation is completed in batches. If A equals B, the computing array is currently only used for vision task 1. If A is less than B, A multipliers and the required accumulators are allocated to vision task 1, while the remaining multipliers and accumulators can be used for vision task 2. Similarly, assuming there are more vision tasks, if there are still idle resources in the computing array after this allocation for vision task 2, then the next priority vision task can continue to participate in the allocation.

[0050] According to an example of the present invention, taking the acceleration of the LeNet model on a 128×128 computation array as an example, for ease of understanding, it is assumed that each convolutional channel of the LeNet model is assigned only one multiplier (in practice, the number of multipliers can be set according to the implementer's needs), and the corresponding accumulator is assigned according to the construction of the convolutional kernel to accumulate the results of the corresponding multipliers. Part of the structure of the LeNet model is as follows:

[0051] Layer 1 (convolutional layer): The input is 32x32x1, which is passed through a 5x5x1x6 convolutional kernel with a stride of 1 and no padding. The output is 28x28x6, and the number of cycles required is 19600.

[0052] Layer 2: The input is 28x28x6, pooling is performed, and the output is 14x14x6, regardless of the number of cycles;

[0053] Layer 3 (convolutional layer): The input is 14x14x6, which is processed by a 5x5x6x16 convolutional kernel with a stride of 1 and no padding. The output is 10x10x16, and the number of cycles required is 2500.

[0054] Layer 4: The input is 10x10x16, pooling is performed, and the output is 5x5x16, regardless of the number of cycles;

[0055] Layer 5 (convolutional layer): The input is 5x5x16, which passes through a 5x5x16x120 convolutional kernel with a stride of 1 and no padding. The output is 1x1x120, and the number of cycles required is 25.

[0056] Layer 6 (convolutional layer): The input is 1x1x120, which is passed through a 1x1x120x84 convolutional kernel with a stride of 1 and no padding. The output is 1x1x84, and the number of cycles required is 1.

[0057] Layer 7 (convolutional layer): The input is 1x1x84, which is passed through a 1x1x84x10 convolutional kernel with a stride of 1 and no padding. The output is 1x1x10, and the number of cycles required is 1.

[0058] In simple terms, assuming the next convolutional layer doesn't start calculating while the previous one is being calculated, the total number of cycles required for the above convolutional layer process is 19600 + 2500 + 25 + 1 + 1 = 22127. Using existing technology, if there are N LeNet models corresponding to visual tasks simultaneously, the accelerator would need 22127 × N cycles to complete all computations. It can be seen that no single convolutional layer in a LeNet model fully utilizes the 128 × 128 computational array, while the technology of this invention allows other visual models to utilize the computational array in parallel, significantly reducing the computation time.

[0059] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0060] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0061] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0062] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multi-visual task accelerator, wherein the visual task refers to the task of processing input images using convolutional neural networks, each image convolutional neural network including at least one convolutional layer, characterized in that, The accelerator includes: A computation array for performing convolution operations on a convolutional layer, wherein the computation array includes multiple multipliers and multiple accumulators; A controller for responding to acceleration requests for one or more vision tasks, controlling a computing array to execute convolution operations corresponding to a vision task, or dividing the computing array into at least two regions to simultaneously execute convolution operations corresponding to at least two vision tasks among multiple vision tasks when idle computing resources exist for executing convolution operations corresponding to a single vision task, the controller is configured to perform the following operations when there are multiple vision tasks to be accelerated: allocating multipliers and accumulators in the computing array to convolutional layers in multiple vision tasks according to a predetermined priority, wherein each convolutional channel in the convolutional layer is allocated a corresponding multiplier and accumulator; and dynamically partitioning the computing array to utilize multipliers and accumulators in different regions to process convolution operations corresponding to convolutional layers of different vision tasks when the corresponding convolutional layer of the vision task that was allocated earlier has not fully utilized all the multipliers in the computing array.

2. The multi-vision task accelerator according to claim 1, characterized in that, Each multiplier performs a multiplication operation between the weight parameters and the corresponding data. Each accumulator corresponds to one or more multipliers to accumulate the results of the corresponding multiplication operations to obtain the convolution sum corresponding to the convolution operation.

3. The multi-vision task accelerator according to claim 2, characterized in that, The accelerator also includes: Data storage unit used to store feature maps that need to be processed by convolutional layers, which are either the input image or an intermediate image obtained by image convolutional neural network; Weight storage units used to store the weight parameters of convolutional layers; A data buffer unit is used to store all the data that the computing array will process next, which is the data in the feature map that is allocated to be processed in the computing array next time; A weight buffer unit is used to store all the weight parameters that the computation array will use next time. These weight parameters are used to multiply the data in the feature map that are assigned to be processed in the computation array next time.

4. The multi-vision task accelerator according to claim 2, characterized in that, The predetermined priority is determined based on the initiation time of the accelerated computing request for the visual task and / or the latency requirements of the visual task.

5. The multi-vision task accelerator according to claim 3, characterized in that, The accelerator also includes: A first multiplexer for filling data from the data buffer unit into the corresponding region of the computing array, and A second multiplexer is used to fill the corresponding region of the computation array with weight parameters from the weight buffer unit. The controller is configured to control the first multiplexer to fill data into the corresponding area from the left and right directions of the computing array and the second multiplexer to fill weight parameters into the corresponding area from the top and bottom directions of the computing array, or to control the first multiplexer to fill data into the corresponding area from the top and bottom directions of the computing array and the second multiplexer to fill weight parameters into the corresponding area from the left and right directions of the computing array.

6. The multi-vision task accelerator according to claim 5, characterized in that, Each multiplier in the computation array includes: A first selector used to send data input from the left or right side of the computation array to the data terminal of the corresponding multiplication unit; A second gate used to send the weight parameters input from above or below the computation array to the weight end of the corresponding multiplication unit; A multiplication unit is used to perform multiplication operations on the data obtained from the data end and the weight parameters obtained from the weight end, and outputs the result of the multiplication operation from the output end of the multiplication unit; The third selector is used to transfer the result of the multiplication operation output by the multiplication unit from the left or right side of the computation array to the accumulator corresponding to the multiplier where the multiplication unit is located. The controller is configured as follows: Based on the divided regions, control the first multiplexer, the second multiplexer, the first gate, the second gate, and the third gate so that the data and weight parameters of one region are not transmitted across regions.

7. The multi-vision task accelerator according to claim 5, characterized in that, The computation array consists of m×m multipliers, where m is an integer greater than or equal to 128. In each computation cycle, when the number of channels of the corresponding convolutional layer of the visual task that was previously assigned is less than the value of m×m, the remaining multipliers in the computation array are assigned to the convolutional channels of the corresponding convolutional layers of other visual tasks.

8. The multi-vision task accelerator according to any one of claims 1-7, characterized in that, The accelerator further includes a data conversion unit for converting images received from sensors, cameras, or other different types of image acquisition devices into data formats that conform to the requirements of the input image convolutional neural network.

9. A control method for multi-vision task processing implemented on the multi-vision task accelerator according to any one of claims 1-8, characterized in that, include: Obtain accelerated computation requests for visual tasks, including the number of visual tasks that need to be executed simultaneously, the number of convolutional channels in each convolutional layer of each visual task, and the priority of each visual task. In response to a request for accelerated computation of one or more vision tasks, the computing array is controlled to perform a convolution operation corresponding to a vision task, or when there are idle computing resources available for the computing array to perform a convolution operation corresponding to a single vision task, the computing array is divided into at least two regions to simultaneously perform convolution operations corresponding to at least two of the multiple vision tasks. The data and weight parameters that participate in filling the computation array are filled into the corresponding area of ​​the computation array. The computation array is used to perform a convolution operation corresponding to a visual task or to perform convolution operations corresponding to at least two visual tasks in parallel to obtain the convolution sum corresponding to the convolution operation.

Images