Dynamic convolution calculation method and architecture for hardware acceleration

Abstract

The invention discloses a dynamic convolution calculation method and architecture for hardware acceleration, relates to the technical field of neural network hardware acceleration, and solves the technical problem that the current calculation method and architecture urgently need a hardware architecture capable of flexibly adjusting a convolution kernel, stride and filling parameters. The method comprises the steps of dynamically adjusting convolution calculation parameters in a convolution calculation task according to input data; decomposing the convolution calculation task into a plurality of subtasks according to the availability of hardware resources and the calculation amount of each convolution operation in the convolution calculation task; distributing the plurality of sub-tasks to different computing units according to a priority scheduling strategy; monitoring the load condition of each calculation unit in real time, and carrying out load balancing on each calculation unit; and executing each sub-task after load balancing in parallel through a plurality of calculation units to obtain a calculation result of the convolution calculation task. According to the invention, convolution calculation parameters can be flexibly adjusted, and the calculation efficiency and the resource utilization rate of the NPU are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural network hardware acceleration, and particularly relates to a dynamic convolution calculation method and architecture for hardware acceleration. BACKGROUND

[0002] Convolutional Neural Networks (CNN) is one of the core computing models in deep learning, and is widely used in computer vision, speech recognition and other fields. The computing performance of CNN is highly dependent on convolution calculation, and the efficiency and amount of convolution calculation directly affect the performance of the entire network. Therefore, a hardware accelerator is usually used to improve the efficiency of convolution calculation. Currently, common hardware accelerators such as Neural Processing Unit (NPU) are used to accelerate the convolution calculation of CNN. However, the existing NPU architecture usually uses fixed convolution kernel, stride and padding parameters, which cannot be dynamically adjusted according to the needs of different input data and network levels. This static design often leads to waste of computing resources and reduction of computing efficiency when facing different computing tasks.

[0003] Currently, the processing of different input data is usually completed by fixed parameters for convolution calculation, which makes some convolution kernels in the computing task too large or too small, and the stride and padding may not meet the needs of a specific task, resulting in excessive computation or insufficient precision, affecting the computing efficiency and effective use of hardware resources. Therefore, some hardware acceleration methods use low-precision floating-point numbers to reduce the amount of calculation and storage requirements, thereby improving the computing efficiency. However, the optimization focus of this method is on data precision, not on adaptive adjustment of hardware architecture, and it is still difficult to effectively solve the flexibility and efficiency of dynamic convolution calculation at the hardware level.

[0004] Therefore, some people propose to use multi-dimensional attention mechanism to dynamically adjust the convolution kernel parameters, such as adaptive weight distribution of convolution kernel, but these methods are essentially still software-level optimization, lack the ability of dynamic allocation of hardware computing resources, resulting in large computing overhead, and are difficult to adapt to the high-throughput hardware acceleration requirements.

[0005] In the field of hardware acceleration of Graph Convolutional Network (GCN), the GRIP (Graph-based Image Processing) architecture proposed by someone adopts a fixed edge or vertex center execution mode, combines matrix calculation optimization, and realizes low-latency inference 17-23 times faster than CPU or GPU. However, the architecture still has problems such as insufficient sparse calculation optimization, limited support for dynamic graphs and heterogeneous graphs, high storage overhead, and uneven memory access, which limit its applicability in complex GNN tasks.

[0006] In the process of implementing the present application, the inventors found that at least the following problems exist in the prior art: There is an urgent need for a hardware architecture that can flexibly adjust the convolution kernel, stride and padding parameters to improve the computing efficiency and resource utilization of the NPU. SUMMARY

[0007] The present application aims to provide a dynamic convolution calculation method and architecture for hardware acceleration to solve the technical problem in the prior art that there is an urgent need for a hardware architecture that can flexibly adjust the convolution kernel, stride and padding parameters to improve the computing efficiency and resource utilization of the NPU.

[0008] The preferred technical solutions in the many technical solutions provided by the present application can produce the many technical effects detailed below.

[0009] To achieve the above-mentioned purpose, the present application provides the following technical solutions: The dynamic convolution calculation method for hardware acceleration provided by the present application comprises the following steps: dynamically adjusting the convolution calculation parameters in the convolution calculation task according to the input data, and outputting the dynamically adjusted convolution calculation task; the convolution calculation parameters include the size of the convolution kernel, the stride and the padding amount; according to the availability of hardware resources and the calculation amount of each convolution operation in the convolution calculation task, the convolution calculation task is decomposed into multiple subtasks; the hardware resources include multiple different computing units; multiple subtasks are assigned to different computing units according to a priority scheduling strategy; the load situation of each computing unit is monitored in real time, and each computing unit is load balanced; the subtasks after load balancing are executed in parallel by multiple computing units to obtain the calculation result of the convolution calculation task.

[0010] Optionally, the dynamically adjusting the convolution calculation parameter in the convolution calculation task according to the input data and outputting the dynamically adjusted convolution calculation task comprises: dynamically adjusting the size of the convolution kernel according to the size of the input data, the output feature map and the stride and the padding amount; dynamically adjusting the stride according to the size of the input data, the output feature map and the size of the convolution kernel and the padding amount; dynamically adjusting the padding amount according to the size of the convolution kernel and the stride; and outputting the dynamically adjusted convolution calculation task after adjusting the size of the convolution kernel, the stride and the padding amount.

[0011] Optionally, the formula for dynamically adjusting the size of the convolution kernel is: ; the formula for dynamically adjusting the stride is: ; and the formula for dynamically adjusting the padding amount is: ; wherein K is the size of the convolution kernel; O is the size of the output feature map of the input data; S is the stride; I is the size of the input data; and P is the padding amount.

[0012] Optionally, the dividing the convolution calculation task into a plurality of subtasks according to the calculation amount of each convolution operation in the convolution calculation task and the availability of hardware resources comprises: calculating the calculation amount of each convolution operation to obtain the load of the convolution calculation task; and dividing the convolution calculation task into a plurality of subtasks according to the load of the convolution calculation task and the availability of hardware resources.

[0013] Optionally, the allocating a plurality of subtasks to different computing units according to a priority scheduling strategy comprises: allocating the subtask with high calculation complexity to the computing unit with abundant computing resources; and allocating the subtask with low calculation complexity to the computing unit with few computing resources.

[0014] Optionally, the real-time monitoring of the load condition of each computing unit and the load balancing of each computing unit comprises: real-time calculation of the load of each computing unit; and real-time adjustment of the subtask allocated to each computing unit according to the load of the computing unit, so as to equalize the load of each computing unit.

[0015] Optionally, the real-time adjustment of the subtask allocated to each computing unit is achieved by adjusting the task scheduling time. wherein, is the number of computing units; is the calculation complexity of the convolution calculation task; is the calculation capability of one computing unit.​​

[0016] An architecture comprises: An adaptive convolution module is configured to dynamically adjust convolution calculation parameters in a convolution calculation task according to input data, and output the dynamically adjusted convolution calculation task; the convolution calculation parameters include the size of a convolution kernel, a stride, and a padding amount; A task decomposition module is configured to decompose the convolution calculation task into a plurality of subtasks according to the availability of hardware resources and the calculation amount of each convolution operation in the convolution calculation task; the hardware resources include a plurality of different calculation units; A task scheduling module is configured to assign the plurality of subtasks to different calculation units according to a priority scheduling strategy; A load balancing module is configured to monitor the load of each calculation unit in real time, and perform load balancing on each calculation unit; A multi-level parallel calculation module is configured to perform each of the load-balanced subtasks in parallel by the plurality of calculation units, to obtain the calculation result of the convolution calculation task.

[0017] Optionally, the adaptive convolution module comprises: A convolution kernel size control unit is configured to dynamically adjust the size of the convolution kernel according to the size of the input data, an output feature map, and the stride and the padding amount; A stride control unit is configured to dynamically adjust the stride according to the size of the input data, the output feature map, and the size of the convolution kernel and the padding amount; A padding control unit is configured to dynamically adjust the padding amount according to the size of the convolution kernel and the stride; A task output unit is configured to output the dynamically adjusted convolution calculation task after adjusting the size of the convolution kernel, the stride, and the padding amount.

[0018] Optionally, the multi-level parallel calculation module adopts a shared memory mechanism to enable the plurality of calculation units to execute the subtasks in parallel, and share data core calculation results through a high-speed communication interface.

[0019] Implementing one of the technical solutions of the present application has the following advantages or beneficial effects: The dynamic convolution calculation method for hardware acceleration provided by the present application dynamically adjusts the convolution kernel, the stride, and the padding amount during convolution calculation, optimizes the efficiency of convolution calculation and the utilization rate of hardware resources, optimizes the performance of convolution calculation through task decomposition, priority scheduling strategy, load balancing mechanism, and the like, ensures efficient utilization and efficient execution of each calculation unit, and finally greatly improves the calculation throughput and reduces the execution time of tasks through multi-level parallel design. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art. The drawings are as follows: Figure 1 is a flow chart of a dynamic convolution calculation method for hardware acceleration according to an embodiment of the present application; Figure 2 is a schematic diagram of a multi-level parallel architecture according to an embodiment of the present application; Figure 3 is a structural block diagram of an architecture according to an embodiment of the present application; Figure 4 is a structural block diagram of an adaptive convolution module according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the objects, technical solutions and advantages of the present application clearer, the various exemplary embodiments to be described below will be described with reference to the corresponding drawings, which constitute a part of the exemplary embodiments and describe various exemplary embodiments that can be used to implement the present application. The same numerals in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the present disclosure. It should be understood that they are only examples of processes, methods and apparatuses, etc. consistent with some aspects of the present disclosure as described in the appended claims, and other embodiments can also be used, or modifications can be made to the embodiments listed herein in structure and function without departing from the scope and spirit of the present application.

[0022] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse" and the like indicate the orientation or positional relationship shown in the drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the elements referred to must have a specific orientation, be constructed and operated in a specific orientation. The terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "a plurality of" means two or more. The terms "connected", "connected" should be broadly understood, for example, it can be fixed connection, detachable connection, integral connection, mechanical connection, electrical connection, communication connection, direct connection, indirect connection through intermediate medium, internal communication of two elements or interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0023] In order to illustrate the technical solutions of the present application, the following will be described by specific examples, only showing the part related to the embodiment of the present application.

[0024] Example one: As Figure 1 shown, the present application provides a dynamic convolution calculation method for hardware acceleration, including the following steps: S1, dynamically adjusting the convolution calculation parameters in the convolution calculation task according to the input data, and outputting the dynamically adjusted convolution calculation task; the convolution calculation parameters include the size of the convolution kernel, the stride and the padding amount; S2, according to the availability of hardware resources and the calculation amount of each convolution operation in the convolution calculation task, the convolution calculation task is decomposed into multiple subtasks; the hardware resources include multiple different computing units; S3, according to the priority scheduling strategy, the multiple subtasks are allocated to different computing units; S4, real-time monitoring the load condition of each computing unit, and load balancing for each computing unit; S5, performing each load balanced subtask in parallel through multiple computing units to obtain the calculation result of the convolution calculation task.

[0025] The dynamic convolution calculation method for hardware acceleration provided by the embodiment optimizes the efficiency of convolution calculation and the utilization rate of hardware resources by dynamically adjusting the parameters such as convolution kernel, stride and padding during convolution calculation; then through task decomposition, priority scheduling strategy, load balancing mechanism, the performance of convolution calculation is optimized, the efficient use and efficient execution of each computing unit are guaranteed; finally, through the design of multi-level parallel, the calculation throughput is greatly improved, and the execution time of the task is reduced.

[0026] Next, combined with Figure 1, to introduce the specific implementation steps of the dynamic convolution calculation method for hardware acceleration provided in this embodiment in detail. First, step S1 is performed, and the convolution calculation parameters in the convolution calculation task are dynamically adjusted according to the input data, and the dynamically adjusted convolution calculation task is output; the convolution calculation parameters include the size of the convolution kernel, the stride kernel padding amount. In the convolution calculation task, the input data (such as an image) is convolved with the convolution kernel to obtain a feature map. This embodiment changes the traditional convolution calculation which only uses a fixed set of convolution calculation parameters, and instead changes these parameters in real time according to the specific characteristics and needs of the input data, to adapt to different neural networks, especially the hardware acceleration process of the graph convolution neural network.

[0027] The size of the convolution kernel refers to the size of the convolution kernel, which is usually represented by a two-dimensional matrix, such as 3x3, 5x5, etc. Different convolution kernel sizes have different receptive fields for input data. A larger convolution kernel can capture more extensive feature information, but the amount of calculation will also be larger; a smaller convolution kernel can focus on more local features, and the amount of calculation is relatively small. The stride refers to the step size of the convolution kernel moving on the input data. The larger the stride, the more elements the convolution operation skips, and the smaller the size of the output feature map; the smaller the stride, the relatively larger the size of the output feature map, which can retain more detailed information. The padding amount is to add additional pixel values (usually 0) to the edge of the input data, the purpose is to control the size of the output feature map, to avoid the loss of edge information in the convolution process. The larger the padding amount, the larger the size of the output feature map, which can retain more edge features.

[0028] Specifically, step S1 includes: dynamically adjusting the size of the convolution kernel according to the size of the input data, the output feature map, and the stride and padding amount; dynamically adjusting the stride according to the size of the input data, the output feature map, and the size of the convolution kernel and the padding amount; dynamically adjusting the padding amount according to the size of the convolution kernel and the stride; after adjusting the size of the convolution kernel, the stride and the padding amount, output the dynamically adjusted convolution calculation task. By dynamically adjusting the size of the convolution kernel, the stride and the padding amount, the flexibility and efficiency of hardware acceleration are improved.

[0029] Further, the formula for dynamically adjusting the size of the convolution kernel is: The formula for dynamically adjusting the stride is: The formula for dynamically adjusting the padding amount is: ; wherein K is the size of the convolution kernel; O is the size of the output feature map of the input data; S is the stride; I is the size of the input data; P is the padding amount.

[0030] The following describes step S1 in detail: the size of the convolution kernel directly determines the complexity of the convolution calculation. In order to optimize the convolution operation, the size of the convolution kernel is dynamically selected in the pipeline control according to the size of the input data and the calculation requirements of the convolution layer, so as to reasonably control the calculation amount.

[0031] Suppose the size of the input data is , the size of the convolution kernel is , the stride is , the padding amount is , and the size of the output feature map is , the calculation amount FLOPs (floating point operation) of the convolution operation is proportional to the size of the convolution kernel . The calculation amount formula is: ; The calculation formula of the size of the output feature map is: ; According to the calculation formula of the size of the output feature map , the formula of the size of the convolution kernel K is: ; Therefore, according to the size of the input data, the size of the output feature map, the stride and the padding amount, the optimal convolution kernel size can be calculated to ensure that the calculation amount of the convolution operation is moderate and meets the specific accuracy requirement. In particular, when the input data size is large, a larger convolution kernel can be selected to speed up the convolution calculation; when the input data size is small, a smaller convolution kernel can be selected to reduce the calculation redundancy.

[0032] The stride determines the step size of the convolution kernel sliding on the input data, and the selection of the stride directly affects the size of the output feature map and the calculation amount. If the stride is too large, the features may be lost, and if the stride is too small, the calculation amount will increase. In order to achieve efficient calculation, the method dynamically adjusts the stride according to the size of the input data, the size of the convolution kernel and the complexity of the calculation task.

[0033] In order to find a balance between calculation amount and calculation accuracy, the size of the stride is dynamically adjusted according to the size requirement of the output feature map. For example, when the input data is large, the stride can be appropriately increased to reduce the redundancy of the convolution calculation; when the input data is small, the stride can be reduced to improve the accuracy of the convolution operation. By dynamically adjusting the size of the stride , the calculation efficiency is maximized while ensuring the accuracy and integrity of the convolution result. According to the size of the output feature map The formula of stride S can also be derived as: ; Through this formula, the stride can change with the dynamic changes of the input size , the size of the convolution kernel , the padding amount , and the size of the output feature map .

[0034] The padding amount P is another important parameter in the convolution operation, which is used to supplement the boundary part of the input data, ensuring that the convolution operation can cover the entire input image, especially the edge part of the image. A suitable padding amount can avoid calculation errors or data loss caused by incomplete application of the convolution kernel to the edge. The calculation formula of the padding amount is: This formula shows that the padding amount is dynamically adjusted according to the size of the convolution kernel , the stride , and the data boundary. In this way, the padding amount can be adaptively adjusted according to the characteristics of the input data and the configuration of the convolution kernel, ensuring the integrity and accuracy of each convolution operation.

[0035] According to the requirements of input data and computing tasks, dynamically adjusting the size of the convolution kernel, stride, and padding amount, etc. parameters can ensure that each link of the convolution operation can be efficiently and accurately executed.

[0036] Then, step S2 is performed, and the convolution calculation task is decomposed into a plurality of sub-tasks according to the calculation amount of each convolution operation in the convolution calculation task and the availability of hardware resources. The hardware resources include a plurality of different computing units. In the fields of deep learning and computer vision, convolution calculation is a very common and computationally intensive operation. Each convolution operation has its specific calculation amount, and the size of the calculation amount depends on factors such as the size of the convolution kernel, the size of the input data, and the like. Hardware resources refer to physical devices used to perform calculation tasks, and in this embodiment, specifically include a plurality of different computing units (Processing Element, PE for short), which may differ in computing power, processing speed, energy consumption, and the like, such as different types of computing units such as CPU, GPU, TPU, and the like. According to the calculation amount of each convolution operation and the availability of the computing units, the original convolution calculation task is split into a plurality of sub-tasks, which can more reasonably allocate the calculation tasks to different computing units, fully utilize the advantages of hardware resources, improve the calculation efficiency, and reduce the calculation time and energy consumption, and the like. For example, convolution operation sub-tasks with small calculation amount and simple logic can be allocated to CPU for processing; and convolution operation sub-tasks that are computationally intensive and require a large amount of parallel calculation can be allocated to computing units such as GPU or TPU that are better at parallel calculation for processing.

[0037] Specifically, step S2 includes: calculating the calculation amount of each convolution operation to obtain the load of the convolution calculation task; and decomposing the convolution calculation task into a plurality of sub-tasks according to the load of the convolution calculation task and the availability of the hardware resources. In deep learning or other computing scenarios involving convolution operations, a large task can generally be split into a plurality of sub-tasks, each containing a number of convolution operations. By calculating the calculation amount required by the convolution calculation task, the calculation amount of all convolution operations in the convolution calculation task is accumulated to obtain the load of the convolution calculation task. The load refers to a measure of the calculation resources (such as calculation time, computing power, etc.) consumed to complete a convolution calculation task. After obtaining the load of the convolution calculation task, the availability of the hardware resources in the system also needs to be considered. According to the load of the convolution calculation task and the resource usage (i.e., resource availability) of each computing unit, the convolution calculation task is decomposed into a plurality of sub-tasks. For example, if there are N available computing units, the convolution calculation task is decomposed into N sub-tasks, and the load of the convolution calculation task is divided by N to obtain the calculation amount required for each sub-task; or each sub-task is allocated a corresponding calculation amount according to the computing power, processing speed, energy consumption, and the like of each computing unit.

[0038] Next, step S3 is performed, in which the plurality of sub-tasks are assigned to different computing units according to a priority scheduling strategy. According to a pre-set priority rule, the plurality of sub-tasks are reasonably assigned to different computing units for processing, and sub-tasks with high priority are processed first, so that the entire system can operate efficiently and orderly.

[0039] Specifically, step S3 includes: assigning sub-tasks with high computing complexity to computing units with abundant computing resources; and assigning sub-tasks with low computing complexity to computing units with less computing resources. According to the complexity of the tasks and the load of the computing units, sub-tasks with high priority (i.e., high computing complexity) are assigned to computing units with abundant computing resources, and sub-tasks with low priority (i.e., low computing complexity) are assigned to computing units with less computing resources, so as to ensure that each computing unit can efficiently execute its tasks.

[0040] Then, step S4 is performed, in which the load of each computing unit is monitored in real time, and load balancing is performed on each computing unit. Through load balancing of the computing units, each computing unit can be dynamically assigned computing tasks to avoid overloading or idling of the computing units.

[0041] Specifically, step S4 includes: calculating the load of each computing unit in real time, and adjusting the sub-tasks assigned to each computing unit in real time according to the load of the computing unit, so that the load of each computing unit is equal. In this way, the computing load of each computing unit can be balanced as much as possible to prevent some computing units from being overloaded while other computing units are idle.

[0042] Specifically, the sub-tasks assigned to each computing unit are adjusted in real time through a task scheduling time, and the task scheduling time is calculated according to the following formula: : wherein, is the number of computing units; is the computing complexity of the convolution calculation task; is the computing capacity of one computing unit.

[0043] In order to achieve load balancing, the load of each computing unit needs to be calculated and the task assignment needs to be adjusted in real time. Assuming that the load of each computing unit is , the goal of load balancing is to ensure that the load of all computing units is close to equal, i.e., ; Through the formula, the task scheduling module can dynamically adjust the task assignment according to the load of the computing units, so as to ensure that the computing tasks can be executed efficiently.

[0044] Finally, step S5 is performed, and each subtask after load balancing is executed in parallel by multiple computing units to obtain the calculation result of the convolution calculation task. After task decomposition, multiple computing units execute each subtask in parallel, which can greatly improve the calculation throughput.

[0045] As shown in Figure 2 The embodiment optimizes the efficiency and performance of convolution calculation through mechanisms such as task decomposition, dynamic task scheduling, and load balancing. Among them, the convolution task is decomposed into multiple subtasks according to the hardware resources, task complexity, and input size, and parallel calculation is performed; task scheduling dynamically adjusts task allocation according to the complexity of the calculation task and the load of the computing unit, ensuring efficient use of computing resources; the load balancing mechanism ensures that the computing load of all computing units is balanced, avoiding waste of hardware resources. Through these designs, the NPU can achieve efficient calculation and resource utilization, thereby improving the overall performance and efficiency in the neural network inference process.

[0046] The embodiment method greatly improves the efficiency of convolution calculation by dynamically adjusting the convolution kernel, stride, and padding amount, especially in the acceleration calculation of graph convolutional neural network (GCN). Through the cooperative work of the multi-level parallel computing mechanism and the task scheduling mechanism, the NPU can achieve efficient calculation task allocation and load balancing, thereby improving the throughput and calculation efficiency of the entire calculation process. For example, when performing convolution operation on image data, the calculation speed is increased by about 40% compared to the processing method of the traditional NPU, and the utilization rate of hardware resources is increased by about 30%. In the graph convolutional neural network (GCN) calculation task, the embodiment can automatically adjust the convolution kernel, stride, and padding amount according to the network level requirements, significantly improving the calculation performance and resource utilization efficiency.

[0047] The embodiment is only one specific example and does not mean that the present application is in this way.

[0048] After each embodiment, the technical effects achieved by the embodiment are described.

[0049] Embodiment two: As shown in Figure 3 The present application also provides an architecture, which includes: An adaptive convolution module is configured to dynamically adjust convolution calculation parameters in a convolution calculation task according to input data and output the dynamically adjusted convolution calculation task; the convolution calculation parameters include the size of the convolution kernel, the stride kernel, and the padding amount; unlike the traditional NPU architecture with static parameter setting, the adaptive convolution module can adjust the convolution kernel, stride, and padding amount in real time, adapt to the needs of different calculation tasks, and improve the calculation efficiency and throughput.

[0050] The task decomposition module decomposes the convolution calculation task into multiple sub-tasks according to the availability of hardware resources and the calculation amount of each convolution operation in the convolution calculation task. The hardware resources include multiple different calculation units. The task decomposition not only depends on the size of the input data, but also considers the calculation complexity of the task and the feature map size of the target output. To this end, the task decomposition module automatically calculates the appropriate sub-task size according to the size of the convolution kernel and the size of the input data. The calculation amount of each sub-task is balanced allocated to the calculation units, thereby ensuring efficient use of hardware.

[0051] The task scheduling module allocates multiple sub-tasks to different calculation units according to a priority scheduling strategy. The task scheduling module can allocate tasks with higher priority to calculation units with more calculation resources, and allocate tasks with lower priority to calculation units with less calculation resources, to ensure that each calculation unit can efficiently execute its task.

[0052] The load balancing module monitors the load of each calculation unit in real time and balances the load of each calculation unit. The goal of the load balancing module is to dynamically adjust the sub-tasks allocated to each calculation unit according to the load of the calculation unit, to ensure that each calculation unit can efficiently execute its task. The task scheduling module needs to monitor the calculation load of each calculation unit in real time and dynamically allocate calculation tasks, to avoid overloading or idling of the calculation unit.

[0053] The multi-level parallel calculation module is used to execute each load-balanced sub-task in parallel through multiple calculation units to obtain the calculation result of the convolution calculation task. Through the multi-level parallel calculation module, the calculation throughput can be greatly improved.

[0054] As an optional implementation, the adaptive convolution module includes: a convolution kernel size control unit for dynamically adjusting the size of the convolution kernel according to the size of the input data, the output feature map, and the stride and padding amount; a stride control unit for dynamically adjusting the stride according to the size of the input data, the output feature map, and the size of the convolution kernel and the padding amount; a padding control unit for dynamically adjusting the padding amount according to the size of the convolution kernel and the stride; and a task output unit for outputting the dynamically adjusted convolution calculation task after adjusting the size of the convolution kernel, the stride, and the padding amount. Figure 4As shown, the convolution adaptive module (APE) integrates three core control units: the convolution kernel size control unit (KSCU), the stride control unit (SCU), and the padding control unit (PCU), each of which is responsible for the adaptive adjustment of convolution operation, stride, and padding amount. These units dynamically calculate through hardware, avoiding the limitations of constant parameters in traditional static convolution design, greatly improving computational efficiency and hardware resource utilization. Each module controls and adjusts specific convolution parameters through hardware circuit, enabling the hardware to adapt to the needs of different computing tasks.

[0055] Through hardware cooperation, the parameters of convolution kernel, stride, and padding are dynamically adjusted according to the input data and the requirements of the computing task. The entire process is scheduled by the control unit to ensure that each link of the convolution operation is efficiently and accurately executed.

[0056] The three core units (convolution kernel size control unit, stride control unit, and padding control unit) in the architecture provided by the embodiment work together through hardware cooperation to optimize the execution process of convolution operation. The control unit, as the core scheduling module, is responsible for coordinating the cooperation between the convolution kernel size control unit, the stride control unit, and the padding control unit, ensuring efficient and accurate execution of convolution operation at each stage. When input data is transmitted into the architecture, the size and characteristics of the input data are first analyzed, and the task requirements are combined to start the workflow of the convolution kernel size control unit, the stride control unit, and the padding control unit. The convolution kernel size control unit calculates the most suitable convolution kernel size based on the input data and task requirements, ensuring the balance between the computational load and feature extraction capability of the convolution operation. At the same time, the stride control unit dynamically adjusts the stride size based on the size of the input data and the computing requirements to ensure the accuracy and efficiency of the convolution operation. The padding control unit calculates and adjusts the appropriate padding amount based on the output of the convolution kernel size control unit and the stride control unit to avoid edge effects and ensure that the convolution kernel can completely cover each part of the input data. During the entire process, the three control units transmit calculation results in real time through hardware buses, ensuring that KSCU, SCU, and PCU handle their respective tasks at the same time, but their results will be accurately coordinated at the hardware level to ensure efficient execution of the convolution operation. The unified scheduling of the control unit through real-time feedback and parameter adjustment ensures that the convolution kernel, stride, and padding amount are always in the optimal configuration throughout the convolution calculation process, thereby avoiding redundant calculations and maximizing computational efficiency. This hardware cooperation mode enables the APE (Application Programming Environment) to flexibly respond to the needs of different input data and computing tasks, optimizes the utilization of hardware resources, and ensures the efficiency and accuracy of the entire convolution operation.

[0057] In this embodiment, the architecture employs an adaptive convolution module, which includes three control modules: a convolution kernel size control unit (KSCU), a stride control unit (SCU), and a padding control unit (PCU). These control modules dynamically adjust the convolution computation process through hardware circuitry. Assuming that the input data is a 256x256 image, the convolution kernel size is 3x3, the stride is 1, and the padding is 1. Through the control units of the APE, the system will dynamically adjust these parameters to adapt to the needs of different convolution layers. When performing convolution operations, the convolution kernel size control unit automatically adjusts the size of the convolution kernel based on the size of the input data and the specific needs of the convolution layer. If the size of the input image is small (e.g., 64x64), the convolution kernel size control unit will select a smaller convolution kernel (e.g., 3x3 or 5x5) to avoid redundant calculations; for larger input images (e.g., 512x512), the convolution kernel size control unit may select a larger convolution kernel (e.g., 7x7) to improve calculation speed and convolution feature extraction capability. Assuming that the target output feature map size is 128x128, the stride control unit dynamically calculates the stride size based on the size of the input data and the convolution kernel size to ensure that the output image size meets the target requirements. The stride control unit will calculate the most suitable stride value according to the formula, thereby avoiding redundant calculations and optimizing convolution operations. The padding control unit dynamically calculates and adjusts the padding based on the size of the convolution kernel, the stride, and the size of the input data. When the stride is 1, the convolution kernel size is 5x5, and the size of the input data is 256x256, the padding control unit will calculate the padding as 2 according to the formula, ensuring that the convolution kernel can completely cover the boundary of the input data and avoid edge effects.

[0058] As an alternative implementation, the multi-level parallel computing module employs a shared memory mechanism to enable multiple computing units to perform sub-tasks in parallel and share data kernel computation results through a high-speed communication interface. The multi-level parallel computing module enables each computing unit to independently perform convolution tasks and share data and computation results through a high-speed communication interface, thereby achieving collaborative work. Each computing unit processes a sub-task, but different computing units need to share computation results and input data. To this end, the architecture employs a shared memory mechanism, and all computing units can access the global memory to quickly exchange data. This memory sharing mechanism optimizes data access through caching to reduce memory access bottlenecks. Through the collaborative work of the hardware scheduling unit, the computing units can efficiently perform convolution calculations, thereby greatly improving the computation throughput and reducing the execution time of the tasks.

[0059] The embodiment adopts multi-level parallel computing modules to improve the computing throughput and resource utilization of the NPU. In specific implementation, the convolution task is decomposed into multiple subtasks and allocated to multiple parallel computing units. These computing units work cooperatively through shared memory and cache mechanism to accelerate the convolution calculation process. When performing convolution calculation, the task decomposition module decomposes the convolution operation into multiple subtasks and allocates these subtasks to different computing units. Assuming that the target of convolution calculation is to generate a 128x128 output feature map, and the calculation amount (FLOPs) of each subtask is 1000, the task decomposition module balances the calculation tasks by calculating the load of each subtask and allocating the calculation tasks to multiple computing units. The task scheduling module dynamically adjusts the allocation of tasks according to the calculation capacity and load of each computing unit. For example, in the processing of high-priority tasks, the scheduling module preferentially allocates resource-rich computing units for calculation, and at the same time uses a load balancing model to ensure that the load of each computing unit is as balanced as possible, avoiding excessive concentration or idling of resources. Multiple computing units execute subtasks in parallel, and exchange data through shared memory mechanism and cache. In the execution process, all computing units share global memory and perform data synchronization and load balancing through a hardware scheduling unit to realize cooperative work. Assuming that there are 16 PEs, each PE is responsible for processing different calculation subtasks, and multiple PEs exchange data quickly through shared memory and cache to maximize the computing throughput.

[0060] The above only describes the preferred embodiments of the present application, and those skilled in the art know that various changes or equivalent replacements can be made to the features and embodiments without departing from the spirit and scope of the present application. In addition, the features and embodiments can be modified to adapt to specific conditions and materials under the guidance of the present application without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the protection scope of the present application.

Claims

1. A dynamic convolution calculation method for hardware acceleration, characterized in that: The following steps are involved: Dynamically adjust the convolution calculation parameters in the convolution calculation task according to the input data, and output the dynamically adjusted convolution calculation task; the convolution calculation parameters include the size, stride and padding of the convolution kernel; Decomposing the convolution computation task into a plurality of subtasks according to the availability of hardware resources and the computational effort of each convolution operation in the convolution computation task; the hardware resources include a plurality of different computational units; Allocating the plurality of subtasks to different computing units according to a priority scheduling strategy; Monitor the load of each computing unit in real time and perform load balancing on each computing unit; The calculation results of the convolution calculation task are obtained by executing the subtasks after load balancing in parallel through multiple computing units.

2. The dynamic convolution calculation method for hardware acceleration according to claim 1, characterized in that: The dynamically adjusting the convolution calculation parameters in the convolution calculation task according to the input data and outputting the dynamically adjusted convolution calculation task includes: Dynamically adjust the size of the convolution kernel according to the size of the input data, the output feature map, the stride, and the padding; Dynamically adjust the stride according to the size of the input data, the output feature map, the size of the convolution kernel, and the padding amount; Dynamically adjusting the padding amount according to the size of the convolution kernel and the stride; After adjusting the size, stride, and padding of the convolution kernel, the dynamically adjusted convolution calculation task is output.

3. The dynamic convolution calculation method for hardware acceleration according to claim 2, characterized in that: The formula for dynamically adjusting the size of the convolution kernel is: ; The formula for dynamically adjusting the stride is: ; The formula for dynamically adjusting the filling amount is: ; Wherein, K is the size of the convolution kernel; O is the size of the output feature map of the input data; S is the stride; I is the size of the input data; and P is the padding amount.

4. The dynamic convolution calculation method for hardware acceleration according to claim 1, characterized in that: The convolution calculation task is decomposed into a plurality of subtasks according to the computational amount of each convolution operation in the convolution calculation task and the availability of hardware resources, including: Calculating the computational effort of each convolution operation to obtain the load of the convolution computation task; Decomposing the convolution computing task into a plurality of subtasks according to the load of the convolution computing task and the availability of the hardware resources.

5. The dynamic convolution calculation method for hardware acceleration according to claim 1, characterized in that: Allocating the plurality of subtasks to different computing units according to a priority scheduling strategy includes: The subtasks with high computational complexity are assigned to the computing units with more computational resources; and the subtasks with low computational complexity are assigned to the computing units with fewer computational resources.

6. The dynamic convolution calculation method for hardware acceleration according to claim 1, characterized in that: The real-time monitoring of the load of each computing unit and load balancing of each computing unit include: The load of each computing unit is calculated in real time, and the subtasks allocated to each computing unit are adjusted in real time according to the load of the computing unit, so that the load of each computing unit is equal.

7. The dynamic convolution calculation method for hardware acceleration according to claim 6, characterized in that: The subtasks assigned to each computing unit are adjusted in real time by task scheduling time. The formula is: ： in, is the number of the computing units; is the computational complexity of the convolution calculation task; is the computing capability of one of the computing units.

8. An architecture characterized by: include: An adaptive convolution module is used to dynamically adjust the convolution calculation parameters in the convolution calculation task according to the input data, and output the dynamically adjusted convolution calculation task; the convolution calculation parameters include the size of the convolution kernel, the stride and the padding of the kernel; a task decomposition module, configured to decompose the convolution computing task into a plurality of subtasks based on the availability of hardware resources and the computational effort of each convolution operation in the convolution computing task; the hardware resources include a plurality of different computing units; A task scheduling module, configured to assign the plurality of subtasks to different computing units according to a priority scheduling strategy; A load balancing module, configured to monitor the load of each computing unit in real time and perform load balancing on each computing unit; A multi-level parallel computing module is used to execute each of the load-balanced subtasks in parallel through multiple computing units to obtain the calculation results of the convolution calculation task.

9. The architecture according to claim 8, characterized in that The adaptive convolution module includes: A convolution kernel size control unit, configured to dynamically adjust the size of the convolution kernel according to the size of the input data, the output feature map, the stride, and the padding; A stride control unit, configured to dynamically adjust the stride according to the size of the input data, the output feature map, the size of the convolution kernel, and the padding amount; A padding control unit, configured to dynamically adjust the padding amount according to the size of the convolution kernel and the stride; The task output unit is used to adjust the size, stride and padding of the convolution kernel and output the dynamically adjusted convolution calculation task.

10. The architecture according to claim 8, wherein: The multi-level parallel computing module adopts a shared memory mechanism to enable multiple computing units to execute the subtasks in parallel and share data core computing results through a high-speed communication interface.

Images