CNN acceleration framework and CNN acceleration method

Abstract

The present application discloses a CNN acceleration framework and a CNN acceleration method. The CNN acceleration method is applied to the CNN acceleration framework, and includes: the processing system sends the first input data to the intermediate mapping layer through the framework layer, and stores the first input data to the memory through the intermediate mapping layer; the intermediate mapping layer of the processing system sends the first storage address of the first input data, the second storage address of the parameter information, and the execution command of at least one operator for executing the CNN model to the programmable logic, wherein the parameter information corresponds to the at least one operator; the programmable logic obtains the first input data from the first storage address and obtains the parameter information from the second storage address; and the programmable logic performs CNN-related operations on the first input data through at least one operator according to the execution command.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a CNN acceleration framework and a CNN acceleration method. Background Art

[0002] In recent years, with the rapid development of the Internet of Things (IoT) and mobile computing, edge computing has attracted widespread attention as an emerging technology. Especially in areas such as visual recognition, convolutional neural networks (CNNs) have become one of the mainstream choices due to their excellent performance. However, the demand for high-performance computing is in sharp contrast to limited hardware resources and strict energy consumption constraints. In this context, programmable logic, with its unique reconfigurable characteristics and excellent energy efficiency, has gradually been regarded as an ideal platform for accelerating CNN reasoning at the edge.

[0003] Among them, programmable logic acceleration CNN has the following advantages:

[0004] (1) Customized performance: Based on the characteristics of a specific CNN model, programmable logic can optimize the computing pipeline by reconfiguring logic blocks, greatly improving processing efficiency.

[0005] (2) Low latency and high throughput: Compared with traditional processors, programmable logic can reduce waiting time while maintaining high-speed data processing.

[0006] (3) Energy-saving benefits: By utilizing the shutdown characteristics of programmable logic, only necessary computing units are activated, greatly reducing overall energy consumption.

[0007] Based on the advantages of programmable logic, the existing technology combines the processing system with programmable logic to achieve CNN acceleration. The programmable logic can quickly infer the CNN model, and the machine learning framework corresponding to the CNN model is deployed in the processing system.

[0008] However, the resources of different programmable logics vary greatly, so the processing system cannot easily switch between different programmable logics, and the processing system and programmable logic are highly coupled.

[0009] With respect to the technical problem that the processing system and the programmable logic are highly coupled in the above-mentioned prior art, no effective solution has been proposed yet. Summary of the invention

[0010] The embodiments of the present application provide a CNN acceleration framework and a CNN acceleration method to at least solve the technical problem in the prior art that the processing system and the programmable logic have high coupling.

[0011] According to one aspect of an embodiment of the present application, a CNN acceleration framework is provided, including: a processing system and programmable logic, wherein the processing system is configured to perform the following operations: sending first input data to an intermediate mapping layer through a framework layer, and storing the first input data in a memory through the intermediate mapping layer; and sending a first storage address of the first input data, a second storage address of parameter information, and an execution command for executing at least one operator of a CNN model to the programmable logic through the intermediate mapping layer, wherein the parameter information corresponds to at least one operator, and the programmable logic is configured to perform the following operations: obtaining the first input data from the first storage address and obtaining the parameter information from the second storage address; and performing CNN-related operation processing on the first input data through at least one operator according to the execution command.

[0012] According to another aspect of an embodiment of the present application, a CNN acceleration method is also provided, which is applied to a CNN acceleration framework, including: a processing system sends first input data to an intermediate mapping layer through a framework layer, and stores the first input data in a memory through the intermediate mapping layer; the intermediate mapping layer of the processing system sends a first storage address of the first input data, a second storage address of parameter information, and an execution command for executing at least one operator of a CNN model to the programmable logic, wherein the parameter information corresponds to at least one operator; the programmable logic obtains the first input data from the first storage address and obtains the parameter information from the second storage address; and the programmable logic performs CNN-related operations on the first input data through at least one operator according to the execution command.

[0013] In the embodiment of the present application, the CNN acceleration framework uses the intermediate mapping layer set in the processing system to interact with the programmable logic, and uses the memory to send the parameter information and input data of the operator of the CNN model to the programmable logic, and then the programmable logic implements CNN acceleration based on the parameter information and input data. As a result, the processor system and the programmable logic in this technical solution implement CNN acceleration in a decoupled state. Compared with the prior art, the dependence between the processing system and the programmable logic is reduced, so that the processing system can use the intermediate mapping layer to connect different programmable logics. This solves the technical problem of the high coupling between the processing system and the programmable logic in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 is a schematic diagram of a CNN acceleration framework according to an embodiment of the present application;

[0016] Figure 2 is a schematic diagram of a programmable logic according to an embodiment of the present application;

[0017] Figure 3 is a schematic diagram of an operator linked list according to an embodiment of the present application;

[0018] Figure 4 is a schematic diagram of an execution operator linked list according to an embodiment of the present application;

[0019] Figure 5 is another schematic diagram of the execution operator linked list according to an embodiment of the present application;

[0020] Figure 6 is a flowchart of a CNN acceleration method according to the second aspect of an embodiment of the present application; and

[0021] Figure 7 It is a sequential flow diagram of CNN acceleration according to the CNN acceleration framework described in the embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] Figure 1 The structural diagram of the CNN acceleration framework described in the embodiment of the present application is shown exemplarily.

[0025] The CNN acceleration framework includes a processing system and programmable logic.

[0026] The processing system includes an application layer, a framework layer and an intermediate mapping layer.

[0027] The application layer is used to interact with users, send input data that needs to be processed by the CNN model to the framework layer, and receive output data finally returned by the framework layer. The input data can be image data.

[0028] The framework layer is used to process input data according to the execution order of operators in the operator chain list. The framework layer processes input data through operators executed by the framework layer in the operator chain list. For operators executed by programmable logic in the operator chain list, the input data processed by the programmable logic operators is sent to the intermediate mapping layer. The framework layer is also used to receive output data returned by the intermediate mapping layer.

[0029] The intermediate mapping layer is used to store the input data in the memory. Then the intermediate mapping layer sends the storage address of the input data, the storage address of the parameter information and the execution command to the programmable logic. The parameter information is stored in the memory by the intermediate mapping layer when constructing the operator linked list.

[0030] refer to Figure 2 As shown, the programmable logic includes a data rearrangement module, a preprocessing module and a CNN accelerator.

[0031] The data rearrangement module is used to rearrange input data and output data.

[0032] The preprocessing module is used to preprocess the image data to be processed (ie, input data), wherein the preprocessing includes operations such as image decoding, image scaling and normalization.

[0033] The programmable logic obtains the input data and parameter information from the memory according to the storage address of the input data and the storage address of the parameter information sent by the intermediate mapping layer, and executes the operator in the CNN accelerator according to the storage address and the parameter information through the CNN accelerator.

[0034] Figure 3 A schematic diagram of the operator linked list is shown, see Figure 3 As shown, the operators in the operator list include subgraph operators and single operators. The framework layer executes the single operators in the operator list. The programmable logic executes the subgraph operators in the operator list. For example, the operator list includes operators 1 to 5. Operators 1 and 5 are single operators, and their operator types match the framework layer. Operators 2 to 4 are subgraph operators, which constitute subgraph operators. Figure 1 The operator types of operators 2 to 4 match the programmable logic and are adjacent, so they can be used as subgraph operators to form a subgraph and executed by the programmable logic. Therefore, according to the execution order of each operator in the operator linked list, the framework layer and the programmable logic execute the operators in the operator linked list in sequence.

[0035] It should be noted that there may be multiple subgraphs in the operator linked list, and a subgraph may include multiple subgraph operators with no limit on the number. This embodiment only exemplarily shows the structure of the operator linked list to illustrate the process of operator execution, which will not be repeated here.

[0036] refer to Figure 1 As shown, a CNN acceleration framework is proposed, including: a processing system and a programmable logic, wherein the processing system is configured to perform the following operations: sending first input data to an intermediate mapping layer through a framework layer, and storing the first input data to a memory through the intermediate mapping layer; and sending a first storage address of the first input data, a second storage address of parameter information, and an execution command for executing at least one operator of a CNN model to the programmable logic through the intermediate mapping layer, wherein the parameter information corresponds to at least one operator, and the programmable logic is configured to perform the following operations: obtaining the first input data from the first storage address and obtaining the parameter information from the second storage address; and performing CNN-related operations on the first input data through at least one operator according to the execution command.

[0037] Specifically, the user sends image data to the application layer of the processing system through an application program set on the terminal device, and the application layer sends the image data to the framework layer after receiving the image data.

[0038] Further, the framework layer uses the received image data as input data 11. Afterwards, the framework layer obtains a pre-established operator list, and determines the operator type of the first operator in the operator list according to the mark pre-added to the operator. The operator types include single operators and subgraph operators. And the operators in the operator list are computing modules of the CNN model. The operator mark is added to each operator when parsing the CNN model to identify the operator type of the operator.

[0039] refer to Figure 3 as well as Figure 4 As shown, operator 1 is the first operator in the operator list, and the framework layer determines that the operator type of operator 1 is a single operator, so the input data 11 is input to operator 1, and operator 1 performs CNN-related operations on the input data 11 to generate output data 12, and the output data 12 is stored locally. The CNN-related operations include convolution operations, pooling operations, or element-by-element operations.

[0040] Furthermore, after the framework layer completes the execution of operator 1, it determines whether the operator type of operator 2 in the operator chain list is a single operator or a subgraph operator. If the framework layer determines that operator 2 is a subgraph operator, the output data 12 is used as input data 21 (i.e., the first input data), and then the input data 21 (i.e., the first input data) is sent to the intermediate mapping layer.

[0041] Further, after receiving the input data 21 (ie, the first input data), the intermediate mapping layer sends the input data 21 (ie, the first input data) to the memory for storage.

[0042] And the middle mapping layer accesses the operator list to determine the child to which operator 2 belongs. Figure 1 , and determine the sub Figure 1 All subgraph operators in . Figure 1 All subgraph operators in are operators 2 to 4 (ie, at least one operator).

[0043] Afterwards, the intermediate mapping layer sends the storage address (i.e., the first storage address) of the input data 21 (i.e., the first input data), the storage address (i.e., the second storage address) of the parameter information corresponding to operators 2 to 4, and the execution command for executing operators 2 to 4 to the programmable logic. The parameter information of the subgraph operator has been pre-stored in the memory.

[0044] Further, when the programmable logic receives the first storage address, the second storage address, and the execution command, it obtains input data 21 (i.e., first input data) from the memory according to the first storage address, and obtains parameter information corresponding to operators 2 to 4 from the memory according to the second storage address. The programmable logic performs CNN-related operations on the input data 21 (i.e., first input data) through operators 2 to 4 to generate output data 22.

[0045] As described in the background technology, based on the advantages of programmable logic, the prior art combines the processing system with programmable logic to achieve CNN acceleration. The programmable logic can quickly infer the CNN model, and the machine learning framework corresponding to the CNN model is deployed in the processing system. However, the resources of different programmable logics vary greatly, so the processing system cannot easily switch between different programmable logics, and the processing system and programmable logic are highly coupled.

[0046] In view of the technical problems described above, through the technical solutions of the embodiments of the present application, the CNN acceleration framework uses the intermediate mapping layer set in the processing system to interact with the programmable logic, and uses the memory to send the parameter information and input data of the operators of the CNN model to the programmable logic, and then the programmable logic implements CNN acceleration based on the parameter information and input data. As a result, the processor system and the programmable logic in this technical solution implement CNN acceleration in a decoupled state. Compared with the prior art, the dependence between the processing system and the programmable logic is reduced, so that the processing system can use the intermediate mapping layer to connect different programmable logics. This solves the technical problem of the high coupling between the processing system and the programmable logic in the prior art.

[0047] Optionally, the operation of performing CNN-related operations on the first input data through at least one operator according to the execution command includes: inputting parameter information of at least one operator and the first input data into a preset CNN accelerator according to the execution command; and using the CNN accelerator to perform CNN-related operations on the first input data in sequence through each operator of at least one operator, wherein at least one operator is used as a sub-graph operator to form a sub-graph.

[0048] Specifically, the programmable logic inputs the parameter information of operators 2 to 4 (i.e., at least one operator) obtained from the memory and input data 21 (i.e., the first input data) to the preset CNN accelerator. Figure 1 , among which Figure 1 Including operators 2 to 4. In this embodiment, the subgraph operators in the subgraph can process data continuously. If a single operator is provided in the programmable logic, the output data generated by each single operator needs to be returned to the processor system.

[0049] Further, refer to Figure 5 As shown, the programmable logic processes the input data 21 using the parameter information through the operator 2 of the CNN accelerator to generate output data 211. Wherein, when the operator is a convolution operator, its parameter information at least includes the weight parameters in the convolution layer. Thus, when the operator 2 is a convolution operator, the input data 21 is convolved according to the parameter information to generate output data 211.

[0050] Further, the programmable logic sends the output data 211 to the memory for storage, and inputs the output data 211 into the operator 3 of the CNN accelerator as input data 212. The programmable logic processes the input data 212 according to the parameter information through the operator 3 of the CNN accelerator to generate output data 213.

[0051] Further, the programmable logic sends the output data 213 to the memory for storage, and uses the output data 213 as the input data 214. Then, the programmable logic processes the input data 214 according to the parameter information through the operator 4 of the CNN accelerator to generate the output data 22. Then, the programmable logic stores the output data 22 in the memory.

[0052] Therefore, the technical solution forms a subgraph by combining operators of the same operator type and continuous operators, so that the subgraph operators in the subgraph of the programmable logic can process the input data in sequence, without returning the output data generated by each operator to the processing system, thus reducing the data transmission time. In addition, the technical solution realizes CNN accelerated processing based on the fast data processing speed of programmable logic.

[0053] Optionally, the operation of storing the first input data into the memory through the intermediate mapping layer includes: calling the programmable logic through the intermediate mapping layer to reorder the first input data into a data format that matches the first operator of the subgraph, and storing the reordered first input data into the memory.

[0054] Specifically, the intermediate mapping layer receives the input data 21 sent by the framework layer, then sends the input data 21 to the memory, and then sends the storage address of the input data 21 to the programmable logic. The intermediate mapping layer then generates a data reordering command for reordering the input data 21, and sends the data reordering command to the programmable logic.

[0055] Further, after receiving the storage address and the data rearrangement command, the programmable logic obtains the input data 21 from the memory according to the storage address, and rearranges the input data 21 through the data rearrangement module, thereby rearranging the input data 21 into Figure 1 The first operator in , namely operator 2, matches the data format. Afterwards, the programmable logic sends the rearranged input data 21 to the memory for storage.

[0056] Furthermore, the intermediate mapping layer automatically obtains the rearranged input data 21 from the memory, and generates a command for executing the sub-process according to the input data 21. Figure 1 Then, the intermediate mapping layer stores the rearranged input data 21 to the memory, and sends the storage address (ie, the first storage address) of the rearranged input data 21 and the execution command to the programmable logic.

[0057] Furthermore, after receiving the first storage address and executing the command, the programmable logic obtains the rearranged input data 21 (ie, the first input data) from the memory, processes the rearranged input data 21 through the CNN accelerator, and generates output data 22.

[0058] Therefore, in this technical solution, before the programmable logic performs CNN acceleration processing on the input data, it will rearrange the input data to obtain input data that matches the data format of the first operator in the subgraph, so that each input data is suitable for the programmable logic, improving the adaptability of the programmable logic to process the input data. In addition, this technical solution rearranges the input data to generate input data that can be applied to different programmable logics, thereby improving the portability of the machine learning framework used to deploy the CNN model in the programmable logic.

[0059] Optionally, the CNN acceleration framework also includes: receiving second input data sent by the application layer through the intermediate mapping layer; calling the programmable logic to preprocess the second input data to generate preprocessed data; and returning the preprocessed data to the application layer as the first input data through the intermediate mapping layer.

[0060] Specifically, the application layer receives the image data (ie, the second input data) sent by the user through the application program, and then the application layer sends the image data (ie, the second input data) to the intermediate mapping layer.

[0061] Further, after receiving the image data (i.e., the second input data), the intermediate mapping layer stores the image data (i.e., the second input data) in the memory, and generates a preprocessing command according to the image data (i.e., the second input data). The preprocessing command includes information such as the input size and data format required by the CNN model. Afterwards, the intermediate mapping layer sends the image data (i.e., the second input data) storage address and the preprocessing command to the programmable logic.

[0062] Furthermore, after receiving the preprocessing command, the programmable logic parses the preprocessing command to determine the size and data format required by the CNN model. The programmable logic then inputs the received image data (i.e., the second input data) into the preprocessing module. The programmable logic then preprocesses the image data (i.e., the second input data) according to the preprocessing command through the preprocessing module to generate preprocessed data. The preprocessing operation at least includes: (1) scaling the image data (i.e., the second input data) to an input size that meets the CNN model; (2) normalizing the image data (i.e., the second input data); and (3) converting the data format of the image data (i.e., the second input data) according to the data format of the input data of the CNN model.

[0063] Furthermore, the programmable logic sends the preprocessed data to the memory for storage. Then the intermediate mapping layer automatically reads the preprocessed data from the memory and sends the preprocessed data to the application layer.

[0064] Furthermore, after receiving the pre-processed data, the application layer sends the pre-processed data as input data 11 (ie, first input data) to the framework layer.

[0065] Therefore, before performing CNN processing on the first input data, the technical solution will also generate pre-processed data (i.e., the first input data) that conforms to the input size and data format of the CNN model through programmable logic, thereby making the first input data more suitable for processing by the CNN model and improving the processing speed of the first input data.

[0066] Optionally, the processing system is also configured to perform the following operations: load the CNN model through the application layer and send the CNN model to the framework layer; analyze the CNN model through the framework layer to determine all operators corresponding to the CNN model; establish an operator linked list according to the execution order of all operators through the framework layer; and store parameter information of at least one operator executed by the programmable logic to the memory through the intermediate mapping layer.

[0067] Specifically, the application layer loads the CNN model and sends the CNN model to the framework layer.

[0068] Furthermore, after receiving the CNN model, the framework layer analyzes the CNN model and determines the CNN model header file. The CNN model header file includes: the input size, number of channels and data format required by the CNN model. The framework layer then sends the CNN model header file to the application layer.

[0069] And the framework layer analyzes the CNN model, determines all operators of the CNN model, and determines parameter information such as the weights of each operator.

[0070] For example, all operators of the CNN model include: operator 1, operator 2, ..., operator 5 in the order of execution.

[0071] Furthermore, the framework layer determines whether the CNN model is executed for the first time. When it is determined that the CNN model is executed for the first time, an operator linked list is established according to the execution order of all operators of the CNN model.

[0072] Figure 3 A schematic diagram of an operator linked list is shown, wherein the first node of the operator linked list is operator 1, and the last node is operator 5. The input data pointer of the first node points to the image data to be inferred, and the input data pointers of other nodes point to the output data of the previous node.

[0073] Furthermore, the framework layer determines parameter information of the subgraph operator from all operators, and sends the parameter information of the subgraph operator to the intermediate mapping layer, and the intermediate mapping layer sends the parameter information to the memory for storage.

[0074] Therefore, the technical solution analyzes the CNN model before executing the CNN model, so as to determine in advance the input size and data format required for preprocessing the input data for the application layer, avoiding temporary determination when the CNN model is executed, and improving the processing speed of the input data.

[0075] Optionally, the operation of storing parameter information of at least one operator executed by programmable logic to a memory through an intermediate mapping layer includes: determining at least one operator executed by programmable logic from all operators through a framework layer; re-arranging the parameter information of at least one operator into a data format matching the at least one operator through the framework layer, and sending the re-arranged parameter information to the intermediate mapping layer; and storing the re-arranged parameter information to a memory through the intermediate mapping layer.

[0076] Specifically, all operators of the CNN model include, in order of execution: operator 1, operator 2, ..., operator 5.

[0077] Furthermore, the framework layer determines the operator type of each operator of Operator 1 to Operator 5, and allocates Operator 1 to Operator 5 according to the operator types supported by the framework layer and the programmable logic respectively, thereby determining the operators executed by the programmable logic and the operators executed by the framework layer.

[0078] For example, the operator type supported by the framework layer is the pooling operator, and the operator type supported by the programmable logic is the convolution operator and the activation operator. Operator 1 and operator 5 are pooling operators, operator 2 and operator 4 are convolution operators, and operator 3 is an activation operator.

[0079] Therefore, the framework layer assigns operators 1 and 5 to the framework layer as single operators executed by the framework layer. And according to the principle of generating subgraphs, operators 2 to 4 are used as subgraph operators to form subgraphs. Figure 1 , and the child Figure 1 Assigned to the programmable logic as a subgraph executed by the programmable logic. The principle of generating a subgraph is: it meets the operator type supported by the programmable logic and multiple subgraph operators are adjacent.

[0080] Furthermore, after determining that Operator 2 to Operator 4 are subgraph operators executed by programmable logic, the framework layer determines parameter information of Operator 2 to Operator 4, and rearranges the parameter information of Operator 2 to Operator 4 to generate parameter information in a data format that conforms to programmable logic.

[0081] Furthermore, the framework layer sends the rearranged parameter information to the intermediate mapping layer, and then the intermediate mapping layer sends the rearranged parameter information to the memory for storage. In the subsequent CNN model execution process, the data stored in the second storage address of the parameter information sent by the intermediate mapping layer to the programmable logic is the rearranged parameter information.

[0082] Therefore, the technical solution uses the operators in the CNN model to establish an operator linked list, thereby predetermining the subgraph and the single operator, so that when the CNN model is executed, it can quickly determine whether the executing device is a processing system or programmable logic, thereby improving the execution speed of the CNN model. In addition, the technical solution rearranges the parameter information data, which can make the parameter information suitable for programmable logic and improve the matching accuracy between the parameter information and the programmable logic.

[0083] Optionally, the CNN acceleration framework also includes: the programmable logic is also configured to perform the following operations: storing output data generated after execution of at least one operator to a memory, and the processing system is also configured to perform the following operations: obtaining output data from the memory through an intermediate mapping layer, and sending the output data to the framework layer; when the output data is data generated by the last operator in the operator linked list, sending the output data to the application layer through the framework layer; and when the output data is data generated by a non-last operator in the operator linked list, using the output data as input data for the next operator in the operator linked list.

[0084] Specifically, refer to Figure 4 As shown, the programmable logic completes execution of operators 2 to 4 according to the execution command, generates output data 22, and then sends the output data 22 to the memory for storage.

[0085] Further, after the output data 22 is stored in the memory, the intermediate mapping layer automatically obtains the output data 22 from the memory. Thereafter, the intermediate mapping layer returns the output data 22 to the framework layer.

[0086] Furthermore, after receiving the output data 22 sent by the intermediate mapping layer, the framework layer determines whether operator 4 is the last operator in the operator linked list.

[0087] More specifically, for example, if the framework layer determines that operator 4 is not the last operator in the operator chain list, the output data 22 is used as the input data 31.

[0088] Furthermore, the framework layer performs CNN-related operations on the input data 31 through the operator 5 to generate output data 32, and stores the output data 32 locally.

[0089] Further, the framework layer determines whether operator 5 is the last operator in the operator chain list. Thus, if the framework layer determines that operator 5 is the last operator in the operator chain list, the output data 33 is sent to the application layer.

[0090] Furthermore, the application layer sends the output data 32 to an application installed on the user's terminal device for display.

[0091] Therefore, this technical solution uses the execution order of operators in the operator chain list to process image data using operators, so that the output data of the previous operator is used as the input data of the next operator, so that single operators and sub-graph operators can be executed continuously, thereby improving the execution speed of the CNN model.

[0092] Optionally, the operation of sending the output data to the framework layer includes: calling the programmable logic through the intermediate mapping layer to rearrange the output data into a data format that matches the operator executed by the framework layer, and sending the rearranged output data to the framework layer.

[0093] Specifically, after the programmable logic stores the output data 22 in the memory, the intermediate mapping layer automatically obtains the output data 22 from the memory.

[0094] Then, the intermediate mapping layer generates a data rearrangement command based on the output data 22 to rearrange the output data 22 into a data format that conforms to the framework layer, and then sends the output data 22 to the memory for storage.

[0095] Furthermore, the intermediate mapping layer sends the storage address of the output data 22 and the data rearrangement command to the programmable logic.

[0096] The programmable logic inputs the output data 22 to the data rearrangement module according to the data rearrangement command, and the data rearrangement module rearranges the output data 22 to generate rearranged output data 22. The rearranged output data 22 has the same data format as the framework layer.

[0097] Furthermore, the programmable logic sends the rearranged output data 22 to the memory for storage. The framework layer automatically reads the rearranged output data 22 from the memory and sends the rearranged output data 22 to the framework layer.

[0098] Furthermore, after receiving the output data 22 , the framework layer uses the rearranged output data 22 as the input data 31 .

[0099] Therefore, this technical solution rearranges the output data of the programmable logic output to obtain output data with the same data format as the framework layer, so that when the next operator is a single operator executed by the framework layer, the rearranged output data can be directly used as input data, thereby improving the execution speed of CNN.

[0100] Optionally, the operation of sending the first input data to the intermediate mapping layer through the framework layer includes: sending the first input data to the intermediate mapping layer through the framework layer using an API interface. Thus, the technical solution utilizes the high interaction efficiency of the API interface to improve the interaction efficiency between the framework layer and the intermediate mapping layer.

[0101] Optionally, the operation of sending the first storage address of the first input data, the second storage address of the parameter information, and the execution command to the programmable logic through the intermediate mapping layer includes: using a cross-hardware interactive bus to send the first storage address of the first input data, the second storage address of the parameter information, and the execution command to the programmable logic through the intermediate mapping layer.

[0102] Specifically, when the processing system is an ARM architecture, the cross-hardware interactive bus may be an AXI bus or an Avalon bus; when the processing system is an x86 architecture, the cross-hardware interactive bus may be a PCI Express bus.

[0103] Therefore, the technical solution makes the processing system and the programmable logic more adaptable by installing different buses to match the system architecture of the processing system.

[0104] Therefore, according to the first aspect of this embodiment, the CNN acceleration framework uses the intermediate mapping layer set in the processing system to interact with the programmable logic, and uses the memory to send the parameter information and input data of the operator of the CNN model to the programmable logic, and then the programmable logic implements CNN acceleration based on the parameter information and input data. As a result, the processor system and the programmable logic in this technical solution implement CNN acceleration in a decoupled state, which reduces the dependence between the processing system and the programmable logic compared to the prior art, so that the processing system can use the intermediate mapping layer to connect different programmable logics. This solves the technical problem of the high coupling between the processing system and the programmable logic in the prior art.

[0105] Under the above operating environment, according to the second aspect of this embodiment, a CNN acceleration method is provided. Figure 1 The CNN acceleration framework implementation shown in . Figure 6 A schematic diagram showing the process of the method is shown in FIG. Figure 6 As shown, the method includes:

[0106] S602: The processing system sends the first input data to the intermediate mapping layer through the framework layer, and stores the first input data to the memory through the intermediate mapping layer;

[0107] S604: The processing system intermediate mapping layer sends a first storage address of the first input data, a second storage address of the parameter information, and an execution command for executing at least one operator of the CNN model to the programmable logic, wherein the parameter information corresponds to the at least one operator;

[0108] S606: The programmable logic obtains first input data from the first storage address and obtains parameter information from the second storage address; and

[0109] S608: The programmable logic performs CNN-related operations on the first input data through at least one operator according to the execution command.

[0110] Therefore, according to the second aspect of this embodiment, the CNN acceleration framework uses the intermediate mapping layer set in the processing system to interact with the programmable logic, and uses the memory to send the parameter information and input data of the operator of the CNN model to the programmable logic, and then the programmable logic implements CNN acceleration based on the parameter information and input data. As a result, the processor system and the programmable logic in this technical solution implement CNN acceleration in a decoupled state, which reduces the dependence between the processing system and the programmable logic compared to the prior art, so that the processing system can use the intermediate mapping layer to connect different programmable logics. This solves the technical problem of the high coupling between the processing system and the programmable logic in the prior art.

[0111] In summary, reference Figure 7 As mentioned above, the sequential steps of CNN acceleration framework for CNN acceleration are as follows:

[0112] 701: The application layer of the processing system reads the CNN model from the disk or the network and sends the CNN model to the framework layer;

[0113] 702: The framework layer parses the CNN model, determines the CNN model header file, and returns the CNN model header file to the application layer;

[0114] 703: The framework layer establishes an operator linked list according to the operators of the CNN model, and rearranges the parameter information of the subgraph operators, and sends the rearranged parameter information to the intermediate mapping layer;

[0115] 704: The intermediate mapping layer stores the rearranged parameter information in a memory;

[0116] 705: the application layer obtains the image data (ie, the second input data), and sends the image data (ie, the second input data) to the intermediate mapping layer;

[0117] 706: The intermediate mapping layer calls the preprocessing module of the programmable logic to preprocess the image data, generates preprocessed data, and returns the preprocessed data to the application layer;

[0118] 707: The application layer uses the preprocessed data as first input data, and sends the first input data to the framework layer;

[0119] 708: The framework layer sends the first input data to the intermediate mapping layer;

[0120] 709: The intermediate mapping layer calls the data reordering module of the programmable logic to reorder the first input data, and stores the reordered first input data in the memory;

[0121] 710: The intermediate mapping layer sends the first storage address of the first input data, the second storage address of the parameter information, and the execution command to the programmable logic;

[0122] 711: The programmable logic obtains first input data from the memory according to the first storage address and obtains parameter information from the memory according to the second storage address;

[0123] 712: The programmable logic inputs the first input data and the parameter information into the CNN accelerator according to the execution command, performs CNN-related operations on the first input data through the subgraph operator of the CNN accelerator to generate output data, and then the programmable logic stores the output data into the memory;

[0124] 713: The intermediate mapping layer obtains the output data from the memory, calls the data reordering module of the programmable logic, reorders the output data, and sends the reordered output data to the framework layer;

[0125] 714: When the output data is output data generated by the last operator in the operator linked list, send the output data to the application layer through the framework layer;

[0126] 715: When the output data is data generated by a non-last operator in the operator linked list, use the output data as input data for the next operator in the operator linked list until the operators in the operator linked list are executed.

[0127] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0129] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0130] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0131] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0133] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0134] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0135] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A CNN acceleration framework, characterized in that: include: processing system and programmable logic, where The processing system is configured to perform the following operations: Sending the first input data to the intermediate mapping layer through the framework layer, and storing the first input data in the memory through the intermediate mapping layer; as well as sending a first storage address of the first input data, a second storage address of parameter information, and an execution command for executing at least one operator of the CNN model to the programmable logic through the intermediate mapping layer, wherein the parameter information corresponds to the at least one operator, and The programmable logic is configured to perform the following operations: Acquire the first input data from the first storage address and acquire the parameter information from the second storage address; and According to the execution command, the first input data is processed by the at least one operator related to CNN, and The programmable logic is further configured to perform the following operations: storing output data generated after executing the at least one operator in the memory, and The processing system is also configured to perform the following operations: Obtaining the output data from the memory through the intermediate mapping layer, and sending the output data to the framework layer; In a case where the output data is data generated by the last operator in the operator linked list, sending the output data to the application layer through the framework layer; as well as In the case where the output data is data generated by a non-last operator in the operator chain list, the output data is used as input data for the next operator in the operator chain list, wherein The operation of sending the output data to the framework layer includes: calling the programmable logic through the intermediate mapping layer to rearrange the output data into a data format that matches the operator executed by the framework layer, and sending the rearranged output data to the framework layer.

2. The CNN acceleration framework according to claim 1, characterized in that: The operation of performing CNN-related operation processing on the first input data by using the at least one operator according to the execution command includes: Inputting the parameter information of the at least one operator and the first input data into a preset CNN accelerator according to the execution command; and The CNN accelerator is used to perform CNN-related operations on the first input data in sequence through each operator in the at least one operator, wherein the at least one operator is used as a sub-graph operator to form a sub-graph.

3. The CNN acceleration framework according to claim 2, characterized in that: The operation of storing the first input data into a memory through the intermediate mapping layer includes: The programmable logic is called through the intermediate mapping layer to reorder the first input data into a data format matching the first operator of the subgraph, and the reordered first input data is stored in the memory.

4. The CNN acceleration framework according to claim 1, characterized in that Also includes: receiving second input data sent by the application layer through the intermediate mapping layer; Calling the programmable logic to preprocess the second input data to generate preprocessed data; as well as The pre-processed data is returned to the application layer as the first input data through the intermediate mapping layer.

5. The CNN acceleration framework according to claim 1, characterized in that: The processing system is also configured to perform the following operations: Loading the CNN model through the application layer and sending the CNN model to the framework layer; Analyze the CNN model through the framework layer to determine all operators corresponding to the CNN model; Establishing an operator linked list according to the execution order of all operators through the framework layer; as well as Parameter information of at least one operator executed by the programmable logic is stored in a memory through the intermediate mapping layer.

6. The CNN acceleration framework according to claim 5, characterized in that: The operation of storing parameter information of at least one operator executed by the programmable logic into the memory through the intermediate mapping layer includes: Determining, by the framework layer, the at least one operator executed by the programmable logic from among all the operators; Rearranging the parameter information of the at least one operator into a data format matching the at least one operator through the framework layer, and sending the rearranged parameter information to the intermediate mapping layer; and The rearranged parameter information is stored in the memory through the intermediate mapping layer.

7. The CNN acceleration framework according to claim 1, characterized in that: The operation of sending the first input data to the intermediate mapping layer through the framework layer includes: sending the first input data to the intermediate mapping layer through the framework layer using an API interface.

8. The CNN acceleration framework according to claim 1, characterized in that: The operation of sending the first storage address of the first input data, the second storage address of the parameter information and the execution command to the programmable logic through the intermediate mapping layer includes: using a cross-hardware interactive bus to send the first storage address of the first input data, the second storage address of the parameter information and the execution command to the programmable logic through the intermediate mapping layer.

9. A CNN acceleration method, applied to a CNN acceleration framework, characterized in that: include: The processing system sends the first input data to the intermediate mapping layer through the framework layer, and stores the first input data to the memory through the intermediate mapping layer; The intermediate mapping layer of the processing system sends a first storage address of the first input data, a second storage address of parameter information, and an execution command for executing at least one operator of the CNN model to the programmable logic, wherein the parameter information corresponds to the at least one operator; The programmable logic obtains the first input data from the first storage address and obtains the parameter information from the second storage address; as well as The programmable logic performs CNN-related operations on the first input data through the at least one operator according to the execution command, and The method further comprises: The programmable logic stores output data generated after the at least one operator is executed in the memory, and The processing system obtains the output data from the memory through the intermediate mapping layer and sends the output data to the framework layer; The processing system sends the output data to the application layer through the framework layer when the output data is data generated by the last operator in the operator linked list; and The processing system uses the output data as input data of the next operator in the operator chain list when the output data is data generated by a non-last operator in the operator chain list, wherein The operation of sending the output data to the framework layer includes: calling the programmable logic through the intermediate mapping layer to rearrange the output data into a data format that matches the operator executed by the framework layer, and sending the rearranged output data to the framework layer.

Images