Data processing method, system, computer device and computer readable storage medium

Abstract

The present application discloses a data processing method, system, computer device and computer-readable storage medium. A data processing method includes: sending first input data to programmable logic through a processing system, wherein the first input data is a floating point number; converting the first input data into second input data through programmable logic using a quantization parameter preset in a subgraph operator of a subgraph, wherein the second input data is a fixed point number; performing a CNN-related operation on the second input data through programmable logic using a subgraph operator to generate first output data, wherein the subgraph is composed of at least one subgraph operator, and the subgraph operators that constitute the subgraph can be executed sequentially and continuously; and returning the first output data to the processing system through programmable logic.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, system, computer device and computer-readable storage medium. Background Art

[0002] In recent years, with the rapid development of the Internet of Things (IoT) and mobile computing, edge computing has attracted widespread attention as an emerging technology. Especially in areas such as visual recognition, convolutional neural networks (CNNs) have become one of the mainstream choices due to their excellent performance. However, the demand for high-performance computing is in sharp contrast to limited hardware resources and strict energy consumption constraints. In this context, programmable logic, with its unique reconfigurable characteristics and excellent energy efficiency, has gradually been regarded as an ideal platform for accelerating CNN reasoning at the edge.

[0003] Among them, programmable logic acceleration CNN has the following advantages:

[0004] (1) Customized performance: Based on the characteristics of a specific CNN model, programmable logic can optimize the computing pipeline by reconfiguring logic blocks, greatly improving processing efficiency.

[0005] (2) Low latency and high throughput: Compared with traditional processors, programmable logic can reduce waiting time while maintaining high-speed data processing.

[0006] (3) Energy-saving benefits: By utilizing the shutdown characteristics of programmable logic, only necessary computing units are activated, greatly reducing overall energy consumption.

[0007] Based on the advantages of programmable logic, the existing technology combines the processing system with programmable logic to achieve CNN acceleration. The programmable logic can quickly infer the CNN model, and the machine learning framework corresponding to the CNN model is deployed in the processing system.

[0008] However, when the CNN model is quantized and compressed, differences will appear in the data types of different operators in the CNN model. For example, the data types of operators in the CNN model are divided into fixed-point type and floating-point type. Among them, programmable logic does not support operators with floating-point data types, but only supports operators with fixed-point data types. Therefore, operators with floating-point data types can only be executed by the processing system. In addition, programmable logic does not have the ability to actively execute. Therefore, before the programmable logic operator is executed, the programmable logic needs to send data information such as input data and execution commands to the programmable logic, and after the programmable logic operator is executed, the output data can only be returned to the processing system for subsequent operations.

[0009] However, in this way, the reasoning task of the CNN model needs to be frequently switched between the processing system and the programmable logic. And each switch involves operations such as data copying and data preprocessing, which greatly increases the overall reasoning time.

[0010] With respect to the technical problem that frequent switching between the processing system and the programmable logic in the above-mentioned prior art causes the reasoning of the CNN model to take too long, no effective solution has been proposed so far. Summary of the invention

[0011] The embodiments of the present application provide a data processing method, system, computer device and computer-readable storage medium to at least solve the technical problem in the prior art that frequent switching between the processing system and the programmable logic causes the inference of the CNN model to take too long.

[0012] According to one aspect of an embodiment of the present application, a data processing method is provided, comprising: sending first input data to a programmable logic through a processing system, wherein the first input data is a floating-point number; converting the first input data into second input data through the programmable logic using a quantization parameter preset in a subgraph operator of a subgraph, wherein the second input data is a fixed-point number; performing CNN-related operations on the second input data through the programmable logic using the subgraph operator to generate first output data, wherein the subgraph is composed of at least one subgraph operator, and the subgraph operators constituting the subgraph can be executed sequentially and continuously; and returning the first output data to the processing system through the programmable logic.

[0013] According to another aspect of an embodiment of the present application, a data processing system is also provided, comprising: a processing system and programmable logic, wherein the processing system is configured to perform the following operations: send first input data to the programmable logic, wherein the first input data is a floating point number, and the programmable logic is configured to perform the following operations: convert the first input data into second input data using a quantization parameter preset in a subgraph operator of a subgraph, wherein the second input data is a fixed point number; perform CNN-related operations on the second input data using the subgraph operator to generate first output data, wherein the subgraph is composed of at least one subgraph operator, and the subgraph operators constituting the subgraph can be executed sequentially and continuously; and return the first output data to the processing system.

[0014] According to another aspect of an embodiment of the present application, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above method.

[0015] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program / instructions are stored, and the steps of the above method are implemented when the computer program is executed by a processor.

[0016] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program / instructions, and the steps of the above method are implemented when the computer program is executed by a processor.

[0017] In an embodiment of the present application, the processing system generates a subgraph, so that the programmable logic executes multiple subgraph operators in the subgraph in sequence. After the subgraph operator is executed, it only needs to send the output data directly to the next subgraph operator, so as to realize the continuous execution of each subgraph operator, avoiding the problem that the programmable logic sends the output data to the processing system after executing each operator, and then the processing system sends the next input data to the programmable logic, resulting in frequent switching between the processing system and the programmable logic. Furthermore, the technical solution can adapt to the situation where the programmable logic can only execute fixed-point type operators by adding quantization parameters to the subgraph operator, thereby increasing more operators executable by the programmable logic, thereby improving the reasoning speed of the CNN model and reducing time consumption. This solves the technical problem of frequent switching between the processing system and the programmable logic in the prior art, which takes too long for the reasoning of the CNN model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a schematic diagram of a data processing system according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of a data processing method according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of an operator linked list according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of an execution operator linked list according to an embodiment of the present application;

[0023] Figure 5 is another schematic diagram of the execution operator linked list according to an embodiment of the present application;

[0024] Figure 6 is another schematic diagram of the operator linked list according to an embodiment of the present application;

[0025] Fig. 7A is a schematic diagram of performing quantization perception training according to an embodiment of the present application;

[0026] Figure 7B is another schematic diagram of performing quantization-aware training according to an embodiment of the present application;

[0027] Figure 8 is a schematic diagram of a CNN acceleration framework according to an embodiment of the present application;

[0028] Fig. 9 is a schematic diagram of a programmable logic according to an embodiment of the present application; and

[0029] Fig.10 It is a sequential flow diagram of CNN acceleration according to the CNN acceleration framework described in the embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Figure 1 is a schematic diagram of a data processing system according to this embodiment. Figure 1 As shown, the system includes: a model training system and a CNN acceleration framework.

[0033] The model training system is used to train the CNN model, and then the CNN model is sent to the CNN acceleration framework.

[0034] The CNN acceleration framework includes a processing system and programmable logic.

[0035] The processing system is used to parse the CNN model sent by the model training system and establish an operator list. The processing system then uses a single operator in the operator list to process the input data and generate output data. The processing system is also used to send the parameter information of the subgraph operator in the operator list and the input data of the CNN-related operation performed by the subgraph operator to the programmable logic.

[0036] The programmable logic is used to use the sub-graph operator to perform CNN-related operations on the input data according to the parameter information sent by the processing system, and convert the output data output by the sub-graph operator from a fixed-point number to a floating-point number through a preset calibration operator, and send the floating-point number to the processing system.

[0037] Figure 3 A schematic diagram of the operator linked list is shown in Figure 3 As shown, the operators in the operator list include subgraph operators and single operators, and the processing system executes the single operator in the operator list. The programmable logic executes the subgraph operators in the operator list. For example, the operator list includes operators 1 to 4. Operator 3 is a single operator, and its operator type matches the processing system. Operator 1, operator 2, and operator 4 are subgraph operators, and their operator types match the programmable logic. Operator 1 and operator 2 form a subgraph operator. Fig.10 , operator 4 forms a subgraph 20. Operator 1 and operator 2 are adjacent, so they can form a subgraph, which is executed by the programmable logic. Therefore, according to the execution order of each operator in the operator chain list, the processing system and the programmable logic execute the operators in the operator chain list in sequence.

[0038] It should be noted that this technical solution only takes the four operators in the CNN model as an example to illustrate the execution process of the operators. In practical applications, the number of operators in the CNN model is not specifically limited.

[0039] Under the above operating environment, according to the first aspect of this embodiment, a data processing method is provided. Figure 1 The CNN acceleration framework implementation shown in . Figure 2 A schematic diagram showing the process of the method is shown in FIG. Figure 2 As shown, the method includes:

[0040] S202: Sending first input data to the programmable logic through the processing system, wherein the first input data is a floating point number;

[0041] S204: converting the first input data into second input data by using a quantization parameter preset in a subgraph operator of the subgraph through a programmable logic, wherein the second input data is a fixed-point number;

[0042] S206: performing a CNN-related operation on the second input data using a subgraph operator through programmable logic to generate first output data, wherein the subgraph is composed of at least one subgraph operator, and the subgraph operators constituting the subgraph can be executed sequentially and continuously; and

[0043] S208: Return the first output data to the processing system through the programmable logic.

[0044] Specifically, refer to Figure 3 as well as Figure 4 As shown, the user sends image data to the processing system through an application program set on the terminal device. The processing system then uses the received image data as input data 11 (ie, first input data).

[0045] Further, the processing system obtains the pre-established operator chain list, and determines whether the operator type of the first operator in the operator chain list, operator 1, is a subgraph operator or a single operator. Thus, the processing system determines that operator 1 is a subgraph operator, and thus sends input data 11 (i.e., the first input data) to the programmable logic. The input data 11 (i.e., the first input data) is a floating point number.

[0046] At the same time, the processing system determines the subgraph to which operator 1 belongs, and determines other subgraph operators in the subgraph, so as to send parameter information of the subgraph operators in the subgraph to the programmable logic, wherein the parameter data may be data such as weight information.

[0047] For example, the processing system determines that the subgraph to which operator 1 belongs is subgraph Fig.10 , and determine the sub Fig.10 The subgraph operators include operator 1 and operator 2, thereby sending parameter information of operator 1 and operator 2 to the programmable logic.

[0048] Further, the programmable logic receives input data 11 (i.e., first input data) and parameter information of operators 1 to 2. Then, the programmable logic inputs the input data 11 (i.e., first input data) into operator 1, and converts the floating-point input data 11 (i.e., first input data) from floating-point numbers to fixed-point numbers using the quantization parameters set in operator 1, thereby generating fixed-point input data 11 (i.e., second input data). Then, the programmable logic uses the parameter information of operator 1 to perform CNN-related operations on the fixed-point input data 11, and generates output data 12. For example, if operator 1 is a convolution operator, the programmable logic uses the parameter information of operator 1 to perform a convolution operation on the input data 12.

[0049] The parameter information of all operators in the operator linked list are floating point numbers, so the generated output data 12 are also floating point numbers. Further, the programmable logic uses the quantization parameters set in operator 1 to convert the output data 12 from floating point numbers to fixed point numbers.

[0050] Furthermore, the programmable logic inputs the output data 12 as input data 21 into the operator 2, and uses the parameter information of the operator 2 to perform CNN-related calculation operations on the input data 21 to generate output data 22.

[0051] So far, son Fig.10 The execution of the subgraph operator in is completed, so that the programmable logic returns the output data 22 to the processing system.

[0052] As described in the background technology, based on the advantages of programmable logic, the prior art combines the processing system with programmable logic to achieve CNN acceleration. The programmable logic can quickly infer the CNN model, and the machine learning framework corresponding to the CNN model is deployed in the processing system. However, when the CNN model is quantized and compressed, the data types between different operators in the CNN model will be different. For example, the data types of the operators in the CNN model are divided into fixed-point type and floating-point type. The programmable logic does not support operators with floating-point data types, but only supports operators with fixed-point data types. Therefore, operators with floating-point data types can only be executed by the processing system. And the programmable logic does not have the ability to actively execute, so before the programmable logic operator is executed, the programmable logic needs to send data information such as input data and execution commands to the programmable logic, and after the programmable logic operator is executed, the output data can only be returned to the processing system, and the processing system performs subsequent operations. However, in this way, the reasoning task of the CNN model needs to be frequently switched between the processing system and the programmable logic. And each switch involves operations such as data copying and data preprocessing, which greatly increases the overall reasoning time.

[0053] In view of the technical problems described above, through the technical solutions of the embodiments of the present application, the processing system generates a subgraph, so that the programmable logic executes multiple subgraph operators in the subgraph in sequence. After the subgraph operator is executed, it only needs to send the output data directly to the next subgraph operator, so as to realize the continuous execution of each subgraph operator, avoiding the problem that the programmable logic sends the output data to the processing system after executing each operator, and then the processing system sends the next input data to the programmable logic, resulting in frequent switching between the processing system and the programmable logic. Furthermore, the technical solution can adapt to the situation where the programmable logic can only execute fixed-point type operators by adding quantization parameters to the subgraph operator, thereby increasing more operators executable by the programmable logic, thereby improving the reasoning speed of the CNN model and reducing time consumption. This solves the technical problem of frequent switching between the processing system and the programmable logic in the prior art, which takes too long for the reasoning of the CNN model.

[0054] Optionally, the operation of performing CNN-related operations on the second input data using a subgraph operator through programmable logic to generate first output data includes: performing CNN-related operations on the second input data according to parameter information of the subgraph operator through programmable logic to generate second output data, wherein the parameter information is a floating point number and the second output data is a fixed point number; and converting the second output data into first output data using a preset calibration operator through programmable logic, wherein the first output data is a floating point number, and wherein the calibration operator is used to convert the fixed point number into a floating point number, and the calibration operator is set in the subgraph.

[0055] Specifically, the programmable logic inputs the output data 12 as the input data 21 into the operator 2, and uses the parameter information of the operator 2 to perform CNN-related operations on the input data 21 to generate output data 23. The output data 23 is a floating point number. Thus, further, the programmable logic uses the quantization parameter set in the operator 2 to convert the output data 22 from a floating point number to a fixed point number, thereby generating fixed-point type output data 23 (i.e., the second output data). And there is only one quantization parameter between two adjacent subgraph operators, so that the quantization parameter can be shared.

[0056] Furthermore, the programmable logic converts the fixed-point output data 23 from a fixed-point number to a floating-point number through a calibration operator to generate output data 22 (i.e., first output data). The calibration operator is set at the end of each subgraph to convert the fixed-point number outputted by the subgraph at the end into a floating-point number for use by the processing system.

[0057] So far, son Fig.10 The execution of the subgraph operator in is completed, so that the programmable logic returns the output data 23 to the processing system.

[0058] Therefore, the technical solution converts the fixed-point number to be returned to the processing system into a floating-point number by setting a calibration operator after the subgraph, so that the processing system can use the floating-point number calculation corresponding to the single operator to reduce the error.

[0059] Optionally, the method also includes: using the first output data sent by the programmable logic as third input data through the processing system; determining the operator type of the next operator of the subgraph in a preset operator linked list through the processing system; when the operator type is a single operator, using the single operator through the processing system to perform CNN-related operations on the third input data; and when the operator type is a subgraph operator, sending the third input data to the programmable logic through the processing system, and using the subgraph operator through the programmable logic to perform CNN-related operations on the third input data.

[0060] Specifically, the processing system receives the output data 22 (i.e., the first output data) returned by the programmable logic, and uses the output data 22 (i.e., the first output data) as the input data 31 (i.e., the third input data). Then, the processing system determines the child in the operator linked list. Fig.10 The operator type of the next operator is a subgraph operator or a single operator. In the case where the operator type is a single operator, the processing system processes the input data 31 (i.e., the third input data) through the single operator. In the case where the operator type is a subgraph operator, the programmable logic processes the input data 31 (i.e., the third input data).

[0061] refer to Figure 3 and Figure 4 As shown, for example, Fig.10 The next operator is operator 3, where operator 3 is a single operator executed by the processing system. The data types of the input data and output data supported by the single operator are both floating point numbers, and the parameter information of the single operator is also a floating point number. And the input data 31 (i.e., the third input data) is also a floating point number, so there is no need to change the data type of the input data 31 (i.e., the third input data) to adapt to the data type of the single operator.

[0062] Thus, the processing system uses the operator 3 to perform a CNN-related operation on the input data 31 (ie, the third input data) to generate output data 32. The data type of the output data 32 is also a floating point number.

[0063] Further, the processing system uses the output data 32 as the input data 41, and then determines from the operator linked list whether the operator type of the operator 4 is a subgraph operator or a single operator. Thus, the processing system determines that the operator 4 is a subgraph operator, and thus sends the input data 41 to the programmable logic. The data type of the input data 41 is a floating point number.

[0064] At the same time, the processing system determines the subgraph to which operator 4 belongs, and determines other subgraph operators in the subgraph, so as to send parameter information of the subgraph operators in the subgraph to the programmable logic, wherein the parameter data may be data such as weight information.

[0065] For example, the processing system determines that the subgraph to which operator 4 belongs is subgraph 20, and determines that there is only one subgraph operator, operator 4, in subgraph 20, so that the parameter information of operator 4 is sent to the programmable logic.

[0066] Further, the programmable logic receives the input data 41 and the parameter information of the operator 4. Then, the programmable logic uses the quantization parameter set in the operator 4 to convert the input data 41 from a floating point number to a fixed point number. Then, the programmable logic uses the parameter information of the operator 1 to perform CNN-related operations on the input data 11 to generate output data 12. The programmable logic only supports operators with fixed-point data types, so the data types of the input data and output data of the subgraph operator executed by the programmable logic are both fixed-point numbers. In addition, the data type of the parameter information of the subgraph operator can be unlimited.

[0067] The parameter information of all operators in the operator linked list are floating point numbers, so the generated output data 42 are also floating point numbers. Further, the programmable logic uses the quantization parameters set in the operator 4 to convert the output data 42 from floating point numbers to fixed point numbers.

[0068] Furthermore, a calibration operator is provided after the subgraph 20, so the programmable logic converts the output data 42 of the fixed-point type into a floating-point number through the calibration operator, so that the processing system can process the output data 42 of the floating-point type.

[0069] Furthermore, the programmable logic sends output data 42 of a floating point data type to the processing system.

[0070] After receiving the output data 42, the processing system determines whether there are any unexecuted operators in the operator chain list. If the processing system determines that all operators in the operator chain list have been executed, the output data 42 is sent to the application program set in the user's terminal device for display.

[0071] Therefore, the technical solution executes a single operator that is not supported by the programmable logic through the processing system, and other operators are accelerated by the programmable logic, so that the execution speed of the operator can be improved in this way.

[0072] Optionally, the method also includes: parsing the first CNN model through the processing system, determining the subgraph operator and the single operator from all operators in the first CNN model, wherein the subgraph operator is executed by the programmable logic and the single operator is executed by the processing system; and constructing an operator linked list through the processing system according to the execution order of the subgraph operator and the single operator.

[0073] Specifically, before the operator is executed, the processing system loads the CNN model (ie, the first CNN model), and then parses the CNN model (ie, the first CNN model) to determine all operators in the CNN model (ie, the first CNN model). For example, all operators are operators 1 to 4.

[0074] Further, the processing system determines the operator type of each operator in all operators (ie, operator 1 to operator 4 ) according to the operators supported by the processing system and the programmable logic.

[0075] For example, the processing system supports pooling operators, and the programmable logic supports activation function operators and convolution operators. The operator type of the operator supported by the processing system is a single operator, and the operator type of the operator supported by the programmable logic is a subgraph operator.

[0076] And operator 1 is the ReLU activation function operator, operator 2 is the convolution operator, operator 3 is the pooling operator, and operator 4 is the convolution operator.

[0077] Thus, the processing system determines that Operator 1, Operator 2, and Operator 4 are subgraph operators supported by the programmable logic, and Operator 3 is a single operator supported by the processing system.

[0078] Furthermore, the processing system establishes an operator linked list according to the execution order of operators 1 to 4 (refer to Figure 3 The execution order of operators 1 to 4 is: operator 1, operator 2, operator 3 and operator 4.

[0079] In addition, the processing system will parse the parameter information of each operator, thereby sending the parameter information of the subgraph operator to the programmable logic.

[0080] Therefore, the technical solution analyzes the CNN model, establishes an operator chain list, and clarifies whether the device that executes each operator is a programmable logic or a processing system. Therefore, when the technical solution executes the operators in sequence through the operator chain list, the operators can be executed according to the predetermined devices, thereby saving time and improving execution efficiency.

[0081] Optionally, the first CNN model is parsed by the processing system, and the operation of the sub-graph operator is determined from all operators in the first CNN model, including: parsing the first CNN model by the processing system to determine the sub-graph operator of the first CNN model; determining the corresponding calibration operator according to the sub-graph operator by the processing system; and adding the calibration operator to the operator linked list.

[0082] Specifically, the processing system parses the CNN model (i.e., the first CNN model) and determines a subgraph operator from all operators, wherein the subgraph operator includes operator 1, operator 2, and operator 4, and wherein operator 1 and operator 2 are adjacent, then the processing system groups operator 1 and operator 2 into a subgraph, i.e., subgraph. Fig.10 The processing system groups the operators 4 into a subgraph, namely, the subgraph 20 .

[0083] Further, the processing system determines a calibration operator from the CNN model, wherein the calibration operator is set at the end of each sub-graph, so that the sub-graph Fig.10 There is a calibration operator after each of the subgraphs 1 and 20. The processing system then sets the calibration operator in the operator linked list according to the position of the calibration operator after the subgraph (refer to Figure 6 shown).

[0084] Therefore, the technical solution can convert the output data of fixed-point numbers into floating-point numbers by setting a calibration operator after the subgraph, and then return them to the processing system to adapt to the floating-point operation of the processing system.

[0085] Optionally, the first CNN model is generated in the following manner: constructing a second CNN model corresponding to the first CNN model through a model training system, wherein the second CNN model is a full-precision model; determining operators supported by programmable logic from the second CNN model according to the operator type of the operator through the model training system as subgraph operators; inserting pseudo-quantization nodes at corresponding positions of each subgraph operator through the model training system; performing quantization-aware training on the second CNN model through the model training system, and after the training is completed, deleting the pseudo-quantization nodes in the second CNN model; determining subgraph operators that can form a subgraph from all subgraph operators through the model training system to form a subgraph; and setting a calibration operator after the subgraph through the model training system to generate the first CNN model, wherein the calibration operator is used to convert fixed-point numbers into floating-point numbers.

[0086] Specifically, before the processing system loads the first CNN model and parses the first CNN model, the model training system first generates the first CNN model, and the steps of generating the first CNN model are as follows:

[0087] The model training system constructs a second CNN model corresponding to the first CNN model. The second CNN model is a full-precision model. The parameter information of the operators in the second CNN model are all floating-point numbers, and the data types of the supported input data and output data are also floating-point types.

[0088] Furthermore, the operators supported by the programmable logic of the model training system determine the subgraph operator from all operators of the second CNN model.

[0089] For example, the second CNN model includes operators 1 to 4. Operator 1 is a ReLU activation function operator, operator 2 is a convolution operator, operator 3 is a pooling operator, and operator 4 is a convolution operator.

[0090] The programmable logic supports activation function operators and convolution operators. Thus, the model training system determines that operator 1, operator 2, and operator 4 are subgraph operators supported by the programmable logic.

[0091] Furthermore, the model training system inserts pseudo quantization nodes before and after each subgraph operator. The subgraph operators include operator 1, operator 2, and operator 4, so pseudo quantization nodes are inserted before and after operator 1, operator 2, and operator 4. Fig. 7A As shown, the pseudo quantization nodes are represented by S1 to S5. Operator 1 and operator 2 are two consecutive subgraph operators, so it is only necessary to insert a pseudo quantization node between operator 1 and operator 2.

[0092] Further, the model training system performs quantization-aware training on the second CNN model according to the pseudo-quantization nodes, thereby generating corresponding quantization parameters at each pseudo-quantization node. The quantization parameters are used to convert floating-point numbers into fixed-point numbers. Quantization-aware training is a commonly used technical means in the prior art and will not be described in detail here.

[0093] Furthermore, after the quantization-aware training is completed, the model training system deletes the inserted pseudo-quantization nodes to obtain a second CNN model with generated quantization parameters (refer to Figure 7B As shown, where the quantization parameters are represented by scale1 to scale5).

[0094] Furthermore, the model training system determines a subgraph operator from all operators, wherein the subgraph operator includes operator 1, operator 2, and operator 4, and wherein operator 1 and operator 2 are adjacent, then the model training system combines operator 1 and operator 2 into a subgraph, i.e., subgraph Fig.10 The model training system groups operator 4 into a subgraph, namely, subgraph 20.

[0095] Furthermore, the model training system sets a calibration operator after each subgraph to generate a first CNN model, wherein the calibration operator is used to convert fixed-point numbers into floating-point numbers.

[0096] Therefore, this technical solution performs quantization-aware training on the second CNN model, thereby converting the floating-point type operators in the full-precision model into a format suitable for programmable logic execution through quantization parameters, so that the operators supported by the programmable logic can be grouped together to achieve continuous execution and improve the execution speed of the operators.

[0097] also, Figure 8 The schematic diagram of the structure of the CNN acceleration framework described in the embodiment of the present application is shown as an example, wherein the CNN acceleration framework includes a processing system and programmable logic.

[0098] The processing system includes an application layer, a framework layer and an intermediate mapping layer.

[0099] The application layer is used to interact with users, send input data that needs to be processed by the CNN model to the framework layer, and receive output data finally returned by the framework layer. The input data can be image data.

[0100] The framework layer is used to process input data according to the execution order of operators in the operator chain list. The framework layer processes input data through operators executed by the framework layer in the operator chain list. For operators executed by programmable logic in the operator chain list, the input data processed by the programmable logic operators is sent to the intermediate mapping layer. The framework layer is also used to receive output data returned by the intermediate mapping layer.

[0101] The intermediate mapping layer is used to store the input data in the memory. Then the intermediate mapping layer sends the storage address of the input data, the storage address of the parameter information and the execution command to the programmable logic. The parameter information is stored in the memory by the intermediate mapping layer when constructing the operator linked list.

[0102] refer to Fig. 9 As shown, the programmable logic includes a data rearrangement module, a preprocessing module and a CNN accelerator.

[0103] The data rearrangement module is used to rearrange input data and output data.

[0104] The preprocessing module is used to preprocess the image data to be processed (ie, input data), wherein the preprocessing includes operations such as image decoding, image scaling and normalization.

[0105] The programmable logic obtains the input data and parameter information from the memory according to the storage address of the input data and the storage address of the parameter information sent by the intermediate mapping layer, and executes the operator in the CNN accelerator according to the storage address and the parameter information through the CNN accelerator.

[0106] Specifically, the user sends image data to the application layer of the processing system through an application program set on the terminal device, and the application layer sends the image data to the framework layer after receiving the image data.

[0107] Further, the framework layer uses the received image data as input data 11. Afterwards, the framework layer obtains a pre-established operator list, and determines the operator type of the first operator in the operator list according to the mark pre-added to the operator. The operator types include single operators and subgraph operators. And the operators in the operator list are computing modules of the first CNN model. The operator mark is added to each operator when parsing the first CNN model to identify the operator type of the operator.

[0108] refer to Figure 3 as well as Figure 4 As shown, operator 1 is the first operator in the operator list, and the framework layer determines that the operator type of operator 1 is a subgraph operator, and then sends the input data 11 to the intermediate mapping layer.

[0109] Furthermore, after receiving the input data 11, the intermediate mapping layer sends the input data 11 to the memory for storage.

[0110] And the middle mapping layer accesses the operator list to determine the child to which operator 1 belongs. Fig.10 , and determine the sub Fig.10 All subgraph operators in . Fig.10 All subgraph operators in are operator 1 to operator 2.

[0111] Then the intermediate mapping layer sends the storage address of the input data 11, the storage address of the parameter information corresponding to operators 1 to 2, and the execution command for executing operators 1 to 2 to the programmable logic. The parameter information of the subgraph operator has been pre-stored in the memory.

[0112] Further, the programmable logic receives the storage address of the input data 11, the storage address of the parameter information corresponding to the operator 1 to the operator 2, and the execution command for executing the operator 1 to the operator 2, and then obtains the input data 11 from the memory according to the storage address of the input data 11, and obtains the parameter information corresponding to the operator 1 to the operator 2 from the memory according to the storage address of the parameter information corresponding to the operator 1 to the operator 2. The programmable logic performs CNN-related operations on the input data 11 through the operator 1 to the operator 2 to generate output data 22.

[0113] Optionally, the programmable logic inputs the parameter information of operators 1 to 2 and the input data 11 obtained from the memory to a preset CNN accelerator. Fig.10 , among which Fig.10Including operator 1 to operator 2. And in this embodiment, the subgraph operators in the subgraph can process data continuously.

[0114] Furthermore, the programmable logic processes the input data 11 using the parameter information through the operator 1 of the CNN accelerator to generate output data 12. When the operator is a convolution operator, its parameter information includes at least the weight parameters in the convolution layer. Thus, when the operator 1 is a convolution operator, the input data 11 is convolved according to the parameter information to generate output data 12.

[0115] Further, the programmable logic sends the output data 12 to the memory for storage, and inputs the output data 12 into the operator 2 of the CNN accelerator as input data 21. The programmable logic processes the input data 21 according to the parameter information through the operator 2 of the CNN accelerator to generate output data 22.

[0116] Further, the programmable logic sends the output data 22 to the memory for storage.

[0117] Therefore, the technical solution forms a subgraph by combining operators of the same operator type and continuous operators, so that the subgraph operators in the subgraph of the programmable logic can process the input data in sequence, without returning the output data generated by each operator to the processing system, thus reducing the data transmission time. In addition, the technical solution realizes CNN accelerated processing based on the fast data processing speed of programmable logic.

[0118] Optionally, the intermediate mapping layer receives the input data 11 sent by the framework layer, then sends the input data 11 to the memory, and then sends the storage address of the input data 11 to the programmable logic. The intermediate mapping layer then generates a data reordering command for reordering the input data 11, and sends the data reordering command to the programmable logic.

[0119] Further, after receiving the storage address and the data rearrangement command, the programmable logic obtains the input data 11 from the memory according to the storage address, and rearranges the input data 11 through the data rearrangement module, thereby rearranging the input data 11 into Fig.10 The first operator in , namely operator 1, matches the data format. Afterwards, the programmable logic sends the rearranged input data 11 to the memory for storage.

[0120] Furthermore, the intermediate mapping layer automatically obtains the rearranged input data 11 from the memory, and generates a command for executing the sub-process according to the input data 11. Fig.10 Then, the intermediate mapping layer stores the rearranged input data 11 in the memory, and sends the storage address of the rearranged input data 11 and the execution command to the programmable logic.

[0121] Furthermore, after the programmable logic receives the storage address of the rearranged input data 11 and the execution command, it obtains the rearranged input data 11 from the memory, processes the rearranged input data 11 through the CNN accelerator, and generates output data 22.

[0122] Therefore, in this technical solution, before the programmable logic performs CNN acceleration processing on the input data, it will rearrange the input data to obtain input data that matches the data format of the first operator in the subgraph, so that each input data is suitable for the programmable logic, improving the adaptability of the programmable logic to process the input data. In addition, this technical solution rearranges the input data to generate input data that can be applied to different programmable logics, thereby improving the portability of the machine learning framework used to deploy the CNN model in the programmable logic.

[0123] Optionally, the application layer receives image data sent by the user through an application program, and then the application layer sends the image data to the intermediate mapping layer.

[0124] Furthermore, after receiving the image data, the intermediate mapping layer stores the image data in a memory and generates a preprocessing command based on the image data. The preprocessing command includes information such as the input size and data format required by the first CNN model. The intermediate mapping layer then sends the image data storage address and the preprocessing command to the programmable logic.

[0125] Furthermore, after receiving the preprocessing command, the programmable logic parses the preprocessing command to determine the size and data format required by the first CNN model. The programmable logic then inputs the received image data into the preprocessing module. The programmable logic then preprocesses the image data according to the preprocessing command through the preprocessing module to generate preprocessed data. The preprocessing operation at least includes: (1) scaling the image data to an input size that meets the first CNN model; (2) normalizing the image data; and (3) converting the data format of the image data according to the data format of the input data of the first CNN model.

[0126] Furthermore, the programmable logic sends the preprocessed data to the memory for storage. Then the intermediate mapping layer automatically reads the preprocessed data from the memory and sends the preprocessed data to the application layer.

[0127] Furthermore, after receiving the pre-processed data, the application layer sends the pre-processed data as input data 11 to the framework layer.

[0128] Therefore, before performing CNN processing on the input data, the technical solution will also generate pre-processed data that conforms to the input size and data format of the first CNN model through programmable logic, so that the input data is more suitable for processing by the first CNN model and the processing speed of the input data is improved.

[0129] Optionally, the application layer loads the first CNN model and sends the first CNN model to the framework layer.

[0130] Furthermore, after receiving the first CNN model, the framework layer analyzes the first CNN model and determines the first CNN model header file. The first CNN model header file includes: the input size, number of channels and data format required by the CNN model. The framework layer then sends the first CNN model header file to the application layer.

[0131] And the framework layer analyzes the first CNN model, determines all operators of the first CNN model, and determines parameter information such as weights of each operator.

[0132] For example, all operators of the first CNN model include: operator 1, operator 2, ..., operator 4 in the order of execution.

[0133] Furthermore, the framework layer determines whether the first CNN model is executed for the first time. When it is determined that the first CNN model is executed for the first time, an operator linked list is established according to the execution order of all operators of the first CNN model.

[0134] Figure 3 A schematic diagram of an operator linked list is shown, wherein the first node of the operator linked list is operator 1, and the last node is operator 4. The input data pointer of the first node points to the image data to be inferred, and the input data pointers of other nodes point to the output data of the previous node.

[0135] Furthermore, the framework layer determines parameter information of the subgraph operator from all operators, and sends the parameter information of the subgraph operator to the intermediate mapping layer, and the intermediate mapping layer sends the parameter information to the memory for storage.

[0136] Therefore, before executing the first CNN model, the technical solution analyzes the first CNN model, so as to determine in advance the input size and data format required for preprocessing the input data for the application layer, avoiding temporary determination when the first CNN model is executed, and improving the processing speed of the input data.

[0137] Optionally, all operators of the first CNN model include, in order of execution: operator 1, operator 2, ..., operator 5.

[0138] Furthermore, the framework layer determines the operator type of each operator of Operator 1 to Operator 4, and allocates Operator 1 to Operator 4 according to the operator types supported by the framework layer and the programmable logic respectively, thereby determining the operators executed by the programmable logic and the operators executed by the framework layer.

[0139] For example, the operator type supported by the framework layer is the pooling operator, and the operator types supported by the programmable logic are the convolution operator and the activation function operator. Operator 1 is the ReLU activation function operator, operator 2 is the convolution operator, operator 3 is the pooling operator, and operator 4 is the convolution operator.

[0140] Therefore, the framework layer assigns operator 3 to the framework layer as a single operator executed by the framework layer. And according to the principle of generating subgraphs, operators 1 to 2 are used as subgraph operators to form subgraphs. Fig.10 , operator 4 is used as a subgraph operator to form subgraph 20, and Fig.10 The subgraph 20 is assigned to the programmable logic as a subgraph executed by the programmable logic. The principle of generating the subgraph is: conforming to the operator type supported by the programmable logic, multiple adjacent subgraph operators or a single subgraph operator.

[0141] Furthermore, after determining that Operator 1 to Operator 2 are subgraph operators executed by programmable logic, the framework layer determines parameter information of Operator 1 to Operator 2, and rearranges the parameter information of Operator 1 to Operator 2 to generate parameter information in a data format that conforms to the programmable logic.

[0142] Further, the framework layer sends the rearranged parameter information to the intermediate mapping layer, and then the intermediate mapping layer sends the rearranged parameter information to the memory for storage. In the subsequent execution of the first CNN model, the data stored in the storage address of the parameter information sent by the intermediate mapping layer to the programmable logic is the rearranged parameter information.

[0143] Thus, the technical solution uses the operators in the first CNN model to establish an operator linked list, thereby predetermining the subgraph and the single operator, so that when the first CNN model is executed, it can quickly determine whether the executing device is a processing system or a programmable logic, thereby improving the execution speed of the first CNN model. In addition, the technical solution rearranges the parameter information data, so that the parameter information can be applied to the programmable logic, thereby improving the matching accuracy between the parameter information and the programmable logic.

[0144] Optionally, after the programmable logic completes execution of operators 1 to 2 according to the execution command and generates output data 22 , the output data 22 is sent to the memory for storage.

[0145] Further, after the output data 22 is stored in the memory, the intermediate mapping layer automatically obtains the output data 22 from the memory. Thereafter, the intermediate mapping layer returns the output data 22 to the framework layer.

[0146] Furthermore, after receiving the output data 22 sent by the intermediate mapping layer, the framework layer determines whether operator 2 is the last operator in the operator linked list.

[0147] More specifically, for example, if the framework layer determines that operator 2 is not the last operator in the operator chain list, the output data 22 is used as the input data 31.

[0148] Then the framework layer determines the operator type of operator 3 in the operator chain list. When the framework layer determines that the operator type of operator 3 is a single operator, the operator 3 performs CNN-related operations on the input data 31 to generate output data 32, and the output data 32 is stored locally.

[0149] Furthermore, the framework layer determines that operator 3 is not the last operator in the operator chain list, and uses output data 32 as input data 41.

[0150] The operator 4 performs CNN-related operations on the input data 41 to generate output data 42, and stores the output data 42 in the memory.

[0151] Further, the framework layer determines whether operator 4 is the last operator in the operator chain list. Thus, the framework layer determines that operator 4 is the last operator in the operator chain list. Then, the framework layer sends the output data 42 to the application layer.

[0152] Furthermore, the application layer sends the output data 42 to an application installed on the user's terminal device for display.

[0153] Therefore, this technical solution uses the execution order of operators in the operator chain list to process image data using operators, so that the output data of the previous operator is used as the input data of the next operator, so that single operators and sub-graph operators can be executed continuously, thereby improving the execution speed of the CNN model.

[0154] Optionally, after the programmable logic stores the output data 22 in the memory, the intermediate mapping layer automatically obtains the output data 22 from the memory.

[0155] Then, the intermediate mapping layer generates a data rearrangement command based on the output data 22 to rearrange the output data 22 into a data format that conforms to the framework layer, and then sends the output data 22 to the memory for storage.

[0156] Furthermore, the intermediate mapping layer sends the storage address of the output data 22 and the data rearrangement command to the programmable logic.

[0157] The programmable logic inputs the output data 22 to the data rearrangement module according to the data rearrangement command, and the data rearrangement module rearranges the output data 22 to generate rearranged output data 22. The rearranged output data 22 has the same data format as the framework layer.

[0158] Furthermore, the programmable logic sends the rearranged output data 22 to the memory for storage. The framework layer automatically reads the rearranged output data 22 from the memory and sends the rearranged output data 22 to the framework layer.

[0159] Furthermore, after receiving the output data 22 , the framework layer uses the rearranged output data 22 as the input data 31 .

[0160] Therefore, this technical solution rearranges the output data of the programmable logic output to obtain output data with the same data format as the framework layer, so that when the next operator is a single operator executed by the framework layer, the rearranged output data can be directly used as input data, thereby improving the execution speed of CNN.

[0161] Optionally, the framework layer sends the input data to the intermediate mapping layer through the API interface. Thus, the technical solution utilizes the high interaction efficiency of the API interface to improve the interaction efficiency between the framework layer and the intermediate mapping layer.

[0162] Optionally, the storage address of the input data, the storage address of the parameter information and the execution command are sent to the programmable logic through the intermediate mapping layer using a cross-hardware interactive bus.

[0163] Specifically, when the processing system is an ARM architecture, the cross-hardware interactive bus may be an AXI bus or an Avalon bus; when the processing system is an x86 architecture, the cross-hardware interactive bus may be a PCI Express bus.

[0164] Therefore, the technical solution makes the processing system and the programmable logic more adaptable by installing different buses to match the system architecture of the processing system.

[0165] In summary, reference Fig.10 As mentioned above, the sequential steps of CNN acceleration framework for CNN acceleration are as follows:

[0166] 1001: The application layer of the processing system reads the CNN model and sends the CNN model to the framework layer;

[0167] 1002: The framework layer parses the CNN model, determines the CNN model header file, and returns the CNN model header file to the application layer;

[0168] 1003: The framework layer establishes an operator linked list according to the operators of the CNN model, and rearranges the parameter information of the subgraph operators, and sends the rearranged parameter information to the intermediate mapping layer;

[0169] 1004: The intermediate mapping layer stores the rearranged parameter information in a memory;

[0170] 1005: the application layer obtains the image data and sends the image data to the intermediate mapping layer;

[0171] 1006: the intermediate mapping layer calls the preprocessing module of the programmable logic to preprocess the image data, generates preprocessed data, and returns the preprocessed data to the application layer;

[0172] 1007: The application layer uses the preprocessed data as input data and sends the input data to the framework layer;

[0173] 1008: The framework layer sends the input data to the intermediate mapping layer;

[0174] 1009: The intermediate mapping layer calls the data reordering module of the programmable logic to reorder the input data, and stores the reordered input data in the memory;

[0175] 1010: The intermediate mapping layer sends the storage address of the input data, the storage address of the parameter information and the execution command to the programmable logic;

[0176] 1011: the programmable logic obtains input data from the memory according to the storage address of the input data and obtains parameter information from the memory according to the storage address of the parameter information;

[0177] 1012: the programmable logic inputs the input data and parameter information into the CNN accelerator according to the execution command, performs CNN-related operations on the input data through the subgraph operator of the CNN accelerator to generate output data, and then the programmable logic stores the output data into the memory;

[0178] 1013: The intermediate mapping layer obtains the output data from the memory, calls the data reordering module of the programmable logic, reorders the output data, and sends the reordered output data to the framework layer;

[0179] 1014: When the output data is output data generated by the last operator in the operator linked list, send the output data to the application layer through the framework layer;

[0180] 1015: When the output data is data generated by a non-last operator in the operator linked list, use the output data as input data for the next operator in the operator linked list until the operators in the operator linked list are executed.

[0181] In addition, according to a second aspect of the present embodiment, a data processing system is provided, comprising: a processing system and programmable logic, wherein the processing system is configured to perform the following operations: send first input data to the programmable logic, wherein the first input data is a floating point number, and the programmable logic is configured to perform the following operations: convert the first input data into second input data using a quantization parameter preset in a sub-graph operator of the sub-graph, wherein the second input data is a fixed point number; perform CNN-related operations on the second input data using the sub-graph operator to generate first output data, wherein the sub-graph is composed of at least one sub-graph operator, and the sub-graph operators that constitute the sub-graph can be executed continuously in sequence; and return the first output data to the processing system.

[0182] In addition, according to a third aspect of this embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above method.

[0183] In addition, according to a fourth aspect of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0184] In addition, according to a fifth aspect of this embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0185] Therefore, according to this embodiment, the processing system generates a subgraph, so that the programmable logic executes multiple subgraph operators in the subgraph in sequence. After the subgraph operator is executed, it only needs to send the output data directly to the next subgraph operator, so as to realize the continuous execution of each subgraph operator, avoiding the problem that the programmable logic sends the output data to the processing system after executing each operator, and then the processing system sends the next input data to the programmable logic, resulting in frequent switching between the processing system and the programmable logic. Furthermore, the technical solution can adapt to the situation where the programmable logic can only execute fixed-point type operators by adding quantization parameters to the subgraph operator, thereby increasing more operators executable by the programmable logic, thereby improving the reasoning speed of the CNN model and reducing time consumption. This solves the technical problem of frequent switching between the processing system and the programmable logic in the prior art, which takes too long for the reasoning of the CNN model.

[0186] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0187] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0188] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0189] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0193] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0194] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Sending first input data to the programmable logic by the processing system, wherein the first input data is a floating point number; The first input data is converted into second input data by the programmable logic using a quantization parameter preset in a subgraph operator of the subgraph, wherein the second input data is a fixed-point number, wherein the programmable logic supports an activation function operator and a convolution operator, and the operator type of the operator supported by the programmable logic is a subgraph operator; Performing a CNN-related operation on the second input data by using the subgraph operator through the programmable logic to generate first output data, wherein the subgraph is composed of at least one subgraph operator, and the subgraph operators constituting the subgraph can be executed sequentially and continuously; as well as returning the first output data to the processing system through the programmable logic, and The method also includes: using the first output data sent by the programmable logic as third input data by the processing system; Determining, by the processing system, the operator type of the next operator of the subgraph in a preset operator chain list; In a case where the operator type is a single operator, performing a CNN-related operation on the third input data by the processing system using the single operator, wherein the processing system supports a pooling operator, and the operator type of the operator supported by the processing system is a single operator; and In the case where the operator type is a subgraph operator, the third input data is sent to the programmable logic through the processing system, and the subgraph operator is used by the programmable logic to perform CNN-related calculation operations on the third input data.

2. The method according to claim 1, characterized in that The operation of performing a CNN-related operation on the second input data by using the subgraph operator through the programmable logic to generate the first output data includes: Performing a CNN-related operation on the second input data according to the parameter information of the subgraph operator by the programmable logic to generate second output data, wherein the parameter information is a floating point number and the second output data is a fixed point number; and The second output data is converted into first output data by the programmable logic using a preset calibration operator, wherein the first output data is a floating point number, and wherein the calibration operator is used to convert a fixed point number into a floating point number, and the calibration operator is set in a subgraph.

3. The method according to claim 1, characterized in that Also includes: Parsing a first CNN model by the processing system, determining the subgraph operator and the single operator from all operators in the first CNN model, wherein the subgraph operator is executed by the programmable logic, and the single operator is executed by the processing system; as well as The operator linked list is constructed by the processing system according to the execution order of the subgraph operator and the single operator.

4. The method according to claim 3, characterized in that Parsing the first CNN model by the processing system, and determining the operation of the subgraph operator from all operators in the first CNN model, includes: Parsing the first CNN model by the processing system to determine a subgraph operator of the first CNN model; determining, by the processing system, a corresponding calibration operator according to the subgraph operator; and The calibration operator is added to the operator linked list.

5. The method according to claim 3, characterized in that: The first CNN model was generated in the following way: Constructing a second CNN model corresponding to the first CNN model through a model training system, wherein the second CNN model is a full-precision model; Determining, by the model training system, an operator supported by programmable logic from the second CNN model according to the operator type of the operator as a subgraph operator; Inserting a pseudo-quantization node at a corresponding position of each subgraph operator through the model training system; Performing quantization-aware training on the second CNN model by the model training system, and after the training is completed, deleting the pseudo-quantization nodes in the second CNN model; Determining, through the model training system, subgraph operators that can form a subgraph from among all subgraph operators, and forming the subgraph; as well as The model training system sets a calibration operator after the subgraph to generate the first CNN model, wherein the calibration operator is used to convert fixed-point numbers into floating-point numbers.

6. A data processing system, characterized in that: include: processing system and programmable logic, where The processing system is configured to perform the following operations: Sending first input data to the programmable logic, wherein the first input data is a floating point number; Using the first output data sent by the programmable logic as third input data; Determine the operator type of the next operator of the subgraph in a preset operator linked list; In a case where the operator type is a single operator, using the single operator to perform a CNN-related operation on the third input data, wherein the processing system supports a pooling operator, and the operator type of the operator supported by the processing system is a single operator; as well as In case the operator type is a subgraph operator, sending the third input data to the programmable logic, and The programmable logic is configured to perform the following operations: Converting the first input data into second input data using a quantization parameter preset in a subgraph operator of the subgraph, wherein the second input data is a fixed-point number, wherein the programmable logic supports an activation function operator and a convolution operator, and the operator type of the operator supported by the programmable logic is a subgraph operator; Using the subgraph operator to perform a CNN-related operation on the second input data to generate first output data, wherein the subgraph is composed of at least one subgraph operator, and the subgraph operators constituting the subgraph can be executed sequentially and continuously; returning the first output data to the processing system; as well as The subgraph operator is used to perform a CNN-related operation on the third input data.

7. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 1 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claim 1 are implemented.

Images