Convolutional neural network acceleration method, system and medium based on Cortex-M processor

Abstract

The present application relates to a convolutional neural network acceleration method, system and medium based on a Cortex‑M processor, wherein the method comprises: setting MCR instructions and CDP instructions according to common basic operators of the convolutional neural network, wherein the common basic operators include convolution operators, Relu activation operators, pooling operators, table lookup operators and quantization operators; configuring the internal registers of the convolutional neural network coprocessor through the MCR instructions, and then starting the common basic operators of the convolutional neural network through the CDP instructions. Through the present application, the problems of inefficiency, high cost and inflexibility of the convolutional neural network algorithm in processor execution are solved, and the basic operators required for the execution of the convolutional neural network are realized through the coprocessor instruction set, which can reduce the cost of reconstructing hardware for application fields with variable algorithms.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a convolutional neural network acceleration method, system and medium based on a Cortex-M processor. Background Art

[0002] With the continuous development of science and technology, artificial intelligence technology is constantly being integrated into people's daily lives. Applications such as target detection and speech recognition make society operate more efficiently and orderly. For example, ImageNet, which is used for image recognition, has achieved an object recognition accuracy rate higher than that of the human eye. As a type of artificial neural network, the convolutional neural network CNN can automatically obtain the features of the original data and obtain the mapping relationship between input and output without manually selecting features or clarifying the relationship between input and output. The basic operations in the convolutional neural network include convolution, pooling, vector operations and Relu activation.

[0003] In response to the bandwidth cost and latency issues of long-distance transmission of large amounts of data in cloud computing, more and more edge devices are beginning to support convolutional neural network related operations (such as convolution, activation, pooling, etc.). In addition to directly using the MCU's central processing unit for operations, various convolutional neural network hardware accelerators equipped on the MCU have also been designed to perform specific operations. However, the typical microcontroller unit MCU is not capable of such huge data operations, which will result in lengthy inference time on the end side; the dedicated hardware accelerator structure is fixed and inflexible, and the development of hardware accelerators for algorithms with various forms will increase development costs.

[0004] Currently, no effective solution has been proposed to address the problems of inefficiency, high cost and inflexibility of convolutional neural network algorithms in processor execution in related technologies. Summary of the invention

[0005] The embodiments of the present application provide a convolutional neural network acceleration method, system and medium based on a Cortex-M processor to at least solve the problems of inefficiency, high cost and inflexibility of convolutional neural network algorithms in processor execution in the related art.

[0006] In a first aspect, an embodiment of the present application provides a convolutional neural network acceleration method based on a Cortex-M processor, the method comprising:

[0007] Setting MCR instructions and CDP instructions according to common basic operators of convolutional neural networks, wherein the common basic operators include convolution operators, Relu activation operators, pooling operators, table lookup operators, and quantization operators;

[0008] The internal registers of the convolutional neural network coprocessor are configured through the MCR instruction, and then the common basic operator of the convolutional neural network is started through the CDP instruction.

[0009] In some embodiments, configuring the internal registers of the convolutional neural network coprocessor through the MCR instruction includes:

[0010] The MCR instruction is used to configure the data address, stride block information and format information of the internal registers of the convolutional neural network coprocessor, wherein the data address is used to read and write data during operation, the stride block information is used to block data during operation, and the format information is used to confirm the operation format and write-back format of the data.

[0011] In some embodiments, configuring the internal register by the MCR instruction and then starting the common basic operator by the CDP instruction includes:

[0012] Through the first MCR instruction, the local cache address of the convolution kernel is configured to the first register, the local cache address of the feature data is configured to the second register, the stride block information is configured to the scale register, and the format information is configured to the control register;

[0013] Starting the convolution operator through the CDP instruction, and determining the preset number of channels and the preset number of groups of the feature data in each operation according to the stride block information;

[0014] According to the total number of channels of the feature data and the preset number of channels, performing multiplication and accumulation operations of the feature data and the convolution kernel in sequence according to the channel direction;

[0015] In each channel of the feature data, according to the total number of groups of the feature data, the preset number of groups and the format information, multiplication and accumulation operations are performed on the feature data and the convolution kernel in sequence in a preset direction until the convolution results of all channels are obtained.

[0016] In some embodiments, configuring the internal register through the MCR instruction and then starting the common basic operator through the CDP instruction further includes:

[0017] By means of the second MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information is configured to the scale register;

[0018] The Relu activation operator of the convolutional neural network is started by the CDP instruction, and the input data is input into the Relu activation function according to the stride block information. In, return the result value;

[0019] The result value is written back to the local cache according to the write-back information.

[0020] In some embodiments, configuring the internal register through the MCR instruction and then starting the common basic operator through the CDP instruction further includes:

[0021] By means of a third MCR instruction, a local cache address of the first vector group is configured to a first register, a local cache address of the second vector group is configured to a second register, a local cache address of write-back information is configured to a third register, and stride block information is configured to a scale register;

[0022] Starting the pooling operator of the convolutional neural network through the CDP instruction, comparing the values ​​in the first vector group and the second vector group one by one according to the stride block information, and returning a vector with a larger value each time the comparison is performed;

[0023] The maximum pooling result obtained by the comparison is written back to the local cache according to the write-back information.

[0024] In some embodiments, configuring the internal register through the MCR instruction and then starting the common basic operator through the CDP instruction further includes:

[0025] By means of the fourth MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information and the table base address information are configured to the scale register;

[0026] Starting a table lookup operator of the convolutional neural network through the CDP instruction, and performing a table lookup operation according to the input data, the stride block information and the table base address information;

[0027] The table lookup results are written back to the local cache according to the write-back information.

[0028] In some embodiments, configuring the internal register through the MCR instruction and then starting the common basic operator through the CDP instruction further includes:

[0029] By means of the second MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information is configured to the scale register;

[0030] The quantization operator of the convolutional neural network is started by the CDP instruction, and the 32-bit single-precision floating-point number that complies with the IEEE-754 standard in the input data is converted into a 16-bit integer number according to the stride block information, or the 16-bit integer number in the input data is converted into a 32-bit single-precision floating-point number that complies with the IEEE-754 standard; the conversion result is written back to the local cache according to the write-back information.

[0031] In some embodiments, the method further comprises:

[0032] Through the fifth MCR instruction, the main memory address is configured to the first register, the local cache address is configured to the second register, and the stride block information is configured to the scale register;

[0033] Initiating a data read operation through the CDP instruction, and reading the data in the main memory address into the local cache according to the stride block information;

[0034] The data write operation is initiated through the CDP instruction, and the locally cached data is written to the main memory address according to the stride block information.

[0035] In a second aspect, an embodiment of the present application provides a convolutional neural network acceleration system based on a Cortex-M processor, the system comprising an instruction set setting module and an instruction set execution module;

[0036] The instruction set setting module sets the MCR instruction and the CDP instruction according to the common basic operators of the convolutional neural network, wherein the common basic operators include a convolution operator, a Relu activation operator, a pooling operator, a table lookup operator and a quantization operator;

[0037] The instruction set execution module configures the internal registers of the convolutional neural network coprocessor through the MCR instruction, and then starts the common basic operator of the convolutional neural network through the CDP instruction.

[0038] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the convolutional neural network acceleration method based on a Cortex-M processor as described in the first aspect above.

[0039] Compared with the related art, the embodiment of the present application provides a convolutional neural network acceleration method, system and medium based on a Cortex-M processor, which sets MCR instructions and CDP instructions according to the common basic operators of the convolutional neural network, wherein the common basic operators include convolution operators, Relu activation operators, pooling operators, table lookup operators and quantization operators; configures the internal registers of the convolutional neural network coprocessor through the MCR instruction, and then starts the common basic operators of the convolutional neural network through the CDP instruction, thereby solving the problems of inefficiency, high cost and inflexibility of the convolutional neural network algorithm in the processor execution, and realizes (1) through the coprocessor instruction (1) The coprocessor instruction set executes the basic operators required for convolutional neural networks, which can reduce the cost of hardware reconstruction for application fields with diverse algorithms; (2) The coprocessor instruction set is used to retrieve data from the local cache, which improves the reuse rate of the local cache data and reduces the bandwidth requirement for the coprocessor to access the main memory, thereby reducing the power consumption and cost of the entire system; (3) The coprocessor is used to process artificial intelligence operations, specifically through the CPU-specific coprocessor interface to transmit instructions, which can avoid the delay problem caused by bus blocking and improve system efficiency; (4) The coprocessor instruction set in the present invention is flexibly designed and has a large reserved space, which is convenient for adding additional instructions when the hardware is upgraded. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 It is a flowchart of the steps of a convolutional neural network acceleration method based on a Cortex-M processor according to an embodiment of the present application;

[0042] Figure 2 It is a flow chart of the steps of executing the convolution operator through the MCR instruction and the CDP instruction;

[0043] Figure 3 It is a specific flow chart of executing the convolution operator through the MCR instruction and the CDP instruction;

[0044] Figure 4 It is a schematic diagram of a specific multiply-accumulate operation without write-back function;

[0045] Figure 5 It is a structural block diagram of a convolutional neural network acceleration method based on a Cortex-M processor according to an embodiment of the present application;

[0046] Figure 6 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application.

[0047] Description of the drawings: 51. Instruction set setting module; 52. Instruction set execution module. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0049] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0050] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0051] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0052] In the prior art, the simplest method is to directly use the MCU processor to process the calculations of these convolutional neural networks. The existing ARM Cortex-M series processors contain a series of independent operation instructions such as addition, multiplication, and multiplication-accumulation, which can handle a small number of operations. Due to the inability to parallelize, the processor is inefficient when processing large amounts of data. For example, it takes at least ten instructions to process the most basic multiplication-accumulation operation (Multiply Accumulate) in the convolution operation. If a complete lenet-5 network is calculated, tens of thousands of instructions will be used. For an edge device, this is difficult to meet the real-time requirements. At the same time, a large number of operations will also occupy the processor's resources, thereby affecting the overall performance of the system.

[0053] On the one hand, there are dedicated hardware accelerators designed to handle these operations. The convolution operation has the largest amount of calculations in convolutional neural networks. The method of using application-specific integrated circuits ASICs to build dedicated deep learning accelerators has certain effects, but dedicated hardware structures are designed for different needs. In today's world where artificial intelligence algorithms emerge in an endless stream, the original hardware structure may not be able to meet the latest algorithm requirements. Repeated customization of hardware will lead to increased costs.

[0054] On the other hand, if cloud computing is used, there will be bandwidth costs and delays for long-distance transmission. In some situations where real-time requirements are very high, such as using deep learning to detect arc occurrence in industry, it is necessary to identify the arc as quickly as possible and cut off the power supply to protect electrical equipment. Excessive delay will increase the occurrence of danger, so cloud computing solutions have certain limitations.

[0055] Therefore, in order to realize a convolutional neural network accelerator with certain flexibility, the present invention proposes an efficient, concise and flexible convolutional neural network coprocessor instruction set, which deletes some unnecessary operations to achieve the purpose of lightweight, and can realize convolution, activation, pooling, element vector operations and quantization operators, and support different convolutional neural network algorithms without redesigning the hardware structure.

[0056] The embodiment of the present application provides a convolutional neural network acceleration method based on a Cortex-M processor. Figure 1 is a flowchart of the steps of a convolutional neural network acceleration method based on a Cortex-M processor according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0057] Step S102, setting MCR instructions and CDP instructions according to common basic operators of the convolutional neural network, wherein the common basic operators include a convolution operator, a Relu activation operator, a pooling operator, a table lookup operator, and a quantization operator;

[0058] Specifically, Table 1 is a CDP instruction set of the convolutional neural network coprocessor. As shown in Table 1, each CDP instruction corresponds to two operands and corresponding instruction functions.

[0059] Table 1

[0060] Operand 1 Operand 2 Function 0000 000 Read main memory data to local cache operation 0000 001 Write local cache data to main memory operation 0001 011 Multiply-accumulate operation without write-back 0001 111 Multiply-accumulate operation with write-back 0010 001 Element-wise vector addition 0010 010 Element-wise vector comparison operations 0011 001 Relu activation operation 0011 010 32-bit single-precision floating point number (FP32) to 16-bit integer number (INT16) calculation 0011 011 16-bit integer (INT16) to 32-bit single-precision floating point (FP32) calculation 0100 000 Table lookup operation with 64 entries 0100 001 Table lookup operation with 128 entries 0100 010 Table lookup operation with 256 entries 0100 011 Table lookup operation with 512 entries

[0061] Step S104, configure the internal registers of the convolutional neural network coprocessor through the MCR instruction, and then start the common basic operator of the convolutional neural network through the CDP instruction.

[0062] Specifically, the MCR instruction is used to configure the data address, stride block information and format information of the internal registers of the convolutional neural network coprocessor, wherein the data address is used to read and write data during operation, the stride block information is used to block data during operation, and the format information is used to confirm the operation format and write-back format of the data.

[0063] Then use the CDP instruction in Table 1 to start the common basic operator of the convolutional neural network.

[0064] Through steps S102 to S104 in the embodiment of the present application, the problems of low efficiency, high cost and inflexibility of the convolutional neural network algorithm in processor execution are solved. The basic operators required for executing the convolutional neural network are realized through the coprocessor instruction set, which can reduce the cost of reconstructing hardware for application fields with variable algorithms; the coprocessor instruction set is used to fetch data from the local cache, which improves the reuse rate of the local cache data and reduces the bandwidth requirement for the coprocessor to access the main memory, thereby reducing the power consumption and cost of the entire system; the coprocessor is used to process artificial intelligence operations, and the instruction transmission is carried out specifically through the CPU-specific coprocessor interface, which can avoid the delay problem caused by bus blocking and improve system efficiency; the coprocessor instruction set is flexibly designed and has a large reserved space, which is convenient for adding additional instructions when the hardware is upgraded.

[0065] In some of these embodiments, Figure 2 It is a flow chart of the steps of executing the convolution operator through the MCR instruction and the CDP instruction, such as Figure 2 As shown, step S104, configuring the internal register through the MCR instruction, and then starting the common basic operator through the CDP instruction specifically includes the following steps:

[0066] Step S202, through the first MCR instruction, configure the local cache address of the convolution kernel to the first register, configure the local cache address of the feature data to the second register, configure the stride block information to the scale register, and configure the format information to the control register;

[0067] Specifically, the local cache address of the convolution kernel (weight data) is configured to the DLA_ADDR1 register through the first MCR instruction; the local cache address of the feature data is configured to the DLA_ADDR2 register; the number of stride blocks and the stride block interval are configured to the DLA_SIZE register; and the operation mode and write-back accuracy are configured to the DLA_Control register.

[0068] The stride block information includes the number of stride blocks, stride block interval and stride block size. The number of stride blocks is DLA_SIZE[15:0], which indicates the number of feature data groups; the stride block interval is DLA_SIZE[23:16], which indicates the interval size between each set of feature data, with a granularity of 128 bits (16 bytes). 0 indicates continuous access, otherwise the actual stride size is (DLA_SIZE[23:16]+1)*16 bytes; the stride block size is fixed to 128 bits (16 bytes). Therefore, the amount of feature data for this operation is the number of stride blocks * stride block size, that is, DLA_SIZE[15:0]*16 bytes. In addition, the number of convolution kernels (weight data) for each operation is fixed to 512 bits (64 bytes).

[0069] The operation mode is DLA_Control[0]. When configured to 0, it indicates that the multiplication and accumulation unit is in the mode of 8-bit integer multiplication and 16-bit integer addition (INT8*INT8+INT16). When configured to 1, it indicates that the multiplication and accumulation unit is in the mode of 16-bit integer multiplication and 32-bit integer addition (INT16*INT16+INT32). The write-back precision is DLA_Control[1]. When configured to 0, it writes back with 8 bits in operation mode 0 and writes back with 16 bits in operation mode 1. When configured to 1, it writes back with 16 bits in operation mode 0 and writes back with 32 bits in operation mode 1.

[0070] Step S204, starting the convolution operator through the CDP instruction, and determining the preset number of channels and the preset number of groups of feature data in each operation according to the stride block information;

[0071] Specifically, Figure 3 This is a specific flow chart of executing the convolution operator through the MCR instruction and the CDP instruction, such as Figure 3 As shown, the operation in the convolution operator is essentially a multiplication-accumulation operation of the convolution kernel and the feature data. The convolution operator is started by the CDP 0001 011 instruction or the CDP 0001 111 instruction. Since the amount of data calculated by a single multiplication-accumulation instruction of the coprocessor is limited, the total convolution operation needs to be split to comply with the working mode of the hardware. The preset number of channels of the feature data in each operation after the split is determined according to the stride block size, and the number of groups of feature data in each operation is determined according to the number of stride blocks.

[0072] Step S206, performing multiplication and accumulation operations on the feature data and the convolution kernel in sequence according to the total number of channels of the feature data and the preset number of channels;

[0073] Specifically, Figure 3 As shown in the figure, according to the total number of channels of the feature data and the preset number of channels, the multiplication and accumulation operations of the feature data and the convolution kernel are performed in sequence in the channel direction. For example, if the preset number of channels for each operation is 8 and the total number of channels is 128, it is necessary to perform 16 multiplication and accumulation operations of the feature data and the convolution kernel in sequence in the channel direction.

[0074] Step S208, in each channel of the feature data, according to the total number of groups of feature data, the preset number of groups and the format information, multiplication and accumulation operations are performed on the feature data and the convolution kernel in sequence in the preset direction until the convolution results of all channels are obtained.

[0075] Specifically, Figure 3As shown, in each channel of the feature data, first traverse in the F direction. The maximum number of feature data groups for the multiplication and accumulation operation is 16. Assuming that the total number of groups of feature data (horizontal size) is 32, two multiplication and accumulation operations are required. After the F direction loop is completed, traverse in the E direction. The last multiplication and accumulation operation uses the CDP 0001 111 instruction to write the result of the current operation back to the local cache, move the convolution kernel, and repeat the above convolution operation until the convolution results of all channels are obtained.

[0076] In addition, it should be noted that the convolution operator (multiplication-accumulation operation) started by the CDP 0001 011 instruction does not have a write-back function, that is, the result will be stored in the temporary cache and will not be written back to the local cache, and can be used as the initial value of the next multiplication-accumulation operation.

[0077] Specific examples are as follows:

[0078] Figure 4 This is a schematic diagram of a specific multiplication-accumulation operation without write-back function, such as Figure 4 As shown, the operation mode DLA_Control[0] is configured to 1 (INT16*INT16+INT32), and the write-back precision DLA_Control[1] is configured to 0 (16 bits). The local cache width is 16 bits, so each address corresponds to a 16-bit data.

[0079] Each operation will start from the given weight data address to take 64Bytes of weight data, that is, 32 numbers (each data is 16bit), and take several groups of feature data with a granularity of 16Bytes from the starting address of the feature data (up to 16 groups, i.e. 256Bytes). Each group (8 numbers) of feature data will be multiplied with the 64Bytes of weight data in sequence and then added to get 4 intermediate results, and finally get [4*number of feature data groups] intermediate results. The intermediate results are stored in a temporary cache and used as the initial value for the next multiplication and accumulation operation.

[0080] Preferably, the overflow mode can also be configured to the DLA_Control register through the first MCR instruction. After configuration, the CDP 0001 111 instruction can be used to start the convolution operator (multiplication-accumulation operation) with write-back function, and write the final calculation result from the temporary cache back to the local cache.

[0081] In some embodiments, step S104, configuring the internal registers through the MCR instruction, and then starting the common basic operator through the CDP instruction further includes:

[0082] By means of the second MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information is configured to the scale register;

[0083] The Relu activation operator of the convolutional neural network is started through the CDP instruction, and the input data is input to the Relu activation function according to the stride block information. In, return the result value;

[0084] Write the result value back to the local cache according to the writeback information.

[0085] Specifically, the local cache address of the input data is configured to the DLA_ADDR1 register through the second MCR instruction, the local cache address of the write-back information is configured to the DLA_ADDR2 register, and the number of stride blocks is configured to the DLA_SIZE register;

[0086] The stride block information includes the number of stride blocks and the stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of feature data groups; the stride block size is fixed to 128 bits (16 bytes). Therefore, the amount of feature data for this operation is the number of stride blocks * stride block size, that is, DLA_SIZE[15:0]*16 bytes.

[0087] The Relu activation operator of the convolutional neural network is started through the CDP 0011 001 instruction, and the input data is input to the Relu activation function according to the number of stride blocks and the stride block size. In the function, the result value is returned, where e is a natural constant in mathematics and x is the input data.

[0088] Write the result value back to the local cache according to the writeback information.

[0089] In some embodiments, step S104, configuring the internal registers through the MCR instruction, and then starting the common basic operator through the CDP instruction further includes:

[0090] By means of a third MCR instruction, a local cache address of the first vector group is configured to a first register, a local cache address of the second vector group is configured to a second register, a local cache address of write-back information is configured to a third register, and stride block information is configured to a scale register;

[0091] The pooling operator of the convolutional neural network is started through the CDP instruction, and the values ​​in the first vector group and the second vector group are compared one by one according to the stride block information, and the vector with the larger value is returned each time the comparison is performed;

[0092] The maximum pooling result obtained by comparison is written back to the local cache according to the write-back information.

[0093] Specifically, the local cache address of the first vector group is configured to the DLA_ADDR1 register, the local cache address of the second vector group is configured to the DLA_ADDR2 register, the local cache address of the write-back information is configured to the DLA_ADDR3 register, and the number of stride blocks is configured to the DLA_SIZE register through the third MCR instruction;

[0094] The stride block information includes the number of stride blocks and the stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of feature data groups; the stride block size is fixed to 128 bits (16 bytes). Therefore, the amount of feature data for this operation is the number of stride blocks * stride block size, that is, DLA_SIZE[15:0]*16 bytes.

[0095] The pooling operator of the convolutional neural network is started by the CDP 0010 010 instruction. The values ​​in the first vector group and the second vector group are compared one by one according to the stride block information. Each comparison returns the vector with the larger value and writes the result back to the local cache. The element vector comparison operation can be used for the maximum pooling operation.

[0096] In addition, based on configuring the internal registers through the third MCR instruction, the CDP 0010001 instruction can be used to add the values ​​in the first vector group and the second vector group one by one according to the stride block information, and write the obtained result back to the local cache.

[0097] In some embodiments, step S104, configuring the internal registers through the MCR instruction, and then starting the common basic operator through the CDP instruction further includes:

[0098] By means of the fourth MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information and the table base address information are configured to the scale register;

[0099] The table lookup operator of the convolutional neural network is started through the CDP instruction, and the table lookup operation is performed according to the input data, stride block information and table base address information;

[0100] The table lookup results are written back to the local cache according to the write-back information.

[0101] Specifically, the local cache address of input data is configured to the DLA_ADDR1 register through the fourth MCR instruction, the local cache address of write-back information is configured to the DLA_ADDR2 register, and the number of stride blocks and table base address information are configured to the DLA_SIZE register;

[0102] The stride block information includes the number of stride blocks and the stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of feature data groups; the stride block size is fixed to 128Bit (16Bytes). Therefore, the amount of feature data for this operation is the number of stride blocks * stride block size, that is, DLA_SIZE[15:0] * 16Bytes; DLA_SIZE[31:16] is the 16-bit table base address.

[0103] The CDP 0100 000 instruction / CDP 0100 001 instruction / CDP 0100 010 instruction / CDP0100011 instruction can be used to respectively start four types of table lookup operators with table entry sizes of 64 items / 128 items / 256 items / 512 items, and perform table lookup operations according to input data, stride block information and table base address information;

[0104] It should be noted that before looking up the table, the table to be looked up needs to be written to a fixed local cache in advance, and then the table lookup operation is performed according to the input data and the table base address, and the result is written back to the local cache. In addition to Relu activation, other activation functions (such as tanh and sigmoid) can be implemented by table lookup operations. The table lookup method can realize a variety of different activation methods, which improves flexibility.

[0105] In some embodiments, step S104, configuring the internal registers through the MCR instruction, and then starting the common basic operator through the CDP instruction further includes:

[0106] By means of the second MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information is configured to the scale register;

[0107] The quantization operator of the convolutional neural network is started through the CDP instruction. The 32-bit single-precision floating-point number in the input data that complies with the IEEE-754 standard is converted into a 16-bit integer number according to the stride block information, or the 16-bit integer number in the input data is converted into a 32-bit single-precision floating-point number that complies with the IEEE-754 standard; the conversion result is written back to the local cache according to the write-back information.

[0108] Specifically, the local cache address of the input data is configured to the DLA_ADDR1 register through the second MCR instruction, the local cache address of the write-back information is configured to the DLA_ADDR2 register, and the number of stride blocks is configured to the DLA_SIZE register;

[0109] The stride block information includes the number of stride blocks and the stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of feature data groups; the stride block size is fixed to 128 bits (16 bytes). Therefore, the amount of feature data for this operation is the number of stride blocks * stride block size, that is, DLA_SIZE[15:0]*16 bytes.

[0110] The quantization operator of the convolutional neural network is started by the CDP 0011 010 instruction or the CDP 0011 011 instruction, and the 32-bit single-precision floating-point number in the input data that complies with the IEEE-754 standard is converted into a 16-bit integer number, or the 16-bit integer number in the input data is converted into a 32-bit single-precision floating-point number that complies with the IEEE-754 standard according to the stride block information.

[0111] Write the conversion result back to the local cache according to the write-back information.

[0112] In some embodiments, the method further comprises:

[0113] Through the fifth MCR instruction, the main memory address is configured to the first register, the local cache address is configured to the second register, and the stride block information is configured to the scale register;

[0114] The data read operation is initiated through the CDP instruction, and the data in the main memory address is read into the local cache according to the stride block information;

[0115] The data write operation is initiated through the CDP instruction, and the locally cached data is written to the main memory address according to the stride block information.

[0116] Specifically, the main memory address is configured to the DLA_ADDR1 register through the fifth MCR instruction; the local cache address is configured to the DLA_ADDR2 register; the number of stride blocks, the stride block interval and the stride block size are configured to the DLA_SIZE register.

[0117] The stride block information includes the number of stride blocks, the stride block interval and the stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of reads / writes; the stride block interval is DLA_SIZE[23:16], which indicates the interval size between reads / writes, with a granularity of 32 bits (4 bytes). 0 indicates continuous access, otherwise the actual stride size is (DLA_SIZE[23:16]+1)*4 bytes; the stride block size is DLA_SIZE[25:24], which indicates the number of reads / writes each time. When DLA_SIZE[25:24] is 2'd00, the block size is 4 bytes, when it is 2'd01, the block size is 8 bytes, and when it is 2'd10, the block size is 16 bytes. Therefore, the characteristic data volume of this read operation / write operation is the number of stride blocks*the stride block size, that is, DLA_SIZE[15:0]*DLA_SIZE[25:24].

[0118] The data read operation is started by the CDP 0000 000 instruction, and the data in the main memory address is read into the local cache according to the stride block information;

[0119] The data write operation is started by the CDP 0000 001 instruction, and the locally cached data is written to the main memory address according to the stride block information.

[0120] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0121] The embodiment of the present application provides a convolutional neural network acceleration system based on a Cortex-M processor, Figure 5 is a structural block diagram of a convolutional neural network acceleration method based on a Cortex-M processor according to an embodiment of the present application, such as Figure 5 As shown, the system includes an instruction set setting module 51 and an instruction set execution module 52;

[0122] The instruction set setting module 51 sets the MCR instruction and the CDP instruction according to the common basic operators of the convolutional neural network, wherein the common basic operators include a convolution operator, a Relu activation operator, a pooling operator, a table lookup operator, and a quantization operator;

[0123] The instruction set execution module 52 configures the internal registers of the convolutional neural network coprocessor through the MCR instruction, and then starts the common basic operator of the convolutional neural network through the CDP instruction.

[0124] Through the instruction set setting module 51 and the instruction set execution module 52 in the embodiment of the present application, the problems of inefficiency, high cost and inflexibility of the convolutional neural network algorithm in processor execution are solved.

[0125] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0126] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0127] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0128] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0129] In addition, in combination with the convolutional neural network acceleration method based on the Cortex-M processor in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by the processor, any one of the convolutional neural network acceleration methods based on the Cortex-M processor in the above embodiment is implemented.

[0130] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a convolutional neural network acceleration method based on a Cortex-M processor is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0131] In one embodiment, Figure 6 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 6 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 6 As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, the computer program is executed by the processor to implement a convolutional neural network acceleration method based on a Cortex-M processor, and the database is used to store data.

[0132] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0134] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A convolutional neural network acceleration method based on Cortex-M processor, characterized in that: The method comprises: Setting MCR instructions and CDP instructions according to common basic operators of convolutional neural networks, wherein the common basic operators include convolution operators, Relu activation operators, pooling operators, table lookup operators, and quantization operators; The MCR instruction is used to configure the data address, stride block information and format information of the internal register of the convolutional neural network coprocessor, wherein the data address is used for reading and writing data in the operation, the stride block information is used for blocking the data in the operation, and the format information is used to confirm the operation format and write-back format of the data; and then the CDP instruction is used to start the common basic operator of the convolutional neural network; Among them, configuring the data address, stride block information and format information of the internal register of the convolutional neural network coprocessor through the MCR instruction; and then starting the common basic operator of the convolutional neural network through the CDP instruction includes: Through the first MCR instruction, the local cache address of the convolution kernel is configured to the first register, the local cache address of the feature data is configured to the second register, the stride block information is configured to the scale register, and the format information is configured to the control register; and then the convolution operator of the convolutional neural network is started through the CDP instruction; Through the second MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information is configured to the scale register; and then the Relu activation operator of the convolutional neural network is started through the CDP instruction; By means of a third MCR instruction, a local cache address of the first vector group is configured to a first register, a local cache address of the second vector group is configured to a second register, a local cache address of write-back information is configured to a third register, and stride block information is configured to a scale register; and then a pooling operator of the convolutional neural network is started by means of the CDP instruction; By means of the fourth MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information and the table base address information are configured to the scale register; and then the table lookup operator of the convolutional neural network is started by means of the CDP instruction; Through the second MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information is configured to the scale register; and then the quantization operator of the convolutional neural network is started through the CDP instruction.

2. The method according to claim 1, characterized in that Then starting the convolution operator of the convolutional neural network through the CDP instruction includes: Starting the convolution operator through the CDP instruction, and determining the preset number of channels and the preset number of groups of the feature data in each operation according to the stride block information; According to the total number of channels of the feature data and the preset number of channels, performing multiplication and accumulation operations of the feature data and the convolution kernel in sequence according to the channel direction; In each channel of the feature data, according to the total number of groups of the feature data, the preset number of groups and the format information, multiplication and accumulation operations are performed on the feature data and the convolution kernel in sequence in a preset direction until the convolution results of all channels are obtained.

3. The method according to claim 1, characterized in that Then starting the Relu activation operator of the convolutional neural network through the CDP instruction includes: The Relu activation operator of the convolutional neural network is started by the CDP instruction, and the input data is input into the Relu activation function according to the stride block information. In the above example, the result value is returned, where e is a natural constant in mathematics and x is the input data; The result value is written back to the local cache according to the write-back information.

4. The method according to claim 1, characterized in that: Then starting the pooling operator of the convolutional neural network through the CDP instruction includes: Starting the pooling operator of the convolutional neural network through the CDP instruction, comparing the values ​​in the first vector group and the second vector group one by one according to the stride block information, and returning a vector with a larger value each time the comparison is performed; The maximum pooling result obtained by the comparison is written back to the local cache according to the write-back information.

5. The method according to claim 1, characterized in that Then starting the table lookup operator of the convolutional neural network through the CDP instruction includes: Starting a table lookup operator of the convolutional neural network through the CDP instruction, and performing a table lookup operation according to the input data, the stride block information and the table base address information; The table lookup results are written back to the local cache according to the write-back information.

6. The method according to claim 1, characterized in that Then starting the quantization operator of the convolutional neural network through the CDP instruction includes: The quantization operator of the convolutional neural network is started by the CDP instruction, and the 32-bit single-precision floating-point number that complies with the IEEE-754 standard in the input data is converted into a 16-bit integer number according to the stride block information, or the 16-bit integer number in the input data is converted into a 32-bit single-precision floating-point number that complies with the IEEE-754 standard; the conversion result is written back to the local cache according to the write-back information.

7. The method according to claim 1, characterized in that The method further comprises: Through the fifth MCR instruction, the main memory address is configured to the first register, the local cache address is configured to the second register, and the stride block information is configured to the scale register; Initiating a data read operation through the CDP instruction, and reading the data in the main memory address into the local cache according to the stride block information; The data write operation is initiated through the CDP instruction, and the locally cached data is written to the main memory address according to the stride block information.

8. A convolutional neural network acceleration system based on Cortex-M processor, characterized in that: The system includes an instruction set setting module and an instruction set execution module; The instruction set setting module sets the MCR instruction and the CDP instruction according to the common basic operators of the convolutional neural network, wherein the common basic operators include a convolution operator, a Relu activation operator, a pooling operator, a table lookup operator and a quantization operator; The instruction set execution module configures the data address, stride block information and format information of the internal register of the convolutional neural network coprocessor through the MCR instruction, wherein the data address is used for reading and writing data in the operation, the stride block information is used for blocking data in the operation, and the format information is used to confirm the operation format and write-back format of the data; and then starts the common basic operator of the convolutional neural network through the CDP instruction; Among them, configuring the data address, stride block information and format information of the internal register of the convolutional neural network coprocessor through the MCR instruction; and then starting the common basic operator of the convolutional neural network through the CDP instruction includes: Through the first MCR instruction, the local cache address of the convolution kernel is configured to the first register, the local cache address of the feature data is configured to the second register, the stride block information is configured to the scale register, and the format information is configured to the control register; and then the convolution operator of the convolutional neural network is started through the CDP instruction; Through the second MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information is configured to the scale register; and then the Relu activation operator of the convolutional neural network is started through the CDP instruction; By means of a third MCR instruction, a local cache address of the first vector group is configured to a first register, a local cache address of the second vector group is configured to a second register, a local cache address of write-back information is configured to a third register, and stride block information is configured to a scale register; and then a pooling operator of the convolutional neural network is started by means of the CDP instruction; By means of the fourth MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information and the table base address information are configured to the scale register; and then the table lookup operator of the convolutional neural network is started by means of the CDP instruction; Through the second MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information is configured to the scale register; and then the quantization operator of the convolutional neural network is started through the CDP instruction.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a convolutional neural network acceleration method based on a Cortex-M processor as described in any one of claims 1 to 7 is implemented.

Images