Recurrent neural network acceleration method, system and medium based on Cortex-M processor

By configuring MCR and CDP instructions on the Cortex-M processor to execute common basic operators of recurrent neural networks, the problems of inefficiency, high cost, and inflexibility of recurrent neural network algorithms in processor execution are solved, achieving flexible hardware acceleration and efficient data processing.

CN114298293BActive Publication Date: 2024-11-29HANGZHOU VANGO TECH
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202111641429.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-29
Publication Date
2024-11-29
Estimated Expiration
2041-12-29

AI Technical Summary

Technical Problem

Existing recurrent neural network algorithms suffer from inefficiency, high cost, and inflexibility when executed by processors, especially in collaborative processing between the cloud and the edge, where data transmission bandwidth and timeliness are low, high-performance MCUs are expensive, and dedicated hardware accelerators have fixed and inflexible structures.

Method used

A recurrent neural network acceleration method based on the Cortex-M processor is adopted. By setting the coprocessor registers through MCR and CDP instructions, common basic operators such as matrix multiplication, vector operation, Sigmoid activation and Tanh activation are executed. Local cache is used to improve data reuse rate, reduce main memory access bandwidth requirements, and transmit instructions through the CPU dedicated coprocessor interface to avoid bus blockage.

Benefits of technology

It enables the reduction of hardware reconfiguration costs, improvement of system efficiency, reduction of power consumption and cost, avoidance of bus latency, and flexible design to support hardware upgrades in application fields with diverse algorithms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114298293B_ABST
    Figure CN114298293B_ABST
Patent Text Reader

Abstract

The present application relates to a recurrent neural network acceleration method, system and medium based on a Cortex‑M processor, wherein the method comprises: setting MCR instructions and CDP instructions according to common basic operators of the recurrent neural network, wherein the common basic operators include matrix multiplication operators, vector operation operators, Sigmoid activation operators, Tanh activation operators and quantization operators; configuring the internal registers of the recurrent neural network coprocessor through the MCR instruction; based on the configured internal registers, starting the common basic operators of the recurrent neural network through the CDP instruction. Through the present application, the problems of low efficiency, high cost and inflexibility of the recurrent neural network algorithm in processor execution are solved, and the basic operators required for the execution of the recurrent neural network are realized through the coprocessor instruction set. For application fields with variable algorithms, the cost of reconstructing hardware can be reduced, and the power consumption and cost of the system are reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a recurrent neural network acceleration method, system and medium based on a Cortex-M processor. Background Art

[0002] With the continuous innovation of science and technology, new artificial intelligence algorithms emerge in an endless stream, which greatly improve the production efficiency of society and facilitate people's daily life. As one of the artificial intelligence network structures, recurrent neural networks have important applications in natural language processing (NLP), such as speech recognition, language modeling, text translation, etc., and are also often used in various time series predictions, such as weather forecasts, stock forecasts, etc. Compared with convolutional neural networks that focus on spatial expansion, that is, all inputs (including outputs) are independent of each other, recurrent neural networks focus on temporal expansion, that is, they can mine the temporal information and semantic information in the data, and each output depends on the previous calculation results to a certain extent. The basic operations in recurrent neural networks include matrix multiplication, vector multiplication, vector addition, Sigmoid activation, and Tanh activation.

[0003] Among the existing technical solutions, there is a method of sending the data to be processed to the cloud, and returning the result to the user end after the operation is completed. Its general workflow includes the steps of edge-side data collection, edge-side data transmission, cloud-side data reception, cloud-side data processing, cloud-side data transmission, and edge-side data reception. There are also methods of directly using high-performance MCU processors to directly process these operations or designing dedicated hardware accelerators. However, the collaborative processing between the cloud and the edge has bandwidth problems for data transmission and low timeliness; the high-performance MCU has high cost of use; and the hardware accelerator structure developed for specific algorithms is fixed and inflexible.

[0004] Currently, no effective solution has been proposed to address the problems of inefficiency, high cost and inflexibility of recurrent neural network algorithms in processor execution in related technologies. Summary of the invention

[0005] The embodiments of the present application provide a recurrent neural network acceleration method, system and medium based on a Cortex-M processor to at least solve the problems of inefficiency, high cost and inflexibility of recurrent neural network algorithms in processor execution in the related art.

[0006] In a first aspect, an embodiment of the present application provides a recurrent neural network acceleration method based on a Cortex-M processor, the method comprising:

[0007] Setting MCR instructions and CDP instructions according to common basic operators of the recurrent neural network, wherein the common basic operators include a matrix multiplication operator, a vector operation operator, a Sigmoid activation operator, a Tanh activation operator, and a quantization operator;

[0008] The internal registers of the recurrent neural network coprocessor are configured by the MCR instruction;

[0009] Based on the configured internal register, the common basic operator of the recurrent neural network is started through the CDP instruction.

[0010] In some embodiments, configuring the internal registers of the recurrent neural network coprocessor through the MCR instruction includes:

[0011] Through the first MCR instruction, the local cache address of the weight data is configured to the first register, the local cache address of the feature data is configured to the second register, the stride block information is configured to the scale register, and the operation mode and write-back precision are configured to the control register;

[0012] By means of a second MCR instruction, a local cache address of the first vector group is configured to the first register, a local cache address of the second vector group is configured to the second register, a local cache address of write-back information is configured to the third register, and stride block information is configured to the scale register;

[0013] Through the third MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information is configured to the scale register.

[0014] In some embodiments, after configuring the internal registers of the recurrent neural network coprocessor by the first MCR instruction, the method further includes:

[0015] Starting the matrix multiplication operator of the recurrent neural network through the CDP instruction, dividing the matrix of the feature data into blocks according to the stride block information, and dividing the matrix of the weight data into blocks according to a preset weight number;

[0016] According to the operation mode, corresponding multiplication and accumulation operations are performed on the blocked feature data matrix and the weight data matrix.

[0017] In some embodiments, after configuring the internal register of the recurrent neural network coprocessor by the second MCR instruction, the method further includes:

[0018] Starting a vector operation operator of the recurrent neural network through the CDP instruction, and adding or multiplying values ​​in the first vector group and the second vector group one by one according to the stride block information;

[0019] The operation result is written back to the local cache according to the write-back information.

[0020] In some embodiments, after configuring the internal registers of the recurrent neural network coprocessor by the third MCR instruction, the method further includes:

[0021] The Sigmoid activation operator of the recurrent neural network is started by the CDP instruction, and the input data is input into the Sigmoid activation function according to the stride block information. In, return the result value;

[0022] The result value is written back to the local cache according to the write-back information.

[0023] In some embodiments, after configuring the internal registers of the recurrent neural network coprocessor by the third MCR instruction, the method further includes:

[0024] The Tanh activation operator of the recurrent neural network is started by the CDP instruction, and the input data is input into the Tanh activation function according to the stride block information. In, return the result value;

[0025] The result value is written back to the local cache according to the write-back information.

[0026] In some embodiments, after configuring the internal registers of the recurrent neural network coprocessor by the third MCR instruction, the method further includes:

[0027] Starting the quantization operator of the recurrent neural network through the CDP instruction, converting the 32-bit single-precision floating-point number in the input data that complies with the IEEE-754 standard into a 16-bit integer number according to the stride block information, or converting the 16-bit integer number in the input data into a 32-bit single-precision floating-point number that complies with the IEEE-754 standard;

[0028] Write the conversion result back to the local cache according to the write-back information.

[0029] In some embodiments, the method further comprises:

[0030] Through the fourth MCR instruction, the main memory address is configured to the first register, the local cache address is configured to the second register, and the stride block information is configured to the scale register;

[0031] Initiating a data read operation through the CDP instruction, and reading the data in the main memory address into the local cache according to the stride block information;

[0032] The data write operation is initiated through the CDP instruction, and the locally cached data is written to the main memory address according to the stride block information.

[0033] In a second aspect, an embodiment of the present application provides a recurrent neural network acceleration system based on a Cortex-M processor, the system comprising an instruction set setting module and an instruction set execution module;

[0034] The instruction set setting module sets the MCR instruction and the CDP instruction according to the common basic operators of the recurrent neural network, wherein the common basic operators include a matrix multiplication operator, a vector operation operator, a Sigmoid activation operator, a Tanh activation operator and a quantization operator;

[0035] The instruction set execution module configures the internal registers of the recurrent neural network coprocessor through the MCR instruction;

[0036] The instruction set execution module starts the common basic operator of the recurrent neural network through the CDP instruction based on the configured internal register.

[0037] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the recurrent neural network acceleration method based on a Cortex-M processor as described in the first aspect above.

[0038] Compared with the related art, the embodiments of the present application provide a recurrent neural network acceleration method, system and medium based on a Cortex-M processor, which sets MCR instructions and CDP instructions according to the common basic operators of the recurrent neural network, wherein the common basic operators include matrix multiplication operators, vector operation operators, Sigmoid activation operators, Tanh activation operators and quantization operators; configures the internal registers of the recurrent neural network coprocessor through the MCR instruction; based on the configured internal registers, starts the common basic operators of the recurrent neural network through the CDP instruction, thereby solving the problems of inefficiency, high cost and inflexibility of the recurrent neural network algorithm in processor execution.

[0039] Technical effect:

[0040] 1. Implemented the basic operators required to execute recurrent neural networks through the coprocessor instruction set, which can reduce the cost of hardware reconstruction for application fields with variable algorithms;

[0041] 2. Fetching data from the local cache through the coprocessor instruction set improves the reuse rate of local cache data, reduces the bandwidth requirement for the coprocessor to access the main memory, and thus reduces the power consumption and cost of the entire system;

[0042] 3. Processing artificial intelligence calculations through coprocessors, specifically transmitting instructions through the CPU's dedicated coprocessor interface, can avoid delays caused by bus congestion and improve system efficiency;

[0043] 4. The coprocessor instruction set in the present invention is flexibly designed and has a large reserved space, which facilitates the addition of additional instructions during hardware upgrades. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 is a flowchart of the steps of a recurrent neural network acceleration method based on a Cortex-M processor according to an embodiment of the present application;

[0046] Figure 2 It is a schematic diagram of a specific multiply-accumulate operation without write-back function;

[0047] Figure 3 It is a schematic diagram of the matrix multiplication operator operation of the recurrent neural network;

[0048] Figure 4 It is a structural block diagram of a recurrent neural network acceleration system based on a Cortex-M processor according to an embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application.

[0050] Description of the drawings: 41. Instruction set setting module; 42. Instruction set execution module. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0052] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0053] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0054] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0055] In the prior art, the simplest method is to directly use the MCU's processor to process the calculations of these recurrent neural networks. The existing ARM instruction set contains some simple independent operation instructions, which can perform some basic processing operations, but it is inefficient for large-scale operations such as matrix multiplication or complex operations such as Tanh activation. For example, each time a matrix multiplication operation is performed, many instructions need to be repeated and cannot be operated in parallel, so the efficiency is very low when processing a large number of operations. For example, it takes more than four hundred clock cycles to calculate Tanh activation (data format is single-precision floating point) using the math.h library.

[0056] On the one hand, there are dedicated hardware accelerators designed to handle these operations. Using application-specific integrated circuits (ASICs) to build dedicated hardware accelerators can significantly improve computing efficiency. Dedicated Tanh hardware accelerators only take dozens of clock cycles to calculate Tanh activations. However, there are many variants of recurrent neural networks (LSTM, GRU, etc.), and different application scenarios require different network structures. Designing corresponding hardware accelerators for each structure will incur high costs.

[0057] On the other hand, the data to be processed is sent to the cloud, and the result is returned to the user after the operation is completed. Its general workflow includes edge data collection, edge data transmission, cloud data reception, cloud data processing, cloud data transmission, edge data reception, etc. However, the use of cloud computing will generate bandwidth costs and delay problems for long-distance transmission. In some occasions with high real-time requirements, such as the use of deep learning in industry to detect arc occurrence, it is necessary to identify the arc as quickly as possible and cut off the power supply to protect electrical equipment. Excessive delay will increase the occurrence of danger, so the cloud computing solution has certain limitations.

[0058] Therefore, in order to realize a recurrent neural network accelerator that can work on an MCU and has a certain degree of flexibility, the present invention proposes a lightweight recurrent neural network coprocessor instruction set, which can realize matrix multiplication, vector multiplication, vector addition, Sigmoid activation, Tanh activation and quantization operators in recurrent neural networks, support different algorithms without redesigning the hardware structure, and meet the MCU's requirements for timeliness.

[0059] The present application embodiment provides a recurrent neural network acceleration method based on a Cortex-M processor. Figure 1 is a flowchart of the steps of the recurrent neural network acceleration method based on the Cortex-M processor according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0060] Step S102, setting MCR instructions and CDP instructions according to common basic operators of the recurrent neural network, wherein the common basic operators include a matrix multiplication operator, a vector operation operator, a Sigmoid activation operator, a Tanh activation operator, and a quantization operator;

[0061] Specifically, Table 1 is a CDP instruction set of the recurrent neural network coprocessor. As shown in Table 1, each CDP instruction corresponds to two operands and corresponding instruction functions.

[0062] Table 1

[0063] Operand 1 Operand 2 Function 0000 000 Read main memory data to local cache operation 0000 001 Write local cache data to main memory operation 0001 011 Multiply-accumulate operation without write-back 0001 111 Multiply-accumulate operation with write-back 0010 001 Vector multiplication 0010 010 Vector addition 0011 001 Sigmoid activation operation 0011 010 Tanh activation operation 0011 011 32-bit single-precision floating point number (FP32) to 16-bit integer number (INT16) calculation 0011 100 16-bit integer (INT16) to 32-bit single-precision floating point (FP32) calculation

[0064] Step S104, configuring the internal registers of the recurrent neural network coprocessor through the MCR instruction;

[0065] Step S106, based on the configured internal registers, start the common basic operator of the recurrent neural network through the CDP instruction.

[0066] Through steps S102 to S106 in the embodiment of the present application, the problems of low efficiency, high cost and inflexibility of the recurrent neural network algorithm in processor execution are solved. The basic operators required for executing the recurrent neural network are realized through the coprocessor instruction set, which can reduce the cost of reconstructing hardware for application fields with variable algorithms; the coprocessor instruction set is used to fetch data from the local cache, which improves the reuse rate of the local cache data and reduces the bandwidth requirement for the coprocessor to access the main memory, thereby reducing the power consumption and cost of the entire system; the coprocessor is used to process artificial intelligence operations, and the instruction transmission is carried out specifically through the CPU-specific coprocessor interface, which can avoid the delay problem caused by bus blocking and improve system efficiency; the coprocessor instruction set is flexibly designed and has a large reserved space, which is convenient for adding additional instructions when the hardware is upgraded.

[0067] In some embodiments, step S104, configuring the internal registers of the recurrent neural network coprocessor through the MCR instruction includes:

[0068] Through the first MCR instruction, the local cache address of the weight data is configured to the first register, the local cache address of the feature data is configured to the second register, the stride block information is configured to the scale register, and the operation mode is configured to the control register.

[0069] Specifically, the local cache address of the weight data is configured to the DLA_ADDR1 register through the first MCR instruction; the local cache address of the feature data is configured to the DLA_ADDR2 register; the number of stride blocks and the stride block interval are configured to the DLA_SIZE register; and the operation mode is configured to the DLA_Control register.

[0070] The stride block information includes the number of stride blocks, stride block interval and stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of feature data groups; the stride block interval is DLA_SIZE[23:16], which indicates the interval size between each set of feature data, with a granularity of 128 bits (16 bytes). Configuring it to 0 indicates continuous access, otherwise the actual stride size is (DLA_SIZE[23:16]+1)*16 bytes; the stride block size is fixed to 128 bits (16 bytes). Therefore, the amount of feature data for this operation is the number of stride blocks * stride block size, that is, DLA_SIZE[15:0]*16 bytes. In addition, the number of weights for each operation is fixed to 512 bits (64 bytes).

[0071] The operation mode is DLA_Control[0]. When configured to 0, it indicates that the multiplication and accumulation unit is in the mode of 8-bit integer multiplication and 16-bit integer addition (INT8*INT8+INT16). When configured to 1, it indicates that the multiplication and accumulation unit is in the mode of 16-bit integer multiplication and 32-bit integer addition (INT16*INT16+INT32). The write-back precision is DLA_Control[1]. When configured to 0, it writes back with 8 bits in operation mode 0 and writes back with 16 bits in operation mode 1. When configured to 1, it writes back with 16 bits in operation mode 0 and writes back with 32 bits in operation mode 1.

[0072] Once configured, you can use the CDP 0001 011 instruction to start a multiply-accumulate operation without write-back.

[0073] It should be noted that the "without write-back function" here means that the result obtained will be stored in the temporary cache and will not be written back to the local cache, and can be used as the initial value of the next multiplication and accumulation operation.

[0074] Specific examples are as follows:

[0075] Figure 2 This is a schematic diagram of a specific multiplication-accumulation operation without write-back function, such as Figure 2 As shown, the operation mode DLA_Control[0] is configured to 1 (INT16*INT16+INT32), and the write-back precision DLA_Control[1] is configured to 0 (16 bits). The local cache width is 16 bits, so each address corresponds to a 16-bit data.

[0076] Each operation will start from the given weight data address to take 64Bytes of weight data, that is, 32 numbers (each data is 16bits), and take several groups of feature data with a granularity of 16Bytes from the starting address of the feature data (up to 16 groups, i.e. 256Bytes). Each group (8 numbers) of feature data will be multiplied with the 64Bytes of weight data in sequence and then added to get 4 intermediate results, and finally get [4*number of feature data groups] intermediate results. The intermediate results are stored in a temporary cache and used as the initial value for the next multiplication and addition operation.

[0077] Preferably, on the basis of the above, the overflow mode can also be configured to the DLA_Control register through the first MCR instruction. After configuration, the CDP 0001 111 instruction can be used to start the multiplication and accumulation operation with write-back function, and the final calculation result can be written back from the temporary cache to the local cache.

[0078] Through the second MCR instruction, the local cache address of the first vector group is configured to the first register, the local cache address of the second vector group is configured to the second register, the local cache address of the write-back information is configured to the third register, and the stride block information is configured to the scale register.

[0079] Specifically, the local cache address of the first vector group is configured to the DLA_ADDR1 register, the local cache address of the second vector group is configured to the DLA_ADDR2 register, the local cache address of the write-back information is configured to the DLA_ADDR3 register, and the number of stride blocks is configured to the DLA_SIZE register through the second MCR instruction;

[0080] The stride block information includes the number of stride blocks and the stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of feature data groups; the stride block size is fixed to 128 bits (16 bytes). Therefore, the amount of feature data for this operation is the number of stride blocks * stride block size, that is, DLA_SIZE[15:0]*16 bytes.

[0081] After configuration, you can use the CDP 0010 001 instruction to start a vector multiplication operation; or you can use the CDP0010010 instruction to start a vector addition operation.

[0082] Through the third MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information is configured to the scale register.

[0083] Specifically, the local cache address of the input data is configured to the DLA_ADDR1 register, the local cache address of the write-back information is configured to the DLA_ADDR2 register, and the number of stride blocks is configured to the DLA_SIZE register through the third MCR instruction;

[0084] The stride block information includes the number of stride blocks and the stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of feature data groups; the stride block size is fixed to 128 bits (16 bytes). Therefore, the amount of feature data for this operation is the number of stride blocks * stride block size, that is, DLA_SIZE[15:0]*16 bytes.

[0085] After configuration, you can use the CDP 0011 001 instruction to start the Sigmoid activation operation; or you can use the CDP0011 010 instruction to start the Tanh activation operation. You can also use the CDP 0011 011 instruction or the CDP 0011 100 instruction to start the quantization operation.

[0086] In some embodiments, after configuring the internal registers of the recurrent neural network coprocessor by the first MCR instruction in step S104, the method further includes:

[0087] The matrix multiplication operator of the recurrent neural network is started through the CDP instruction, the matrix of feature data is divided into blocks according to the stride block information, and the matrix of weight data is divided into blocks according to the preset weight number;

[0088] According to the operation mode, corresponding multiplication and accumulation operations are performed on the blocked feature data matrix and weight data matrix.

[0089] Specifically, Figure 3 This is a schematic diagram of the matrix multiplication operator operation of the recurrent neural network, such as Figure 3 As shown, the matrix multiplication operator of the recurrent neural network is started by the CDP0001 011 instruction or the CDP 0001 111 instruction. Since the amount of data calculated by a single multiplication and accumulation instruction of the coprocessor is limited, the operation needs to be split to conform to the working mode of the hardware.

[0090] Matrix 1 is the weight data, and matrix 2 is the feature data. The size of each data in the two matrices is 32 bits. Since the stride block size (feature block size) is fixed at 128 bits, it is necessary to divide it into blocks with a granularity of 4. Matrix 2 is divided into 4*1 to obtain sixteen matrix blocks X11, X12...X27, X28; since the number of weights for each multiplication and addition operation is fixed at 512 bits, matrix 1 is divided into 4*4 to obtain four matrix blocks W11, W12, W21, and W22. The 4*4 matrix blocks are multiplied and added with the 4*1 matrix blocks in turn to obtain sixteen matrix blocks Z11, Z12...Z27, Z28, which are the final results of the matrix multiplication operator operation.

[0091] In some embodiments, after configuring the internal registers of the recurrent neural network coprocessor by the second MCR instruction in step S104, the method further includes:

[0092] Starting the vector operation operator of the recurrent neural network through the CDP instruction, adding or multiplying the values ​​in the first vector group and the second vector group one by one according to the stride block information;

[0093] Write the calculation results back to the local cache according to the write-back information.

[0094] Specifically, the vector addition operator of the recurrent neural network is started by the CDP 0010 010 instruction; or the vector multiplication operator of the recurrent neural network is started by the CDP0010 001 instruction;

[0095] According to the stride block information, the values ​​in the first vector group and the second vector group are added or multiplied one by one. The stride block information includes the number of stride blocks and the stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of feature data groups; the stride block size is fixed to 128Bits (16Bytes). Therefore, the amount of feature data for this operation is the number of stride blocks * stride block size, that is, DLA_SIZE[15:0]*16Bytes;

[0096] Write the calculation results back to the local cache according to the write-back information.

[0097] In some embodiments, after configuring the internal registers of the recurrent neural network coprocessor by the third MCR instruction in step S104, the method further includes:

[0098] The Sigmoid activation operator of the recurrent neural network is started through the CDP instruction, and the input data is input to the Sigmoid activation function according to the stride block information. In the above example, the result value is returned, where e is a natural constant in mathematics and x is the input data;

[0099] Write the result value back to the local cache according to the writeback information.

[0100] Specifically, the Sigmoid activation operator of the recurrent neural network is started through the CDP 0011 001 instruction;

[0101] Input data to the Sigmoid activation function according to the stride block information The result value is returned. The stride block information includes the number of stride blocks and the stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of feature data groups; the stride block size is fixed to 128Bits (16Bytes). Therefore, the amount of feature data for this operation is the number of stride blocks * stride block size, that is, DLA_SIZE[15:0]*16Bytes;

[0102] Write the result value back to the local cache according to the writeback information.

[0103] In some embodiments, after configuring the internal registers of the recurrent neural network coprocessor by the third MCR instruction in step S104, the method further includes:

[0104] The Tanh activation operator of the recurrent neural network is started through the CDP instruction, and the input data is input into the Tanh activation function according to the stride block information. In the above example, the result value is returned, where e is a natural constant in mathematics and x is the input data;

[0105] Write the result value back to the local cache according to the writeback information.

[0106] Specifically, the Tanh activation operator of the recurrent neural network is started through the CDP 0011 010 instruction;

[0107] Input data to the Tanh activation function according to the stride block information The result value is returned. The stride block information includes the number of stride blocks and the stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of feature data groups; the stride block size is fixed to 128Bits (16Bytes). Therefore, the amount of feature data for this operation is the number of stride blocks * stride block size, that is, DLA_SIZE[15:0]*16Bytes;

[0108] Write the result value back to the local cache according to the writeback information.

[0109] In some embodiments, after configuring the internal registers of the recurrent neural network coprocessor by the third MCR instruction in step S104, the method further includes:

[0110] The quantization operator of the recurrent neural network is started through the CDP instruction, and the 32-bit single-precision floating-point number in the input data that complies with the IEEE-754 standard is converted into a 16-bit integer number, or the 16-bit integer number in the input data is converted into a 32-bit single-precision floating-point number that complies with the IEEE-754 standard according to the stride block information;

[0111] Write the conversion result back to the local cache according to the write-back information.

[0112] Specifically, the quantization operator of the recurrent neural network is started by a CDP 0011 011 instruction or a CDP 0011 100 instruction;

[0113] According to the stride block information, the 32-bit single-precision floating-point number in the input data that complies with the IEEE-754 standard is converted into a 16-bit integer, or the 16-bit integer in the input data is converted into a 32-bit single-precision floating-point number that complies with the IEEE-754 standard. The stride block information includes the number of stride blocks and the stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of groups of feature data; the stride block size is fixed to 128Bits (16Bytes). Therefore, the amount of feature data for this operation is the number of stride blocks * stride block size, that is, DLA_SIZE[15:0]*16Bytes;

[0114] Write the conversion result back to the local cache according to the write-back information.

[0115] In some embodiments, the method further comprises:

[0116] Through the fourth MCR instruction, the main memory address is configured to the first register, the local cache address is configured to the second register, and the stride block information is configured to the scale register;

[0117] The data read operation is initiated through the CDP instruction, and the data in the main memory address is read into the local cache according to the stride block information;

[0118] The data write operation is initiated through the CDP instruction, and the locally cached data is written to the main memory address according to the stride block information.

[0119] Specifically, the fourth MCR instruction configures the main memory address to the DLA_ADDR1 register; configures the local cache address to the DLA_ADDR2 register; and configures the number of stride blocks, stride block interval, and stride block size to the DLA_SIZE register.

[0120] The stride block information includes the number of stride blocks, the stride block interval and the stride block size, where the number of stride blocks is DLA_SIZE[15:0], which indicates the number of reads / writes; the stride block interval is DLA_SIZE[23:16], which indicates the interval size between reads / writes, with a granularity of 32 bits (4 bytes). 0 indicates continuous access, otherwise the actual stride size is (DLA_SIZE[23:16]+1)*4 bytes; the stride block size is DLA_SIZE[25:24], which indicates the number of reads / writes each time. When DLA_SIZE[25:24] is 2'd00, the block size is 4 bytes, when it is 2'd01, the block size is 8 bytes, and when it is 2'd10, the block size is 16 bytes. Therefore, the characteristic data volume of this read operation / write operation is the number of stride blocks*the stride block size, that is, DLA_SIZE[15:0]*DLA_SIZE[25:24].

[0121] The data read operation is started by the CDP 0000 000 instruction, and the data in the main memory address is read into the local cache according to the stride block information;

[0122] The data write operation is started by the CDP 0000 001 instruction, and the locally cached data is written to the main memory address according to the stride block information.

[0123] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0124] The embodiment of the present application provides a recurrent neural network acceleration system based on a Cortex-M processor, Figure 4 is a structural block diagram of a recurrent neural network acceleration system based on a Cortex-M processor according to an embodiment of the present application, such as Figure 4 As shown, the system includes an instruction set setting module 41 and an instruction set execution module 42;

[0125] The instruction set setting module 41 sets the MCR instruction and the CDP instruction according to the common basic operators of the recurrent neural network, wherein the common basic operators include a matrix multiplication operator, a vector operation operator, a Sigmoid activation operator, a Tanh activation operator and a quantization operator;

[0126] The instruction set execution module 42 configures the internal registers of the recurrent neural network coprocessor through the MCR instruction;

[0127] The instruction set execution module 42 starts the common basic operator of the recurrent neural network through the CDP instruction based on the configured internal registers.

[0128] Through the instruction set setting module 41 and the instruction set execution module 42 in the embodiment of the present application, the problems of low efficiency, high cost and inflexibility of the recurrent neural network algorithm in the processor execution are solved. The basic operators required for executing the recurrent neural network are realized through the coprocessor instruction set, which can reduce the cost of reconstructing the hardware for application fields with variable algorithms; the coprocessor instruction set is used to fetch data from the local cache, which improves the reuse rate of the local cache data, reduces the bandwidth requirement of the coprocessor to access the main memory, and thus reduces the power consumption and cost of the entire system; the coprocessor is used to process artificial intelligence operations, and the instruction transmission is carried out specifically through the CPU-specific coprocessor interface, which can avoid the delay problem caused by bus blocking and improve system efficiency; the coprocessor instruction set is flexibly designed and has a large reserved space, which is convenient for adding additional instructions when the hardware is upgraded.

[0129] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0130] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0131] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0132] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0133] In addition, in combination with the recurrent neural network acceleration method based on the Cortex-M processor in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by the processor, any one of the recurrent neural network acceleration methods based on the Cortex-M processor in the above embodiment is implemented.

[0134] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a recurrent neural network acceleration method based on a Cortex-M processor is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0135] In one embodiment, Figure 5 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 5 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 5 As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, the computer program is executed by the processor to implement a recurrent neural network acceleration method based on a Cortex-M processor, and the database is used to store data.

[0136] Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0138] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A recurrent neural network acceleration method based on Cortex-M processor, characterized in that: The method comprises: Setting MCR instructions and CDP instructions according to common basic operators of the recurrent neural network, wherein the common basic operators include a matrix multiplication operator, a vector operation operator, a Sigmoid activation operator, a Tanh activation operator, and a quantization operator; Through the first MCR instruction, the local cache address of the weight data is configured to the first register, the local cache address of the feature data is configured to the second register, the stride block information is configured to the scale register, and the operation mode and write-back precision are configured to the control register; By means of a second MCR instruction, a local cache address of the first vector group is configured to the first register, a local cache address of the second vector group is configured to the second register, a local cache address of write-back information is configured to the third register, and stride block information is configured to the scale register; By means of the third MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information is configured to the scale register; Through the fourth MCR instruction, the main memory address is configured to the first register, the local cache address is configured to the second register, and the stride block information is configured to the scale register; Based on the configured internal registers of the recurrent neural network coprocessor, the common basic operator of the recurrent neural network is started through the CDP instruction.

2. The method according to claim 1, characterized in that After configuring the internal registers of the recurrent neural network coprocessor by using the first MCR instruction, the method further includes: Starting the matrix multiplication operator of the recurrent neural network through the CDP instruction, dividing the matrix of the feature data into blocks according to the stride block information, and dividing the matrix of the weight data into blocks according to a preset weight number; According to the operation mode, corresponding multiplication and accumulation operations are performed on the blocked feature data matrix and the weight data matrix.

3. The method according to claim 1, characterized in that After configuring the internal register of the recurrent neural network coprocessor by the second MCR instruction, the method further includes: Starting a vector operation operator of the recurrent neural network through the CDP instruction, and adding or multiplying values ​​in the first vector group and the second vector group one by one according to the stride block information; The operation result is written back to the local cache according to the write-back information.

4. The method according to claim 1, characterized in that: After configuring the internal registers of the recurrent neural network coprocessor by the third MCR instruction, the method further includes: The Sigmoid activation operator of the recurrent neural network is started by the CDP instruction, and the input data is input into the Sigmoid activation function according to the stride block information. In the above example, the result value is returned, where e is a natural constant in mathematics and x is the input data; The result value is written back to the local cache according to the write-back information.

5. The method according to claim 1, characterized in that After configuring the internal registers of the recurrent neural network coprocessor by the third MCR instruction, the method further includes: The Tanh activation operator of the recurrent neural network is started by the CDP instruction, and the input data is input into the Tanh activation function according to the stride block information. In the above example, the result value is returned, where e is a natural constant in mathematics and x is the input data; The result value is written back to the local cache according to the write-back information.

6. The method according to claim 1, characterized in that After configuring the internal registers of the recurrent neural network coprocessor by the third MCR instruction, the method further includes: Starting the quantization operator of the recurrent neural network through the CDP instruction, converting the 32-bit single-precision floating-point number in the input data that complies with the IEEE-754 standard into a 16-bit integer number according to the stride block information, or converting the 16-bit integer number in the input data into a 32-bit single-precision floating-point number that complies with the IEEE-754 standard; Write the conversion result back to the local cache according to the write-back information.

7. The method according to claim 1, characterized in that The method further comprises: Initiating a data read operation through the CDP instruction, and reading the data in the main memory address into the local cache according to the stride block information; The data write operation is initiated through the CDP instruction, and the locally cached data is written to the main memory address according to the stride block information.

8. A recurrent neural network acceleration system based on Cortex-M processor, characterized in that: The system includes an instruction set setting module and an instruction set execution module; The instruction set setting module sets the MCR instruction and the CDP instruction according to the common basic operators of the recurrent neural network, wherein the common basic operators include a matrix multiplication operator, a vector operation operator, a Sigmoid activation operator, a Tanh activation operator and a quantization operator; The instruction set execution module configures the local cache address of the weight data to the first register, configures the local cache address of the feature data to the second register, configures the stride block information to the scale register, and configures the operation mode and write-back precision to the control register through the first MCR instruction; By means of a second MCR instruction, a local cache address of the first vector group is configured to the first register, a local cache address of the second vector group is configured to the second register, a local cache address of write-back information is configured to the third register, and stride block information is configured to the scale register; By means of the third MCR instruction, the local cache address of the input data is configured to the first register, the local cache address of the write-back information is configured to the second register, and the stride block information is configured to the scale register; Through the fourth MCR instruction, the main memory address is configured to the first register, the local cache address is configured to the second register, and the stride block information is configured to the scale register; The instruction set execution module starts the common basic operator of the recurrent neural network through the CDP instruction based on the internal register of the configured recurrent neural network coprocessor.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a recurrent neural network acceleration method based on a Cortex-M processor as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Programmable device, operating method thereof and computer usable medium

    CN108804139A

  • Neural network hardware accelerator

    CN111915003A

  • Lightweight artificial intelligence acceleration module

    CN112559043A

  • Loop-oriented neural network compilation

    US11144291B1

  • Hierarchical partitioning of operators

    US20210158131A1