Automobile cabin coprocessor based on heterogeneous multi-core processor

Abstract

The invention relates to an automobile cabin coprocessor based on a heterogeneous multi-core processor, so that the coprocessor occupies less resources, and the flexibility and universality are enhanced. The coprocessor comprises a convolutional neural network model, and the convolutional neural network model comprises a convolutional layer, a pooling layer, an activation function calculation layer and a full connection layer; the convolution layer is used for extracting the features of the input image and outputting a new feature map as the input image of the next layer, and the calculation process is that convolution is performed firstly and then an activation function is performed to obtain a result; the convolution layer is calculated by using a direct convolution method, and the direct convolution uses a convolution kernel to carry out multiplication and accumulation calculation with an image which is intercepted from the image and has the same size.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to an in-vehicle cockpit coprocessor based on a heterogeneous multi-core processor. Background Art

[0002] In the intelligent cockpit of a vehicle, a central control screen is used to display rich vehicle information, such as vehicle speed, engine speed, battery power, tire pressure, surround view image, etc. Usually, a coprocessor is used to process this information on the central control screen. However, information inside the vehicle, such as temperature, seat belt, driver posture, etc., is not processed to ensure driving safety.

[0003] A coprocessor is a processor developed and applied to assist the central processing unit in performing processing tasks that it cannot execute or execute with low efficiency and poor effectiveness.

[0004] With the explosion of big data and the leapfrog improvement of computer performance in recent years, the research on artificial intelligence has become popular again. Neural network algorithms have been widely applied to scenarios such as image processing, face recognition, and speech recognition. As the neural network structure continues to deepen, the requirements for the resources and computing power of the computing platform are getting higher and higher, making it increasingly difficult for general-purpose processors to perform operations on network models. Currently, the main solutions for accelerating neural network models are as follows:

[0005] 1) Using a graphics processing unit (GPU) to accelerate model calculations, but it has disadvantages such as high power consumption and difficulty in integration, and is only suitable for large computing platforms or cloud servers;

[0006] 2) Using an application-specific integrated circuit (ASIC) to accelerate model calculations, which has the characteristics of higher performance and lower power consumption, but has disadvantages such as a long design cycle and high development cost;

[0007] 3) Using a field-programmable gate array (FPGA) to accelerate model calculations, which has the characteristics of low power consumption, short development cycle, and high flexibility, and has gradually become a popular platform for current research on neural network acceleration.

[0008] These research works mainly focus on the exploration of the parallel design space, attempting to improve the parallelism of the design by using multi-computation units for calculation, data chunking and parallelism, pipelined computing, etc. These methods often consume a large amount of resources without fully considering the rational utilization of FPGA resources. There are mainly two types of neural network acceleration architectures based on FPGA: the data flow processing architecture and the computing unit architecture. The data flow processing architecture usually implements all layers of the neural network model on the FPGA. When data flows through the circuits implementing these layers, a calculation is completed. Since the circuit structure of each layer is carefully designed, resources can be flexibly configured for each layer to improve the computing performance. However, when calculating different neural network models, the FPGA needs to be reconfigured, and this method has high resource requirements for FPGA devices. It is difficult to deploy large neural network models. The accelerator of the computing unit architecture realizes the calculations of different layers in the neural network by configuring the computing units. The entire neural network calculation can be achieved according to a certain configuration order. When calculating different neural network models, only the configuration order needs to be changed. Compared with the data flow processing architecture, although it cannot achieve the best computing performance, it occupies less resources and has strong flexibility and generality.

[0009] How to provide a design scheme of a neural network processor based on a custom instruction set on the FPGA platform is a technical problem to be solved in this field. Summary of the Invention

[0010] The technical problem to be solved by the present invention is to provide an in-vehicle cockpit coprocessor based on a heterogeneous multi-core processor, so that the coprocessor can achieve less resource occupation and enhanced flexibility and generality.

[0011] A technical solution proposed by the present invention to solve the above technical problem is: to provide an in-vehicle cockpit coprocessor based on a heterogeneous multi-core processor, including: establishing a convolutional neural network model, and the convolutional neural network model includes: a convolutional layer, a pooling layer, activation function calculation, and a fully connected layer; the input of each layer of the convolutional neural network is a three-dimensional matrix, which is respectively defined as: length, width, and depth.

[0012] The convolutional layer is used to extract the features of the input image and output a new feature map as the input image of the next layer. The calculation process is to perform convolution first and then pass through an activation function to obtain the result; the convolutional layer uses the direct convolution method for calculation. The direct convolution uses a convolution kernel to perform multiplication and accumulation calculations with an image of the same size intercepted on the image. The mathematical expression for the calculation is:

[0013]

[0014] where f(*) is a non-linear activation function, β firepresents the offset value, O(x, y) represents the value at the coordinates (x, y) of the input feature map, w(kx, ky) represents the weight value on the coordinates (kx, ky) of the convolution kernel, and In(x + kx, y + ky) represents the value at the coordinates (x + kx, y + ky) of the output feature map; kx, ky represent the size of the convolution kernel, fi represents the i-th input feature map, and N fi represents the number of input feature maps.

[0015] The downsampling layer is placed after the convolutional layer and is divided into max pooling and average pooling according to different sampling specifications. The role of the pooling layer is to extract the local features of the feature map and reduce the dimensionality of the feature map, thereby reducing the computational time of the model. The expression for max pooling is Equation (2), and the expression for average pooling is Equation (3):

[0016]

[0017]

[0018] The f in Equation (3) out represents the value at the position (i, j) of the output image, and f in . represents the value at the position (i, j) in the input image, and P represents the size of the pooling kernel.

[0019] The activation function can perform a non-linear transformation on the input excitation. It is usually followed by the convolutional layer. Using the activation function can make the expression ability of the network more powerful, rather than simply performing a linear expression. Commonly used activation functions include the Sigmoid function, Tanh function, and ReLu function, and their expressions are Equation (4), Equation (5), and Equation (6) respectively:

[0020]

[0021]

[0022] Relu = max(0, x) (6).

[0023] The convolutional layer uses the method of direct convolution for calculation. Direct convolution requires windowing operations on the input image, and then multiplying and accumulating each element of the matrix intercepted by the window with the convolutional kernel point by point. The final accumulated value is used as the output value of one convolution. As the window slides over the image, the convolution calculation of the image is finally completed. The pooling calculation is similar to the convolution calculation and also requires windowing operations on the input image. However, the intercepted window matrix does not perform multiplication and accumulation calculations, but calculates the value of the window matrix according to the corresponding sampling rules. Since both convolution and pooling involve the windowing operation, the present invention multiplexes the circuit and controls the convolution and pooling functions through signals. To implement the windowing operation, the input image needs to be line-buffered first. After buffering the number of rows required for the window size, and then outputting by column, the windowing operation can be achieved. As the image data is serially input, the effect of windowing is finally achieved. Since the size of the input image for convolution and pooling calculations is uncertain, a conventional strip shift register cannot be used for line buffering. In the present invention, a dual-port RAM is used to implement the construction of the window. Figure 2 is the design block diagram of the convolution and pooling operation module. The dual-port RAMs are connected end to end. When one RAM buffers the number of data equal to the width of the input image, it starts to read data from the current RAM and write it into the next RAM. When the last RAM also buffers the number of data equal to the width of the input image, the data in all RAMs is output column by column one by one. At the same time, the image continues to be serially input, thus implementing the windowing operation. Since the image data is serially input, the data matrix after windowing is inverted with respect to the matrix that actually needs to be calculated. Therefore, the data matrix intercepted by windowing needs to be remapped once. The remapped data matrix is the correctly windowed matrix. The convolutional kernel configuration is controlled by the writing module according to relevant signals to configure the number of registers to form a convolutional kernel matrix, and the weight values read from the external memory are written into the register convolutional kernel matrix in sequence. When both the convolutional kernel and the data in the convolutional domain are stored in the corresponding registers, the register matrix storing the weights and the register matrix storing the image data are multiplied point by point, and the calculated values are stored in an adder tree register matrix. When the point-by-point multiplication operation is completed, a radix-2 adder tree operation is performed on the adder tree matrix, that is, the values of every two adjacent registers are added and placed in the adder tree register of the next layer. Through layer-by-layer addition operations, the sum of all values in the adder tree matrix is finally obtained, which is the result of one window convolution. Combining with the previous windowing operation, the convolution calculation of the image is finally achieved.

[0024] For the pooling operation, its windowing operation is the same as that of the convolution operation. Only by configuring the values in the convolution kernel can the pooling operation be realized. When performing the max pooling operation, the values in the convolution kernel register are all configured to 1 through the convolution kernel configuration module. At the same time, the radix-2 adder tree operation array no longer performs the operation of (a + b), but executes the operation max{a, b}; when performing the average pooling operation, the values in the convolution kernel register are all configured to 1 through the convolution kernel configuration module, and the radix-2 adder tree array performs the operation of (a + b). The final sum is divided by a fixed-point divider to obtain the quotient as the final pooling result. When performing convolution calculation and max pooling operation, the divisor of the divider is set to 1.

[0025] In order to make the representation ability of the convolutional neural network stronger, an activation function is usually used for non-linear mapping after convolution calculation. Common activation functions include: Sigmoid function, Tanh function, and ReLU function. The present invention designs an activation function calculation module with a pipeline structure based on the CORDIC algorithm. The most complex part in the activation function calculation process is the exponential calculation. Taking the exp exponential calculation as an example, the design principle of the activation function calculation module with a pipeline structure based on the CORDIC algorithm is described. The iterative expression of the CORDIC algorithm is Equation (7). Where θ k represents the angle of the k-th rotation, dk represents the direction of the k-th rotation, x and y represent coordinates, and z represents the angle value.

[0026]

[0027] The present invention has positive effects: The present invention proposes a design scheme of a neural network processor with a custom instruction set based on the FPGA platform. The instruction set includes 16 arithmetic instructions. The processor uses 128-bit-wide instructions to control and implement different calculation operations, multiplexes the calculation circuits for the convolution and pooling series, reduces the consumption of FPGA resources, uses the pipelined CORDIC algorithm [131 to implement the calculation of the activation function. At the same time, the present invention designs a multi-port read / write control module, and the processor realizes the read / write of bus data through this module. The present invention proposes a scheme for implementing a neural network coprocessor with an instruction set architecture on the FPGA platform. By parsing different neural network structures, calculation instructions can be generated, and the accelerated calculation of complex neural networks on the FPGA platform can be flexibly realized, shortening the deployment time of neural networks; and through the time-sharing multiplexing of the hardware circuit structure, the consumption of resources is reduced. In the scheme, Python and UVM verification methodologies are used to build a verification platform, and it is verified that the error of the instruction calculation is at the 10 -4 level, meeting the requirements of the operation. Through testing on the PYNQ-Z2 development board, the operation energy efficiency is 19.87 GOPS. W -1Reach the mainstream level of the same level of design, and the resources consumed are reduced by 80% on average compared with products of the same level. The data bit width used in the operations in this solution is 32 bits, in order to better reduce power consumption and improve the parallelism of calculations. Description of the Drawings

[0028] Figure 1 It is a block diagram designed for the convolution and pooling modules;

[0029] Figure 2 It is a block diagram for implementing the pipelined CORDIC algorithm;

[0030] Figure 3 It is the overall design framework of the coprocessor. Detailed Implementation Manner

[0031] Example 1

[0032] An in-vehicle cockpit coprocessor based on a heterogeneous multi-core processor, including

[0033] An in-vehicle cockpit coprocessor based on a heterogeneous multi-core processor, including: establishing a convolutional neural network model, and the convolutional neural network model includes: a convolutional layer, a pooling layer, activation function calculation, and a fully connected layer; the input of each layer of the convolutional neural network is a three-dimensional matrix, which is respectively defined as: length, width, and depth.

[0034] The convolutional layer is used to extract the features of the input image and output a new feature map as the input image of the next layer. The calculation process is to first perform convolution and then pass through an activation function to obtain the result; the convolutional layer uses the direct convolution method for calculation, and the direct convolution uses the convolution kernel to perform multiplication and accumulation calculations with the image of the same size intercepted on the image. The mathematical expression for the calculation is:

[0035]

[0036] where f(*) is a non-linear activation function, β fi represents the offset value, O(x, y) represents the value at the coordinate (x, y) of the input feature map, w(kx, ky) represents the weight value on the coordinate (kx, ky) of the convolution kernel, In(x + kx, y + ky) represents the value on the coordinate (x + kx, y + ky) of the output feature map; kx, ky represent the size of the convolution kernel, fi represents the i-th input feature map, and N fi represents the number of input feature maps.

[0037] The downsampling layer is placed after the convolutional layer and is divided into max pooling and average pooling according to different sampling specifications. The role of the pooling layer is to extract the local features of the feature map and reduce the dimension of the feature map, reducing the calculation time of the model. The expression for max pooling is Equation (2), and the expression for average pooling is Equation (3):

[0038]

[0039]

[0040] f in Equation (3) out represents the value at the position (i, j) of the output image, f in . represents the value at the position (i, j) in the input image, and P represents the size of the pooling kernel.

[0041] The activation function can perform non-linear transformation on the input excitation. It is usually followed after the convolutional layer. Using the activation function can make the network's expressive ability become more powerful, rather than simply being able to perform linear expression. Commonly used activation functions include the Sig-moid function, Tanh function, and ReLu function, and their expressions are Equation (4), Equation (5), and Equation (6) respectively:

[0042]

[0043]

[0044] Relu = max(0, x) (6).

[0045] The convolutional layer uses the direct convolution method for calculation. Direct convolution requires windowing operations on the input image, and then the matrix intercepted by the window is multiplied and accumulated point by point with the convolutional kernel. The final accumulated value is used as the output value of one convolution. As the window slides on the image, the convolution calculation of the image is finally completed. The pooling calculation is similar to the convolution calculation and also requires windowing operations on the input image, but the intercepted window matrix is no longer multiplied and accumulated, but the value of the window matrix is calculated according to the corresponding sampling rules. Since both convolution and pooling contain the windowing operation, the present invention multiplexes the circuit and controls through signals to implement the convolution and pooling functions. To implement the windowing operation, first, the input image needs to be line-buffered. After buffering the number of rows required for the window size, and then outputting by column, the windowing operation can be achieved. As the image data is serially input, the windowing effect is finally achieved. Since the size of the input image for convolution and pooling calculations is uncertain, a conventional strip shift register cannot be used for line buffering. In the present invention, a dual-port RAM is used to implement the construction of the window. Figure 2It is a design block diagram of the convolution and pooling operation module. The dual-port RAMs are connected end to end. After one RAM caches the data of the input image width, it starts to read the data from the current RAM and write it into the next RAM. After the last RAM also caches the data of the input image width, the data in all RAMs is output column by column one by one. At the same time, the image continues to be serially input, thus realizing the windowing operation. Since the data of the image is serially input, the data matrix after windowing is inverted with respect to the matrix that actually needs to be calculated. Therefore, the data matrix intercepted by windowing needs to be remapped once, and the data matrix after remapping is the correctly windowed matrix. The convolution kernel configuration is controlled by the writing module according to relevant signals to configure the number of registers to form a convolution kernel matrix, and the weight values read from the external memory are written into the register convolution kernel matrix in sequence. When the data of the convolution kernel and the convolution domain are stored in the corresponding registers, the register matrix storing the weights and the register matrix storing the image data are subjected to point-by-point multiplication calculation, and the calculated values are stored in an adder tree register matrix. After the point-by-point multiplication operation is completed, a radix-2 adder tree operation is performed on the adder tree matrix, that is, the values of every two adjacent registers are added and placed in the adder tree register of the next layer. Through layer-by-layer addition operations, the sum of all values in the adder tree matrix is finally obtained, which is the result of one window convolution. Combined with the previous windowing operation, the convolution calculation of the image is finally realized.

[0046] For the pooling operation, its windowing operation is the same as that of the convolution operation, and only the values in the convolution kernel need to be configured to realize the pooling operation. When performing the max pooling operation, the values in the convolution kernel register are all configured to 1 through the convolution kernel configuration module, and at the same time, the radix-2 adder tree operation array no longer performs the (a + b) operation, but performs the operation max{a, b}; when performing the average pooling operation, the values in the convolution kernel register are all configured to 1 through the convolution kernel configuration module, the radix-2 adder tree array performs the (a + b) operation, and the final accumulated sum is divided by a fixed-point divider to obtain the quotient as the final pooling result. When performing convolution calculation and max pooling operation, the divisor of the divider is set to 1.

[0047] In order to make the representation ability of the convolutional neural network stronger, an activation function is usually used for non-linear mapping after convolution calculation. Common activation functions include: Sigmoid function, Tanh function, and ReLU function. The present invention designs an activation function calculation module with a pipeline structure based on the CORDIC algorithm. The most complex part in the activation function calculation process is the exponential calculation. Taking the exp exponential calculation as an example, the design principle of the activation function calculation module with a pipeline structure based on the CORDIC algorithm is described. The iterative expression of the CORDIC algorithm is Equation (7). Where θ krepresents the angle of the k-th rotation, dk represents the direction of the k-th rotation, x and y represent coordinates, and z represents the angle value.

[0048]

[0049] The activation function calculation module with a pipeline structure based on the CORDIC algorithm includes: a rotation angle ROM table, an output correction coefficient ROM table, system control, and coordinate rotators at all levels. Among them, the rotation angle ROM table stores the rotation angles during CORDIC rotation. The output correction coefficient ROM table stores the correction coefficients for the output results. The module is mainly composed of a state machine. After receiving a reset signal, it first loads the rotation angles and correction coefficients in the ROM table. The coordinate rotators at all levels implement the rotation transformation of angles in the CORDIC algorithm. Through the coordinate rotations at different levels, the pipelining process of the CORDIC algorithm can be realized. By configuring the initialized x k , y k and the rotation angle values of each level in the CORDIC rotation angle table, the calculation of the exponential function can be realized.

[0050] The Sigmoid function and Tanh function in the convolutional neural network can be obtained through the exponential operations in Equation (4) and Equation (5). Therefore, in order to realize the pipelined calculation of the Sigmoid function and Tanh function, an additional pipelined divider is required. For different levels of pipelines, the resources consumed by calculating the activation function through the CORDIC algorithm and the achieved precision are different, and a trade-off needs to be made between precision and consumed resources. According to experimental tests, it is found that the effect of using the CORDIC algorithm with 10-level pipelines to implement the Sigmoid function and Tanh function is the best.

[0051] Design of the coprocessor for the instruction set architecture: In order to be able to accelerate the calculation of different types of convolutional neural networks, a neural network coprocessor based on the instruction set architecture is designed. Figure 3 is the overall block diagram of the processor. The processor uses instructions for control and calculation. The operation result of each instruction is in the form of a two-dimensional matrix. The operation results of each row are cached in the on-chip _ and finally written into the off-chip memory. The operation instructions of the processor need to be solidified in the on-chip RAM and can be modified and configured through the bus. The weight values, input image data, and intermediate operation results in the convolutional neural network are stored in the off-chip memory through the multi-port read-write control module.

[0052] Instruction Set and Instruction Executor Design: In a convolutional neural network, most operations can be classified into vector operations, scalar operations, and matrix operations. For example, operations such as addition, multiplication, and activation functions can be regarded as vector operations, matrix multiplication and dot product can be regarded as matrix operations, and addition and subtraction of immediate numbers can be regarded as scalar operations. Therefore, the present invention specifically breaks down the calculations of the convolutional neural network into instruction operations such as matrix addition and subtraction, multiplication, non-linear function mapping, convolution and pooling operations of two-dimensional images, and addition and subtraction of matrices and immediate numbers. For the convenience of calculation, all operation data in the present invention are 32-bit signed fixed-point operations, where the upper 16 bits represent the integer part and the lower 16 bits represent the fractional part. The instruction is designed to be 128 bits, and the instruction structure is shown in Table 1.

[0053] Table 1 Coprocessor instruction structure description

[0054] Table 1 Coprocessor instruction structure description

[0055]

[0056] Among them, [127:124] represents the operation code; [123:92] represents the starting address of the first input data parameter of the instruction;

[0057] [91:60] represents the starting address of the second input data parameter of the instruction; [59:28] represents the starting address of the output data parameter 3; [27:0] is used to represent operation parameters such as the size of the image and the size of the convolution kernel according to different instructions.

[0058] The present invention designs an instruction executor to control the sequential execution of instructions, without the need to access the external processor every time an instruction is executed. After the system is reset, only one instruction to start the operation needs to be sent to the neural network coprocessor, and the processor can execute all the instructions in the instruction RAM in sequence. The implementation of the instruction sequencer is divided into two steps: 1) Store the instructions sent from the outside into the instruction RAM; 2) After receiving the instruction to start the operation, obtain the instructions in the instruction RAM in sequence. When all the instructions are executed, an operation end signal is issued. The entire instruction sequencer design mainly consists of an instruction analysis module and a sequential execution state machine. The instruction analysis module needs to analyze the received instructions. If it is a start instruction, it notifies the sequential execution state machine to start the instruction scheduling of the neural network calculation module. After receiving the start command from the instruction analysis module, the instruction execution state machine will loop to give the address of the instruction, obtain the instruction, and schedule the calculation unit to start the calculation until all the instructions are executed.

[0059] The sequential execution state machine consists of 6 states: IDLE, READ, SEND, DELAY, WAIT, and ADDR. The descriptions of each state and the jump conditions are as follows: 1) When the system is reset, it enters the idle state IDLE; 2) In the IDLE state, if a start operation signal is received, it will jump to the READ state; 3) In the READ state, wait for 3 clocks to complete the instruction reading task. If the instruction is an invalid instruction, it will jump to the IDLE state, otherwise it will enter the SEND state; 4) In the SEND state, send the instruction to the calculation unit and give an instruction valid signal, and then enter the DELAY state; 5) In the DELAY state, wait for 5 clock cycles to allow the calculation unit to receive the instruction and the instruction valid signal, and then enter the WAIT state; 6) In the WAIT state, wait for the calculation unit to complete the calculation. After receiving the completion signal given by the calculation unit, it will enter the ADDR state; 7) In the ADDR state, increment the addressing address of the instruction RAM by 1, and then enter the READ state. 3.2 Design of the multi-port read / write controller Since the external DDR memory usually has only one bus interface, and there are multiple units in the system that need to read and write to the memory, and the access speeds of different units to the memory may also be different. The present invention designs a multi-port read / write controller based on an asynchronous FIFO, which multiplexes the single port of the DDR control in time division, so as to realize the read / write control function of multiple ports to one port, and there is a priority between different ports. The controller framework for 2 write ports and 2 read / write ports. To implement more ports, only the corresponding FIFO needs to be added. The write port buffer and the read port address buffer have 3 input signals, which are: address signal, request signal, and buffer margin signal; the read port data buffer has two output signals, which are data valid signal and data read complete signal. For the write port, an asynchronous FIFO with a width of 64 bits and a depth of 8 is used for data caching. In the FIFO, each storage space stores a 32-bit address and a 32-bit data; for the read port, an asynchronous FIFO with a width of 32 bits and a depth of 8 is used for caching. In the FIFO, each storage space stores the DDR address to be read. An asynchronous FIFO with a width of 32 bits and a depth of 64 is used to cache the data read from the DDR.

[0060] The read / write port selection control module is mainly composed of a state machine, and the jump logic of the state machine is as follows: 1) After the system is reset, the state machine enters state 0; 2) In state 0, first enable all read / write ports, and scan different read / write ports in sequence. If the asynchronous FIFO of a port is not empty, jump to different states 1 to 4 according to the port number; 3) In state 1, process the write request from write port #0, generate the corresponding write address and write request signal of DDR, and after completing the write operation, cancel the DDR write request, then shield port #0, enable other ports, and scan whether there are requests for read / write ports; 4) In state 2, process the read request from read port #1, generate the corresponding read address and read request signal of DDR, and after completing the data reading, cancel the DDR read request, then shield port #1, enable other ports, Scan whether there are requests for read / write ports; 5) In state 3, process the write request from write port #2, generate the corresponding write address and write request signal of DDR, cancel the DDR write request after completing the write operation, then shield port #2, enable other ports, and scan whether there are requests for read / write ports; 6) In state 4, process the read request from read port #3, generate the corresponding read address and read request signal of DDR, cancel the DDR read request after completing the data reading, then shield port #3, enable other ports, and scan whether there are requests for read / write ports; In order to distinguish whether the read data signal and the read data valid signal correspond to read port #1 or read port, a 4-bit wide and 64-deep asynchronous FIFO is used to cache the read port selection, and the operating clock is the same as the port selection control module clock. Use a multi-port controller to expand the read / write port, requiring each read / write port to initiate read / write requests with continuous addresses as much as possible to achieve the maximum DDR read / write bandwidth utilization.

[0061] Neural Network Computing Unit Design: The arithmetic logic in the coprocessor is implemented in the neural network computing unit. The computing unit consists of parts such as arithmetic control logic, instruction parsing logic, instruction arithmetic logic, and DDR read / write interface. If the computing unit is ready, it will send out a Ready signal indicating that it can receive arithmetic instructions. After receiving the instruction and the instruction valid signal, it parses the execution type of the instruction and the control logic controls the computing unit to start reading data and performing arithmetic operations on the instruction. Neural network computing unit design. The read / write address generation module is used to generate the addresses of the data in DDR during read / write operations and is implemented by the arithmetic control logic. The arithmetic control logic is mainly composed of a finite state machine and includes sub-state machines for performing different tasks. When the system is reset, the control logic is in an idle state. After receiving an instruction, it jumps to the corresponding sub-state machine of the task according to the encoding of the Opcode, and controls the reading, writing, and arithmetic operations on the data through the sub-state machine. To improve the arithmetic efficiency, read-write operations are performed while reading each task. When performing the double matrix point-by-point arithmetic task, first read a row of data of parameter 1, then read a row of data of parameter 2. At the same time, for each data of parameter 2 read, perform point-by-point arithmetic with the corresponding data of parameter 1 and cache the result. When all the data in a row is calculated, write the result back to DDR, and then start reading the next row of data until the entire matrix is completely read. When performing the single matrix point-by-point arithmetic task, just read the data of parameter 1 first, perform corresponding pipeline arithmetic, perform point-by-point arithmetic on a row of data and write it back to DDR at the same time. When a row of data is calculated, continue to read the next row of data until all the data of parameter 1 is calculated. When performing the matrix transpose task, to ensure that the written-back data is continuous by row, when reading data, preferentially read the data indicated by parameter 1 by column and directly write it back to DDR by row until the entire matrix is completely transposed. When performing the matrix multiplication task, first read a certain row of the data indicated by parameter 1, and then sequentially read each column of the data indicated by parameter 2. While reading the data indicated by parameter 2, perform multiply-accumulate arithmetic and cache the result. When all the columns of the data indicated by parameter 2 are read, write the arithmetic result back to DDR and start reading the next row of the data indicated by parameter 1 until all the data indicated by parameter 1 is read.

[0062] Obviously, the above embodiments are merely examples for clearly illustrating the embodiments of the present invention and are not limitations on the embodiments of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. And these obvious changes or modifications derived from the spirit of the present invention are still within the protection scope of the present invention.

Claims

1. An automotive cockpit coprocessor based on heterogeneous multi-core processors, Features: Including establishing a convolutional neural network model, which includes: convolution layer, pooling layer, activation function calculation, and fully connected layer; The convolution layer is used to extract the features of the input image and output a new feature map as the input image of the next layer. The calculation process is to first perform convolution and then pass an activation function to obtain the result; The convolution layer uses the direct convolution method for calculation. Direct convolution uses the convolution kernel to perform cumulative multiplication and accumulation calculations with images of the same size intercepted from the image. The mathematical expression for the calculation is: ； where f(*) is a non-linear activation function, β fi represents the offset value, O(x, y) represents the value at the input feature map coordinates (x, y), w(kx, ky) represents the weight value on the convolutional kernel coordinates (kx, ky), In(x + kx, y + ky) represents the value at the output feature map coordinates (x + kx, y + ky); kx, ky represent the size of the convolutional kernel, fi represents the i-th input feature map, N fi represents the number of input feature maps; The downsampling layer is placed after the convolution layer and is divided into maximum pooling and average pooling according to different sampling specifications. The function of the pooling layer is to extract local features of the feature map and reduce the dimension of the feature map to reduce the calculation time of the model. The expression of maximum pooling is formula (2), and the expression of average pooling is formula (3): ； f in Equation (3) out represents the value at the position (i, j) of the output image, f in . represents the value at the position (i, j) in the input image, and P represents the size of the pooling kernel.

2. The automotive cockpit coprocessor based on heterogeneous multi-core processor according to claim 2, Features: The activation function can perform nonlinear transformation on the input stimulus. It is usually followed by the convolution layer. Using the activation function can make the network's expression ability more powerful, rather than simply performing linear expression. Commonly used activation functions include Sigmoid function, Tanh function and ReLu function. Their expressions are respectively: (4), (5) and (6) 。 3. The automotive cockpit coprocessor based on heterogeneous multi-core processor according to claim 2, Features: The convolution kernel configuration is controlled by the write module according to the relevant signals, and the number of registers is configured to form a convolution kernel matrix. The weight values ​​read from the external memory are written into the register convolution kernel matrix in order. After the data of the convolution kernel and the convolution domain are stored in the corresponding registers, the register matrix storing the weights and the register matrix storing the image data are multiplied point by point, and the calculated values ​​are stored in an addition tree register matrix. After the point-by-point multiplication operation is completed, the addition tree matrix is ​​subjected to a radix 2 addition tree operation, that is, the values ​​of each two adjacent registers are added and placed in the addition tree register of the next layer. Through layer-by-layer addition operations, the sum of all values ​​in the addition tree matrix is ​​finally obtained, which is the result of a window convolution. With the previous window operation, the convolution calculation of the image is finally realized.

4. The automotive cockpit coprocessor based on heterogeneous multi-core processor according to claim 2, Features: For the pooling operation, its windowing operation is the same as that of the convolution operation. Only by configuring the values in the convolution kernel can the pooling operation be realized. When performing the max pooling operation, the values in the convolution kernel register are all configured to 1 through the convolution kernel configuration module. At the same time, the base-2 adder tree operation array no longer performs the operation of (a + b), but executes the operation max{a, b}; when performing the average pooling operation, the values in the convolution kernel register are all configured to 1 through the convolution kernel configuration module. The base-2 adder tree array performs the operation of (a + b), and the quotient of the final accumulated sum is obtained using a fixed-point divider as the final pooling result. When performing convolution calculation and max pooling operation, the divisor of the divider is set to 1.

Images