Neural network hardware deployment component, architecture and multi-chip neural network hardware architecture

Abstract

The embodiment of the invention relates to a neural network hardware deployment component and architecture and a multi-chip neural network hardware architecture. The component comprises N acceleration modules, N data input paths and N data output paths, the acceleration module comprises a control signal input port, a data input port, a data output port and an arithmetic unit. The neural network hardware deployment component and architecture provided by the embodiment of the invention can be adapted to various neural network structures, and all layers of the neural network are connected and directly deployed on hardware, so that the flexibility of neural network deployment and the task execution efficiency of a hardware accelerator are improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of neural network technology, and in particular to a neural network hardware deployment component, architecture, and multi-chip neural network hardware architecture. Background Art

[0002] The edge deployment of neural networks is of great significance, especially for mobile smart devices that cannot connect to the Internet due to force majeure. The demand for embedded deployment of neural networks is even stronger.

[0003] Neural networks are made up of many "layers" connected together, and the connections between "layers" are flexible and diverse, mainly reflected in the fact that in addition to sequential connections, there is also a structure in which the results of multi-layer operations are first integrated and then sent to a layer for acceleration. Different "layer" connections will form different network structures, and different network structures have very different effects on the same task.

[0004] To adapt to the flexibility of the neural network structure, most hardware deployment work usually only deploys one layer of the network on the hardware, and then deploys the network layers in sequence on this single layer of hardware structure to perform accelerated operations. Specifically, first deploy the first layer of parameters on the hardware structure, load the input data to complete the first layer of operations; then reload the second layer of parameters on the hardware, load the second layer of input data to complete the second operation, and so on, until all layers are completed to complete a complete neural network operation process. Only then can the neural network operation of the second frame input be performed.

[0005] Although the above method can realize various network connection structures. For the serial structure, the operation results of the current layer are stored in the memory. After the parameters of the next layer are configured, the data in the memory is sent to the acceleration module as input for the acceleration operation of the next layer. For the integrated structure, the operation results of the layer to be connected are temporarily stored in the memory. When the integrated layer needs to be executed, the operation results of the previous layers stored in the external memory are pieced together as input to perform a layer of convolution operation. The disadvantages of this deployment method are also very obvious: first, it takes too long to change parameters; second, the layer-by-layer operation causes the entire network to be calculated before the calculation of the next frame task can be started; third, the data to be integrated will be completely stored in the external memory, and the input and output feature map data of the convolution layer is very large, which will consume a lot of storage space. In short, the efficiency of this deployment method is very low. Summary of the invention

[0006] Based on the above situation of the prior art, the purpose of the embodiments of the present invention is to provide a neural network hardware deployment component, architecture, and multi-chip neural network hardware architecture. By using this component and architecture, all layers of the neural network can be connected and directly deployed on the hardware, thereby improving the efficiency of the neural network hardware accelerator in performing tasks.

[0007] To achieve the above object, according to a first aspect of the present invention, a neural network hardware deployment component is provided, comprising N acceleration modules, N data input paths and N data output paths; the N data input paths are used to receive external input data, and the N data output paths correspond one to one to the N acceleration modules;

[0008] The acceleration module includes a control signal input port, a data input port, a data output port and an operation unit;

[0009] The data input port is configured to select at least one data input from the N data input paths and the data output ports of the N acceleration modules according to the configuration signal received by the control signal input port;

[0010] The operation unit is configured to operate on the input data of the data input port according to a preset first operation logic to obtain operation result data;

[0011] The data output port is configured to output the operation result data, and select whether to output the operation result data to the corresponding data output path according to the configuration signal received by the control signal input port;

[0012] Wherein, N is a natural number greater than or equal to 2.

[0013] Furthermore, the acceleration module also includes a cache unit, and the cache unit is used to store input data of the data input port.

[0014] Furthermore, the operation unit is also configured to, when there are multiple input data at the data input port, integrate the multiple input data, and operate on the integrated data according to a preset first operation logic to obtain operation result data.

[0015] Furthermore, the acceleration module also includes an enable port, and the enable port is used to control the enable state of the acceleration module according to an enable signal.

[0016] According to a second aspect of the present invention, there is provided a neural network hardware deployment architecture, comprising a data input interface, a data output interface and M neural network hardware deployment components, wherein the M neural network hardware deployment components are cascade-connected;

[0017] Among the M neural network hardware deployment components, the data input path of the first neural network hardware deployment component is connected to the data input interface, the data output path of the last neural network hardware deployment component is connected to the data output interface, and the data input paths and data output paths of any two adjacent neural network hardware deployment components are connected one-to-one;

[0018] Wherein, M is a natural number greater than or equal to 2, and the neural network hardware deployment component includes the neural network hardware deployment component as described in the first aspect of the present invention.

[0019] Further, it also includes a bus matrix control component and at least one cross-layer memory component;

[0020] The bus matrix control component is used to control the storage or reading of interaction data of any neural network hardware deployment component in the architecture in the at least one cross-layer memory component. Further, the neural network hardware deployment component also includes a cross-layer data input path;

[0021] The cross-layer memory component is configured to select at least one data to read and store from the data input path and the data output path of the M neural network hardware deployment components according to the first control information of the bus matrix control component; and output the stored data to at least one cross-layer data input path of the M neural network hardware deployment components according to the second control information of the bus matrix control component.

[0022] Furthermore, the data input port is also configured to select at least one data input from the N data input paths, cross-layer data input paths and data output ports of the N acceleration modules according to the configuration signal received by the control signal input port.

[0023] According to a third aspect of the present invention, there is provided a multi-chip neural network hardware deployment architecture, comprising at least two chips; each of the chips comprises a set of neural network hardware deployment architectures, a data receiving module and a data sending module; the data receiving module is used to receive external data, and the data sending module is used to output data to the outside; the output of the data receiving module is connected to the neural network hardware deployment architecture, and the input of the data sending module is connected to the neural network hardware deployment architecture;

[0024] The at least two chips are connected in cascade, and the data sending module of each chip is connected to the data receiving module of the next chip;

[0025] Wherein, the neural network hardware deployment architecture includes the neural network hardware deployment architecture as described in the second aspect of the present invention.

[0026] Further, the data receiving module is configured to transmit the received external data to the data input interface and / or any one or more cross-layer memory components of the neural network hardware deployment architecture according to the third control information of the bus matrix control component;

[0027] The data sending module is configured to output data of the data output interface of the neural network hardware deployment architecture and / or data of any one or more cross-layer memory components according to fourth control information of the bus matrix control component.

[0028] In summary, the embodiment of the present invention provides a neural network hardware deployment component, architecture, and multi-chip neural network hardware architecture, the component includes N acceleration modules, N data input paths and N data output paths; the acceleration module includes a control signal input port, a data input port, a data output port and an operation unit; the data input port is configured to select at least one data input from the N data input paths and the data output ports of the N acceleration modules according to the configuration signal received by the control signal input port; the operation unit is configured to operate the input data of the data input port according to the preset first operation logic to obtain the operation result data; the data output port is configured to output the operation result data, and according to the configuration signal received by the control signal input port, select whether to output the operation result data to the corresponding data output path. The neural network hardware deployment component and architecture provided by the embodiment of the present invention can adapt to various neural network structures, and directly deploy them on hardware by connecting all layers of the neural network, thereby improving the flexibility of neural network deployment and the efficiency of hardware accelerators in executing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of the structure of a neural network hardware deployment component provided by an embodiment of the present invention;

[0030] Figure 2 It is a schematic diagram of group convolution operation;

[0031] Figure 3 It is a mapping diagram of implementing a group convolution operation using a neural network hardware deployment component according to an embodiment of the present invention;

[0032] Figure 4 It is a schematic diagram of residual algorithm operation;

[0033] Figure 5 It is a mapping diagram of implementing a residual algorithm using a neural network hardware deployment component according to an embodiment of the present invention;

[0034] Figure 6is a schematic diagram of the structure of the neural network hardware deployment architecture provided by an embodiment of the present invention;

[0035] Figure 7 It is a mapping diagram of implementing a group convolution and residual block combination algorithm using a neural network hardware deployment architecture according to an embodiment of the present invention;

[0036] Figure 8 is a structural diagram of a neural network hardware deployment architecture provided by another embodiment of the present invention;

[0037] Fig. 9 It is a structural diagram of a neural network hardware deployment architecture using a bus matrix control component and a cross-layer memory component provided by an embodiment of the present invention;

[0038] Fig.10 It is a schematic diagram of the structure of a multi-chip neural network hardware deployment architecture provided by an embodiment of the present invention;

[0039] Fig.11 It is a schematic diagram of the algorithm operation of residual structure 1;

[0040] Fig.12 is a schematic diagram of a mapping of a residual structure 1 implemented using a neural network hardware deployment architecture having a bus matrix control component and a cross-layer memory component;

[0041] Fig.13 It is a schematic diagram of the mapping of residual structure 1 using a neural network hardware deployment architecture;

[0042] Fig.14 It is a schematic diagram of the algorithm operation of residual structure 2;

[0043] Fig.15 is a schematic diagram of a mapping of a residual structure 2 using a neural network hardware deployment architecture having a bus matrix control component and a cross-layer memory component;

[0044] Fig.16 It is a schematic diagram of the mapping of residual structure 2 using a neural network hardware deployment component;

[0045] Fig.17 It is a schematic diagram of mapping residual structure 2 using the acceleration module as a cache;

[0046] Fig.18 This is a schematic diagram of the c2f structural block operation in the Yolov8 network;

[0047] Fig.19 It is a schematic diagram of mapping the c2f structural blocks in the Yolov8 network using a neural network hardware deployment architecture having a bus matrix control component and a cross-layer memory component;

[0048] Fig. 20This is a schematic diagram of the ELAN structural block operation in the Yolov7 network;

[0049] Fig.21 It is a schematic diagram of mapping the ELAN building blocks in the Yolov7 network using a neural network hardware deployment architecture having a bus matrix control component and a cross-layer memory component;

[0050] Fig. 22 This is a schematic diagram of the RVB structural block operation in the Yolov6 network;

[0051] Fig.23 It is a schematic diagram of the mapping of the RVB structural blocks in the Yolov6 network using the neural network hardware deployment architecture. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0053] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in one or more embodiments of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0054] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. An embodiment of the present invention provides a neural network hardware deployment component, which includes N acceleration modules, N data input paths and N data output paths; the N data input paths are used to receive external input data, and the N data output paths correspond to the N acceleration modules one by one; wherein N is a natural number greater than or equal to 2. Those skilled in the art can set the value of N according to actual needs to realize the corresponding neural network function. Figure 1 FIG. 1 is a schematic diagram of the structure of the neural network hardware deployment component of an embodiment of the present invention. This embodiment takes a neural network hardware deployment component including four acceleration modules as an example to illustrate the composition structure of the neural network hardware deployment component of an embodiment of the present invention. Figure 1 As shown in FIG. 1 , the neural network hardware deployment component includes 4 acceleration modules (NL1, NL2, NL3 and NL4), 4 data input paths (i_dat1, i_dat2, i_dat3 and i_dat4) and 4 data output paths (o_dat1, o_dat2, o_dat3 and o_dat4). The 4 data input paths are used to receive external input data, and the 4 data output paths correspond to the 4 acceleration modules one by one, as shown in FIG. Figure 1 In the embodiment of the present invention, the data output path o_dat1 corresponds to the output of the acceleration module NL1, the data output path o_dat2 corresponds to the output of the acceleration module NL2, the data output path o_dat3 corresponds to the output of the acceleration module NL3, and the data output path o_dat4 corresponds to the output of the acceleration module NL4. In the embodiment of the present invention, the structure of each acceleration module is the same, including a control signal input port, a data input port, a data output port and an operation unit. The control signal input port is used to receive a control signal i_cfg, and the control signal i_cfg contains the configuration signal of all acceleration modules in the neural network hardware deployment component. Among them, the data input port is configured to select at least one data input from all data input paths and data output ports of all acceleration modules according to the configuration signal received by the control signal input port; the operation unit is configured to operate the input data of the data input port according to the preset first operation logic to obtain the operation result data; the data output port is configured to output the operation result data, and according to the configuration signal received by the control signal input port, select whether to output the operation result data to the corresponding data output path.

[0055] The data input port of the acceleration module selects at least one as input data of the data input port from all the data input paths of the neural network hardware deployment component and the data output ports of all the acceleration modules according to the configuration signal provided by the control signal i_cfg to the acceleration module. For example, in this embodiment, there are 4 data input paths and 4 acceleration modules. The data input port of each acceleration module can select at least one data as input data from the 4 data input paths (i_dat1, i_dat2, i_dat3 and i_dat4) and the data output ports of the 4 acceleration modules (NL1, NL2, NL3 and NL4), and perform acceleration operation on the input data through the computing unit of the acceleration module.

[0056] The operation units of the acceleration module are all deployed with a preset first operation logic, which is, for example, a basic operation, data processing, arithmetic logic, and a coding table. Among them, the basic operation can be an adder or a multiplier, or a combination of an adder and a multiplier, etc.; the data processing can be a processing splicing combination, an arithmetic operation combination, and a logical operation combination, etc.; the arithmetic logic can be a logical operation or an arithmetic operation, and a combination of a logical operation and an arithmetic operation, etc. The operation unit can operate on the input data of the data input port according to the preset first operation logic to obtain the operation result data.

[0057] The data output port of the acceleration module is used to output the operation result data. The data output port also selects whether to output the operation result data to the corresponding data output path according to the configuration signal received by the control signal input port. That is to say, the operation result data output by the data output port of each acceleration module may be transmitted in two directions: one direction is output from the corresponding data output path as the output data of the neural network hardware deployment component; the other direction is output to other acceleration modules inside the same neural network hardware deployment component, or the data input port of the acceleration module itself as input data. For example, the operation result data of the acceleration module NL1 can be used as the input source of the acceleration module NL2, the acceleration module NL3 and the acceleration module NL4; the operation result data of the acceleration module NL2 can be used as the input source of the acceleration module NL3 and the acceleration module NL4; the operation result data of the acceleration module NL3 can be used as the input source of the acceleration module NL4, etc. In this way, conventional neural network layer serial connections such as the acceleration module NL1 to the acceleration module NL2, the acceleration module NL2 to the acceleration module NL3, the acceleration module NL3 to the acceleration module NL4, and the connection of the acceleration module NL4 to the acceleration module NL3 can be realized.

[0058] According to some optional embodiments, the operation unit is further configured to, when there are multiple input data at the data input port, integrate the multiple input data, and perform operations on the integrated data according to a preset first operation logic to obtain operation result data. That is, when multiple input data are input to a certain data input port, the multiple input data can be integrated first, and then the corresponding acceleration operation can be performed on the integrated data. In various embodiments of the present invention, the integration of data includes data splicing and / or addition operations.

[0059] According to some optional embodiments, the acceleration module may further include a cache unit, which is used to store input data of the data input port. Therefore, the acceleration module can not only accelerate the network layer, but also be used as a cache function alone.

[0060] According to certain optional embodiments, the acceleration module may further include an enable port, which is used to control the enable state of the acceleration module according to an enable signal. The enable port can be used to enable the clock signal. For acceleration modules that are not needed, the clock signal of the acceleration module can be turned off through the enable port to reduce system power consumption. In applications that consider versatility, after the corresponding acceleration modules are assigned to each layer of the neural network, there will still be acceleration modules remaining in the hardware architecture. At this time, the clocks of these remaining acceleration modules are turned off through the enable port to ensure the low power consumption characteristics of the lightweight network. At the same time, the clock of the acceleration module assigned to the network layer does not need to be always on. It only needs to turn on the clock through the enable port at the time of participating in the operation, and keep the clock off during the waiting operation, so as to further reduce the power consumption of the hardware system. In the entire neural network, the size of the input data is the largest, and the middle network layer often has a downsampling process, which will result in a smaller amount of calculation for the later network layer. That is, the amount of calculation in the front layer is large, and the amount of calculation in the back layer is small. Therefore, the back layer needs to wait for the front layer to transmit enough data before it can start the calculation. So, under normal circumstances, the front layer can theoretically calculate continuously, but the back layer will have to wait for data. These acceleration modules can also reduce power consumption by enabling the port to turn off the clock while waiting for data.

[0061] The neural network hardware deployment component provided by the embodiment of the present invention can realize basic structures such as serial connection, parallel connection, and cross-layer connection. The combined application of these basic structures can support a variety of neural network connection structures. The following is a specific application example to illustrate the application implementation of the neural network hardware deployment component.

[0062] Application Example 1: Typical parallel and integrated computing combination application form (group convolution)

[0063] Taking group convolution operation as an example, Figure 2 The schematic diagram of the convolution operation is shown in Figure 2 As shown in the figure, the group convolution in this example includes two 3×3 convolution operations and one 1×1 convolution operation. The first input data input1 is subjected to a 3×3 convolution operation through the first layer Layer1, and the second input data input2 is subjected to a 3×3 convolution operation through the second layer Layer2. The operation results of the two convolution operations are subjected to a 1×1 convolution operation through the third layer Layer3 to obtain the operation result data output.

[0064] When the neural network hardware deployment component of the embodiment of the present invention is used for implementation, the mapping diagram of the above group convolution operation is as follows: Figure 3 As shown (the neural network algorithm mapping diagrams involved in each embodiment of the present invention only retain valid data flow connections). Figure 3As shown, three operations will use three acceleration modules, among which the first two 3×3 operations are accelerated in parallel by acceleration module NL1 and acceleration module NL2 respectively, and then the output results of these two acceleration modules are integrated in acceleration module NL3 to perform 1×1 operation.

[0065] Application Example 2: Typical combined application of serial, parallel, integrated, and cross-layer connections (residual blocks)

[0066] Figure 4 A typical residual algorithm operation diagram is shown in Figure 4 As shown, the input data input in the residual algorithm is not only input to the first layer Layer1, but also integrated with the output of the second layer Layer2 as the input of the third layer Layer3. In other words, the third layer Layer3 will not use the received input data input immediately, but needs to receive the output of the second layer Layer2 before starting the operation. While waiting for the output result of the second layer Layer2, the third layer Layer3 should store the received "input data" and start the operation after the output data of the second layer Layer2 is generated. If the amount of "input data" exceeds the storage capacity of the internal cache of the third layer Layer3, then before the second layer Layer2 generates output data, the cache of the third layer Layer3 will have an overflow error, and the above residual algorithm cannot be implemented.

[0067] When the neural network hardware deployment component of the embodiment of the present invention is used to implement the above residual algorithm, the above problems can be solved. Figure 5 FIG. 4 is a schematic diagram showing a mapping of a residual algorithm using a neural network hardware deployment component according to an embodiment of the present invention. Figure 5 As shown, in the neural network hardware deployment component, the first layer of the residual block is deployed in the acceleration module NL1, the second layer is deployed in the acceleration module NL2, and the third layer is deployed in the acceleration module NL3. The acceleration module NL4 is used to temporarily store the input data input. That is, the input data input in the residual block is simultaneously input to the acceleration module NL1 and the acceleration module NL4, the operation result data of the acceleration module NL1 is output to the acceleration module NL2, and the data in the acceleration module NL4 is sent to the acceleration module NL3 as needed and integrated with the output of the acceleration module NL2, thereby completing the network layer acceleration operation inside the acceleration module NL3 and realizing the deployment of the residual block. The data output port of the acceleration module NL3 outputs the operation result data o_dat3, and outputs the operation result data o_dat3 to the data output path corresponding to the neural network hardware deployment component.

[0068] An embodiment of the present invention further provides a neural network hardware deployment architecture. The neural network hardware deployment architecture provided by this embodiment of the present invention includes a data input interface, a data output interface and M neural network hardware deployment components, and the M neural network hardware deployment components are cascaded. Wherein, M is a natural number greater than or equal to 2, and the neural network hardware deployment component is, for example, the neural network hardware deployment component involved in the above embodiment of the present invention. In the neural network hardware deployment architecture, the M neural network hardware deployment components are cascaded and deployed to form a complete neural network. Among the M neural network hardware deployment components, the data input path of the first neural network hardware deployment component is connected to the data input interface, and the data output path of the last neural network hardware deployment component is connected to the data output interface. The data input path and the data output path of any two adjacent neural network hardware deployment components are connected one-to-one. Figure 6 The schematic diagram of the neural network hardware deployment architecture is shown in Figure 6 As shown, taking four neural network hardware deployment components as an example, the data input path of neural network hardware deployment component 601 is connected to the data input interface, which serves as the input of the neural network hardware deployment architecture; the data output path of neural network hardware deployment component 604 is connected to the data output interface, which serves as the output of the neural network hardware deployment architecture; the data output path of neural network hardware deployment component 601 is connected to the data input path of neural network hardware deployment component 602 in a one-to-one correspondence; the data output path of neural network hardware deployment component 602 is connected to the data input path of neural network hardware deployment component 603 in a one-to-one correspondence; the data output path of neural network hardware deployment component 603 is connected to the data input path of neural network hardware deployment component 604 in a one-to-one correspondence.

[0069] By cascading multiple neural network hardware deployment components to form the above neural network hardware deployment architecture, a neural network with multi-layer acceleration function can be realized. Taking the group convolution and residual blocks involved in the above embodiment as an example, the above neural network hardware deployment architecture can be used to combine the group convolution and residual blocks and deploy them as a complete neural network structure. Figure 7 FIG. 4 shows a schematic diagram of a mapping of a group convolution and residual block combination algorithm using a neural network hardware deployment architecture according to an embodiment of the present invention. Figure 7 As shown, a neural network hardware deployment component 701 and a neural network hardware deployment component 702 are used to implement a combination of group convolution and residual blocks, and the data output path of the neural network hardware deployment component 701 is correspondingly connected to the data input path of the neural network hardware deployment component 702. The data input path of the neural network hardware deployment component 701 is connected to the data input interface, and the data output path of the neural network hardware deployment component 702 is connected to the data output interface, thereby implementing a neural network structure of a combination of group convolution and residual blocks.

[0070] The functional implementation of the neural network also involves the integration of large cross-layer data, that is, when the above-mentioned neural network hardware deployment architecture is used, there is a data connection between two non-adjacent neural network hardware deployment components. For example, the implementation of the above-mentioned residual block involves the need to temporarily store cross-layer data when cross-layer data integration is involved. The more acceleration modules are separated between the generation and use of the integrated data, the greater the amount of data that needs to be stored for the integrated data. Therefore, the neural network hardware deployment architecture of this embodiment may also include a bus matrix control component and at least one cross-layer memory component. The bus matrix control component is used to control the storage or reading of the interactive data of any neural network hardware deployment component in the architecture in at least one cross-layer memory component. The cross-layer memory component is configured to select at least one data from the data input path and data output path of the M neural network hardware deployment components to read and store according to the first control information of the bus matrix control component; and, according to the second control information of the bus matrix control component, output the stored data to at least one cross-layer data input path of the M neural network hardware deployment components. Figure 8 FIG. 4 is a schematic diagram showing a structure of a neural network hardware deployment architecture provided by another embodiment of the present invention. Figure 8As shown, the neural network hardware deployment architecture includes four neural network hardware deployment components (801, 802, 803, 804), a bus matrix control component 805 and two cross-layer memory components (806, 807). Each neural network hardware deployment component also includes a cross-layer data input path i_kdat, which is used to transmit data read from the cross-layer memory component. Each neural network hardware deployment component and the cross-layer memory component are connected to the bus matrix control component 805, and data interaction is completed in the bus matrix control component 805, that is, the neural network hardware deployment component realizes the function of reading the cross-layer memory component through the bus matrix control component 805. The bus matrix control component 805 can determine which cross-layer memory component's data is read out and sent to which neural network hardware deployment component's cross-layer data input path according to the configuration information. Similarly, it can also determine which data in which neural network hardware deployment component output is written into the corresponding cross-layer memory component according to the configuration information. The configuration information can be configured by external software, for example. The connection between the neural network hardware deployment component and the cross-layer memory component is configured through the bus matrix control component 805, and the read and write signals of the cross-layer memory component are controlled, so as to realize data interaction between different neural network hardware deployment components. In this embodiment, the data received by the data input port of each acceleration module, in addition to the data of the data input path and the data output port of each acceleration module, can also select input data from the cross-layer data input path, that is, the data input port of each acceleration module can be configured to select at least one data input from N data input paths, cross-layer data input paths and data output ports of N acceleration modules according to the configuration signal received by the control signal input port.

[0071] Fig. 9 A schematic diagram of the structure of the neural network hardware deployment architecture using a bus matrix control component and a cross-layer memory component is shown in FIG. Fig. 9 As shown, the neural network hardware deployment architecture includes three neural network hardware deployment components (901, 902, 903), a bus matrix control component 904, and a cross-layer memory component 905. Under the control of the bus matrix control component 904, the cross-layer memory component 905 selects a data from the data output path of the neural network hardware deployment component 901 to read and store, and outputs the stored data to a cross-layer data input path of the neural network hardware deployment component 903.

[0072] Too many neural network layers in actual applications may cause the chip area to be too large. The neural network hardware deployment architecture involved in the above embodiments can be deployed in multiple chips, and the chips can be connected in series to achieve the deployment of the entire neural network. An embodiment of the present invention also provides a multi-chip neural network hardware deployment architecture, including at least two chips; each chip includes a set of neural network hardware deployment architecture, a data receiving module and a data sending module, and the neural network hardware deployment architecture can be the neural network hardware deployment architecture involved in the above embodiments of the present invention; at least two chips are cascaded, and the data sending module of each chip is connected to the data receiving module of the next chip; the data receiving module is used to receive external data, and the data sending module is used to output data to the outside; the output of the data receiving module is connected to the neural network hardware deployment architecture, and the input of the data sending module is connected to the neural network hardware deployment architecture. Fig.10 FIG. 1 is a schematic diagram showing a multi-chip neural network hardware deployment architecture according to an embodiment of the present invention. Fig.10 As shown, the multi-chip neural network hardware deployment architecture includes N chips (CHIP_1, ..., CHIP_N), each chip includes a set of neural network hardware deployment architecture, data receiving module and data sending module. Taking the first chip CHIP_1 as an example, a neural network hardware deployment architecture, a data receiving module 1401 and a data sending module 1402 are set on it, wherein the neural network hardware deployment architecture includes M neural network hardware deployment components, a bus matrix control component 1001 and multiple cross-layer memory components (1301, ..., 1302). The data receiving module and the data sending module are used to receive and send data between chips respectively. Considering the chip area, the above chip cascade structure can be used to deploy the neural network. The data receiving module receives the data of the previous chip and can selectively output the data to the first neural network hardware deployment component on the chip or the cross-layer memory component that stores data; the data sending module can selectively send the data generated by the last neural network hardware deployment component on the chip or the data in the cross-layer memory component to the next chip. That is, the data receiving module is configured to transmit the received external data to the data input interface and / or any one or more cross-layer memory components of the neural network hardware deployment architecture according to the third control information of the bus matrix control component; the data sending module is configured to output the data of the data output interface and / or any one or more cross-layer memory components of the neural network hardware deployment architecture according to the fourth control information of the bus matrix control component.

[0073] In practical applications, the hardware deployment of the neural network is to map the trained neural network to the hardware structure provided by the embodiment of the present invention. Since the neural network structure is diverse, the mapping to the hardware is also diverse. Not every acceleration module in the hardware structure will be used, and it needs to be mapped according to the specific neural network algorithm structure. The mapping of different residual blocks is a typical mapping case. For unnecessary acceleration modules, their clock signals can be turned off by enabling the port to reduce system power consumption.

[0074] The neural network hardware deployment architecture provided by the above embodiments can realize more complex and flexible neural network hardware deployment compared to the neural network hardware deployment components.

[0075] Application Example 3, Residual Structure 1

[0076] Fig.11 The schematic diagram of the algorithm operation of the residual structure 1 is shown in Figure 4 Compared with the residual structure 1 shown in , this example has one more layer of operation. The input data input is not only input to the first layer Layer1, but also integrated with the output of the third layer Layer3 as the input of the fourth layer Layer4.

[0077] The residual structure 1 can be implemented by a neural network hardware deployment architecture having a bus matrix control component and a cross-layer memory component. Fig.12 , a schematic diagram of the mapping of the residual structure 1 is shown in FIG. 1 using a neural network hardware deployment architecture having a bus matrix control component and a cross-layer memory component. Fig.12 As shown, the architecture includes a neural network hardware deployment component 1200, a cross-layer memory component 1202, and a bus matrix control component 1201, that is, the large cross-layer mode is used for deployment. Specifically, the input data input can be input to the acceleration module NL1 of the neural network hardware deployment component 1200 for the acceleration operation of the first layer Layer1, and at the same time input to the cross-layer memory component 1202 for temporary storage, the second layer Layer2 is deployed to the acceleration module NL2, the third layer Layer3 is deployed to the acceleration module NL3, and the operation result data of the acceleration module NL3 and the input data input cached in the cross-layer memory component 1202 are output to the acceleration module NL4 for the acceleration operation of the fourth layer Layer4.

[0078] The residual structure 1 can also be implemented by another neural network hardware deployment architecture. Fig.13 A schematic diagram of the mapping of the residual structure 1 using the neural network hardware deployment architecture is shown in FIG. Fig.13As shown, the neural network hardware deployment architecture includes a neural network hardware deployment component 1301 and a neural network hardware deployment component 1302. The input data input is input to the acceleration module NL1 of the neural network hardware deployment component 1301 for accelerating the operation of the first layer Layer1, and is also input to the acceleration module NL4 of the neural network hardware deployment component 1301 for temporary storage. The second layer Layer2 is deployed to the acceleration module NL2 of the neural network hardware deployment component 1301, and the third layer Layer3 is deployed to the acceleration module NL3 of the neural network hardware deployment component 1301. The operation result data of the acceleration module NL3 of the neural network hardware deployment component 1301 and the input data input cached in the acceleration module NL4 are output to the acceleration module NL1 of the neural network hardware deployment component 1302, and the fourth layer Layer4 is deployed in the acceleration module NL1 of the neural network hardware deployment component 1302.

[0079] Application Example 4, Residual Structure 2

[0080] Fig.14 The schematic diagram of the algorithm operation of the residual structure 2 is shown in the figure. The input data is input to the first layer Layer1 for accelerated operation. At the same time, after the accelerated operation of the fourth layer Layer4, it is integrated with the operation result data of the second layer Layer2 and used as the input of the third layer Layer3.

[0081] The residual structure 2 can be implemented by a neural network hardware deployment architecture having a bus matrix control component and a cross-layer memory component. Fig.15 , a schematic diagram of the mapping of the residual structure 2 is shown in FIG. 1 , which is implemented by a neural network hardware deployment architecture having a bus matrix control component and a cross-layer memory component. Fig.15 As shown, the architecture includes a neural network hardware deployment component 1500, a cross-layer memory component 1502, and a bus matrix control component 1501, that is, the large cross-layer mode is used for deployment. Specifically, the input data input is simultaneously input to the acceleration module NL1 and the acceleration module NL2, and the network layer acceleration operations of the fourth layer Layer4 (deployed to the acceleration module NL1) and the first layer Layer1 (deployed to the acceleration module NL2) are performed in parallel. The operation result data of the acceleration module NL1 is stored in the cross-layer memory component 1502, and the operation result data of the acceleration module NL2 is output to the acceleration module NL3 (deployment of the acceleration operation of the second layer Layer2), and the operation result data of the acceleration module NL3 and the operation result data of the acceleration module NL1 stored in the cross-layer memory component 1502 are input together into the acceleration module NL4 to complete the acceleration operation of the third layer Layer3.

[0082] If the cache space of the acceleration module is sufficient, the residual structure 2 can also be implemented through the neural network hardware deployment component. Fig.16 A schematic diagram of the mapping of the residual structure 2 using a neural network hardware deployment component is shown in FIG. Fig.16 As shown, the input data input is simultaneously input to the acceleration module NL1 and the acceleration module NL4 of the neural network hardware deployment component 1600, and the network layer acceleration operations of the first layer Layer1 (deployed to the acceleration module NL1) and the fourth layer Layer4 (deployed to the acceleration module NL4) are performed in parallel. The output data of the acceleration module NL4 is output to the acceleration module NL3 for temporary caching and waits for the output result of the acceleration module NL2, and then the acceleration operation of the third layer Layer3 is completed in the acceleration module NL3.

[0083] The deployment of the above residual structure reflects to a certain extent the flexibility of the neural network hardware deployment components and architecture connections of the embodiment of the present invention, and gives examples to illustrate the application methods of residuals (small cross-layers) and large cross-layers. Among them, large cross-layers refer to the need to use cross-layer memory components to achieve data interaction, and small cross-layers do not require cross-layer memory components but only require the cache unit inside the acceleration module to achieve data interaction. The number of cross-layers of large cross-layer connections is far greater than that of small cross-layer connections, but the deployment method is the same as the idea of ​​using cross-layers to implement hardware deployment of small cross-layers.

[0084] If the cache space of the acceleration module is insufficient, in addition to adopting the connection method using cross-layer memory components in the first implementation method mentioned above, the acceleration module can also be used as a cache. Fig.17 FIG. 2 shows a schematic diagram of a mapping of the residual structure 2 using an acceleration module as a cache. Fig.17 As shown, the neural network hardware deployment frame includes a neural network hardware deployment component 1701 and a neural network hardware deployment component 1702. The first layer Layer1, the second layer Layer2 and the fourth layer Layer4 are respectively deployed in the acceleration module NL1, the acceleration module NL2 and the acceleration module NL4 of the neural network hardware deployment component 1701, the operation result data of the fourth layer Layer4 is input into the acceleration module NL4 of the neural network hardware deployment component 1702 for caching, and the operation result data of the second layer Layer2 and the cache data of the acceleration module NL4 in the neural network hardware deployment component 1702 are input into the acceleration module NL3 of the neural network hardware deployment component 1702 for integration and network layer operation.

[0085] Application Example 5: c2f Building Blocks in Yolov8 Network

[0086] Fig.18 The schematic diagram of the c2f structural block operation in the Yolov8 network is shown in Fig.18As shown in the figure, the input data first passes through a convolution layer Layer1, and then the convolution result is split into two parts, one part is connected across layers via path ①, and the other part is connected across layers via paths ② and ③ and is also used as the input of the second layer Layer2. In the embodiment of the present invention, the "plus sign" and "concat" in the figure are generally referred to as cross-layer connections, and these operations can be designed in the acceleration module. The embodiment of the present invention only considers the connection relationship of the network layer, and does not consider the specific operation method in the acceleration module.

[0087] Fig.18 The c2f building block operation in the Yolov8 network shown in can be achieved by Fig.19 A neural network hardware deployment architecture with a bus matrix control component and a cross-layer memory component is used in the implementation, such as Fig.19As shown, the neural network hardware deployment architecture includes four neural network hardware deployment components (1901, 1902, 1903, 1904), a bus matrix control component 1905 and two cross-layer memory components (1906, 1907). Since the result of the convolution layer Layer1 needs to be split, it can be equivalently split in the operation stage and deployed in the two acceleration modules NL1 and NL2 of the neural network hardware deployment component 1901, and each acceleration module outputs half of the operation result data. The operation result data o_dat2 of the acceleration module NL2 in the neural network hardware deployment component 1901 is output to the acceleration module NL1 of the neural network hardware deployment component 1904 via the cross-layer memory component 1906, completing the connection of path ①; the operation result data o_dat1 of the acceleration module NL1 in the neural network hardware deployment component 1901 is output to the acceleration module NL1 of the neural network hardware deployment component 1904 via the cross-layer memory component 1907, completing the connection of path ②. The second layer Layer2 is deployed to the acceleration module NL1 of the neural network hardware deployment component 1902. The output data o_dat1 of the neural network hardware deployment component 1901 is not only used as the input of the acceleration module NL1 in the neural network hardware deployment component 1902, but also cached in the acceleration module NL4 of the neural network hardware deployment component 1902. The third layer Layer3 is deployed to the acceleration module NL2 of the neural network hardware deployment component 1902. The operation result data of the acceleration module NL2 of the neural network hardware deployment component 1902 and the cache data of the acceleration module NL4 are input to the acceleration module NL3 for integration and accelerated operation, and the operation result data o_dat3 is output to the input of the acceleration module NL1 and the acceleration module NL4 of the neural network hardware deployment component 1903. The acceleration modules NL1 and NL2 of the neural network hardware deployment component 1903 are used to deploy the fourth layer Layer4 and the fifth layer Layer5 respectively. The acceleration module NL4 is used to cache the data output data o_dat4. The operation result data of the acceleration modules NL2 and NL4 are output to the acceleration module NL3 for integration and accelerated operation, and the operation result data o_dat3 is output. The data of the last few cross-layer connection paths, i.e., the output data of the cross-layer memory component 1906, the output data of the cross-layer memory component 1907, the operation result data o_dat3, and the cached data o_dat4 are output to the acceleration module NL1 (deployment of the sixth layer Layer6) of the neural network hardware deployment component 1905 for integration and accelerated operation.

[0088] Application Example 6: ELAN Building Blocks in Yolov7 Network

[0089] Fig. 20The diagram of the ELAN block operation in the Yolov7 network is shown in Figure 1. The input data is split into two parts and enters the network block. One part is connected across layers after passing through the sixth layer Layer6, and the other part is used as the input of the first layer Layer1. The other connection relationships are as follows: Fig. 20 shown.

[0090] The ELAN building block can be implemented through a neural network hardware deployment architecture having a bus matrix control component and a cross-layer memory component. Fig.21 FIG. 1 shows a schematic diagram of a mapping of the ELAN structural blocks in the Yolov7 network using a neural network hardware deployment architecture having a bus matrix control component and a cross-layer memory component. Fig.21As shown, the architecture includes three neural network hardware deployment components (2101, 2102, 2103), two cross-layer memory components (2105, 2106), and a bus matrix control component 2104. Part of the input data input is input via the data input path i_dat1 of the neural network hardware deployment component 2101, and the other part is input via the data input path i_dat2 interface of the neural network hardware deployment component 2101. The first layer Layer1, the second layer Layer2, the third layer Layer3 and the sixth layer Layer6 are respectively deployed in the acceleration module NL1, the acceleration module NL2, the acceleration module NL3 and the acceleration module NL4 of the neural network hardware deployment component 2101. The operation result data o_dat1 of the acceleration module NL1 in the neural network hardware deployment component 2101 is output to the acceleration module NL3 of the neural network hardware deployment component 2103 via the cross-layer memory component 2106. The operation result data o_dat2 of the acceleration module NL4 in the neural network hardware deployment component 2101 is output to the acceleration module NL3 of the neural network hardware deployment component 2103 via the cross-layer memory component 2105. The fourth layer Layer4 and the fifth layer Layer5 are respectively deployed in the acceleration module NL1 and the acceleration module NL2 of the neural network hardware deployment component 2102. The operation result data o_dat3 of the acceleration module NL2 in the neural network hardware deployment component 2102 is output to the acceleration module NL3 of the neural network hardware deployment component 2103. The output of the acceleration module NL3 in the neural network hardware deployment component 2101 is cached in the acceleration module NL4 of the neural network hardware deployment component 2102. The data of the acceleration module NL4 of the neural network hardware deployment component 2102 is cached again in the acceleration module NL4 of the neural network hardware deployment component 2103, and then sent to the acceleration module NL3 (deployment of the seventh layer Layer7) of the neural network hardware deployment component 2103 together with other data, namely the output data of the cross-layer memory component 2105, the output data of the cross-layer memory component 2106, and the operation result data o_dat3 of the acceleration module NL2 for integration and accelerated calculation.

[0091] Application Example 7: RVB Building Blocks in Yolov6 Network

[0092] Fig. 22 The figure shows the operation diagram of the RVB structural block in the Yolov6 network. The input data input is simultaneously input to the first layer Layer1 and the second layer Layer2 for accelerated operation. After the operation result data is integrated, it is simultaneously output to the third layer Layer3 and the fourth layer Layer4 for accelerated operation. The accelerated operation result data is integrated together to obtain the output data output of the structural block.

[0093] The RVB building block can be implemented through the neural network hardware deployment architecture. Fig.23 The schematic diagram of the mapping of the RVB structural blocks in the Yolov6 network using the neural network hardware deployment architecture is shown in FIG. Fig.23 As shown, the architecture includes a neural network hardware deployment component 2301 and a neural network hardware deployment component 2302, and the first layer Layer1, the second layer Layer2, the third layer Layer3 and the fourth layer Layer4 are respectively deployed in the acceleration module NL1, the acceleration module NL2, the acceleration module NL3 and the acceleration module NL4 of the neural network hardware deployment component 2301. The operation result data o_dat3 and o_dat4 of the acceleration module NL3 and the acceleration module NL4 are input into the acceleration module NL1 of the neural network hardware deployment component 2302 for integration and output.

[0094] It can be seen from the above examples that the neural network connection structure provided by the embodiment of the present invention is very flexible and can realize various neural network algorithms. The above examples are for illustration only. Based on the technical solution provided by the embodiment of the present invention, the embodiment of the present invention can also support other network structures. In practical applications, the specific neural network algorithm structure can be reasonably mapped to the hardware connection structure of the embodiment of the present invention.

[0095] In summary, the embodiments of the present invention relate to a neural network hardware deployment component, architecture, and multi-chip neural network hardware architecture. The technical solution provided by the embodiments of the present invention aims to adapt to various neural network structures. The components and architecture provided by the embodiments of the present invention can flexibly deploy various neural network structures to the hardware architecture provided by the embodiments of the present invention through different combinations of various basic functions such as serial connection, parallel connection, cross-layer connection, cache in the acceleration module, cross-layer storage, etc. As for whether it is necessary to design a cache inside the acceleration module on the hardware, and the number of cross-layer storage components, they are all determined according to the actual network. Normally, the amount of data cached by the acceleration module is sufficient to support its network layer operation, but in actual applications, if you want the acceleration module to play a greater cache function, you can set the cache larger.

[0096] In tasks that consider versatility, it is often necessary to consider the trade-off between the versatility of the hardware structure and resource utilization. Once the hardware structure is determined, it cannot be changed. Therefore, in order to ensure that more neural network structures can be adapted as much as possible, situations where resource consumption is relatively large are usually considered. For example, considering the maximum number of layers of common neural networks, the hardware usually sets enough network layer units to ensure that neural networks with a variable number of layers can be adapted; considering the various connection structures of common neural networks, the hardware hopes to find a structure that can meet the connection requirements of the neural network algorithm as much as possible, so as to find a more universal hardware module when the complexity of the hardware structure control is not too large. The hardware deployment components and architectures provided in the embodiments of the present invention are such a hardware design with strong versatility. When the number is determined, the hardware is fixed. Different neural networks only need to configure appropriate parameters to map to the above architecture. For those acceleration modules that do not need to participate in the calculation, only their clocks need to be turned off. If only one neural network is involved in an application, the hardware structure can completely cut the connection of the hardware design according to this unique network structure without the need to switch the network, thereby ensuring high utilization of hardware resources. The above are two typical application ideas in practical applications, one focuses on the flexibility of hardware adaptation to neural network algorithms, and the other considers customizing hardware structures for unique application scenarios. The embodiments of the present invention support different application scenarios.

[0097] It should be understood that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present invention (including claims) is limited to these examples; under the idea of ​​the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of one or more embodiments of the present invention as described above, which are not provided in detail for the sake of simplicity. The above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the scope of protection of the present invention. In addition, the claims attached to the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries.

Claims

1. A neural network hardware deployment component, characterized in that: The system comprises N acceleration modules, N data input paths and N data output paths; the N data input paths are used to receive external input data, and the N data output paths correspond to the N acceleration modules one by one; The acceleration module includes a control signal input port, a data input port, a data output port and an operation unit; The data input port is configured to select at least one data input from the N data input paths and the data output ports of the N acceleration modules according to the configuration signal received by the control signal input port; The operation unit is configured to operate on the input data of the data input port according to a preset first operation logic to obtain operation result data; The data output port is configured to output the operation result data, and select whether to output the operation result data to the corresponding data output path according to the configuration signal received by the control signal input port; Wherein, N is a natural number greater than or equal to 2.

2. The assembly according to claim 1, characterized in that The acceleration module further includes a cache unit, which is used to store input data of the data input port.

3. The assembly according to claim 2, characterized in that The operation unit is further configured to, when there are multiple input data at the data input port, integrate the multiple input data, and operate on the integrated data according to a preset first operation logic to obtain operation result data.

4. The assembly according to any one of claims 1 to 3, characterized in that The acceleration module further includes an enable port, and the enable port is used to control the enable state of the acceleration module according to an enable signal.

5. A neural network hardware deployment architecture, characterized in that: It includes a data input interface, a data output interface and M neural network hardware deployment components, wherein the M neural network hardware deployment components are cascade-connected; Among the M neural network hardware deployment components, the data input path of the first neural network hardware deployment component is connected to the data input interface, the data output path of the last neural network hardware deployment component is connected to the data output interface, and the data input paths and data output paths of any two adjacent neural network hardware deployment components are connected one-to-one; Wherein, M is a natural number greater than or equal to 2, and the neural network hardware deployment component includes the neural network hardware deployment component as described in any one of claims 1-4.

6. The architecture according to claim 5, characterized in that Also included is a bus matrix control component and at least one cross-layer memory component; The bus matrix control component is used to control the storage or reading of interaction data of any neural network hardware deployment component in the architecture in the at least one cross-layer memory component.

7. The architecture according to claim 6, characterized in that The neural network hardware deployment component also includes a cross-layer data input path; The cross-layer memory component is configured to select at least one data to read and store from the data input path and the data output path of the M neural network hardware deployment components according to the first control information of the bus matrix control component; and output the stored data to at least one cross-layer data input path of the M neural network hardware deployment components according to the second control information of the bus matrix control component.

8. The architecture according to claim 7, characterized in that The data input port is also configured to select at least one data input from the N data input paths, the cross-layer data input paths, and the data output ports of the N acceleration modules according to the configuration signal received by the control signal input port.

9. A multi-chip neural network hardware deployment architecture, characterized in that: The invention comprises at least two chips; each of the chips comprises a set of neural network hardware deployment architecture, a data receiving module and a data sending module; the data receiving module is used to receive external data, and the data sending module is used to output data to the outside; the output of the data receiving module is connected to the neural network hardware deployment architecture, and the input of the data sending module is connected to the neural network hardware deployment architecture; The at least two chips are connected in cascade, and the data sending module of each chip is connected to the data receiving module of the next chip; The neural network hardware deployment architecture comprises the neural network hardware deployment architecture as described in any one of claims 6 to 8.

10. The architecture according to claim 9, characterized in that The data receiving module is configured to transmit the received external data to the data input interface and / or any one or more cross-layer memory components of the neural network hardware deployment architecture according to the third control information of the bus matrix control component; The data sending module is configured to output data of the data output interface of the neural network hardware deployment architecture and / or data of any one or more cross-layer memory components according to fourth control information of the bus matrix control component.

Images