Computing core and data processing method

Abstract

The embodiment of the invention provides a calculation core and a data processing method, relates to the technical field of chips, and solves the problems that an existing storage and calculation integrated device is low in calculation resource utilization rate and high in hardware cost. According to the specific scheme, a calculation core comprises a plurality of convolution circuits and a plurality of multiplexers, each multiplexer comprises an output end and at least two input ends, the output ends of the multiplexers correspond to the input ends of the convolution circuits in a one-to-one mode, the output end of each convolution circuit in the convolution circuits is coupled with the input ends of the L multiplexers, l is an integer greater than or equal to 2, at least one of the L multiplexers further comprises an external data output end, and at least one of the L convolution circuits corresponding to the L multiplexers further comprises an external data input end. Wherein the multiplexer is used for conducting the output end of at least one convolution circuit coupled with the multiplexer to the output end of the multiplexer.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of chip technology, and in particular to a computing core and a data processing method. Background Art

[0002] With the development of artificial intelligence (AI) technology, more and more AI products are entering people's lives, providing people with various intelligent services.

[0003] The core of artificial intelligence technology includes two aspects: one is the neural network algorithm, and the other is the processor that provides massive hardware computing power. With the upgrade and iteration of deep learning neural network algorithms, especially the emergence of algorithms such as chat generative pre-trained transformer (ChatGTP) and transformer, the demand for hardware computing power has increased exponentially. The existing neural process unit (NPU) architecture adopts centralized computing and centralized storage, such as Figure 1 As shown, Figure 1 The figure shows the computing array and memory. The computing array stores the computing results in the memory, but the physical memory bandwidth limits the ability to read and write large bandwidth data at the same time, that is, the memory wall problem is serious. The memory wall problem limits the upper limit of the computing power of existing neural network processors, and existing neural network processors are difficult to meet the computing power requirements of future artificial intelligence technology. Summary of the invention

[0004] The embodiments of the present application provide a computing core and a data processing method, which can flexibly meet the depth of the convolutional neural network by coupling the output end of each convolution circuit with the input ends of L multiplexers, thereby improving the utilization rate of computing resources and reducing hardware costs.

[0005] In order to achieve the above-mentioned purpose, the embodiment of the present application adopts the following technical solution.

[0006] In a first aspect, an embodiment of the present application provides a computing core, which includes: multiple convolution circuits and multiple multiplexers, the multiplexer includes an output end and at least two input ends, the output end of the multiplexer corresponds to the input end of the convolution circuit one-to-one, the output end of each convolution circuit in the multiple convolution circuits is coupled to the input end of L multiplexers, L is an integer greater than or equal to 2, at least one of the L multiplexers also includes an external data output end, and at least one of the L convolution circuits corresponding to the L multiplexers also includes an external data input end; a multiplexer, used to connect the output end of at least one convolution circuit coupled to the multiplexer to the output end of the multiplexer; the external data output end of the multiplexer, used to output the convolution results of the convolution circuits corresponding to the L multiplexers respectively; the external data input end of the convolution circuit, used to input the feature map to be processed.

[0007] Thus, the output end of each convolution circuit in the computing core provided by the embodiment of the present application is coupled to the input end of at least two multiplexers. Compared with the matrix-type storage-computation integrated network structure, the computing core can flexibly meet the computing core's demand for computing resources in different dimensions such as network depth, number of input and output channels, and feature map size through the selection function of the multiplexer, thereby improving the computing resource utilization of the computing core and reducing the circuit area and cost of the chip.

[0008] In one possible design, the computing core also includes multiple adders, the multiple convolution circuits include a first convolution circuit and a second convolution circuit, the first input end of the first adder in the multiple adders is coupled to the output end of the first convolution circuit, the second input end of the first adder is coupled to the output end of the second convolution circuit, and the output end of the first adder is coupled to the input ends of L multiplexers.

[0009] In this design, the computing core can achieve a 2-fold expansion of input channels, can flexibly meet the requirements of the computing core for input channels, and increase the applicability of the computing core to different network algorithms.

[0010] In one possible design, the multiple convolution circuits further include a third convolution circuit and a fourth convolution circuit, the multiple adders further include a second adder and a third adder, the first input terminal of the second adder is coupled to the output terminal of the third convolution circuit, the second input terminal of the second adder is coupled to the output terminal of the fourth convolution circuit, the output terminal of the second adder is coupled to the first input terminal of the third adder, the second input terminal of the third adder is coupled to the output terminal of the first adder, and the output terminal of the third adder is coupled to the input terminals of L multiplexers.

[0011] In this design, the computing core can achieve a 4-fold expansion of input channels, which can flexibly meet the requirements of the computing core for input channels and increase the applicability of the computing core to different network algorithms. In addition, the computing core can also achieve other multiples of input channel expansion.

[0012] In one possible design, the multiple convolution circuits include a fifth convolution circuit, and the multiple multiplexers include a first multiplexer and a second multiplexer; the first multiplexer is used to connect the output end of the fifth convolution circuit to the output end of the first multiplexer; the second multiplexer is used to connect the output end of the fifth convolution circuit to the output end of the second multiplexer.

[0013] In this design, the computing core can realize the expansion of the output channel, can flexibly meet the requirements of the computing core for the output channel, and increase the applicability of the computing core to different network algorithms.

[0014] In one possible design, the convolution circuit includes a first cache module and a convolution calculation module; the first cache module is used to cache input data and weight parameters of other convolution circuits; and the convolution calculation module is used to perform a convolution operation based on the input data and weight parameters.

[0015] In this design, the convolution circuit is also a storage and computing integrated architecture. The input data passes through different convolution circuits in sequence inside the computing core and is only output to the computing core at the end, thereby saving power consumption in data access and reducing processing delays.

[0016] In one possible design, the multiple convolution circuits further include a sixth convolution circuit and a seventh convolution circuit, the input data of the multiple convolution circuits include a first feature map, the first feature map includes a first data block and a second data block, the multiplexer is also used to transmit the first data block to the sixth convolution circuit at a first moment, and to transmit the second data block to the seventh convolution circuit at a second moment, the sixth convolution circuit is used to start performing the convolution operation of the first data block at a first moment, and the seventh convolution circuit is used to start performing the convolution operation of the second data block at a second moment. The interval between the first moment and the second moment is negligible compared to the time it takes for the convolution circuit to perform the convolution operation on the data block. Therefore, for larger feature maps, the computing core can realize parallel execution of different data blocks of the same feature map, thereby improving the computing resource utilization of the computing core.

[0017] In one possible design, the computing core also includes a hardware multi-thread control circuit, the convolution circuit is used to execute at least one hardware thread, the at least one hardware thread includes a first hardware thread, and the hardware multi-thread control circuit is used to: after the convolution circuit obtains the third data block, control the convolution circuit to execute the first hardware thread; wherein, at least one hardware thread corresponds one-to-one to the convolution operations of different convolution layers, and the third data block is a data block of a feature map of the convolution layer corresponding to the first hardware thread.

[0018] In one possible design, the hardware multi-thread control circuit is also used to: after the convolution circuit obtains the fourth data block, control the convolution circuit to repeatedly execute the first hardware thread; wherein the fourth data block is also the data block of the feature map of the convolution layer corresponding to the first hardware thread.

[0019] In one possible design, at least one hardware thread also includes a second hardware thread, the convolution circuit includes an eighth convolution circuit, and the hardware multi-thread control circuit is further used to: control the eighth convolution circuit to execute the first hardware thread; and when the first hardware thread is completed, control the eighth convolution circuit to continue to execute the second hardware thread calculation.

[0020] In this design, the computing core can time-share multiple hardware threads, and multiple hardware threads can correspond to the same convolution layer, or multiple hardware threads can correspond to different convolution layers. That is, the computing core can time-share the same convolution layer operation, or it can time-share the different convolution layer operations, which improves the computing resource utilization.

[0021] In one possible design, the hardware multi-thread control circuit is specifically used to: when the first hardware thread is completed, determine whether the second hardware thread meets the switching condition, the switching condition including that the eighth convolution circuit has obtained the data block of the feature map of the convolution layer corresponding to the second hardware thread, the first cache module corresponding to the output data block of the second hardware thread stored in the eighth convolution circuit is empty and the eighth convolution circuit is in an idle state; if the second hardware thread meets the switching condition, control the convolution circuit to continue to execute the second hardware thread calculation.

[0022] In this design, after the convolution circuit completes the calculation of the first hardware thread, the hardware multi-thread control circuit will determine whether the second hardware thread meets the switching condition, thereby determining whether the convolution circuit continues to execute the second hardware thread. The second hardware thread that meets the switching condition can ensure that the second hardware thread can be immediately executed, thereby improving the computing resource utilization of the computing core.

[0023] In one possible design, the hardware multi-thread control circuit is further used to: after the second hardware thread meets the switching condition, control the convolution circuit to continue to execute the second hardware thread calculation according to the weight parameter and priority of the second hardware thread.

[0024] In this design, if there are multiple second hardware threads that meet the switching condition, the second hardware thread that has obtained the corresponding weight parameter and the highest priority among the multiple second hardware threads should be processed first, which can improve the efficiency of the computing core.

[0025] In one possible design, an output terminal of each of the plurality of convolution circuits is coupled to input terminals of the plurality of multiplexers.

[0026] In this design, when the number of convolution circuits is small, the output of each convolution circuit in the multiple convolution circuits can be coupled to the input of the multiplexer, thereby forming a full-select architecture, which can improve the flexibility of the computing core.

[0027] In one possible design, the computing core includes n convolution circuits, where n is an integer greater than 1. The computing core also includes a switch and n second cache modules. The switch includes n input terminals and n output terminals. The output terminals of the n second cache modules correspond one-to-one to the n input terminals of the switch, and the n output terminals of the switch correspond one-to-one to the n convolution circuits respectively.

[0028] In this design, a second cache module can be added before the convolution circuit to facilitate centralized management. In addition, a switch is added to the computing core to improve the flexibility of computing core data transmission.

[0029] In the second aspect, an embodiment of the present application also provides a data processing method, which is applied to a computing core, wherein the computing core includes multiple convolution circuits and multiple multiplexers, the multiplexer includes an output end and at least two input ends, the output end of the multiplexer corresponds to the input end of the convolution circuit one-to-one, the output end of each convolution circuit in the multiple convolution circuits is coupled to the input end of L multiplexers, L is an integer greater than or equal to 2, at least one of the L multiplexers also includes an external data output end, and at least one of the L convolution circuits corresponding to the L multiplexers also includes an external data input end; the method includes: controlling the multiplexer to connect the output end of at least one convolution circuit coupled to the multiplexer to the output end of the multiplexer; controlling the external data output end of the multiplexer to output the convolution results of the convolution circuits corresponding to the L multiplexers respectively; and controlling the external data input end of the convolution circuit to input the feature map to be processed.

[0030] The beneficial effects of the second aspect can refer to the description of the first aspect.

[0031] In one possible design, the computing core also includes multiple adders, the multiple convolution circuits include a first convolution circuit and a second convolution circuit, the first input end of the first adder in the multiple adders is coupled to the output end of the first convolution circuit, the second input end of the first adder is coupled to the output end of the second convolution circuit, and the output end of the first adder is coupled to the input ends of L multiplexers.

[0032] In one possible design, the multiple convolution circuits further include a third convolution circuit and a fourth convolution circuit, the multiple adders further include a second adder and a third adder, the first input terminal of the second adder is coupled to the output terminal of the third convolution circuit, the second input terminal of the second adder is coupled to the output terminal of the fourth convolution circuit, the output terminal of the second adder is coupled to the first input terminal of the third adder, the second input terminal of the third adder is coupled to the output terminal of the first adder, and the output terminal of the third adder is coupled to the input terminals of L multiplexers.

[0033] In one possible design, the multiple convolution circuits include a fifth convolution circuit, and the multiple multiplexers include a first multiplexer and a second multiplexer; the method includes: controlling the first multiplexer to connect the output end of the fifth convolution circuit to the output end of the first multiplexer; and controlling the second multiplexer to connect the output end of the fifth convolution circuit to the output end of the second multiplexer.

[0034] In one possible design, the convolution circuit includes a first cache module and a convolution calculation module; the method also includes: controlling the first cache module to cache input data and weight parameters of other convolution circuits; and controlling the convolution calculation module to perform a convolution operation based on the input data and weight parameters.

[0035] In one possible design, the multiple convolution circuits also include a sixth convolution circuit and a seventh convolution circuit, the input data of the external data input terminals of the multiple convolution circuits include a first feature map, the first feature map includes a first data block and a second data block, and the method also includes: controlling the multiplexer to transmit the first data block to the sixth convolution circuit at a first moment, and transmitting the second data block to the seventh convolution circuit at a second moment; controlling the sixth convolution circuit to start performing the convolution operation of the first data block at the first moment, and controlling the seventh convolution circuit to start performing the convolution operation of the second data block at the second moment.

[0036] In one possible design, the computing core also includes a hardware multi-thread control circuit, the convolution circuit is used to execute at least one hardware thread, and the at least one hardware thread includes a first hardware thread. The method also includes: after the convolution circuit obtains the third data block, controlling the hardware multi-thread control circuit to control the convolution circuit to execute the first hardware thread; wherein, at least one hardware thread corresponds one-to-one to the convolution operations of different convolution layers, and the third data block is a data block of a feature map of the convolution layer corresponding to the first hardware thread.

[0037] In one possible design, the method also includes: after the convolution circuit obtains the fourth data block, controlling the hardware multi-thread control circuit to control the convolution circuit to repeatedly execute the first hardware thread; wherein the fourth data block is also a data block of a feature map of the convolution layer corresponding to the first hardware thread.

[0038] In one possible design, at least one hardware thread also includes a second hardware thread, the convolution circuit includes an eighth convolution circuit, and the method also includes: controlling the hardware multi-thread control circuit to control the eighth convolution circuit to execute the first hardware thread; when the first hardware thread is completed, controlling the hardware multi-thread control circuit to control the eighth convolution circuit to continue to execute the second hardware thread calculation.

[0039] In one possible design, when the first hardware thread is completed, the hardware multi-thread control circuit is controlled to control the eighth convolution circuit to continue to execute the second hardware thread calculation, including: when the first hardware thread is completed, it is determined whether the second hardware thread meets the switching condition, the switching condition including that the eighth convolution circuit has obtained the data block of the feature map of the convolution layer corresponding to the second hardware thread, the first cache module corresponding to the output data block of the second hardware thread stored in the eighth convolution circuit is empty and the eighth convolution circuit is in an idle state, and the hardware multi-thread control circuit is controlled to control the convolution circuit to continue to execute the second hardware thread calculation.

[0040] In one possible design, the method further includes: after the second hardware thread meets the switching condition, according to the weight parameter and priority of the second hardware thread, controlling the hardware multi-thread control circuit to control the convolution circuit to continue to execute the second hardware thread calculation.

[0041] In one possible design, an output terminal of each of the plurality of convolution circuits is coupled to input terminals of the plurality of multiplexers.

[0042] In one possible design, the computing core includes n convolution circuits, where n is an integer greater than 1. The computing core also includes a switch and n second cache modules. The switch includes n input terminals and n output terminals. The output terminals of the n second cache modules correspond one-to-one to the n input terminals of the switch, and the n output terminals of the switch correspond one-to-one to the n convolution circuits respectively.

[0043] In a third aspect, an embodiment of the present application provides a chip, the chip includes a processor and a computing core of the first aspect, and the computing core and the processor are electrically connected.

[0044] In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a printed circuit board and a computing core of the first aspect, wherein the computing core and the printed circuit board are electrically connected.

[0045] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the data processing method in any of the above aspects and any possible implementation methods.

[0046] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer or a processor, enables the computer or the processor to execute the data processing method in any of the above aspects and any possible implementation manners.

[0047] In a seventh aspect, an embodiment of the present application provides a system, which may include a wireless access device in any possible implementation of any of the above aspects and at least one electronic device. The electronic device and the wireless access device may execute the data processing method in any of the above aspects and any possible implementation.

[0048] It can be understood that any of the computing cores, chips, electronic devices, computer-readable storage media or computer program products provided above can be applied to the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods and will not be repeated here.

[0049] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 An architectural diagram of a neural network processor provided in an embodiment of the present application;

[0051] Figure 2 A grid architecture diagram based on storage and computing integration provided in an embodiment of the present application;

[0052] Figure 3 A line graph of the number of weights of different convolutional layers and the number of pixels of a feature map of a network algorithm provided in an embodiment of the present application;

[0053] Figure 4 A multiplier resource allocation diagram when running a network algorithm based on a storage-computing integrated network architecture provided in an embodiment of the present application;

[0054] Figure 5 A diagram of the execution efficiency of a multiplier when running a network algorithm based on a storage-computation integrated network architecture provided in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of an execution device provided in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of a chip provided in an embodiment of the present application;

[0057] Figure 8 A schematic diagram of the structure of a computing core provided in an embodiment of the present application;

[0058] Fig. 9 A schematic diagram of continuous convolution of a computing kernel provided in an embodiment of the present application;

[0059] Fig.10 A schematic diagram of a computing core input channel expansion provided in an embodiment of the present application;

[0060] Fig.11 A schematic diagram of a computing core output channel expansion provided in an embodiment of the present application;

[0061] Fig.12 A schematic diagram of a computing core executing multiple data blocks in parallel provided in an embodiment of the present application;

[0062] Fig.13 A mapping diagram of a computing core and an AI algorithm network provided in an embodiment of the present application;

[0063] Fig.14 A schematic diagram of data block division of a feature graph provided in an embodiment of the present application;

[0064] Fig.15 A timing diagram of a hardware thread performing different convolution layer calculations in time-sharing manner provided in an embodiment of the present application;

[0065] Fig.16 A schematic diagram of a full selection architecture provided in an embodiment of the present application;

[0066] Fig.17 A schematic diagram of the structure of another computing core provided in an embodiment of the present application;

[0067] Fig.18 A flowchart of a data processing method provided in an embodiment of the present application;

[0068] Fig.19 A schematic diagram of computing resource utilization of a network algorithm based on a storage-computing integrated architecture provided in an embodiment of the present application;

[0069] Fig. 20 A schematic diagram of computing resource utilization based on computing cores provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] For ease of understanding, some examples of concepts related to the embodiments of the present application are provided for reference, as shown below:

[0071] Convolutional neural networks (CNN) are a type of feedforward neural networks (FNN) that contain convolution calculations and have a deep structure. They are one of the representative algorithms of deep learning.

[0072] Convolutional neural networks contain a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution calculation can be regarded as using a trainable filter to convolve an input image or a convolution feature plane (feature map), which can also be called a feature map. The convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature maps, each of which can be composed of some rectangular neurons. Neurons in the same feature map share weights, and the weight matrix corresponding to the shared weights here is the convolution kernel. Shared weights can be understood as the way to extract image information is independent of position. Taking an image as an example, the implicit principle is that the statistical information of a part of the image is the same as that of other parts, which means that the image information learned in a part can also be used in another part. Therefore, for all positions on the image, the same learning can be used to obtain image information. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0073] Storage and computing integration: embedding algorithms in the memory, transferring the calculations in the computer from the central processing unit (CPU) to the memory, and realizing calculations in the storage and computing unit (cell), which can greatly reduce data exchange time and data access energy consumption during the calculation process.

[0074] There are two ways to implement the integrated storage and computing device, namely, using analog devices (such as resistive random-access memory (ReRAM)) to build a computing array, and using digital devices (such as static random-access memory (SRAM)) to build a computing array.

[0075] Computation array (crossbar, XB): constructed by storage computing units, each computation array contains several rows and several columns.

[0076] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0077] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0078] In order to solve the problem of memory wall, the existing technology is Figure 2 In the proposed mesh architecture based on in memory computing (IMC), 8 layers of convolution operations are shown. In the in memory computing (IMC) mesh structure, the compute / storage units are connected end to end to form a matrix. When allocating computing resources, each column is allocated to the same layer of the network algorithm, and hardware computing resources are allocated in sequence according to the order of the convolution layers, so that the entire network algorithm can be allocated to the grid array by layer. During the calculation process, the intermediate layer data flows internally without leaving the memory computing core, which not only solves the problem of memory wall, but also greatly improves the hardware computing power.

[0079] However, the grid structure based on storage and computing integration has the problem of low computing resource utilization in practical applications. In one example, Table 1 shows the number of weights of each convolution layer and the image size of the feature map (expressed in pixels) of a network algorithm. In addition, the corresponding line graph of Table 1 is shown in Figure 3 shown.

[0080] Table 1

[0081] Convolutional Layer Number of weights The number of pixels in the feature map 1 1152 1534464 2 1152 383616 3 9216 383616 4 2048 95904 5 18432 95904 6 6144 23976 7 73728 23976 8 24576 5994 9 18432 5994 10 9216 23976 11 1024 23976 12 5120 23976 13 9216 95904 14 1024 95904 15 2560 95904 16 2304 383616 17 256 383616 18 1024 383616 19 1728 1534464

[0082] From Table 1 and Figure 3 It can be seen that the number of weights in each convolution layer varies greatly. For example, the number of weights in the 7th convolution layer is about 64 times that in the 1st convolution layer. Since the number of weights determines the number of multipliers required, the number of multipliers required in the 7th convolution layer is about 64 times that in the 1st convolution layer. Figure 4 As shown, there will be a problem that the multiplier resources of the 7th convolution layer are insufficient, while the multiplier resources of the 1st convolution layer are largely surplus. As a result, the utilization rate of the hardware algorithm is very low.

[0083] In addition, the storage-computation integrated grid structure still has the problem of low time efficiency in practical applications. It can be seen from Table 1 that the number of pixels in the feature map of each convolution layer also varies greatly. For example, the number of pixels in the feature map of the first convolution layer is about 256 times that of the feature map of the eighth convolution layer. Since the number of pixels in the feature map determines the computational requirements of the convolution layer, the computational requirements of a single channel of the first convolution layer are about 256 times that of a single channel of the eighth convolution layer. When the storage-computation integrated grid structure uses the network algorithm for calculations, such as Figure 5 As shown in the figure, the execution time of a single channel of the first convolutional layer is 2 orders of magnitude longer than that of the eighth convolutional layer. Even if the first convolutional layer adopts a parallel execution strategy to increase computing power, it cannot make up for the gap of 2 orders of magnitude. As a result, the multipliers of other convolutional layers will be idle for a large amount of time during the entire running time cycle, and the execution efficiency of the multipliers is low.

[0084] In addition to the above two problems, the grid structure based on storage and computing integration is also difficult to be compatible with deep convolutional network algorithms. Among them, different network algorithms have different requirements for the number of convolutional layers and the computing power requirements for each convolutional layer. For example, some networks have fewer layers, but each convolutional layer requires a lot of computing power, or some networks have more layers, but each convolutional layer requires less computing power. The grid structure based on storage and computing integration is difficult to be compatible with different combinations of requirements for the number of layers and computing power of each layer of different network algorithms. In summary, the grid structure based on storage and computing integration is inefficient, wastes a lot of hardware resources, and has high hardware area costs.

[0085] Therefore, an embodiment of the present application provides a computing core, which includes multiple convolution circuits and multiple multiplexers, wherein the output end of each convolution circuit is coupled to the input ends of at least two multiplexers. Compared with the matrix-type storage and computing integrated network structure, the computing core can flexibly meet the demand for computing resources in different dimensions such as network depth, number of input and output channels, and feature map size through the selection function of the multiplexer, thereby improving the computing resource utilization of the computing core and reducing the circuit area and cost of the chip.

[0086] In the above scenario, the computing core and data processing method of the embodiment of the present application can be applied to different systems or devices, such as Figure 6 The execution device shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality device (AR) ( Figure 6Not shown), virtual reality equipment (virtual reality, VR) ( Figure 6 Not shown) and the vehicle terminal ( Figure 6 The execution device is provided with an input / output (I / O) interface for data interaction with an external device. For example, a user can input data to the I / O interface through a client device. The input data in the embodiment of the present application may include: an input image, which may be an image collected by the execution device through a data acquisition device, an image in a database of the execution device, or an image from a client device.

[0087] In some embodiments, the computing core in the embodiments of the present application can be understood as a chip, for example, the chip is a system-on-a-chip (SoC). Figure 7 As shown, the SoC includes a processor, a memory, and an I / O interface, etc. The processor may be a single-core processor or a multi-core processor. After the processor loads the data and application in the memory, it processes the data, such as performing the convolution processing in this application. For example, a convolution calculation is performed on the input image to obtain the convolution calculation result of the input image.

[0088] The present application embodiment provides a computing core 80, such as Figure 8 As shown, Figure 8 A schematic diagram of the structure of a computing core provided in an embodiment of the present application. The computing core 80 includes a plurality of convolution circuits 81 and a plurality of multiplexers 82 (multiplexers, MUX), the multiplexer 82 includes an output end and at least two input ends, the output end of the multiplexer 82 corresponds to the input end of the convolution circuit 81 one by one, the output end of each convolution circuit 81 in the plurality of convolution circuits 81 is coupled to the input end of L multiplexers 82, L is an integer greater than or equal to 2, at least one of the L multiplexers 82 also includes an external data output end, and at least one of the L convolution circuits 81 corresponding to the L multiplexers 82 also includes an external data input end.

[0089] The multiplexer 82 is used to conduct the output end of at least one convolution circuit 81 coupled to the multiplexer 82 to the output end of the multiplexer 82. The external data output end of the multiplexer 82 is used to output the convolution results of the convolution circuits 81 corresponding to the L multiplexers 82, respectively, and the external data input end of the convolution circuit 81 is used to input the feature map to be processed.

[0090] in, Figure 8The multiplexer 82 shown in FIG. 8 includes one output terminal and five input terminals, wherein the output terminal of each convolution circuit 81 is coupled to five input terminals of the multiplexer 82 , that is, L=5.

[0091] The convolution circuit 81 may also be referred to as a convolution unit (CU), and the convolution circuit 81 may include multiple input channels, multiple multipliers, and adders. The multiple input channels are parallel input channels, and the input data of each input channel includes a feature map and a weight. The multiple multipliers are respectively used to multiply the feature map and the weight in the input data of each input channel, and the adder is used to add the output results of the multiple multipliers to obtain the convolution result.

[0092] The multiplexer 82 may also be referred to as a data selector or a selector, etc. The multiplexer 82 may include multiple input terminals and an output terminal, and is used to select a signal from multiple input signals as an output signal. In the embodiment of the present application, the input terminal of each multiplexer 82 is coupled to the output terminals of the L convolution circuits 81, and the multiplexer 82 may select an output signal from the output signals of the L convolution circuits 81 as an output signal.

[0093] For ease of understanding, Figure 8 The convolution circuits 81 in the embodiment are numbered as convolution circuit a, convolution circuit b, ..., convolution circuit w, and Figure 8 The multiplexers 82 in the figure are numbered in the manner of MUX1, MUX2, ..., MUX22. Subsequent figures are numbered in the same manner. Figure 8 The convolution circuit 81 and the multiplexer 82 in the circuit form a spiral shape. Figure 8 The circuit structure in FIG. 8 is only a schematic diagram, wherein the output end of the convolution circuit a can be coupled to the input end of MUX22, so that the computing core 80 is a spiral ring. In addition, the output end of the convolution circuit a may not be coupled to the input end of MUX22.

[0094] The convolution circuit 81 may also include an external data input terminal, from which Figure 8 It can be seen that convolution circuit a, convolution circuit e, convolution circuit j, convolution circuit o and convolution circuit t may include external data input terminals. Among them, convolution circuit 81 including the external data input terminal may be used as the first convolution layer of the network algorithm. In addition, multiplexer 82 may also include an external data output terminal, from Figure 8It can be seen that MUX1, MUX6, MUX11, MUX16 and MUX21 may include external data output terminals. Thus, the convolution results of convolution circuit a, convolution circuit b and convolution circuit c can be output through the external data output terminal of MUX1. The convolution results of convolution circuit d, convolution circuit e, convolution circuit f, convolution circuit g and convolution circuit h can be output through the external data output terminal of MUX6. By analogy, convolution circuit a to convolution circuit w all have corresponding external data output terminals of multiplexer 82 for result output. That is, when the current convolution circuit 81 is the last layer of convolution operation of the network algorithm, the current convolution circuit 81 has a corresponding external data output terminal of multiplexer 82 for convolution result output.

[0095] like Fig. 9 As shown, Fig. 9 A schematic diagram of a computing kernel performing continuous convolution according to an embodiment of the present application. In which, multiple convolution circuits 81 can execute different convolution layers in parallel. In one example, it is assumed that convolution circuit e is used to execute the convolution operation of the first convolution layer, convolution circuit f is used to execute the convolution operation of the second convolution layer, and convolution circuit g is used to execute the convolution operation of the third convolution layer. In which, MUX4 corresponding to convolution circuit e receives the first convolution result (e.g. Fig. 9 Path 9-1 in the figure), MUX4 transmits the first convolution result to convolution circuit e (eg Fig. 9 Path 9-2 in the convolution circuit e performs the convolution operation of the first convolution layer to obtain a second convolution result. The convolution circuit e transmits the second convolution result to MUX5 (e.g. Fig. 9 Path 9-3 in the figure), MUX5 transmits the second convolution result to the convolution circuit f (e.g. Fig. 9 The convolution circuit f performs the convolution operation of the second convolution layer to obtain a third convolution result. The convolution circuit f transmits the third convolution result to MUX6 (e.g. Fig. 9 Path 9-5 in the figure), MUX6 transmits the third convolution result to the convolution circuit g (e.g. Fig. 9 The convolution circuit g performs the convolution operation of the third convolution layer to obtain the fourth convolution result, and transmits it to the next convolution circuit 81 (e.g. Fig. 9 Path 9-7).

[0096] Among them, the computing core 80 can realize input channel expansion, such as Fig.10 As stated, Fig.10A schematic diagram of a computing core input channel expansion provided in an embodiment of the present application. The computing core 80 may also include multiple adders 83, and the multiple convolution circuits 81 include a first convolution circuit and a second convolution circuit. The first input end of the first adder 831 in the multiple adders 83 is coupled to the output end of the first convolution circuit, the second input end of the first adder 831 is coupled to the output end of the second convolution circuit, and the output end of the first adder 831 is coupled to the input end of L multiplexers 82.

[0097] In one example, it is assumed that each convolution circuit 81 includes N input channels, where N is an integer greater than or equal to 1. It is assumed that convolution circuit e is a first convolution circuit, and convolution circuit f is a second convolution circuit. Convolution circuit e performs a convolution operation on the input feature graphs of the N input channels to obtain a fifth convolution result, and convolution circuit e transmits the fifth convolution result to the first input terminal of the first adder (e.g. Fig.10 In addition, the convolution circuit f also performs a convolution operation on the input feature maps of the N input channels to obtain a sixth convolution result, and the convolution circuit f transmits the sixth convolution result to the second input terminal of the first adder 831 (e.g. Fig.10 The first adder 831 adds the fifth convolution result and the sixth convolution result to obtain a seventh convolution result, wherein the seventh convolution result is equivalent to the feature map obtained after performing the convolution operation on the input feature maps of the 2N input channels. The first adder 831 transmits the seventh convolution result to the next multiplexer 82 (e.g. Fig.10 Path 10-3) in for subsequent convolution operations.

[0098] In addition, the multiple convolution circuits 81 may further include a third convolution circuit and a fourth convolution circuit, and the multiple adders 83 may further include a second adder 832 and a third adder 833. A first input terminal of the second adder 832 is coupled to an output terminal of the third convolution circuit 813, a second input terminal of the second adder 832 is coupled to an output terminal of the fourth convolution circuit 814, an output terminal of the second adder 832 is coupled to a first input terminal of the third adder 833, a second input terminal of the third adder 833 is coupled to an output terminal of the first adder 831, and an output terminal of the third adder 833 is coupled to input terminals of the L multiplexers 82.

[0099] Continue to view Fig.10 , assuming that the convolution circuit g is the third convolution circuit and the convolution circuit h is the fourth convolution circuit. In one example, the convolution circuit g performs a convolution operation on the input feature graphs of the N input channels to obtain an eighth convolution result, and the convolution circuit g transmits the eighth convolution result to the first input terminal of the second adder 832 (e.g. Fig.10In addition, the convolution circuit h also performs a convolution operation on the input feature maps of the N input channels to obtain a ninth convolution result, and the convolution circuit h transmits the ninth convolution result to the second input terminal of the second adder 832 (e.g. Fig.10 The second adder 832 adds the eighth convolution result and the ninth convolution result to obtain a tenth convolution result, wherein the tenth convolution result is equivalent to the feature map obtained after performing the convolution operation on the input feature maps of the 2N input channels. The second adder 832 transmits the tenth convolution result to the first input terminal (e.g., Fig.10 The first adder 831 transmits the seventh convolution result to the second input terminal of the third adder 833 (eg, Fig.10 The third adder 833 adds the seventh convolution result and the tenth convolution result to obtain an eleventh convolution result, wherein the eleventh convolution result is equivalent to the feature map obtained after performing the convolution operation on the input feature maps of the 4N input channels. The third adder 833 transmits the eleventh convolution result to the next multiplexer 82 (e.g. Fig.10 Path 10-8) in for subsequent convolution operations.

[0100] Similarly, the computing core 80 can also achieve expansion of 8N input channels or even more input channels.

[0101] The computing core 80 can also realize output channel expansion, such as Fig.11 As shown, Fig.11 A schematic diagram of a computing core output channel expansion provided in an embodiment of the present application. The multiple convolution circuits 81 may also include a fifth convolution circuit, and the multiple multiplexers 82 may include a first multiplexer and a second multiplexer. The first multiplexer is used to conduct the output end of the fifth convolution circuit to the output end of the first multiplexer, and the second multiplexer is used to conduct the output end of the fifth convolution circuit to the output end of the second multiplexer.

[0102] In one example, it is assumed that the convolution circuit d is the fifth convolution circuit, MUX5 is the first multiplexer, and MUX6 is the second multiplexer. The fifth convolution circuit can be connected to the output end of MUX5, and the fifth convolution circuit can also be connected to the output end of MUX6. Assuming that the number of output channels of each convolution circuit 81 is also N, the convolution circuit d transmits the convolution results of the N output channels to MUX5 (for example Fig.11 Path 11-1 in the convolution circuit d), MUX5 conducts the output end of the convolution circuit d and the output end of MUX5, and transmits the convolution results of the N output channels to the convolution circuit e (for example Fig.11In addition, the convolution circuit d transmits the convolution results of the other N output channels to MUX6 (e.g. Fig.11 Path 11-3 in the convolution circuit d), MUX6 conducts the output end of the convolution circuit d and the output end of MUX6, and transmits the convolution results of the N output channels to the convolution circuit f (for example Fig.11 Thus, the convolution circuit d can realize the transmission of the convolution results of 2N output channels.

[0103] Similarly, the computing core 80 can also achieve expansion of 4N output channels or even more output channels.

[0104] Therefore, the computing core provided in the embodiment of the present application can achieve different input channel number expansions and output channel number expansions, can meet the needs of more network algorithms, and increase the flexibility and applicability of the computing core.

[0105] Optionally, the convolution circuit 81 may include a first cache module 811 and a convolution calculation module 812, the first cache module 811 may be a buffer register (buffer), the first cache module 811 is used to cache input data and weight parameters of other convolution circuits, and the convolution calculation module 812 is used to perform convolution operations based on the input data and weight parameters.

[0106] Therefore, the convolution circuit is also a storage and computing integrated architecture. The input data passes through different convolution circuits in sequence inside the computing core and is only output to the computing core at the end, thereby saving power consumption of data access and reducing processing delay.

[0107] Optionally, the multiple convolution circuits 81 may also include a sixth convolution circuit and a seventh convolution circuit, the input data of the multiple convolution circuits 81 include a first feature map, the first feature map includes a first data block and a second data block, the multiplexer 82 is also used to transmit the first data block to the sixth convolution circuit at a first moment, and to transmit the second data block to the seventh convolution circuit at a second moment, the sixth convolution circuit is used to start performing a convolution operation on the first data block at the first moment, and the seventh convolution circuit is used to start performing a convolution operation on the second data block at the second moment.

[0108] Please see Fig.12 , Fig.12 A schematic diagram of a computing core executing multiple data blocks in parallel provided in an embodiment of the present application. Assume that convolution circuit e is the sixth convolution circuit, the input end of convolution circuit e is coupled to the output end of MUX5, and convolution circuit f is the seventh convolution circuit, the input end of convolution circuit f is coupled to the output end of MUX6. Assume that MUX5 turns on convolution circuit c whose convolution result is the first data block (for example Fig.12Path 12-1 in the figure), the convolution circuit e corresponding to MUX5 performs a convolution operation on the first data block at the first moment (e.g. Fig.12 Path 12-2 in the example), and output the convolution result to MUX7 (e.g. Fig.12 At the same time, MUX6 turns on the convolution circuit d whose convolution result is the second data block (e.g. Fig.12 Path 12-4 in the figure), the convolution circuit f corresponding to MUX6 performs a convolution operation on the second data block at the second moment (e.g. Fig.12 Path 12-5 in the convolution operation, and output the convolution result to MUX8 (e.g. Fig.12 The interval between the first moment and the second moment is negligible relative to the time for the convolution circuit to perform the convolution operation, which can be understood as the convolution operation is performed on the first data block and the second data block in parallel.

[0109] In another implementation, the first data block and the second data block can be input into different convolution circuits through different external data input terminals, and different convolution circuits perform parallel calculations of different data blocks in the same feature map. In one example, the first data block is input into convolution circuit a through the external data input terminal, and the second data block is also input into convolution circuit e through the external data input terminal, so that convolution circuit a and convolution circuit e can respectively perform convolution operations on the first data block and the second data block.

[0110] Therefore, the computing core 80 uses the data block as the minimum processing unit, and the computing core 80 can realize parallel execution of multiple data blocks in the same feature graph, which can improve computing efficiency.

[0111] Optionally, the computing core 80 may further include a hardware multi-thread control circuit 84, and the convolution circuit 81 is used to execute at least one hardware thread, and the at least one hardware thread includes a first hardware thread. The hardware multi-thread control circuit 84 is used to: after the convolution circuit 81 obtains the third data block, control the convolution circuit 81 to execute the first hardware thread. Among them, at least one hardware thread corresponds to the convolution operation of different convolution layers one by one, and the third data block is a data block of the feature map of the convolution layer corresponding to the first hardware thread.

[0112] Please see Fig.13 , Fig.13 A mapping diagram of a computing core and an AI algorithm network provided in an embodiment of the present application. The AI ​​algorithm network may include a placeholder, multiple convolutional layers, a rectified linear unit (relu), and a padding unit, etc. The multiple convolutional layers include the first convolutional layer to the Nth convolutional layer. Fig.13The computing core 80 shown in includes a convolution circuit a and a convolution circuit b, wherein the convolution circuit a can execute multiple hardware threads, and the multiple hardware threads can include hardware thread 0 to hardware thread m. The convolution operation of each convolution layer in the AI ​​algorithm network can be mapped to one hardware thread or multiple hardware threads in the convolution circuit. For example, the convolution operation of the first convolution layer can be mapped to hardware thread 0 in the convolution circuit a, and the convolution operation of the fifth convolution layer can be mapped to hardware thread 0 in the convolution circuit b. Among them, the convolution circuit 81 can time-share multiple hardware threads and perform different convolution layer calculations in time. The convolution circuit 81 can process a data block after each hardware thread is started, and then switch to other hardware threads to process other convolution layer calculations. In one example, each convolution layer of the AI ​​algorithm network is mapped to each hardware thread of the convolution unit for execution, wherein different hardware threads switch the weight parameters corresponding to the convolution layer they execute, thereby making the convolution circuit 81 run at full load as much as possible to improve the utilization efficiency of the convolution circuit 81.

[0113] Please see Fig.14 , Fig.14 A schematic diagram of data block division of a feature map provided in an embodiment of the present application. Wherein, the feature map is divided into n data blocks, including data block 0, data block 1, ..., data block n. Wherein, data block = rows*i, i is an integer greater than or equal to 1, that is, the data block can be multiple rows or one row. The convolution circuit 81 performs convolution calculations on one data block at a time, and then switches to the next hardware thread to perform the convolution operation on the next data block. That is, the convolution circuit 81 first executes the first hardware thread, and continues to execute the second hardware thread after the first hardware thread is executed.

[0114] The hardware multithreading control circuit 84 is further used to control the convolution circuit 81 to repeatedly execute the first hardware thread after the convolution circuit acquires the fourth data block, wherein the fourth data block is also a data block of the feature map of the convolution layer corresponding to the first hardware thread.

[0115] Exemplarily, when mapping the AI ​​algorithm network and the hardware threads, the convolution operations of the same convolution layer can be mapped to different hardware threads. For example, the convolution operations of the first convolution layer are mapped to the first hardware thread. Assuming that the feature map of the first convolution layer includes the third data block and the fourth data block, the third data block of the feature map of the first convolution layer can be used as the input data of the first hardware thread, and the fourth data block of the feature map of the first convolution layer can also be used as the input data of the first hardware thread.

[0116] Among them, at least one hardware thread also includes a second hardware thread, the convolution circuit 81 includes an eighth convolution circuit, and the hardware multi-thread control circuit 84 is also used to: control the eighth convolution circuit to execute the first hardware thread, and when the first hardware thread is completed, control the eighth convolution circuit to continue to execute the second hardware thread calculation.

[0117] Exemplarily, when mapping the AI ​​algorithm network and the hardware threads, the convolution operations of different convolutional layers can be mapped to different hardware threads. For example, the convolution operation of the first convolutional layer is mapped to the first hardware thread, and the data block of the feature map of the first convolutional layer is used as the input data of the first hardware thread. And the convolution operation of the second convolutional layer is mapped to the second hardware thread, and the data block of the feature map of the second convolutional layer is used as the input data of the second hardware thread.

[0118] Optionally, the hardware multithread control circuit 84 is specifically used to: when the first hardware thread is completed, determine whether the second hardware thread meets the switching condition. The switching condition includes that the convolution circuit 81 has obtained the data block of the feature map of the convolution layer corresponding to the second hardware thread, the memory space corresponding to the output data block of the second hardware thread stored in the convolution circuit 81 is empty, and the convolution circuit 81 is in an idle state. If the second hardware thread meets the switching condition, the convolution circuit 81 is controlled to continue to execute the second hardware thread calculation.

[0119] Exemplarily, the hardware multithreading control circuit 84 can be used for scheduling hardware threads, see Fig.15 , Fig.15 A timing diagram of a hardware thread that performs different convolution layer calculations in time-sharing is provided for an embodiment of the present application. Wherein, the data block is represented by Blk, for example, data block 0 is represented by Blk0. In addition, condition A is that the convolution circuit in the switching condition has obtained the data block of the feature map of the convolution layer corresponding to the second hardware thread, condition B is that the memory space corresponding to the output data block of the second hardware thread stored in the convolution circuit is empty, and condition C is that the convolution circuit is in an idle state. For hardware thread 0, if condition A and condition C are met, the convolution operation of data block 0 is performed. After the convolution operation of data block 0 is executed, if hardware thread 1 meets condition A and condition C, hardware thread 1 executes the convolution operation of data block 0, and so on. If hardware thread 3 completes the convolution operation of data block 0, if hardware thread 0 meets condition B, hardware thread 0 continues to execute the convolution operation of data block 1.

[0120] Furthermore, the hardware multi-thread control circuit 84 is also used to: after the second hardware thread meets the switching condition, control the convolution circuit 81 to continue to execute the second hardware thread calculation according to the weight parameter and priority of the second hardware thread.

[0121] Exemplarily, the switching conditions may include conditions D and F in addition to conditions A, B, and C, where condition D is that the second hardware thread has obtained the corresponding weight parameter, and condition F is that the second hardware thread has the highest priority. For example, if there are multiple second hardware threads that meet conditions A, B, and C, and the second hardware threads have priorities, the second hardware thread with the highest priority should be processed first to improve the efficiency of the computing core.

[0122] Optionally, an output end of each convolution circuit 81 in the plurality of convolution circuits 81 is coupled to an input end of a plurality of multiplexers 82 .

[0123] Exemplarily, the coupling mode of the output end of the convolution circuit 81 and the input end of the multiplexer 82 can be understood as a full selection architecture, wherein the convolution result of the current convolution circuit 81 can be transmitted to the input end of each convolution circuit 81, including the current convolution circuit 81, through the multiplexer 82. That is, the convolution result of the current convolution circuit 81 can be input to the current convolution circuit 81 as the input data of the next layer of convolution after being selected by the multiplexer 82.

[0124] like Fig.16 As shown, Fig.16 An example of eight convolution circuits 81 and one multiplexer 82 is shown in FIG. The computing core may include a convolution circuit 81, a multiplexer 82, and a hardware multithreading control circuit 84. The convolution circuit 81 may include a first cache module 811 and a convolution calculation module 812. The first cache module 811 may be a buffer register (buffer). The first cache module 811 is used to cache input data and weight parameters of other convolution circuits. The convolution calculation module 812 is used to perform a convolution operation based on the input data and weight parameters. The convolution circuit 81 may include a first input terminal and a second input terminal, wherein the first input terminal is coupled to the output terminal of the multiplexer 82, the first input terminal of the convolution circuit 81 is used to input the convolution result, and the second input terminal is coupled to the cache ( Fig.16 The convolution circuit 81 is coupled with the input terminal of the convolution circuit 81 (represented by WBuff in FIG. 8 ), and the second input terminal of the convolution circuit 81 is used to input weight parameters, etc.

[0125] In one example, when the convolution calculation module 812 performs the convolution operation of the current convolution layer, its convolution result is selected by the multiplexer 82 to the first cache module 811 of the convolution circuit 81 responsible for the next convolution layer, and each first cache module 811 stores the input data and weight parameters corresponding to M hardware threads. The hardware multi-thread control circuit 84 is used to control the convolution circuit 81 and the multiplexer 82. Among them, when a hardware thread meets the switching condition, the hardware multi-thread control circuit 84 is controlled to start the hardware thread for calculation, and the timing of the convolution circuit 81 is controlled to complete the start-run-end process. When multiple convolution circuits 81 execute input / output channels or execute multiple data blocks in parallel, the hardware multi-thread control circuit 84 controls multiple convolution circuits 81 at the same time.

[0126] In addition, the computing core 80 may also include a memory manager 85 for controlling the first cache module 811, including controlling the read and write control, ring buffer, empty and full status judgment, and access conflict management of the first cache module 811.

[0127] Optionally, the computing core 80 may include n convolution circuits, where n is an integer greater than 1. The computing core 80 also includes a switch 86 and n second cache modules 87. The switch 86 includes n input terminals and n output terminals. The output terminals of the n second cache modules 87 correspond one-to-one to the n input terminals of the switch, and the n output terminals of the switch correspond one-to-one to the n convolution circuits respectively.

[0128] For example, Fig.17 As shown, Fig.17 A schematic diagram of the structure of another computing core provided in an embodiment of the present application. Fig.17 8 convolution circuits 81, 1 multiplexer 82, hardware multi-thread control circuit 84, 1 switch 86 and 8 second cache modules 87 are shown. Fig.16 The full selection architecture of the computing core in the embodiment of the present invention adds a second cache module 87 and a switch 86, which can increase the flexibility of transmitting the convolution result to the convolution circuit 81. For example, the convolution result 1 can be transmitted to any convolution circuit from CU0 to CU7 through the switch 86.

[0129] Applied to the computing core shown above, the data processing method provided in the embodiment of the present application is introduced below.

[0130] The present application embodiment provides a data processing method, such as Fig.18 As shown, Fig.18 A flowchart of a data processing method provided in an embodiment of the present application. The method includes the following process.

[0131] Step 1801: The computing core controls the multiplexer to connect the output end of at least one convolution circuit coupled to the multiplexer to the output end of the multiplexer.

[0132] Step 1802: The external data output terminal of the multiplexer controlled by the computing core outputs the convolution results of the convolution circuits corresponding to the L multiplexers respectively.

[0133] Step 1803: The external data input terminal of the calculation kernel control convolution circuit inputs the feature map to be processed.

[0134] For example, compared with the matrix-type storage-computing integrated network structure, the computing core provided in the embodiment of the present application can flexibly meet the computing core's demand for computing resources in different dimensions such as network depth, number of input and output channels, and feature map size through the selection function of the multiplexer, thereby improving the computing resource utilization of the computing core and reducing the circuit area and cost of the chip. The specific implementation methods of steps 1801 to 1803 can be referred to the above-mentioned Figure 8 The description will not be repeated here.

[0135] Optionally, in order to expand the output channel of the computing core, the computing core controls the first multiplexer to connect the output end of the fifth convolution circuit to the output end of the first multiplexer; and controls the second multiplexer to connect the output end of the fifth convolution circuit to the output end of the second multiplexer. The specific output channel expansion method can be referred to the above-mentioned Fig.11 The description will not be repeated here.

[0136] Optionally, in order to save data access time, the method may also include: the computing core controls the first cache module to cache input data and weight parameters of other convolution circuits, and controls the convolution computing module to perform convolution operations based on the input data and weight parameters.

[0137] Optionally, in order to improve the computing efficiency of the computing core, the computing core can execute different data blocks in the same feature map in parallel. Therefore, the data processing method may also include: the computing core controls the multiplexer to transmit the first data block to the sixth convolution circuit at the first moment, and transmits the second data block to the seventh convolution circuit at the second moment; and the computing core controls the sixth convolution circuit to start executing the convolution operation of the first data block at the first moment, and controls the seventh convolution circuit to start executing the convolution operation of the second data block at the second moment. For specific parallel execution methods, please refer to the above-mentioned method based on Fig.12 The description will not be repeated here.

[0138] Optionally, in order to improve the utilization efficiency of the convolution circuit, the computing core can time-share multiple hardware threads. Therefore, the data processing method may include: after the convolution circuit obtains the third data block, controlling the convolution circuit to execute the first hardware thread, and after the convolution circuit obtains the fourth data block, controlling the convolution circuit to repeatedly execute the first hardware thread. Alternatively, the data processing method may also include: the computing core controls the eighth convolution circuit to execute the first hardware thread, and when the first hardware thread is completed, controls the eighth convolution circuit to continue to execute the second hardware thread calculation. The specific time-share multiplexing method can be referred to the above-mentioned based on Fig.15 The description will not be repeated here.

[0139] Specifically, the time-sharing multiplexing method is as follows: when the first hardware thread is completed, it is determined whether the second hardware thread meets the switching conditions, and the switching conditions include that the eighth convolution circuit has obtained the data block of the feature map of the convolution layer corresponding to the second hardware thread, and the first cache module corresponding to the output data block of the second hardware thread stored in the eighth convolution circuit is empty and the eighth convolution circuit is idle; if the second hardware thread meets the switching conditions, the convolution circuit is controlled to continue to execute the second hardware thread calculation.

[0140] Furthermore, the time-sharing multiplexing method may also include: after the second hardware thread meets the switching condition, the computing core controls the hardware multi-thread control circuit according to the weight parameter and priority of the second hardware thread to control the convolution circuit to continue to execute the second hardware thread calculation.

[0141] Apply the above data processing method in the computing kernel, such as Fig.19 and Fig. 20 As shown, Fig.19 A schematic diagram of computing resource utilization of a network algorithm based on a storage-computing integrated architecture provided in an embodiment of the present application, Fig. 20 A schematic diagram of computing resource utilization based on a computing core provided for an embodiment of the present application. Among them, the architecture based on storage and computing integration includes 8 convolution circuits, each convolution circuit includes 16 input / output channels, and the computing core includes 8 convolution circuits, each convolution circuit includes 16 input / output channels and 32 hardware threads. In addition, the computing core data block is the smallest processing unit, and the convolution calculation of all convolution layers is executed in the order of upper layer first and lower layer later according to the convolution network algorithm, thereby reducing the intermediate data storage capacity. The data flow passes through different convolution units in the computing core in turn, and the computing core is output only at the end, which greatly saves data access power consumption and reduces processing delay.

[0142] Depend on Fig.19 and Fig. 20It can be seen that the computing resource utilization rate of the computing core-based algorithm is higher than that of the storage-computing integrated network algorithm. Taking the first convolution layer as an example, when the network algorithm based on the computing core calculates 4 data blocks in parallel at the first layer, all 8 convolution circuits are in the execution state. However, in the storage-computing integrated network algorithm, only 2 convolution circuits are in the execution state, and the remaining 6 convolution circuits are in the idle state.

[0143] An embodiment of the present application also provides a chip, which includes a processor and a computing core, and the computing core is electrically connected to a printed circuit board.

[0144] An embodiment of the present application also provides an electronic device, which includes a printed circuit board and a computing core, and the computing core and the printed circuit board are electrically connected.

[0145] It is understandable that in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0146] The embodiment of the present application also provides an electronic device, including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the above-mentioned related method steps to implement the data processing method in the above-mentioned embodiment.

[0147] An embodiment of the present application further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the data processing method in the above-mentioned embodiment.

[0148] The embodiments of the present application further provide a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the data processing method executed by the electronic device in the above-mentioned embodiments.

[0149] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor may execute the computer execution instructions stored in the memory so that the chip executes the data processing method performed by the electronic device in the above-mentioned method embodiments.

[0150] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0151] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0156] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A computing core, characterized in that: include: A plurality of convolution circuits and a plurality of multiplexers, wherein the multiplexer comprises an output terminal and at least two input terminals, the output terminal of the multiplexer corresponds to the input terminal of the convolution circuit one by one, the output terminal of each convolution circuit in the plurality of convolution circuits is coupled to the input terminals of L multiplexers, L is an integer greater than or equal to 2, at least one of the L multiplexers further comprises an external data output terminal, and at least one of the L convolution circuits corresponding to the L multiplexers further comprises an external data input terminal; The multiplexer is used to conduct the output end of at least one convolution circuit coupled to the multiplexer to the output end of the multiplexer; The external data output terminal of the multiplexer is used to output the convolution results of the convolution circuits respectively corresponding to the L multiplexers; The external data input terminal of the convolution circuit is used to input the feature map to be processed.

2. The computing core according to claim 1, characterized in that: The computing core also includes multiple adders, and the multiple convolution circuits include a first convolution circuit and a second convolution circuit. The first input end of the first adder among the multiple adders is coupled to the output end of the first convolution circuit, the second input end of the first adder is coupled to the output end of the second convolution circuit, and the output end of the first adder is coupled to the input ends of the L multiplexers.

3. The computing core according to claim 2, characterized in that: The multiple convolution circuits also include a third convolution circuit and a fourth convolution circuit, and the multiple adders also include a second adder and a third adder, a first input terminal of the second adder is coupled to an output terminal of the third convolution circuit, a second input terminal of the second adder is coupled to an output terminal of the fourth convolution circuit, an output terminal of the second adder is coupled to a first input terminal of the third adder, a second input terminal of the third adder is coupled to an output terminal of the first adder, and an output terminal of the third adder is coupled to an input terminal of the L multiplexers.

4. The computing core according to claim 1, characterized in that: The plurality of convolution circuits include a fifth convolution circuit, the plurality of multiplexers include a first multiplexer and a second multiplexer; The first multiplexer is used to connect the output end of the fifth convolution circuit to the output end of the first multiplexer; The second multiplexer is used to connect the output end of the fifth convolution circuit to the output end of the second multiplexer.

5. The computing core according to any one of claims 1 to 4, characterized in that: The convolution circuit includes a first cache module and a convolution calculation module; The first cache module is used to cache input data and weight parameters of other convolution circuits; The convolution calculation module is used to perform a convolution operation based on the input data and weight parameters.

6. The computing core according to any one of claims 1 to 5, characterized in that: The multiple convolution circuits also include a sixth convolution circuit and a seventh convolution circuit. The input data of the external data input terminals of the multiple convolution circuits include a first feature map. The first feature map includes a first data block and a second data block. The multiplexer is also used to transmit the first data block to the sixth convolution circuit at a first moment, and to transmit the second data block to the seventh convolution circuit at a second moment. The sixth convolution circuit is used to start performing a convolution operation on the first data block at the first moment, and the seventh convolution circuit is used to start performing a convolution operation on the second data block at the second moment.

7. The computing core according to any one of claims 1 to 5, characterized in that: The computing core further includes a hardware multi-thread control circuit, the convolution circuit is used to execute at least one hardware thread, the at least one hardware thread includes a first hardware thread, and the hardware multi-thread control circuit is used to: After the convolution circuit acquires the third data block, controlling the convolution circuit to execute the first hardware thread; Among them, the at least one hardware thread corresponds one-to-one to the convolution operations of different convolutional layers, and the third data block is a data block of the feature map of the convolutional layer corresponding to the first hardware thread.

8. The computing core according to claim 7, characterized in that: The hardware multi-thread control circuit is also used for: After the convolution circuit acquires the fourth data block, controlling the convolution circuit to repeatedly execute the first hardware thread; Among them, the fourth data block is also a data block of the feature map of the convolution layer corresponding to the first hardware thread.

9. The computing core according to claim 7, characterized in that: The at least one hardware thread further includes a second hardware thread, the convolution circuit includes an eighth convolution circuit, and the hardware multi-thread control circuit is further configured to: Controlling the eighth convolution circuit to execute the first hardware thread; When the first hardware thread is completed, the eighth convolution circuit is controlled to continue to execute the second hardware thread calculation.

10. The computing core according to claim 9, characterized in that: The hardware multi-thread control circuit is specifically used to: when the first hardware thread is completed, determine whether the second hardware thread meets the switching condition, and the switching condition includes that the eighth convolution circuit has obtained the data block of the feature map of the convolution layer corresponding to the second hardware thread, and the first cache module corresponding to the output data block of the second hardware thread stored in the eighth convolution circuit is empty and the eighth convolution circuit is in an idle state; if the second hardware thread meets the switching condition, control the convolution circuit to continue to execute the second hardware thread calculation.

11. The computing core according to claim 10, characterized in that: The hardware multi-thread control circuit is also used to: after the second hardware thread meets the switching condition, control the convolution circuit to continue to execute the second hardware thread calculation according to the weight parameter and priority of the second hardware thread.

12. The computing core according to claim 1, characterized in that: An output terminal of each of the plurality of convolution circuits is coupled to input terminals of the plurality of multiplexers.

13. The computing core according to claim 1, characterized in that: The computing core includes n convolution circuits, where n is an integer greater than 1. The computing core also includes a switch and n second cache modules. The switch includes n input terminals and n output terminals. The output terminals of the n second cache modules correspond one-to-one to the n input terminals of the switch, and the n output terminals of the switch correspond one-to-one to the n convolution circuits respectively.

14. A data processing method, characterized in that: The data processing method is applied to a computing core, the computing core includes a plurality of convolution circuits and a plurality of multiplexers, the multiplexer includes an output end and at least two input ends, the output end of the multiplexer corresponds to the input end of the convolution circuit one by one, the output end of each convolution circuit in the plurality of convolution circuits is coupled to the input ends of L multiplexers, L is an integer greater than or equal to 2, at least one of the L multiplexers further includes an external data output end, and at least one of the L convolution circuits corresponding to the L multiplexers further includes an external data input end; the method includes: Controlling the multiplexer to conduct the output end of at least one convolution circuit coupled to the multiplexer to the output end of the multiplexer; Controlling the external data output terminal of the multiplexer to output the convolution results of the convolution circuits respectively corresponding to the L multiplexers; The external data input terminal of the convolution circuit is controlled to input the feature map to be processed.

15. The method according to claim 14, characterized in that The computing core also includes multiple adders, and the multiple convolution circuits include a first convolution circuit and a second convolution circuit. The first input end of the first adder among the multiple adders is coupled to the output end of the first convolution circuit, the second input end of the first adder is coupled to the output end of the second convolution circuit, and the output end of the first adder is coupled to the input ends of the L multiplexers.

16. The method according to claim 15, characterized in that The multiple convolution circuits also include a third convolution circuit and a fourth convolution circuit, and the multiple adders also include a second adder and a third adder, a first input terminal of the second adder is coupled to an output terminal of the third convolution circuit, a second input terminal of the second adder is coupled to an output terminal of the fourth convolution circuit, an output terminal of the second adder is coupled to a first input terminal of the third adder, a second input terminal of the third adder is coupled to an output terminal of the first adder, and an output terminal of the third adder is coupled to an input terminal of the L multiplexers.

17. The method according to claim 14, characterized in that The plurality of convolution circuits include a fifth convolution circuit, the plurality of multiplexers include a first multiplexer and a second multiplexer; and the method includes: Controlling the first multiplexer to connect the output end of the fifth convolution circuit to the output end of the first multiplexer; The second multiplexer is controlled to connect the output end of the fifth convolution circuit to the output end of the second multiplexer.

18. The method according to any one of claims 14 to 17, characterized in that: The convolution circuit includes a first cache module and a convolution calculation module; the method further includes: Controlling the first cache module to cache input data and weight parameters of other convolution circuits; The convolution calculation module is controlled to perform a convolution operation based on the input data and weight parameters.

19. The method according to any one of claims 14 to 18, characterized in that: The plurality of convolution circuits further include a sixth convolution circuit and a seventh convolution circuit, input data of the plurality of convolution circuits include a first feature map, the first feature map includes a first data block and a second data block, and the method further includes: Control the multiplexer to transmit the first data block to the sixth convolution circuit at a first moment, and to transmit the second data block to the seventh convolution circuit at a second moment; The sixth convolution circuit is controlled to start performing a convolution operation on the first data block at the first moment, and the seventh convolution circuit is controlled to start performing a convolution operation on the second data block at the second moment.

20. The method according to any one of claims 14 to 18, characterized in that: The computing core further includes a hardware multi-thread control circuit, the convolution circuit is used to execute at least one hardware thread, the at least one hardware thread includes a first hardware thread, and the method further includes: After the convolution circuit acquires the third data block, controlling the hardware multi-thread control circuit to control the convolution circuit to execute the first hardware thread; Among them, the at least one hardware thread corresponds one-to-one to the convolution operations of different convolutional layers, and the third data block is a data block of the feature map of the convolutional layer corresponding to the first hardware thread.

21. The method according to claim 20, characterized in that The method further comprises: After the convolution circuit acquires the fourth data block, controlling the hardware multi-thread control circuit to control the convolution circuit to repeatedly execute the first hardware thread; Among them, the fourth data block is also a data block of the feature map of the convolution layer corresponding to the first hardware thread.

22. The method according to claim 20, characterized in that The at least one hardware thread further includes a second hardware thread, the convolution circuit includes an eighth convolution circuit, and the method further includes: Controlling the hardware multi-thread control circuit to control the eighth convolution circuit to execute the first hardware thread; When the first hardware thread is completed, the hardware multi-thread control circuit is controlled to control the eighth convolution circuit to continue to execute the second hardware thread calculation.

23. The method according to claim 22, characterized in that The method of controlling the hardware multi-thread control circuit to control the eighth convolution circuit to continue to execute the second hardware thread calculation when the first hardware thread is completed includes: when the first hardware thread is completed, determining whether the second hardware thread satisfies a switching condition, the switching condition including that the eighth convolution circuit has obtained a data block of a feature map of a convolution layer corresponding to the second hardware thread, a first cache module corresponding to an output data block of the second hardware thread stored in the eighth convolution circuit is empty, and the eighth convolution circuit is in an idle state, and controlling the hardware multi-thread control circuit to control the convolution circuit to continue to execute the second hardware thread calculation.

24. The method according to claim 23, characterized in that The method further includes: after the second hardware thread satisfies the switching condition, according to the weight parameter and priority of the second hardware thread, controlling the hardware multi-thread control circuit to control the convolution circuit to continue to execute the second hardware thread calculation.

25. The method according to claim 14, characterized in that An output terminal of each of the plurality of convolution circuits is coupled to input terminals of the plurality of multiplexers.

26. The method according to claim 14, characterized in that The computing core includes n convolution circuits, where n is an integer greater than 1. The computing core also includes a switch and n second cache modules. The switch includes n input terminals and n output terminals. The output terminals of the n second cache modules correspond one-to-one to the n input terminals of the switch, and the n output terminals of the switch correspond one-to-one to the n convolution circuits respectively.

27. A chip, characterized in that: The chip includes a processor and a computing core as described in any one of claims 1 to 13, and the computing core is electrically connected to the processor.

28. An electronic device, characterized in that: The electronic device comprises a printed circuit board, and the computing core as described in any one of claims 1 to 13, wherein the computing core is electrically connected to the printed circuit board.

29. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 14 to 26.

Images