Neural network circuit, system and method of controlling data flow

Abstract

The application discloses a neural network circuit, a system and a method for controlling data flow, relates to the technical field of artificial intelligence, can be applied to a recurrent neural network, and helps to realize that data flow in the neural network is executed in a preserved order. The neural network circuit comprises a first physical engine which is virtualized into one or more logical engines. A first logical engine in the one or more logical engines is used for obtaining a first data frame, and performing calculation on the first data frame when a state of the first physical engine is a first state and a state of the first logical engine is an idle state. Wherein, the state of the first physical engine being the first state is used for indicating that there is no logical engine performing calculation on the first physical engine, and the state of the first logical engine being the idle state is used for indicating that a second logical engine can receive data, and the second logical engine is used for performing calculation on output data of the first logical engine.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to neural network circuits, systems, and methods for controlling data flow. Background Technology

[0002] Natural Language Processing (NLP) is one of the most challenging problems in artificial intelligence. Currently, significant progress has been made in areas such as speech recognition, voice wake-up, and translation. NLP techniques have also gradually shifted from traditional Hidden Markov Models (HMMs) to neural network models. Among these, Recurrent Neural Networks (RNNs) have been widely applied in NLP. RNNs are a type of neural network with memory capabilities; neurons can receive information not only from other neurons but also from themselves, forming a network structure with loops, making them well-suited for processing sequential data. In a RNN system, each loop performs the same computations. Furthermore, RNNs have strict sequential dependencies; consecutive loops cannot be executed in parallel. Within a single loop, computations also have a strict execution order. Therefore, a technical solution for controlling data flow in RNNs is needed. Summary of the Invention

[0003] This application provides neural network circuits, systems, and methods for controlling data flow, which can be applied to recurrent neural networks, and of course, to other neural networks, helping to ensure that data flow in neural networks is executed in an ordered manner.

[0004] To achieve the above objectives, the embodiments of this application provide the following technical solutions:

[0005] Firstly, a neural network circuit is provided, including a first physical engine, which is virtualized into one or more logical engines. The first logical engine among these one or more logical engines is used to: obtain a first data frame; and perform computation on the first data frame when the first physical engine is in a first state and the first logical engine is in an idle state. The first physical engine being in a first state indicates that no logical engine on the first physical engine is performing computation, and the first logical engine being in an idle state indicates that a second logical engine can receive data. The second logical engine is used to perform computation on the output data of the first logical engine. The first physical engine can be any physical engine in the neural network circuit, and the first logical engine can be a logical engine on the first physical engine. The first data frame is the data required for the first logical engine to initiate a computation. The second logical engine belongs to the first physical engine or is a physical engine in the neural network circuit other than the first physical engine.

[0006] In this technical solution, the physical engine is virtualized into one or more logical engines, and the first logical engine is conditionally controlled to start computation. Specifically, after the first logical engine receives a data frame, computation can only start if there is no logical engine performing computation on its corresponding physical engine; that is, at most one logical engine can perform computation on the first physical engine at any given time. Furthermore, computation can only start if the second logical engine (i.e., the logical engine that receives the output data from the first logical engine) can receive the data (i.e., the buffer space corresponding to the second logical engine is free). Thus, after the first logical engine completes its computation, the result can be immediately sent to the second logical engine. Combining these two aspects helps to achieve ordered computation between logical engines in a neural network circuit, thereby facilitating the ordered execution of data streams in the neural network.

[0007] Furthermore, at most one logical engine can perform calculations on the first physical engine at any given time. This facilitates time-division multiplexing of physical resources among multiple logical engines deployed on the same physical engine, thereby improving the utilization rate of physical resources. Once the first logical engine completes its calculation, the result can be immediately sent to the second logical engine. In other words, the data obtained after the first logical engine's calculation does not need to be cached and can be directly used for calculations in the second logical engine, thus saving cache space.

[0008] In one possible design, the first logic engine and the second logic engine are logic engines that perform computations within a sub-execution process. That is, the first logic engine is not the last logic engine to perform computations within a sub-execution process. For a logic engine that performs computations last within a sub-execution process, its receiving node can be considered to always be able to receive data. Therefore, it is not necessary to set a state for this logic engine. This helps save on the bit width of the state register, thereby saving resource overhead; furthermore, it helps reduce the complexity of maintaining the state of the logic engine.

[0009] Optionally, regarding the steps performed by the last logic engine to perform computation within a sub-execution process, please refer to the description of the fourth logic engine below.

[0010] In a recurrent neural network (RNN), each loop in the first m-1 loops of the recurrent layer is a sub-execution process. The m-th loop of the recurrent layer, together with the non-recurrent process, constitutes a sub-execution process of the neural network circuit. Here, m is the number of loops in the recurrent layer, m is greater than or equal to 2, and m is an integer. For a non-recurrent network, a sub-execution process is equivalent to an execution process.

[0011] In one possible design, the first and second logic engines are logic engines that perform computations within a sub-execution process, and the second logic engine does not belong to the first physical engine. Since each physical engine corresponds to a cache queue, when any logic engine on the same physical engine performs computation, that cache queue is already empty. Therefore, the receiving node of that logic engine (i.e., the logic engine that receives its output data) can receive the data. Thus, it is not necessary to set a state for this logic engine, which helps save the bit width of the state register, thereby saving resource overhead; and it also helps reduce the complexity of maintaining the state of the logic engine. Optionally, regarding the steps executed by a logic engine when it and its receiving node belong to the same physical engine within a sub-execution process, refer to the description of the sixth logic engine below.

[0012] In one possible design, the first logic engine is also used to: set the first logic engine's state to a busy state when the number of calculations performed by the first logic engine reaches a preset number. The busy state of the first logic engine indicates that the second logic engine cannot receive data. The preset number of calculations is the total number of calculations performed by the first logic engine in a sub-execution process. In other words, if the number of executions has not reached the preset number, the current logic engine's state is not set to busy. This technical solution is proposed considering that "if the current logic engine's state is set to busy at this time, and no logic engine has yet cleared its state, then the current logic engine cannot start the next calculation, and the neural network circuit will lock up."

[0013] In one possible design, the computation performed by the first logic engine on the first data frame constitutes a sub-execution process. The first logic engine is the last logic engine to perform the computation on the first physical engine within this sub-execution process. In this case, the first logic engine is further configured to: when the number of computations performed by the first logic engine reaches a preset number, send a first indication message to the third logic engine. The third logic engine is configured to send data to the first target logic engine, which is the logic engine on the first physical engine that performed the first computation in the sub-execution process. The first indication message instructs the third logic engine to set its own state to an idle state. The idle state of the third logic engine instructs the first target logic engine to receive data. This technical solution is proposed considering that "other logic engines on the same physical engine will not start the computation of the next sub-execution process if a sub-execution process on the same physical engine has not been completed," or that "after a sub-execution process on the same physical engine is completed, the physical engine can start the next sub-execution process (i.e., the neural network circuit performs pipelined computation at the physical engine granularity)."

[0014] In one possible design, the computation performed by the first logic engine on the first data frame is part of a sub-execution process. The first logic engine is not the last logic engine to perform computation on the first physical engine within this sub-execution process. In this case, the first logic engine is further configured to: when the number of computations performed by the first logic engine reaches a preset number, send a first indication message to the third logic engine. This first indication message instructs the third logic engine to set its own state to an idle state. The third logic engine then sends data to a first target logic engine located on the first physical engine, which is the next logic engine to perform computation after the first logic engine performs computation in the sub-execution process. In this way, the data required for computation by the next logic engine on the first physical engine can arrive during the computation performed by the first logic engine. That is, the computation delay of the first logic engine masks the transmission delay of the next logic engine on its physical engine, allowing the next logic engine to start computation immediately after the first logic engine completes its computation, without waiting for data to arrive.

[0015] In one possible design, the neural network circuit further includes a fourth logic engine. The fourth logic engine and the first logic engine are logic engines within the same sub-execution process, and the fourth logic engine is the last logic engine to perform computation within that sub-execution process. The fourth logic engine is used to obtain the second data frame. The fourth logic engine is used to compute the second data frame when the state of the second physical engine to which it belongs is in the first state. The first state of the second physical engine indicates that no logic engine is currently performing computation on the second physical engine. For a logic engine that is the last to perform computation in a sub-execution process, it can be assumed that the receiving node of that logic engine is always able to receive data. Therefore, it is not necessary to set a state for that logic engine. This helps save the bit width of the state register, thereby saving resource overhead; and it also helps reduce the complexity of maintaining the state of the logic engine.

[0016] In one possible design, the fourth logic engine is also used to send a second instruction message to the fifth logic engine, which in turn sends data to the second target logic engine, the first logic engine to perform computation in the sub-execution process on the second physical engine. The second instruction message indicates that the second target logic engine can receive the data. This helps the first logic engine performing computation on the second physical engine to begin executing the next sub-execution process, thereby facilitating the pipelined computation of the neural network circuit at the physical engine granularity.

[0017] In one possible design, the neural network circuit further includes a sixth logic engine and a seventh logic engine. The seventh logic engine receives the output data from the sixth logic engine. The sixth, seventh, and first logic engines are logic engines within the same sub-execution process, and the sixth and seventh logic engines belong to the third physical engine. The sixth logic engine is used to obtain the third data frame. When the third physical engine is in its first state, the sixth logic engine performs computation on the third data frame. The first state of the third physical engine indicates that no logic engine is currently performing computation on that physical engine. Since each physical engine corresponds to a buffer queue, when any logic engine performs computation on the same physical engine, the buffer queue is already empty. Therefore, the receiving node of that logic engine can receive data. Thus, setting a state for that logic engine is unnecessary, which helps save the bit width of the state register, thereby saving resource overhead; and it also helps reduce the complexity of maintaining the state of the logic engine.

[0018] In one possible design, the first logic engine is further configured to: after the first logic engine starts computation, set the state of the first physical engine to a second state, which indicates that the physical resources of the first physical engine are occupied; and after the first logic engine sends the computation result to the second logic engine, set the state of the first physical engine to a first state. This facilitates time-sharing reuse of the physical engine's resources among logic engines on the same physical engine.

[0019] Secondly, a method for controlling data flow in a neural network circuit is provided. The neural network circuit includes a first physical engine, which is virtualized into one or more logical engines. The method is applied to the first logical engine among the one or more logical engines. The method includes: obtaining a first data frame; and performing computation on the first data frame when the state of the first physical engine is a first state and the state of the first logical engine is an idle state. The first state of the first physical engine indicates that no logical engine is performing computation on the first physical engine, and the idle state of the first logical engine indicates that a second logical engine can receive data. The second logical engine performs computation on the output data of the first logical engine.

[0020] In one possible design, the method further includes: when the number of calculations by the first logic engine reaches a preset number, setting the state of the first logic engine to a busy state, wherein the busy state of the first logic engine is used to indicate that the second logic engine cannot receive data.

[0021] In one possible design, the calculation performed by the first logic engine on the first data frame is a sub-execution process, and the first logic engine is the last logic engine to perform the calculation on the first physical engine in the sub-execution process. The method further includes: when the number of calculations performed by the first logic engine reaches a preset number, sending a first indication message to the third logic engine, wherein the third logic engine is used to send data to the first target logic engine, the first target logic engine is the logic engine that performed the first calculation in the sub-execution process on the first physical engine, and the first indication message is used to instruct the third logic engine to set its own state to an idle state, the idle state of the third logic engine indicating that the first target logic engine can receive data.

[0022] In one possible design, the computation performed by the first logic engine on the first data frame belongs to a sub-execution process. The first logic engine is not the last logic engine to perform computation on the first physical engine within the sub-execution process. The method further includes: when the first logic engine's computation count reaches a preset number, sending a first indication message to the third logic engine. This first indication message instructs the third logic engine to set its own state to an idle state. The third logic engine then sends data to a first target logic engine, which is located on the first physical engine and is the next logic engine to perform computation after the first logic engine performs computation within the sub-execution process.

[0023] In one possible design, the method further includes: after the first logic engine starts calculation, setting the state of the first physical engine to a second state, the second state being used to indicate that the physical resources of the first physical engine are occupied; and after the first logic engine sends the calculation result to the second logic engine, setting the state of the first physical engine to a first state.

[0024] The explanation of the relevant content and the description of the beneficial effects in the second aspect and its possible designs can be found in the description of the first aspect or its corresponding design, and will not be repeated here.

[0025] Thirdly, a neural network system is provided, comprising: a processor and neural network circuitry as provided in the first aspect or any possible design of the first aspect, wherein the processor is configured to send one or more data frames to the neural network circuitry. The one or more data frames include a first data frame.

[0026] Fourthly, a computer-readable storage medium is provided for storing a computer program that, when run on a computer, causes the computer to perform the methods provided by the second aspect or any possible design of the second aspect.

[0027] Fifthly, a computer program product is provided that, when run on a computer, causes the method provided in the second aspect above or any possible design thereof to be executed.

[0028] It is understood that any of the methods, neural network systems, computer-readable storage media, or computer program products provided above are used to execute the corresponding neural network circuits provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding neural network circuits, which will not be repeated here. Attached Figure Description

[0029] Figure 1 This application provides a schematic diagram of the structure of a neural network system according to an embodiment of the present application.

[0030] Figure 2 This application provides a schematic diagram of the logical structure of a recurrent neural network system.

[0031] Figure 3 This is a schematic diagram illustrating the result of an embodiment of a smart earphone provided in this application;

[0032] Figure 4 This is a schematic diagram of a neural network layer in a neural network circuit provided in an embodiment of this application;

[0033] Figure 5 A schematic diagram illustrating a method for deploying a neural network circuit according to an embodiment of this application;

[0034] Figure 6A A schematic diagram illustrating the deployment result of a neural network circuit provided in an embodiment of this application;

[0035] Figure 6B A schematic diagram illustrating the deployment result of another neural network circuit provided in an embodiment of this application;

[0036] Figure 7 A flowchart illustrating a method for controlling data flow in a neural network circuit, provided as an embodiment of this application;

[0037] Figure 8 A flowchart illustrating another method for controlling data flow in a neural network circuit provided in an embodiment of this application;

[0038] Figure 9 A flowchart illustrating another method for controlling data flow in a neural network circuit provided in an embodiment of this application;

[0039] Figure 10 A flowchart illustrating another method for controlling data flow in a neural network circuit provided in an embodiment of this application;

[0040] Figure 11This is a schematic diagram of the structure of a neural network circuit provided in an embodiment of this application;

[0041] Figure 12 This is a schematic diagram of another neural network circuit provided in an embodiment of this application;

[0042] Figure 13 This is a schematic diagram of another neural network circuit provided in an embodiment of this application;

[0043] Figure 14 This is a schematic diagram of another neural network circuit provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions provided in this application can be applied to artificial neural networks (ANNs). Artificial neural networks, also known as neural networks (NNs) or neural network-like systems, are mathematical or computational models in machine learning and cognitive science that mimic the structure and function of biological neural networks (such as the central nervous system of animals, especially the brain), used to estimate or approximate functions. Artificial neural networks can include convolutional neural networks (CNNs), deep neural networks (DNNs), time-delay neural networks (TDNNs), multilayer perceptrons (MLPs), and recurrent neural networks (RNNs), among others.

[0045] Figure 1 This is a schematic diagram of the structure of a neural network system provided in an embodiment of this application. Figure 1 As shown, the neural network system 100 may include a processor 105 and a neural network circuit 110. The neural network circuit 110 is connected to the processor 105 via an interface. This interface may include a Serial Peripheral Interface (SPI) and a Peripheral Component Interconnect Express (PCIe) interface, etc. Figure 1 As shown, the neural network circuit 110 can be connected to the processor 105 via the PCIe bus 106. Therefore, data can be input to the neural network circuit 110 via the PCIe bus 106, and processed data can be received from the neural network circuit 110 via the PCIe bus 106. Furthermore, the processor 105 can also monitor the operating status of the neural network circuit 110 through the interface.

[0046] Processor 105 may include multiple processor cores. Processor 105 may be a very large-scale integrated circuit. The processor core may be a central processing unit (CPU), or it may be another application-specific integrated circuit (ASIC) or DSP, etc.

[0047] The neural network circuit 110 includes an input / output interface (TxRx) 1252 and a switching device 1254. The input / output interface 1252 receives data transmitted to the neural network circuit 110 via the PCIe bus 106 and sends processed data from the neural network circuit 110 back to the PCIe bus 106. The neural network circuit 110 may include one or more process elements (PEs) 1256, which perform neural network calculations on the data input to the neural network circuit 110. The calculation results of the process elements 1256 can be sent to other process elements 1256 via the switching device 1254. In practical applications, a process element 1256 may include modules implementing convolution, pooling, or other neural network operations. The specific circuitry or function of the process element 1256 is not limited here.

[0048] A processing unit 1256 may include one or more hardware engines (HE) 1302, and a hardware engine 1302 may include one or more logical engines 1304. A hardware engine 1302 is a processing unit within the processing unit 1256 that has independent hardware resources. For example, a hardware engine 1302 may be a matrix multiplication and addition module (such as a crossbar), an activation module, or a dot product module. Different hardware engines 1302 can perform parallel computations. A logical engine 1304 is a logically independently executable processing unit, and multiple logical engines 1304 on a hardware engine 1302 can time-share the hardware resources of that hardware engine 1302.

[0049] Due to the advantages of integrated storage and computation, resistive random-access memory (ReRAM) or NOR flash memory have been widely used in neural network systems in recent years. For example, a ReRAM crossbar composed of multiple memristor cells can be used to perform matrix multiplication and addition operations in a neural network system. The physics engine 1302 can be implemented using a ReRAM crossbar.

[0050] Optionally, the technical solutions provided in this application can be applied to recurrent neural networks, such as long short-term memory (LSTM) networks, gated recurrent unit (GRU) networks, convolutional recurrent neural networks (CRNN), and variations thereof. The following description uses the application of the technical solutions provided in this application to recurrent neural networks as an example. It is understood that a non-recurrent neural network can be considered a special type of recurrent neural network that executes one loop. Therefore, the embodiments of this application can also be applied to non-recurrent neural networks. This will be consistently explained here and will not be repeated below.

[0051] like Figure 2 The diagram shown is a logical structure schematic of a recurrent neural network system provided in an embodiment of this application. Figure 2 As shown in Figure a, the recurrent neural network system includes an input layer 201, a hidden layer 202, and an output layer 203. The hidden layer 202 performs iterative calculations on the data received from the input layer 201 and outputs the results to the output layer 203. As an example, combined with... Figure 1 , Figure 1 The processor 105 and neural network circuit 110 in the middle can be used by Figure 2 The hidden layer 202 is implemented in the middle.

[0052] Recurrent neural networks introduce the concept of "memory." The "recurrence" arises from the fact that each iteration of hidden layer 202 performs the same computation, and the output depends on the input (i.e., the data frame x input to the current iteration). t ) and "memory" (i.e., the calculation result h of the previous iteration of the current iteration) t-1 Therefore, it can be seen that recurrent neural networks have strict sequential dependencies, and the preceding and following loops cannot be executed in parallel. Furthermore, within a single loop, the computations also have a strict execution order.

[0053] Taking the GRU network as an example of a recurrent neural network, the recurrent process obtained after the hidden layers are unfolded in chronological order is as follows: Figure 2 As shown in Figure b. Specifically, during the (t-1)th loop, data frame x t-1 The calculation result h of the (t-2)th cycle process t-2 The result h is obtained through calculation (e.g., multiplying by W). t-1 During the t-th loop, data frame x t The calculation result h of the (t-1)th cycle process t-1 The result h is obtained through calculation (e.g., multiplying by W). t During the (t+1)th iteration, data frame x t+1 The calculation result h of the t-th cycle process t The result h is obtained through calculation (e.g., multiplying by W). t+1 Where t is an integer greater than or equal to 2. It is understandable that when t = 2, the result of the 0th loop can be a predefined value.

[0054] The technical solutions provided in this application can be applied to scenarios involving natural language processing. For example, the neural network system provided in this application can be used to implement speech recognition, voice wake-up, and speech translation. In speech recognition, for example, it can be applied to scenarios such as "speech-to-text conversion" and voice control in automobiles. In voice wake-up, for example, it can be applied to voice wake-up in robots, mobile phones, wearable devices, smart homes, and in-vehicle systems. In speech translation, for example, it can be applied to neural machine translation.

[0055] Example 1: In practical applications, the output of some tasks is related to the content before them, such as cloze tests. To fill in the blank in "There was once a sincere feeling placed before me, but I did not ___", you not only need to know all the preceding words, but also the order of the words.

[0056] If this sentence were written using a traditional language model, the options would be: 'Beijing', 'school', 'cherish', 'go to work', etc. Traditional language models are based on statistical models, and can only utilize a very limited amount of information preceding the blank. Currently, the most commonly used traditional language models might utilize preceding information such as: 'went to…', 'didn't go to…', 'I didn't go to…', and anything longer would consume a massive amount of storage space.

[0057] Clearly, even if the available information is "I didn't go...", the probability of choosing "cherish" wouldn't be greater than choosing "Beijing"; however, a recurrent neural network would choose "cherish," and with a much higher probability than "Beijing" or "going to work." This is because a recurrent neural network not only utilizes information like "I didn't go..." but also information like "presented to me" and "once had a sincere relationship." Once this information is utilized, it becomes clear that words like "going to work" and "school" don't fit the meaning when placed on the blanks.

[0058] Example 2: The technical solution provided in this application can be applied to voice wake-up tasks for end-side devices (such as mobile phones, smart headphones, smart speakers, smart TVs, and tablets). Taking voice wake-up of smart headphones as an example, for instance... Figure 3 The diagram shows a hardware structure of a smart headset 70. The smart headset includes a microphone (MIC) 701, a digital signal processor (DSP) 702, and a computer-in-memory (CIM) chip 703. The microphone 701, DSP 702, and CIM chip 703 can be connected via SPI.

[0059] As an example, combined Figure 1 , Figure 1 The neural network system 100 in the middle can be Figure 3 Implemented using DSP702 and CIM chip703, among which, Figure 1 The processor 105 in the middle can be used Figure 3 Implemented using DSP702, Figure 1 The neural network circuit 110 can be located in the CIM chip 703.

[0060] Microphone 701 can be used to acquire speech signals and send them to DSP 702. DSP 702 can process the received speech signals to obtain multiple speech frames, and input each of these speech frames as a data frame for a loop process of CIM chip 703. CIM chip 703 is an in-memory computing chip used to perform multiple loop calculations on the received speech frames to obtain the speech recognition result.

[0061] In one implementation, the CIM chip 703 can determine whether the speech recognition result matches a predefined wake-up word (if the predefined wake-up word is included, the two are considered to match), and send the determination result to the DSP 702. The DSP 702 then sends the matching result to the woken-up device (such as a mobile phone or speaker connected to smart headphones). The woken-up device then performs the wake-up action. In another implementation, the CIM chip 703 can send the speech recognition result to the DSP 702, and the DSP 702 can determine whether the speech recognition result matches a predefined wake-up word. Here, the wake-up action is the action indicated by the speech recognition result. For example, if the speech recognition result is "open WeChat," the wake-up action could be the action of launching the WeChat application on a mobile phone.

[0062] The above examples illustrate the application of recurrent neural networks (RNNs) in natural language processing; however, actual implementations are not limited to this. For instance, RNNs can also be applied to machine translation, generating image descriptions, and video tagging.

[0063] The following is a brief introduction to the technologies and terms involved in the embodiments of this application.

[0064] 1) Data frames, voice frames

[0065] A data frame is a set of data that can initiate a single computation. For example, for a logic engine, if initiating a convolution computation requires 40-dimensional data as input, then a data frame for performing the convolution computation is 40-dimensional.

[0066] A speech frame is a type of data frame, specifically a data frame obtained after processing a speech signal.

[0067] 2) Neural Network Layer

[0068] A neural network system can include multiple neural network layers. A neural network layer is a logical concept; one neural network layer represents one instance of a neural network operation. Each layer's computation is performed by computational nodes. Neural network layers can include convolutional layers, pooling layers, etc. Figure 4 As shown, the neural network system 300 may include six neural network layers (also referred to as a 6-layer neural network, or 6 layers): layer 302, layer 304, layer 306, layer 308, layer 310, and layer 312. Layer 302 can perform convolution operations, layer 304 can perform pooling operations on the output data of layer 302, layer 306 can perform convolution operations on the output data of layer 304, layer 308 can perform convolution operations on the output of layer 306, and layer 310 can perform summation operations on the output data of layer 204 and layer 308, etc. Figure 4 This is just a simple example and explanation of the neural network layers in a neural network system. It does not restrict the specific operations of each neural network layer. For example, the fourth layer 308 can also be a pooling operation, and the fifth layer 310 can also be a convolution operation or a pooling operation, or other neural network operations.

[0069] In recurrent neural network systems, neural network layers can be divided into recurrent layers and non-recurrent layers. For example, in... Figure 4 In the neural network system shown, the system can obtain a single output by performing m recurrent layer calculations and one non-recurrent layer calculation, as shown. Figure 4 As shown in the figure. Where m is an integer greater than or equal to 2.

[0070] For example, in speech recognition, after the neural network system acquires the speech to be recognized, it performs m calculations from "Layer 1 302 to Layer 5 310" to obtain the output of the recurrent layer. In the first m-1 loops, the calculation result of Layer 5 310 and a speech frame are used as the input data for Layer 1 302 in the next loop. After the m-th loop, the calculation result of Layer 5 310 is used as the output of the recurrent layer. The output of the recurrent layer is then fed to Layer 6 312, which performs calculations based on the output of the recurrent layer to obtain the output of the neural network circuit, i.e., the speech recognition result.

[0071] 3) Sub-layers of neural networks

[0072] A neural network layer can include one or more neural network sublayers. As an example, a neural network sublayer can be one or more operators (such as multiple operators with dependencies) within a neural network layer. Taking a GRU layer as an example, a single computation in the GRU layer requires multiple formulas, each of which can be considered an operator, and each operator is a neural network sublayer.

[0073] It is understandable that if a neural network layer includes a neural network sublayer, then the concepts of the neural network layer and the neural network sublayer are the same and can be used interchangeably. This will be explained in a unified manner here and will not be repeated below.

[0074] 4) Dependency relationships

[0075] Two neural network layers with a dependency relationship refer to two neural network layers whose input data includes the output data of the other neural network layer. For example, ... Figure 4As shown, the output data of the first layer 302 is the input data of the second layer 304; therefore, the first layer 302 and the second layer 304 are dependent. The output data of the second layer 304 is the input data of the third layer 306, and the input data of the fifth layer 310 includes the output data of the second layer 304. Therefore, the second layer 304 and the third layer 306 are dependent, and the second layer 304 and the fifth layer 310 are also dependent. Two neural network sublayers with a dependency relationship refer to a situation where the input data of one neural network sublayer includes the output data of the other neural network sublayer.

[0076] 5) One execution process, one loop process, and one sub-execution process

[0077] For ease of description, in the embodiments of this application, the process from when the neural network system acquires input data (e.g., a speech to be executed) to when it obtains the output result of the neural network system based on the input data (e.g., obtaining the speech recognition result of the speech to be executed) is referred to as "an execution process" of the neural network system.

[0078] In the execution process of a neural network system, a recurrent layer needs to execute multiple loop processes, such as those based on... Figure 4 The example in the example requires executing m loop processes, with each loop performing calculations on "layer 1 302, layer 2 304, layer 306, layer 4 308, and layer 5 310". For non-loop layers, one non-loop process needs to be executed, such as based on... Figure 4 In the example shown, layer 6, 312, requires one non-cyclic process.

[0079] Within a loop, each layer of a neural network can perform one or more computations. For example, based on... Figure 4 In the example, within a loop, layer 1 (302) performs 5 calculations and then sends the result to layer 2 (304). Layer 2 (304) performs 1 calculation and then sends the result to layer 3 (306) and layer 5 (310). Layer 3 (306) performs 2 calculations and then sends the result to layer 4 (308). Layer 4 (308) performs 2 calculations and then sends the result to layer 5 (310). After layer 5 (310) performs 1 calculation, the loop ends.

[0080] It is understandable that when m=1, all neural network layers in the neural network system are non-recurrent layers.

[0081] For ease of description, in this embodiment, an execution process of a neural network system is divided into multiple sub-execution processes. Specifically, each loop process in the first m-1 loops of the recurrent layer is a sub-execution process, and the m-th loop process of the recurrent layer, together with the non-recurrent processes, forms a single sub-execution process. For example, based on... Figure 4 In the example, assuming m = 20, each of the first 19 loop processes can be a sub-execution process, in which the first to fifth levels participate in the calculation; the 20th loop process and the non-loop process together form a sub-execution process, in which the first to sixth levels participate in the calculation.

[0082] For a non-recurrent neural network, a sub-execution process is equivalent to an execution process.

[0083] 6) The sending node and receiving node of the logic engine.

[0084] The dependencies between logic engines correspond to the dependencies between sub-layers deployed on a logic engine. These dependencies can be represented by the execution order of the logic engines. For example, if the output data of one sub-layer can be used as the input data of another sub-layer, then the output data of the logic engine containing that sub-layer can be used as the input data of the logic engine containing that other sub-layer. These two logic engines have a dependency relationship, and the execution order of the logic engine containing the first sub-layer precedes that of the logic engine containing the second sub-layer.

[0085] Based on the execution order among the logic engines, the logic engine that performs the calculation first, the logic engine that performs the calculation second, and so on until the logic engine that performs the calculation last can be determined.

[0086] If the output data of a sublayer deployed on the current logic engine can be used as the input data of another sublayer, then the current logic engine is the sending node of the logic engine containing that other sublayer, and the logic engine containing that other sublayer is the receiving node of the current logic engine. For example, suppose the neural network circuit includes logic engines 1 to 7, and the execution order of logic engines 1 to 7 is from logic engine 1 to logic engine 7. Then, the first logic engine to perform calculations is logic engine 1, and the last logic engine to perform calculations is logic engine 7. Furthermore, the sending node of logic engine i+1 is logic engine i, and the receiving node of logic engine i is logic engine i+1, where 1 ≤ i ≤ 6, and i is an integer.

[0087] The sending node of the first logic engine performing computation in the neural network circuit (such as logic engine 1 in this example) is the sending node of the neural network circuit, i.e., the device / module that sends out the data frame. The receiving node of the last logic engine performing computation in the neural network circuit (such as logic engine 7 in this example) can be the receiving node of the neural network circuit. The technical solution provided in the embodiments of this application is applied to... Figure 3 Taking the smart earphone shown as an example, logic engines 1 to 7 can be located in the CIM chip 703, and the transmitting node of logic engine 1 and the receiving node of logic engine 7 can both be DSP 702.

[0088] 7) Indication information, clear status command

[0089] A sends an instruction message to B, which indicates that B's receiving node can receive data, or that B can send data to its receiving node.

[0090] A clear status command is a specific implementation of an instruction message. For example, A sends a clear status command to B, which instructs B to set its own status to idle, indicating that B's receiving node is ready to receive data. If B's ​​status is idle at any given time, it can maintain its current status after receiving the clear status command; if B's ​​status is busy at any given time, it can change its status from busy to idle after receiving the clear status command.

[0091] For ease of understanding, the following embodiments are all illustrated using the command indicating a clear status.

[0092] Understandably, if the module / device receiving the clear status command is not a logic engine (as it is), Figure 3 If the DSP702 is not a logic engine, then the module / device / device is not a logic engine (e.g., DSP702). Figure 3 The DSP702 in the example can be left unset in terms of status. Therefore, in this case, the clear status command and the indication information have the same meaning and can be used interchangeably.

[0093] 8) Other terms

[0094] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0095] In the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0096] The following describes the deployment method of the neural network circuit provided in the embodiments of this application. The entity executing the deployment method of the neural network circuit provided below may be... Figure 1 The processor 105 in it.

[0097] like Figure 5 The diagram shown is a schematic representation of a method for deploying a neural network circuit according to an embodiment of this application. Figure 5 The method shown may include the following steps:

[0098] S101: The processor acquires metadata information of the neural network circuit. This metadata information characterizes the attributes of the neural network system. For example, the metadata information of the neural network circuit may include one or more of the following: which layers the neural network system includes; whether there are dependencies between layers; whether each layer is a recurrent or acyclic layer; how many times the recurrent layer needs to be executed in one execution process; how many computations each recurrent layer needs to perform in one recurrent process; which sublayers each layer includes; and whether there are dependencies between sublayers within the same layer.

[0099] S102: Based on the dependencies between layers in the neural network circuit, and the dependencies between sub-layers within the same layer, the processor may optionally combine optimization objectives (such as high utilization of the physical engine and / or low computational latency) to deploy each sub-layer in the neural network system on a logic engine. Each logic engine corresponds one-to-one with a sub-layer.

[0100] In one implementation, for any recurrent or acyclic layer, the processor can deploy sub-layers belonging to different layers on different physical engines; deploy dependent sub-layers within the same layer on the same physical engine, and deploy non-dependent sub-layers within the same layer on different physical engines. Since different physical engines can execute in parallel, this deployment can improve the overall computational efficiency of the neural network circuit.

[0101] See Figure 6AThis is a schematic diagram illustrating the deployment result of a neural network circuit provided in an embodiment of this application. The neural network circuit includes physical engines 1 to 4, wherein logic engines 1, 3, and 4 are deployed on physical engine 1, logic engines 2 and 5 are deployed on physical engine 2, logic engine 6 is deployed on logic engine 3, and logic engine 7 is deployed on physical engine 4. The layers in which the sub-layers deployed in logic engines 1 to 6 are located are circular layers, and the layers in which the sub-layers deployed in logic engine 7 are located are non-circular layers.

[0102] In another implementation, acyclic and circular layers can be deployed on the same physics engine or on different physics engines; dependent sub-layers within the same layer are deployed on the same physics engine, while non-dependent sub-layers within the same layer are deployed on different physics engines. This deployment saves physics engine resources, i.e., saves hardware resources.

[0103] See Figure 6B This is a schematic diagram illustrating the deployment result of a neural network circuit provided in an embodiment of this application. The neural network circuit includes physical engines 1 to 3, wherein logic engines 1, 3, and 4 are deployed on physical engine 1, logic engines 2 and 5 are deployed on physical engine 2, and logic engines 6 and 7 are deployed on logic engine 3. The sub-layers deployed in logic engines 1 to 6 are located in a recurrent layer, and the sub-layers deployed in logic engine 7 are located in a non-recurrent layer.

[0104] The above are merely examples and do not limit the deployment of each sublayer in the neural network circuit applicable to the embodiments of this application. The technical solutions provided in the embodiments of this application can be applied to the deployment of any neural network system.

[0105] S103: The processor establishes a mapping between physical engines (e.g., each physical engine) and status registers. For example, it establishes a mapping between each physical engine and a first status register and a second status register.

[0106] A first status register corresponding to the physical engine is used to record whether the current state of the physical engine is an available state (i.e., the first state in this application) or an unavailable state (i.e., the second state in this application). An available state indicates that there is no logic engine performing computation on the physical engine. An unavailable state indicates that there is a logic engine performing computation on the physical engine. For example, the available state is represented by the binary number "0", and the unavailable state is represented by the binary number "1".

[0107] The second status register corresponding to the physical engine is used to record the current state of the logical engines deployed on that physical engine: busy or idle. A busy state indicates that the receiving node of that logical engine cannot receive data. An idle state indicates that the receiving node of that logical engine can receive data. For example, a binary number "0" represents the idle state, and a binary number "1" represents the busy state.

[0108] In software, a status register can be understood as a bitmap. For example, the bitmap used to implement the first status register can have 1 bit. For example, if a physical engine has N logical engines deployed, the bitmap used to implement the second status register allocated to that physical engine can have N bits, with each bit corresponding to one logical engine and representing its state.

[0109] In practice, the first and second status registers corresponding to a physical engine can be managed separately or uniformly. Uniform management can be understood as the first and second status registers being the same register. For example, one bit in a bitmap can represent the status of the physical engine, and each bit in the other bits of the bitmap can represent the status of a logical engine on that physical engine.

[0110] Optionally, if a logic engine and its receiving node are deployed on the same physical engine, the processor may not need to set a state for that logic engine. This is because the state of a logic engine indicates whether its receiving node can receive data. A physical engine corresponds to a cache queue, and on the same physical engine, when any logic engine performs computation, the cache queue is already empty. At this point, the receiving node of that logic engine can receive data, so setting a state for that logic engine is unnecessary. This helps save on the bit width of the status register, thus saving resource overhead; and it also helps reduce the complexity of maintaining the state of the logic engines.

[0111] Optionally, for the logic engine that performs the last computation in each sub-execution process, it can be assumed that the receiving node of that logic engine can always receive data. Therefore, the processor does not need to set the state for that logic engine. This helps to save the bit width of the status register, thereby saving resource overhead; and it also helps to reduce the complexity of maintaining the state of the logic engine.

[0112] It should be noted that the logic engine that performs the final calculation in a neural network circuit (such as...) Figure 6A or Figure 6BThe flow control mechanism between the logic engine 7) and its receiving node can adopt other flow control mechanisms in the prior art, and this application embodiment does not limit this.

[0113] S104: Based on the metadata information of the neural network circuit and the deployment status of each sub-layer in the neural network circuit (such as which logic engine each sub-layer is deployed on), as well as the status bit settings in the status register (such as which physical engine corresponds to which status register, which logic engine corresponds to which status bit, etc.), the processor determines the execution order (or dependency relationship) between logic engines in the neural network circuit and configures the execution logic for each logic engine.

[0114] The execution logic of a logic engine can include: whether the logic engine checks its own status; if so, when and which status bit to check, when to set its own status to busy; when and to whom to send a clear status command; and how many calculations the logic engine performs in a loop. This execution logic can be stored in a configuration register. During the computation process of the neural network circuit, each logic engine can read its own execution logic from the configuration register and perform calculations based on that logic. For more information on the execution logic of the logic engines, please refer to the following... Figures 7 to 10 .

[0115] (Optional) S105: The processor rearranges the weights of loop layers that perform multiple calculations in a loop process based on the metadata information of the neural network circuit, so that each loop layer performs a calculation once in a loop process.

[0116] Optionally, the technical solution including S105 can be applied to scenarios where "the neural network circuit includes recurrent layers and non-recurrent layers, and in one loop, some or all of the recurrent layers need to perform multiple calculations." This helps to improve the computation speed of the neural network system, thereby saving computation time.

[0117] Taking a Convolutional Recurrent Neural Network (CRNN) as an example, a CRNN includes: a conv layer, a gru1 layer, a gru2 layer, a fc1 layer, and a fc2 layer. The conv, gru1, and gru2 layers are recurrent layers, while the fc1 and fc2 layers are non-recurrent layers. If, during a loop, the conv layer performs 7 computations, while the gru1 and gru2 layers each perform 1 computation, the processor can copy the weights of the conv layer 7 times, concatenate and rearrange these 7 weights into a large matrix, and then deploy this large matrix on a physics engine (such as a crossbar). Here, this "large matrix" is relative to the weight matrix of the conv layer itself.

[0118] Those skilled in the art will understand that performing multiple calculations on certain data based on a weight matrix is ​​equivalent to performing a single calculation on the same data based on another weight matrix. The embodiments of this application do not limit the specific method of obtaining this other weight matrix (such as the large matrix mentioned above), and for example, existing technologies can be referenced.

[0119] The execution order of S103, S104 and S105 is not limited in this embodiment.

[0120] The following describes a method for controlling data flow in a neural network circuit provided in the embodiments of this application. Specifically, for a logic engine within a sub-execution process, the following situations can be considered:

[0121] 1) For logic engines that are not the last in the sub-execution process:

[0122] If the logic engine and its receiving node are deployed on the same physical engine, then during the deployment phase, the processor does not need to set the state for the logic engine; during the execution phase, the logic engine can execute actions such as... Figure 7 The method shown.

[0123] If the logic engine and its receiving node are not deployed on the same physical engine, and the logic engine is not the last logic engine executed on its physical engine, then during the deployment phase, the processor can set the state for the logic engine, and during the execution phase, the logic engine can execute actions such as... Figure 8 The method shown.

[0124] If the logic engine and its receiving node are not deployed on the same physical engine, and the logic engine is the last logic engine executed on its physical engine, then during the deployment phase, the processor can set the state for the logic engine, and during the execution phase, the logic engine can execute actions such as... Figure 9 The method shown.

[0125] 2) For the last logic engine in this sub-execution process, the processor may not set a state for this logic engine during the deployment phase. During the execution phase, this logic engine can execute, for example... Figure 10 The method shown.

[0126] Figure 7 The method for controlling the data flow in the neural network circuit shown may include the following steps:

[0127] S201: When the current logic engine receives a data frame, it checks the status of its own physical engine (i.e., the current physical engine). If the current physical engine is in an available state, then S202 is executed. If the current physical engine is in an unavailable state, then it waits for the current physical engine's status to become available. For example, the current logic engine can return to execute S201 when the first preset time period starting from the moment the judgment result of S201 is "no" has elapsed. This application embodiment does not limit the specific value or the method of determining the first preset time period.

[0128] S202: The current logic engine starts calculation and sets the current physics engine to an unavailable state.

[0129] Executing S202 helps prevent other logical engines on the current physical engine from starting calculations while the current logical engine is calculating, thus helping to achieve time-sharing reuse of the physical resources of different logical engines on the same physical engine.

[0130] S203: After the current logic engine has finished sending the calculation results, set the current physical engine's state to available.

[0131] implement Figure 7 In the method shown, the current logic engine and its receiving node are deployed on the same physical engine. In this case, the current logic engine does not need to have its state set, and therefore, it does not need to query its own state. Furthermore, the sending node of the receiving node of the current logic engine is the current logic engine itself, and since the current logic engine does not have its state set, its configuration does not need to send a clear state command. This helps simplify the control flow, reduce maintenance costs, and save on the bit width of the status register.

[0132] Figure 8 The method for controlling the data flow in the neural network circuit shown may include the following steps:

[0133] S301: When the current logic engine receives a data frame, it checks the status of its own physical engine (i.e., the current physical engine). If the current physical engine is in an available state, then execute S302. If the current physical engine is in an unavailable state, then wait for the current physical engine to become available.

[0134] S302: The current logic engine checks whether its own state is idle.

[0135] If yes, then execute S303. If no, the current logic engine can wait for its own state to become idle. For example, the current logic engine can return to execute S302 when the second preset time period starting from the moment when the judgment result of S302 is "no" arrives. This application embodiment does not limit the specific value and the method of value selection for the second preset time period.

[0136] S303: The current logic engine starts calculation and sets the current physics engine to an unavailable state.

[0137] S304: When the current logic engine reaches the preset number of calculations, it sets its own state to busy and sends a clear state command to the sending end node "the current physical engine, the logic engine that will perform the calculation after the current logic engine performs the calculation".

[0138] Understandably, in a sub-execution process, if the current logic engine needs to perform multiple calculations, its state should not be set to busy until the preset number of calculations has been reached. This is a technical solution proposed to address the issue that "if the current logic engine's state is set to busy at this point, and no other logic engine has yet cleared its state, then the current logic engine cannot start the next calculation, and the neural network system will lock up." For example, based on... Figure 6A In the example shown, suppose logic engine 4 generates 10 data points per calculation, and logic engine 5 requires 50 data points to start a calculation. Therefore, logic engine 4 needs to execute 5 times before it can start a calculation. If logic engine 4 is set to a busy state after performing one calculation, logic engine 5 cannot start a calculation. At this point, "data cannot flow." Furthermore, since logic engine 4 needs to be in an idle state to start the next calculation, and no logic engine is clearing its state, logic engine 4 cannot start the next calculation, causing the neural network system to lock up; that is, the data is "stuck."

[0139] Optionally, if the current logic engine performs only one calculation in a sub-execution process, then when executing S304, the processor can configure the current logic engine not to check whether the number of calculations has reached the preset number, but to directly set its own state to busy. For example, if S105 is executed during the deployment phase, each logic engine in a sub-execution process can choose not to execute "check whether the number of calculations has reached the preset number".

[0140] Understandably, before starting computation, the current logic engine determines its state to be idle; and after starting computation, it sets its state to busy. Since the current logic engine does not check its busy state again after starting computation within a sub-execution process, it can send the computation result to its receiving node immediately after starting computation.

[0141] Understandably, after the current logic engine starts computation, it sends a clear status command to the sending node "on the current physical engine, during this sub-execution, and the next logic engine to perform computation after the current logic engine performs computation". In this way, the data needed for the next logic engine to perform computation on the current physical engine can arrive during the current logic engine's computation. That is, the computation latency of the current logic engine masks the transmission latency of the next logic engine to perform computation on its physical engine, allowing the next logic engine to start computation immediately after the current logic engine completes its computation, without waiting for the data to arrive.

[0142] S305: After the current logic engine has finished sending the calculation results, set the current physical engine's state to available.

[0143] Figure 9 The method for controlling the data flow in the neural network circuit shown may include the following steps:

[0144] S401: When the current logic engine receives a data frame, it checks the status of its own physical engine (i.e., the current physical engine). If the current physical engine is in an available state, it executes S402. If the current physical engine is in an unavailable state, it waits for the current physical engine to become available.

[0145] S402: The current logic engine checks whether its own state is idle.

[0146] If yes, then execute S403. If no, the current logic engine can wait for its own state to become idle.

[0147] S403: The current logic engine starts calculation and sets the current physics engine to an unavailable state.

[0148] S404: When the current logic engine determines that the number of calculations of the current logic engine has reached the preset number, the current logic engine's state is set to busy state, and a clear state command is sent to the sending end node of the first logic engine executing the calculation in the current sub-execution process on the current physical engine.

[0149] The technical solution proposed regarding "sending a clear status command to the sending node of the first logic engine that performed the calculation in the current sub-execution process on the current physical engine when the current logic engine's calculation count reaches the preset number" takes into account that "other logic engines on the same physical engine will not start the calculation of the next sub-execution process if a sub-execution process on that physical engine has not been completed." This helps to ensure pipelined execution between sub-execution processes.

[0150] If the first logic engine to perform computation in the current physics engine is the first logic engine in the neural network system, then the sending node of that logic engine can be a module / device / device that sends data frames (e.g., ...). Figure 3 (e.g., DSP702 in the DSP702), at this time, the clear status command is used to indicate to the module / device / device that sent the data frame that "the first logic engine in the loop layer that performs the calculation is ready to receive data".

[0151] S405: After the current logic engine has finished sending the calculation results, set the current physical engine's state to available.

[0152] Figure 10 The method for controlling the data flow in the neural network circuit shown may include the following steps:

[0153] S501: When the current logic engine receives a data frame, it checks the status of the physical engine to which it belongs (i.e., the current physical engine). If the current physical engine is in an available state, then S502 is executed. If the current physical engine is in an unavailable state, then it waits for the current physical engine to become available.

[0154] S502: The current logic engine starts calculation and sets the current physics engine to an unavailable state.

[0155] S503: When the current logic engine determines that the number of calculations of the current logic engine has reached the preset number, it sends a clear status command to the sending end node of the first logic engine that performs the calculation in the current sub-execution process on the current physical engine.

[0156] S504: After the current logic engine has finished sending the calculation results, set the current physical engine's state to available.

[0157] With the above Figure 9 Compared to the method shown, execution Figure 10 The logic engine of the method shown does not need to check its own state during execution because it does not set a state. Explanations of the other steps can be found above and will not be repeated here.

[0158] It should be noted that for the last logic engine in the loop layer, assuming m loop processes need to be executed, then in the first m-1 sub-execution processes (i.e., the first m-1 loop processes), this logic engine is the last logic engine in that sub-execution process. Therefore, the execution... Figure 10 The method shown. In the m-th sub-execution process (i.e., the m-th loop process + non-loop process), if the logic engine is not the last logic engine in that sub-execution process, it can execute according to the deployment situation. Figure 7 or Figure 8 or Figure 9 The method shown.

[0159] The following are respectively based on Figure 6A and Figure 6B For example, regarding the above Figures 7-10 The provided technical solutions will be explained.

[0160] Example 1

[0161] This embodiment is based on Figure 6A For example, regarding the above Figures 7-10 The provided technical solution is described below. In this embodiment, it is assumed that the recurrent layer of the neural network system loops 20 times. In each loop, logic engine 4 performs 5 calculations, and the other logic engines each perform 1 calculation. Then:

[0162] For the first sub-execution process (i.e., the first loop process):

[0163] The logic engines for the first sub-execution process are logic engines 1 to 6, among which logic engines 3 and 6 are not set to any state. At the initial moment of the execution phase: physical engines 1 to 4 are in an available state, while logic engines 1, 2, 4, and 5 are in an idle state.

[0164] Logic Engine 1 can execute Figure 8 The method is shown. When executing S304, logic engine 1 sets its own state to busy and sends a clear state command to the sending node (i.e., logic engine 2) of the next logic engine (i.e., logic engine 3) that will perform calculations after logic engine 1 in the current sub-execution process. After execution, logic engine 1 is in a busy state, while logic engines 2, 4, and 5 are in an idle state.

[0165] Logic Engine 2 can execute Figure 8The method is illustrated. During execution S304, logic engine 2 sets its own state to busy and sends a clear state command to the sending node (i.e., logic engine 4) of the logic engine that will perform the next computation after logic engine 2 (i.e., logic engine 5) in the sub-execution process of the current physical engine (i.e., physical engine 2). After execution, logic engines 1 and 2 are in busy state, while logic engines 4 and 5 are in idle state.

[0166] Logic Engine 3 can execute Figure 7 The method is shown. After execution, logic engines 1 and 2 are in a busy state, while logic engines 4 and 5 are in an idle state.

[0167] Logic Engine 4 can execute Figure 9 The method is shown. When executing S404, logic engine 4 performs the first four calculations, then directly executes the next calculation. During the fifth calculation, it sets its own state to busy and sends a message to the sending node (e.g., the first logic engine 1 executing a calculation in the current sub-execution process) on the current physical engine (i.e., physical engine 1). Figure 3 The DSP702 in the system sends a clear status command. After execution, logic engines 1, 2, and 4 are in a busy state, and logic engine 5 is in an idle state.

[0168] Logic Engine 5 can execute Figure 9 The method is shown below. When executing S404, logic engine 5 sets its own state to busy and sends a clear state command to the sending node (logic engine 1) of the first logic engine (logic engine 2) performing computation in the current sub-execution process on the current physical engine (i.e., physical engine 2). After execution, logic engines 2, 4, and 5 are in busy state, and logic engine 1 is in idle state.

[0169] Logic Engine 6 can execute Figure 10 The method is shown. When executing S503, logic engine 6 sends a clear status command to the sending node (i.e., logic engine 5) of the first logic engine (i.e., logic engine 6) performing computation on the current physical engine (i.e., physical engine 3). After execution, logic engines 2 and 4 are in a busy state, while logic engines 1 and 5 are in an idle state.

[0170] Optionally, when a logic engine is in an idle state, other logic engines may not send a clear status command to that engine. For example, in the example above, logic engine 1 may not send a clear status command to logic engine 2. This helps save signaling overhead. In actual implementation, the logic engine may not check whether the receiving node of the clear status command is idle, but will send the clear status command regardless of whether it is idle or not, which helps simplify the design complexity.

[0171] For the second sub-execution process (i.e., the second loop process):

[0172] The logic engines for the second sub-execution process are logic engines 1 through 6, among which logic engines 3 and 6 are not set to any state. At the initial moment of the execution phase, logic engines 2 and 4 are in a busy state, while logic engines 1 and 5 are in an idle state.

[0173] Logic Engine 1 can execute Figure 8 The method is shown. After execution, logic engines 1 and 4 are in a busy state, while logic engines 2 and 5 are in an idle state.

[0174] Logic Engine 2 can execute Figure 8 The method is shown. After execution, logic engines 1 and 2 are in a busy state, while logic engines 4 and 5 are in an idle state.

[0175] The execution process and results of logic engines 3 to 6 can be found in the first sub-execution process, and will not be repeated here.

[0176] For the 3rd to 19th sub-execution processes (i.e., the 3rd to 19th loop processes):

[0177] The descriptions of any of the sub-execution processes from the 3rd to the 19th sub-execution process can refer to the description in the 2nd sub-execution process mentioned above.

[0178] For the 20th sub-execution process (i.e., the 20th loop process + non-loop process):

[0179] The logic engines in the 20th sub-execution process are logic engines 1 through 7. Among them, logic engines 3 and 7 are not set to any state. At the beginning of the execution phase, logic engines 2 and 4 are in a busy state, while logic engines 1, 5, and 6 are in an idle state.

[0180] The execution process and results of logic engines 1 to 5 can be found in the second sub-execution process, and will not be repeated here.

[0181] Logic Engine 6 can execute Figure 9 The method is illustrated. During execution S404, logic engine 6 sets its own state to busy and sends a clear state command to the sending node (i.e., logic engine 5) of the first logic engine (i.e., logic engine 6) to perform calculations in the current sub-execution process on the current physical engine (i.e., physical engine 3). After execution, logic engines 2, 4, and 6 are in busy state, while logic engines 1 and 5 are in idle state.

[0182] Logic Engine 7 can execute Figure 10 The method is shown. During execution S503, logic engine 7 sends a clear status command to the sending node (i.e., logic engine 6) of the first logic engine (i.e., logic engine 7) performing computation in the current sub-execution process on the current physical engine (i.e., logic engine 4). After execution, logic engines 2 and 4 are in a busy state, while logic engines 1, 5, and 6 are in an idle state.

[0183] Understandably, in this embodiment, for the last logic engine (i.e., logic engine 6) in the loop layer, logic engine 6 does not set a state during the first 19 sub-execution processes, but sets its state to busy during the last sub-execution process. This implementation is equivalent to: setting the state of logic engine 6 during the first 19 sub-execution processes, but not setting its state to busy during execution; and only setting its state to busy during the last sub-execution process.

[0184] In other words, if the last logic engine executing computation in the loop layer has not reached the preset number of iterations, its state will not be set to busy. This technical solution is proposed considering that "if the logic engine's state is set to busy at this time, the first logic engine executing computation in the non-loop layer will not be able to start. Without a logic engine to clear its state, it will be unable to start computation, causing the neural network system to lock up." For example, based on... Figure 6A If logic engine 6 sets its state to busy during each of the first 19 executions of the loop layer, logic engine 7 cannot start computation. Therefore, no logic engine clears the state for logic engine 6, which will cause logic engine 6 to fail to start computation, thus causing the neural network system to lock up.

[0185] Example 2

[0186] This embodiment is based on Figure 6B For example, regarding the above Figures 7-10 The provided technical solution is described below. In this embodiment, it is assumed that the recurrent layer of the neural network system loops 20 times. In each loop, logic engine 4 performs 5 calculations, and the other logic engines each perform 1 calculation. Then:

[0187] For the 1st to 19th sub-execution processes (i.e., the 1st to 19th loop processes):

[0188] The descriptions of any of the sub-execution processes from the 1st to the 19th sub-execution process can be found in the above descriptions based on... Figure 6A The execution process described includes sub-execution processes 1 through 19, which will not be elaborated upon here.

[0189] For the 20th sub-execution process (i.e., the 20th loop process + non-loop process):

[0190] The logic engines for the 20th sub-execution process are logic engines 1 through 7, among which logic engines 3, 6, and 7 are not set to any state. At the initial moment of the execution phase, logic engines 2 and 4 are in a busy state, while logic engines 1 and 5 are in an idle state.

[0191] The execution process and results of logic engines 1 to 5 can be found in the second sub-execution process, and will not be repeated here.

[0192] Logic Engine 6 can execute Figure 7 The method is shown. After execution, logic engines 2 and 4 are in a busy state, while logic engines 1 and 5 are in an idle state.

[0193] Logic Engine 7 can execute Figure 10 The method is shown. In S503, logic engine 7 sends a clear status command to the sending node (i.e., logic engine 5) of the logic engine (i.e., logic engine 6) that performs the first calculation in the current sub-execution process on the current physical engine (i.e., physical engine 3).

[0194] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the method steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0195] This application embodiment can divide the neural network system into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0196] like Figure 11The diagram shown is a structural schematic of a neural network circuit 11 provided in an embodiment of this application. This neural network circuit 11 is used to execute the method provided in any of the embodiments described above. The neural network circuit 11 may include: a first physical engine 111, which is virtualized into one or more logic engines. The first logic engine 121 among the one or more logic engines is used to: obtain a first data frame; and perform calculations on the first data frame when the state of the first physical engine 111 is a first state and the state of the first logic engine 121 is an idle state. The first state of the first physical engine 111 indicates that no logic engine is performing calculations on the first physical engine 111, and the idle state of the first logic engine 121 indicates that a second logic engine 122 can receive data. The second logic engine 122 is used to perform calculations on the output data of the first logic engine 121. Figure 11 The example given is that the second logic engine 122 does not belong to the first physical engine 111; however, the actual implementation is not limited to this. For example, combining... Figure 8 The first logic engine 121 can be Figure 8 The current logic engine can be used to execute S301 to S303. For example, combined with... Figure 9 The first logic engine 121 can be Figure 9 The current logic engine in the system can be used to execute S401 to S403.

[0197] Optionally, the first logic engine 121 is further configured to: when the number of calculations performed by the first logic engine 121 reaches a preset number, set the state of the first logic engine 121 to a busy state, wherein the busy state of the first logic engine 121 is used to indicate that the second logic engine cannot receive data. For example, in combination with Figure 8 or Figure 9 The first logic engine 121 can be used to execute steps set to a busy state in S304 or S404.

[0198] Optionally, the calculation performed by the first logic engine 121 on the first data frame is a sub-execution process calculation, and the first logic engine 121 is the last logic engine to perform the calculation on the first physical engine 111 in this sub-execution process. The first logic engine 121 is also used to: when the number of calculations performed by the first logic engine 121 reaches a preset number, send a first indication message to the third logic engine 123, wherein the third logic engine 123 is used to send data to the first target logic engine 124, the first target logic engine 124 is the logic engine on the first physical engine 111 that performed the first calculation in this sub-execution process, and the first indication message is used to instruct the third logic engine 123 to set its own state to an idle state, and the idle state of the third logic engine 123 is used to indicate that the first target logic engine 124 can receive data. Figure 12 This is a schematic diagram of the structure of the neural network circuit 11 applicable under this optional implementation. The first logic engine 121 and the first target logic engine can be the same logic engine or different logic engines. Figure 12 The example used is that the two are different logic engines. For example, combining... Figure 6A The first logic engine 121 can be logic engine 5. In this case, the first target logic engine is logic engine 2, and the third logic engine is logic engine 1.

[0199] Optionally, the calculation performed by the first logic engine 121 on the first data frame belongs to a sub-execution process. The first logic engine 121 is not the last logic engine to perform calculations on the first physical engine in this sub-execution process. The first logic engine 121 is also used to: when the number of calculations performed by the first logic engine 121 reaches a preset number, send a first indication message to the third logic engine 123, wherein the first indication message is used to instruct the third logic engine 123 to set its own state to an idle state. The third logic engine 123 is used to send data to the first target logic engine 124, which is located on the first physical engine 111. The first target logic engine is the next logic engine to perform calculations after the first logic engine 121 performs calculations in this sub-execution process. Figure 13 This is a schematic diagram of the structure of the neural network circuit 11 applicable to this optional implementation. For example, combined with... Figure 6A The first logic engine 121 can be logic engine 2. In this case, the first target logic engine is logic engine 5, and the third logic engine is logic engine 4.

[0200] Optionally, the neural network circuit 11 further includes a fourth logic engine 125; the fourth logic engine 125 and the first logic engine 121 are logic engines in the same sub-execution process, and the fourth logic engine 125 is the last logic engine to perform calculations in this sub-execution process. The fourth logic engine 125 is used to obtain the second data frame; the fourth logic engine 125 is used to perform calculations on the second data frame when the state of the second physical engine 112 to which the fourth logic engine 125 belongs is in the first state, wherein the state of the second physical engine 112 being in the first state is used to indicate that there is no logic engine performing calculations on the second physical engine 112. Figure 14 This is a schematic diagram of the structure of the neural network circuit 11 applicable under this optional implementation. The second logic engine 122 and the fourth logic engine 125 may or may not be the same logic engine. Figure 14 This is illustrated using the example that the two are not based on the same logic engine. For example, combining... Figure 6A When the sub-execution process is not the last sub-execution process, the fourth logic engine 125 can be logic engine 6.

[0201] Optionally, the fourth logic engine 125 is further configured to: send a second instruction message to the fifth logic engine 126, wherein the fifth logic engine 126 is configured to send data to the second target logic engine 127, the second target logic engine 127 being the first logic engine to perform calculations during the sub-execution process on the second physical engine 112, and the second instruction message being configured to indicate that the second target logic engine 127 can receive data. Figure 14 This is a schematic diagram of the structure of the neural network circuit 11 applicable to this optional implementation. For example, combined with... Figure 6A When the sub-execution process is not the last sub-execution process, the fourth logic engine 125 can be logic engine 6, the second target logic engine 127 can be logic engine 6, and the fifth logic engine 126 can be logic engine 5.

[0202] Optionally, the first logic engine 121 is further configured to: after the first logic engine 121 starts calculation, set the state of the first physical engine 111 to a second state, the second state being used to indicate that the physical resources of the first physical engine 111 are occupied; and after the first logic engine 121 sends the calculation result to the second logic engine 122, set the state of the first physical engine 111 to a first state. For example, based on Figures 7 to 10 The current logic engine can be used to execute S203, S305, S405 and S504 respectively.

[0203] In one example, see Figure 1 The neural network circuit 11 can be Figure 1 The neural network circuit 110 in the neural network circuit 11. Any physics engine 1 in the neural network circuit 11 (such as the first physics engine 111, etc.) can be... Figure 1 The physical engine 1302 and any logic engine in the neural network circuit 11 (such as the first logic engine 121, etc.) can be... Figure 1 The logic engine 1304 in the system.

[0204] For a detailed description of the above-mentioned optional methods, please refer to the aforementioned method embodiments, which will not be repeated here. Furthermore, the explanation of any of the neural network circuits 11 provided above, as well as the description of their beneficial effects, can be found in the corresponding method embodiments described above, and will not be repeated here.

[0205] It should be noted that the actions executed by each logic engine mentioned above are merely specific examples. The actual actions executed by each module are as described above. Figures 7 to 10 The actions or steps mentioned in the description of the embodiments.

[0206] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the actions or steps mentioned in any of the embodiments provided above.

[0207] This application also provides a chip. This chip integrates circuitry for implementing the functions of the aforementioned neural network system 11 and one or more interfaces. Optionally, the functions supported by this chip may include those based on… Figures 7 to 10 The processing actions described in the embodiments will not be repeated here. Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, random access memory, etc. The processing unit or processor mentioned above can be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (DSP), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0208] This application also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform any of the methods described in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or may include one or more data storage devices such as servers or data centers that can be integrated with the medium. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.

[0209] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of this application, such as but not limited to the memory, computer-readable storage medium and communication chip, are all non-transitory.

[0210] In implementing the claimed application, those skilled in the art can understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may describe certain measures, this does not mean that these measures cannot be combined to produce good results. Although the application has been described in conjunction with specific features and embodiments, various modifications and combinations can be made without departing from the spirit and scope of the application. Accordingly, this specification and drawings are merely exemplary descriptions of the application as defined by the appended claims and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the application.

Claims

1. A neural network circuit, characterized in that, include: The first physical engine is virtualized into one or more logical engines; The first logic engine in the one or more logic engines is used for: Obtain the first data frame; When the first physical engine is in the first state and the first logic engine is in the idle state, the first data frame is calculated; wherein, the first physical engine being in the first state indicates that there is no logic engine performing calculation on the first physical engine, and the first logic engine being in the idle state indicates that the second logic engine can receive data, and the second logic engine is a logic engine used to calculate the output data of the first logic engine.

2. The neural network circuit according to claim 1, characterized in that, The first logic engine is further configured to: when the number of calculations performed by the first logic engine reaches a preset number, set the state of the first logic engine to a busy state, wherein the busy state of the first logic engine is used to indicate that the second logic engine cannot receive data.

3. The neural network circuit according to claim 1 or 2, characterized in that, The calculation performed by the first logic engine on the first data frame is a calculation of a sub-execution process. The first logic engine is the last logic engine to perform the calculation on the first physical engine in the sub-execution process. The first logic engine is further configured to: when the number of calculations performed by the first logic engine reaches a preset number, send a first indication message to the third logic engine, wherein the third logic engine is configured to send data to the first target logic engine, the first target logic engine being the logic engine that performs the first calculation in the sub-execution process on the first physical engine, and the first indication message being configured to instruct the third logic engine to set its own state to an idle state, the idle state of the third logic engine being configured to instruct the first target logic engine to receive data.

4. The neural network circuit according to claim 1 or 2, characterized in that, The calculation performed by the first logic engine on the first data frame belongs to the calculation of a sub-execution process. The first logic engine is not the logic engine that last performed the calculation on the first physical engine in the sub-execution process. The first logic engine is further configured to: when the number of calculations performed by the first logic engine reaches a preset number, send a first indication message to the third logic engine, wherein the first indication message is configured to instruct the third logic engine to set its own state to an idle state, and the third logic engine is configured to send data to a first target logic engine, the first target logic engine being located on the first physical engine, and the first target logic engine being the next logic engine to perform calculations after the first logic engine performs calculations during the sub-execution process.

5. The neural network circuit according to claim 1 or 2, characterized in that, The neural network circuit further includes: a fourth logic engine; the fourth logic engine and the first logic engine are logic engines in the same sub-execution process, and the fourth logic engine is the logic engine that performs the last calculation in the sub-execution process; The fourth logic engine is used to obtain the second data frame; The fourth logic engine is used to perform calculations on the second data frame when the state of the second physical engine to which the fourth logic engine belongs is in the first state. The state of the second physical engine being in the first state indicates that there is no logic engine performing calculations on the second physical engine.

6. The neural network circuit according to claim 5, characterized in that, The fourth logic engine is further configured to: send a second instruction message to the fifth logic engine, wherein the fifth logic engine is configured to send data to the second target logic engine, the second target logic engine being the first logic engine to perform calculations in the sub-execution process on the second physical engine, and the second instruction message being configured to indicate that the second target logic engine can receive data.

7. The neural network circuit according to claim 1 or 2, characterized in that, The first logic engine is further configured to: after the first logic engine starts calculation, set the state of the first physical engine to a second state, the second state being used to indicate that the physical resources of the first physical engine are occupied; and after the first logic engine sends the calculation result to the second logic engine, set the state of the first physical engine to the first state.

8. A method for controlling data flow in a neural network circuit, characterized in that, The neural network circuit includes a first physical engine, which is virtualized into one or more logical engines. The method is applied to a first logic engine among the one or more logic engines; the method includes: Obtain the first data frame; When the first physical engine is in the first state and the first logic engine is in the idle state, the first data frame is calculated; wherein, the first physical engine being in the first state indicates that there is no logic engine performing calculation on the first physical engine, and the first logic engine being in the idle state indicates that the second logic engine can receive data, and the second logic engine is a logic engine used to calculate the output data of the first logic engine.

9. The method according to claim 8, characterized in that, The method further includes: When the first logic engine reaches a preset number of calculations, the state of the first logic engine is set to busy state, wherein the busy state of the first logic engine is used to indicate that the second logic engine cannot receive data.

10. The method according to claim 8 or 9, characterized in that, The calculation performed by the first logic engine on the first data frame is a calculation of a sub-execution process. The first logic engine is the last logic engine to perform the calculation on the first physical engine in the sub-execution process. The method further includes: When the first logic engine reaches a preset number of calculations, it sends a first indication message to the third logic engine. The third logic engine is used to send data to the first target logic engine, which is the logic engine that performs the first calculation in the sub-execution process on the first physical engine. The first indication message is used to instruct the third logic engine to set its own state to an idle state. The idle state of the third logic engine is used to indicate that the first target logic engine can receive data.

11. The method according to claim 8 or 9, characterized in that, The calculation performed by the first logic engine on the first data frame belongs to the calculation of a sub-execution process. The first logic engine is not the logic engine that last performed the calculation on the first physical engine in the sub-execution process. The method further includes: When the first logic engine reaches a preset number of calculations, it sends a first instruction message to the third logic engine. The first instruction message is used to instruct the third logic engine to set its own state to an idle state. The third logic engine is used to send data to the first target logic engine, which is located on the first physical engine. The first target logic engine is the logic engine that will perform calculations after the first logic engine performs calculations during the sub-execution process.

12. The method according to claim 8 or 9, characterized in that, The method further includes: After the first logic engine starts calculation, the state of the first physical engine is set to the second state, which is used to indicate that the physical resources of the first physical engine are occupied; and after the first logic engine sends the calculation result to the second logic engine, the state of the first physical engine is set to the first state.

13. A neural network system, characterized in that, The neural network system includes: The processor and the neural network circuit as described in any one of claims 1 to 7, wherein, The processor is configured to send one or more data frames to the neural network circuit, wherein the one or more data frames include the first data frame.

14. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when run on a computer, causes the computer to perform the method according to any one of claims 8 to 12.

Images