Loop Transformer Inference Circuit and its Control Method
By designing a cyclic Transformer inference circuit and using an inlet selector and an outlet selector to form a feedback loop, the problems of pipeline bubbles and high control overhead in hardware accelerators are solved, thereby improving the hardware computing performance of edge-side LLM.
Patent Information
- Application Number
- CN202611612070.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
Existing hardware accelerators suffer from pipeline bubbles and high control overhead when processing large language models, which limits hardware performance improvement, especially when GPU memory resources are limited in edge-side LLM deployments.
Design a cyclic Transformer inference circuit that forms a feedback loop through an inlet selector, an outlet selector, and a loop control circuit to achieve adaptive iterative computation at the token granularity, avoiding reliance on software control or a global clock, and reducing pipeline bubbles and control overhead.
It improves hardware-level computing performance, achieves more efficient model calculation through adaptive control, reduces pipeline bubbles and control overhead, and improves computing efficiency.
Smart Images

Figure CN123117202A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a loop Transformer inference circuit and its control method. Background Technology
[0002] As Large Language Models (LLMs) are deployed to edge devices, the bottleneck of off-chip memory access bandwidth and high power consumption caused by billions of parameters are becoming increasingly severe. In order to run LLMs with extremely limited GPU memory resources, some current large model architectures adopt deep layer weight sharing technology, retaining only a very small number of physical blocks (the basic computational units within the large language model), and maintaining the high expressive power of the model by iterating in a single block.
[0003] However, model architectures employing iterative approaches face significant challenges when deployed on hardware: different tokens have varying semantic difficulties, requiring different numbers of iterations. Existing hardware accelerators typically use software state machines or global clocks for shutdown control, which introduces substantial controller overhead and pipeline bubbles, leading to a significant increase in hardware costs and limiting the effective improvement of model computational performance.
[0004] Therefore, overcoming the pipeline bubbles and high control overhead in the existing hardware architecture is the core bottleneck in the current development of edge-side LLM dedicated chips. Summary of the Invention
[0005] The purpose of this disclosure is to provide a loop Transformer inference circuit and its control method to overcome the pipeline bubbles and high control overhead caused by existing adaptive computation depth mechanisms that rely on software control or a global clock, thereby improving the hardware-level computational performance of the model. The specific technical solution is as follows: In a first aspect, this disclosure provides a looping Transformer inference circuit, comprising: an entry selector, a Transformer computation circuit, an exit selector, and a loop control circuit; the entry selector has a data receiving end, a data output end, a feedback receiving end, and a first control end; the data receiving end is connected to an upstream data source, and the data output end is connected to the input end of the Transformer computation circuit; the exit selector has a result receiving end, a result output end, a feedback output end, and a second control end; the result receiving end is connected to the output end of the Transformer computation circuit, and the feedback output end is connected to the feedback receiving end; the output end of the Transformer computation circuit is also connected to the input end of the loop control circuit, and the output end of the loop control circuit is connected to the first control end and the second control end respectively; The Transformer calculation circuit is configured to perform inference calculations on the data output from the data output terminal and output the calculation results. The loop control circuit is configured to: after the Transformer calculation circuit completes each round of calculation, if the preset termination loop condition is not met, output a first control level to the first control terminal and the second control terminal, so that the exit selector routes the calculation result of that round to the feedback output terminal, and the entry selector routes the calculation result received at the feedback receiving terminal to the data output terminal; if the termination loop condition is met, output a second control level to the first control terminal and the second control terminal, so that the exit selector routes the calculation result of that round to the result output terminal, and the entry selector routes the subsequent unprocessed data received at the data receiving terminal to the data output terminal.
[0006] In a second aspect, this disclosure provides a control method for an inference circuit, applied to the loop Transformer inference circuit of the first aspect mentioned above. The method includes: obtaining model description information of a target Transformer model, and generating configuration information for the loop Transformer inference circuit based on the model description information; loading the configuration information into the loop Transformer inference circuit so that the loop Transformer inference circuit performs model calculation.
[0007] The beneficial effects of this disclosure are: The cyclic Transformer inference circuit disclosed herein forms a feedback loop between the input and output of the Transformer computation circuit for implementing iterative computation. Inlet and outlet selectors configured upstream and downstream of the Transformer computation circuit enable switching between the feedback loop and external input / output paths. A loop control circuit controls the selection states of the inlet and outlet selectors. When the loop control circuit determines that the termination condition is not met, it controls the inlet and outlet selectors to select the feedback loop for data backflow, reusing a single Transformer computation circuit to achieve iterative computation for the current token. When the loop control circuit determines that the termination condition is met, it controls the inlet and outlet selectors to cut off the feedback loop to directly output the computation result, thus exiting the iterative computation for the current token. Based on this, the cyclic Transformer inference circuit overcomes the pipeline bubbles and high control overhead caused by existing adaptive computation depth mechanisms that rely on software control or a global clock, achieving adaptive halting and routing at the token level, which helps improve the hardware-level computational performance of the model.
[0008] Of course, implementing any product or method of this disclosure does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description
[0009] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other embodiments can be obtained based on these accompanying drawings.
[0010] Figure 1 This is a logical diagram of a loop Transformer algorithm; Figure 2 A schematic diagram of a loop Transformer inference circuit provided in an embodiment of this disclosure; Figure 3 A schematic diagram of the structure of a Transformer computing circuit provided in an embodiment of this disclosure; Figure 4 Another schematic diagram of the structure of the loop Transformer inference circuit provided in the embodiments of this disclosure; Figure 5 A schematic diagram of the state assessment circuit provided in an embodiment of this disclosure; Figure 6 A schematic diagram of an asynchronous latch circuit provided in an embodiment of this disclosure; Figure 7 A schematic diagram of the decision sub-circuit provided in an embodiment of this disclosure; Figure 8 A schematic diagram of a hidden state convergence evaluation module provided in an embodiment of this disclosure; Figure 9 A schematic diagram of the structure of the FLEQ instruction branch module provided in an embodiment of this disclosure; Figure 10 A schematic diagram of a loop step control module provided in an embodiment of this disclosure; Figure 11 A timing waveform diagram of signals within a cyclic Transformer inference circuit provided in an embodiment of this disclosure; Figure 12 A schematic flowchart of a control method for an inference circuit provided in an embodiment of this disclosure; Figure 13 This is a schematic diagram illustrating the process of Transformer model compilation mapping and hardware execution provided in the embodiments of this disclosure.
[0011] Figure label: 100 - Input Selector, 200 - Transformer Calculation Circuit, 300 - Output Selector, 400 - Loop Control Circuit, 410 - State Evaluation Circuit, 420 - Asynchronous Latch Circuit, 411 - Decision Sub-circuit, 412 - Delay Matching Sub-circuit, 4111 - Hidden State Convergence Evaluation Module, 4112 - FLEQ Instruction Branch Module, 4113 - Loop Step Control Module, 4114 - Multi-Input OR Gate, 41111 - Differential Subtractor, 41112 - Accumulator, 41113 - First Digital Comparator, 41114 - First AND Gate, 41121 - Sign AND Zero Decision Controller, 41122 - PC Selector, 41123 - Stop Instruction Decoder, 41124 - Second AND Gate, 41131 - Step Accumulator Counter, 41132 - Second Digital Comparator, 41133 - Third AND Gate. Detailed Implementation
[0012] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art based on this disclosure are within the scope of protection of this disclosure.
[0013] In subsequent embodiments of this disclosure, a loop Transformer (a deep learning architecture based on self-attention mechanism) inference circuit is provided that eliminates the global clock and enables adaptive halting and routing at the token granularity at the hardware level, in order to overcome the pipeline bubbles and high control overhead caused by existing adaptive computation depth mechanisms that rely on software control or a global clock.
[0014] To facilitate understanding of the subsequent embodiments of this disclosure, the following will first refer to... Figure 1 The cyclic Transformer algorithm upon which this disclosure is based is illustrated by example: like Figure 1 As shown, the input token sequence is stored in the on-chip unified storage space at the top of the system. For example, the on-chip unified storage space can be implemented using input queue SRAM (Static Random-Access Memory) or a register file, serving as shared storage for joint inputs. The on-chip unified storage space can be logically divided into three parts: a temporary storage area, a data area, and an instruction area. The temporary storage area stores the program counter's count value p_PC, as well as temporary calculations and intermediate states of multiple operators; the data area stores the input feature data tensors required for model inference, which may specifically include input scalars, vectors, or matrices; and the instruction area stores instructions containing jump conditions and pointers.
[0015] Logically, the input token sequence is equivalent to a computational block of N shared parameter matrices θ in the computational depth dimension. Figure 1 The computation blocks 1 to N shown in the diagram are calculated iteratively in sequence. And as... Figure 1 As illustrated, in each iteration, an early exit can be performed based on a conditional decision made according to the convergence degree of the hidden state or other preset conditions, so as to output the results ahead of time. For example, an early exit can be performed when the convergence degree of the hidden state is satisfied, or when other preset conditions configured to implement the early exit are met.
[0016] The cyclic Transformer inference circuit provided in this disclosure will now be described with reference to specific embodiments. See also Figure 2 The loop Transformer inference circuit includes an inlet selector 100, a Transformer calculation circuit 200, an outlet selector 300, and a loop control circuit 400.
[0017] The input selector 100 has a data receiver IN1, a data output OUT, a feedback receiver IN2, and a first control terminal SEL1. Under the control of a selection variable applied to the first control terminal SEL1, the input selector 100 can select a specific input terminal from the data receiver IN1 and the feedback receiver IN2, and route the data from that specific input terminal to the data output terminal OUT. In one example, the input selector 100 can be implemented using a multiplexer (MUX).
[0018] The output selector 300 has a result receiving terminal IN, a result output terminal OUT1, a feedback output terminal OUT2, and a second control terminal SEL2. Under the control of a selection variable applied to the second control terminal SEL2, the output selector 300 can select a specific output terminal from the result output terminal OUT1 and the feedback output terminal OUT2, routing the data from the result receiving terminal IN to that specific output terminal. In one example, the output selector 300 can be implemented using a demultiplexer (DEMUX).
[0019] In this embodiment, the data receiver IN1 of the ingress selector 100 is equivalent to the input of the entire cyclic Transformer inference circuit. This data receiver IN1 is connected to an upstream data source to obtain the token sequence to be processed from the upstream data source. As an example, see [reference needed]. Figure 1 The schematic deployment is based on building an on-chip unified storage space using input queue SRAM or register file, and the corresponding on-chip unified storage space is used as the upstream data source. The result output terminal OUT1 of the output selector 300 is equivalent to the output terminal of the entire loop Transformer inference circuit, used to output the LLM processing result to the outside; in one example, an output buffer unit can be configured downstream of the loop Transformer inference circuit, so that the data routed by the output selector 300 to its result output terminal OUT1 flows into the buffer unit.
[0020] The Transformer calculation circuit 200 is equivalent to a hardware-level Transformer block, used to perform single-step iterative calculations on the data received from the input selector 100 and output the single-step calculation results to the output selector 300. Its input terminal is connected to the data output terminal OUT of the input selector 100, and its output terminal is connected to the result receiving terminal IN of the output selector 300 to form a single-step calculation path; furthermore, the feedback output terminal OUT2 of the output selector 300 is also connected to the feedback receiving terminal IN2 of the input selector 100 to form a feedback loop.
[0021] During the reasoning and computation process for each token, the initial input state provided by the upstream data source... x The input state 0 is fed into the Transformer computation circuit 200 through the input selector 100 to perform a single-step iterative computation, and the hidden state is reused in a closed loop in the time iteration dimension of the Transformer computation circuit 200 based on the feedback loop. Specifically, when performing each round of iterative computation for the current token, the Transformer computation circuit 200 can use the current iteration state output from the input selector 100. x i Perform reasoning and output the new iteration state to the exit selector 300. x i+1 If the preset termination loop condition is not met after completing this round of iterative calculation, the exit selector 300 will choose to return the iteration state to the entry selector 100 through the feedback loop. x i+1 So that the Transformer computation circuit 200 is in a new iteration state x i+1 Based on this, the next round of iteration calculation is performed for the current Token; if the preset termination loop condition is met after completing this round of iteration calculation, then the exit selector 300 selects to output the result to OUT1 for the new iteration state. x i+1 Proceed directly to exit the loop iteration for the current token.
[0022] The architecture of the Transformer computing circuit 200 can be configured according to actual needs, and this disclosure does not impose specific limitations. An example can be found by referring to... Figure 3 The following example demonstrates the configuration of the Transformer computation circuit 200: Figure 3 The Transformer computation circuit 200 shown includes a Multi-Head Attention (MHA) computation block and an FFN (Feed-Forward Network) computation block, and the recurrent Transformer inference circuit is configured with an on-chip shared parameter storage for use by the Transformer computation circuit 200. The on-chip shared parameter storage stores the MHA weights required by the MHA computation block, and the FFN weights and function operator library required by the FFN computation block. The MHA weights may include query weights W. q Key weight W k Sum weight W vThe MHA matrix projection and calculation are implemented. FFN weights are used to implement nonlinear activation and algebraic operations. In one example, the shared parameters of the model can be statically loaded into the on-chip SRAM all at once to eliminate the memory wall bottleneck of traditional deep models accessing off-chip memory layer by layer.
[0023] The multi-head attention computation block performs layer normalization on the data, then applies multi-head attention processing to the normalized data, and adds residuals from residual branch 1 to maintain residual stability. The FFN computation block performs layer normalization on the data, then performs feedforward computation on the normalized data through a feedforward network, and adds residuals from residual branch 2 to maintain residual stability. Based on this architecture, the state of the Transformer computation circuit 200 is entered. x i With the support of statically residing weights and shared parameter storage on-chip, the multi-head attention matrix projection calculation and feedforward network calculation are completed sequentially, and the residual branch is superimposed to output a new state. x i+1 .
[0024] The loop termination condition is used to determine whether the loop iteration for the current token needs to be terminated. This loop termination condition can be configured according to actual needs, and this disclosure does not impose specific restrictions. In one example, the loop termination condition may include one or more of the following: the hidden state meets the preset convergence condition; the instruction address of the next instruction to be executed matches the stop address; the iteration calculation rounds for the current token reach the preset maximum number of steps.
[0025] In this embodiment of the disclosure, the selection control of the inlet selector 100 and the outlet selector 300 is specifically achieved through a loop control circuit 400. For example... Figure 2 As shown, the input terminal of the loop control circuit 400 is connected to the output terminal of the Transformer calculation circuit 200, and the output terminal of the loop control circuit 400 is connected to the first control terminal SEL1 of the inlet selector 100 and the second control terminal SEL2 of the outlet selector 300. The loop control circuit 400 can output a control signal V_ACK to the first control terminal SEL1 and the second control terminal SEL2 through its output terminal, so as to realize the selection control of the inlet selector 100 and the outlet selector 300 based on the control signal V_ACK.
[0026] The loop control circuit 400 is configured such that after the Transformer calculation circuit 200 completes each round of calculation, if the termination loop condition is not met, it outputs a first control level to the first control terminal SEL1 and the second control terminal SEL2, so that the output selector 300 routes the calculation result of that round to the feedback output terminal OUT2, and the input selector 100 routes the calculation result received at the feedback receiving terminal IN2 to the data output terminal OUT.
[0027] In other words, if the termination loop condition is not met, the output selector 300 can select the output state of the Transformer calculation circuit 200 under the control of the first control level. x i+1 The data is routed to the feedback loop; the input selector 100, under the control of the first control level, can select to send the data in the feedback loop to the Transformer calculation circuit 200. Thus, based on the control process, the calculation results of the current round can be... x i+1 The data to be inferred in the next round is fed into the Transformer calculation circuit 200 to realize the next round of iterative calculation for the current Token.
[0028] The loop control circuit 400 is also configured to: after the Transformer calculation circuit 200 completes each round of calculation, if the loop termination condition is met, output a second control level to the first control terminal SEL1 and the second control terminal SEL2, so that the output selector 300 routes the calculation result of that round to the result output terminal OUT1, and the input selector 100 routes the subsequent unprocessed data received at the data receiving terminal IN1 to the data output terminal OUT.
[0029] In other words, when the termination loop condition is met, the output selector 300 can select the output state of the Transformer calculation circuit 200 under the control of the second control level. x i+1 The data is directly routed to the output terminal OUT1. Under the control of the second control level, the input selector 100 can select to send the subsequent data to be processed provided by the upstream data source into the Transformer calculation circuit 200. Based on this control process, it is possible to exit the iterative calculation for the current Token. Furthermore, if there are other tokens to be processed after the current Token, the Transformer calculation circuit 200 can also receive the data to be processed corresponding to the next Token from the input selector 100, thereby switching to the iterative calculation for the next token to be processed.
[0030] The levels of the first and second control levels can be set according to actual needs, and this disclosure does not impose specific limitations. In one example, the first control level can be configured to be low and the second control level to be high; that is, the loop control circuit 400 is configured to output V_ACK=0 when the loop termination condition is not met, and output V_ACK=1 when the loop termination condition is met.
[0031] As can be seen from the above, the cyclic Transformer inference circuit provided in this embodiment forms a feedback loop between the input and output terminals of the Transformer calculation circuit 200 for implementing cyclic iterative calculation. The selection and switching between the feedback loop and external input / output paths are achieved through the entry selector 100 and exit selector 300 configured upstream and downstream of the Transformer calculation circuit 200. A loop control circuit 400 controls the selection state of the entry selector 100 and exit selector 300. When the loop control circuit 400 determines that the termination loop condition is not met, it controls the entry selector 100 and exit selector 300 to select the feedback loop for data return, thereby reusing the Transformer calculation circuit 200 to achieve cyclic iterative calculation for the current Token. When the loop control circuit 400 determines that the termination loop condition is met, it controls the entry selector 100 and exit selector 300 to cut off the feedback loop to directly output the calculation result, thereby exiting the cyclic iterative calculation for the current Token. Based on this, the recurrent Transformer inference circuit can overcome the pipeline bubbles and high control overhead caused by existing adaptive computation depth mechanisms that rely on software control or global clocks, and achieve adaptive halting and routing at the token level, which helps to improve the hardware-level computing performance of the model.
[0032] In some embodiments of this disclosure, see Figure 4 The loop control circuit 400 includes a state evaluation circuit 410 and an asynchronous latch circuit 420 connected in sequence. The state evaluation circuit 410 includes two branches: a decision sub-circuit 411 and a delay matching sub-circuit 412. The output of the decision sub-circuit 411 is connected to the first input P1 of the asynchronous latch circuit 420, and the output of the delay matching sub-circuit 412 is connected to the second input P2 of the asynchronous latch circuit 420.
[0033] The decision sub-circuit 411 is configured to output a valid stop enable signal E_stop to the first input terminal P1 when it is determined that the termination loop condition is met.
[0034] Specifically, when the looping Transformer inference circuit first begins executing the iterative calculation for the current Token, the stop enable signal E_stop is initially in an invalid state. Once the loop termination condition for the current Token is met, the stop enable signal E_stop switches to an active state. In one example, the stop enable signal E_stop can be set to active when it is high (E_stop=1) and inactive when it is low (E_stop=0).
[0035] As mentioned earlier, the termination condition can include the hidden state satisfying a preset convergence condition. For example, to facilitate the decision sub-circuit 411 in determining whether the hidden state satisfies the preset convergence condition, please refer to... Figure 4 The diagram illustrates the configuration of a delay register within the loop Transformer inference circuit. This delay register stores the calculation results obtained during iterative calculations of the current Token. Thus, the decision sub-circuit 411 receives the latest round's calculation result from the Transformer calculation circuit 200. x i+1 Then, it can be compared with the calculation result of the previous round stored in the delay register. x i In comparison, it is determined whether the hidden state satisfies the preset convergence condition.
[0036] The delay matching sub-circuit 412 is configured to: in response to the accessed computing circuit ready signal switching to the active state, perform delay time matching between the computing circuit ready signal and the stop enable signal E_stop, and output a valid handshake pulse of the computing completion handshake signal Req_ready to the second input terminal P2, so that the valid handshake pulse of the computing completion handshake signal Req_ready is used to drive the asynchronous latch circuit 420 together with the active stop enable signal E_stop.
[0037] Specifically, the computational circuit ready signal generates a valid level pulse when the Transformer computational circuit 200 completes each round of computation, and returns to an invalid state after the valid level pulse ends. In one example, the computational completion ready signal can be configured to generate a high-level pulse as a valid level pulse when the Transformer computational circuit 200 completes each round of computation, and return to a low level after the valid level pulse ends to enter an invalid state.
[0038] Specifically, when the looping Transformer inference circuit processes the current Token for one round, the Transformer calculation circuit 200 may have completed the calculation for that round (the calculation circuit ready signal has generated a valid level pulse), but the decision sub-circuit 411 has not yet determined whether the end of that round will result in the termination loop condition being met (the stop enable signal E_stop has not yet switched to the valid state). This causes the calculation circuit ready signal and the stop enable signal E_stop to be unable to be in a valid state synchronously. Therefore, in this embodiment, the delay matching sub-circuit 412 can be used to align the delay time of the valid stop enable signal E_stop relative to the valid level pulse of the calculation circuit ready signal. The delay matching sub-circuit 412 outputs a valid handshake pulse corresponding to the valid level pulse and time-aligned with the valid stop enable signal E_stop, so that the valid handshake pulse indicating that the current round of calculation has ended and the valid stop enable signal E_stop indicating that the termination loop condition has been met can be synchronously input to the asynchronous latch circuit 420.
[0039] In this embodiment, the specific implementation method of delay time matching in the delay matching sub-circuit 412 is not limited, and can be selected according to actual needs. In one example, a preset delay duration adapted to the decision duration of the decision sub-circuit 411 can be pre-set according to the actual situation of the circuit. T, and specifically configure the delay matching sub-circuit 412 as follows: starting from when the accessed computing circuit ready signal switches to the valid state, wait for a preset delay time. After T, a valid handshake pulse for the calculated handshake signal Req_ready is output to the second input terminal P2, so that the valid handshake pulse for the calculated handshake signal Req_ready can be used together with the valid stop enable signal E_stop to drive the asynchronous latch circuit 420.
[0040] In one example, the delay matching sub-circuit 412 can be configured to output a high-level pulse on the handshake completion signal Req_ready, i.e., Req_ready=1 as a valid handshake pulse; and cause the handshake completion signal Req_ready to return to a low level after the valid handshake pulse ends, i.e., Req_ready=0 to enter an invalid state.
[0041] The asynchronous latch circuit 420 is configured to: output a first control level to the first control terminal SEL1 and the second control terminal SEL2 in the initial state of the loop Transformer inference circuit starting the loop iteration calculation for the current Token; in response to the shutdown enable signal E_stop and the calculation completion handshake signal Req_ready both switching to the valid state, switch the control level output to the first control terminal SEL1 and the second control terminal SEL2 to the second control level, and latch the second control level.
[0042] In other words, during the iterative calculation of each Token by the cyclic Transformer inference circuit, the asynchronous latch circuit 420 follows the following working logic: when the cyclic Transformer inference circuit just starts the iterative calculation of the current Token, it outputs a first control level to the first control terminal SEL1 and the second control terminal SEL2 as its initial control level; when the input calculation completion handshake signal Req_ready and the stop enable signal E_stop switch to the valid state (the calculation completion handshake signal Req_ready is valid during the validity of the handshake pulse), the output of the first control level to the first control terminal SEL1 and the second control terminal SEL2 will be switched to the second control level, and the second control level will be latched.
[0043] In this embodiment, the asynchronous latch circuit 420 is used to control the selection of the input selector 100 and the output selector 300. When the input conditions are inconsistent—that is, when the effective states of the handshake signal Req_ready and the stop enable signal E_stop are inconsistent—the output control signal V_ACK is maintained in its original state, thus eliminating metastability and glitches during critical section state switching. The specific circuit structure of the asynchronous latch circuit 420 can be designed according to actual needs, and this disclosure does not impose specific limitations. In one example, the asynchronous latch circuit 420 can be implemented based on a Muller C element (a circuit unit specifically designed for asynchronous designs).
[0044] In some embodiments of this disclosure, see Figure 5 The asynchronous latch circuit 420 also has a reset control terminal R, which is connected to the global reset signal Rst_n.
[0045] The global reset signal Rst_n is active during the computation preparation phase before performing iterative calculations on each Token, and inactive at other times. In other words, when performing iterative calculations on each Token using the looping Transformer inference circuit, the global reset signal Rst_n must be set to active before actually starting the iterative calculation for the current Token to complete the computation preparation, including resetting; during the actual execution of the iterative calculation for the current Token, the global reset signal Rst_n remains inactive. For example, the global reset signal Rst_n can be configured to be active when it is low (Rst_n=0) and inactive when it is high (Rst_n=1).
[0046] The asynchronous latch circuit 420 is also configured to: in response to the global reset signal Rst_n switching to the active state, reset the control level output to the first control terminal SEL1 and the second control terminal SEL2 to the first control level.
[0047] In other words, when the global reset signal Rst_n is active, the control level output by the asynchronous latch circuit 420 is the first control level. Conversely, when the global reset signal Rst_n is inactive, the control level output by the asynchronous latch circuit 420 is jointly driven by the handshake completion signal Req_ready and the stop enable signal E_stop. The driving logic follows these rules: when both the handshake completion signal Req_ready and the stop enable signal E_stop are inactive, the first control level is output; when both are active, the second control level is output; when one is active and the other is inactive, the output control level is latched to the original control level.
[0048] Figure 6 An example circuit structure of an asynchronous latch circuit 420 is provided. The asynchronous latch circuit 420 in this example is implemented using Muller C components. The first two layers of the circuit structure are a risk-free self-locking structure consisting of three AND gates and one OR gate, driven by the computation completion handshake signal Req_ready and the stop enable signal E_stop. The last layer is a reset structure consisting of a single AND gate, connected to the global reset signal Rst_n to implement the reset function for the control signal V_ACK.
[0049] As mentioned earlier, the preset termination loop conditions may include the following three items: the hidden state satisfies the preset convergence condition; the instruction address of the next instruction to be executed matches the stop address; and the iteration calculation rounds for the current token reach the preset maximum number of steps. To determine these three termination loop conditions, in some embodiments of this disclosure, reference can be made to... Figure 7 The schematic configuration of the decision sub-circuit 411 is shown. For example... Figure 7 As shown, the decision sub-circuit 411 includes a hidden state convergence evaluation module 4111, an FLEQ instruction branch module 4112, a loop step control module 4113, and a multi-input OR gate 4114. The outputs of the hidden state convergence evaluation module 4111, the FLEQ instruction branch module 4112, and the loop step control module 4113 are respectively connected to different inputs of the multi-input OR gate 4114.
[0050] The hidden state convergence evaluation module 4111 is used to determine whether the hidden state meets the preset convergence condition. It is configured to evaluate the calculation results of the current round. x i+1 Compared with the historical calculation results of the previous round x i The comparison is performed to determine whether the preset convergence condition is met; if the preset convergence condition is met, a stop indication level is output.
[0051] For example, when determining whether the hidden state meets the preset convergence condition, the calculation results of the current round can be used first. x i+1 Compared with the historical calculation results of the previous round x i Perform element-wise difference calculations to obtain the difference calculation results. x i+1 - x i Then, the modulus of the difference calculation results is accumulated element by element to obtain the state change of the current round. i+1 The hidden state convergence evaluation module 4111 can achieve convergence if the state change does not exceed a preset convergence threshold ε, i.e., ∥. i+1 When ∥≤ε, the hidden state is determined to satisfy the preset convergence condition; otherwise, the hidden state is determined not to satisfy the preset convergence condition.
[0052] In one example, the shutdown indication level can be configured to be high. That is, when the hidden state convergence evaluation module 4111 determines that the hidden state meets the preset convergence condition, it outputs Cond_valid1=1; otherwise, it outputs Cond_valid1=0.
[0053] The FLEQ instruction branch module 4112 is used to determine whether the instruction address of the next instruction to be executed matches the stop address HALT_ADDR. It is configured to: determine the instruction address p_next of the next instruction to be executed, and output the stop indication level when the instruction address matches the stop address.
[0054] For example, when determining the address of the next instruction to be executed, the branch status flag mem[flag] can be read first. mem[flag] is a value retrieved from the memory location indexed as flag, used to indicate whether a jump occurs at the address of the next instruction to be executed. It is generally calculated based on the FLEQ (Floating-point Less than or EqualQuiet) instruction. The FLEQ instruction branch module 4112 can determine the jump target p_p provided by the conditional jump as the address p_next of the next instruction to be executed when the branch status flag is less than or equal to 0, i.e., mem[flag] ≤ 0; otherwise, it determines the program counter count value p_PC (those skilled in the art will understand that the program counter updates its count value in an auto-incrementing manner) as the address p_next of the next instruction to be executed.
[0055] In one example, the stop indication level can be configured to be high. That is, when the FLEQ instruction branch module 4112 determines that the instruction address of the next instruction to be executed matches the stop address, i.e., p_next = HALT_ADDR, it outputs Cond_valid2 = 1; otherwise, it outputs Cond_valid2 = 0.
[0056] The loop step control module 4113 is used to determine whether the iteration calculation round for the current Token has reached the preset maximum number of steps. It is configured to: count the iteration calculation rounds of the Transformer calculation circuit 200 for the current Token, and output a stop indication level when the iteration calculation round n for the current Token reaches the preset maximum iteration step N_max.
[0057] In one example, the stop indication level can be configured to be high. That is, when the iteration calculation round for the current token reaches the preset maximum number of iteration steps, i.e., n≥N_max, the loop step control module 4113 outputs Cond_valid3=1; otherwise, it outputs Cond_valid3=0.
[0058] The multi-input OR gate 4114 is configured to output a valid stop enable signal to the first input P1 when at least one of the hidden state convergence evaluation module 4111, FLEQ instruction branch module 4112 and loop step control module 4113 outputs a stop indication level.
[0059] In other words, when any one of the three termination loop conditions—"the hidden state meets the preset convergence condition," "the instruction address of the next instruction to be executed matches the stop address," and "the iteration calculation rounds for the current token reach the preset maximum number of steps"—is met, the multi-input OR gate 4114 will output a valid stop enable signal E_stop. That is, by merging the three outputs of the hidden state convergence evaluation module 4111, the FLEQ instruction branch module 4112, and the loop step control module 4113 into the multi-channel OR gate in parallel, a triple decision mechanism can be formed, consisting of adaptive convergence exit, instruction branch stop, and hardware deadlock prevention.
[0060] For example, when the stop indication level is configured to be high, the multi-input OR gate 4114 can output a high-level stop enable signal E_stop=1 as a valid stop enable signal when any one or more of the signals Cond_valid1, Cond_valid2 and Cond_valid3 are high.
[0061] In some embodiments of this disclosure, such as Figure 8 As shown, the hidden state convergence evaluation module 4111 may include a difference subtractor 41111, an accumulator 41112 and a first digital comparator 41113 connected in sequence.
[0062] The differential subtractor 41111 is configured to perform calculations on the result of the current round. x i+1 Compared with the historical calculation results of the previous round x i Perform element-wise difference calculations to obtain the element-wise difference calculation results. x i+1 - x i ).
[0063] The cumulative calculator 41112 is configured to perform element-wise modulus accumulation calculations on the difference calculation results to obtain the state change amount in the current round. i+1 ∥.
[0064] The first digital comparator 41113 is configured to: convert the state change variable / / i+1 The state change is compared with the preset convergence threshold ε. i+1 When the convergence threshold ε is not exceeded, output the shutdown indication level.
[0065] In one example, it can be like Figure 8 As shown, the convergence threshold ε is configured in the CSR (Control and Status Register) register so that the first digital comparator 41113 reads the convergence threshold ε based on the CSR register and compares it with the state change quantity ∥ i+1 Comparison between ∥.
[0066] In one example of this disclosure, the output signal of the first digital comparator 41113 can be directly used as a logic input signal into the multi-input OR gate 4114 to achieve logic control of the multi-input OR gate 4114.
[0067] In another preferred example of this disclosure, it may also be as follows: Figure 8 As shown, a first AND gate 41114 is further configured within the hidden state convergence evaluation module 4111. One input of the first AND gate 41114 is connected to the output of the first digital comparator 41113, and the other input is connected to the convergence decision enable signal En_conv. The output signal of the first AND gate 41114 is used as the logic input signal Cond_valid1 within the multi-input OR gate 4114.
[0068] The convergence decision enable signal En_conv is a flag signal used to indicate whether the Transformer computing circuit 200 has completed the current round of calculation. Specifically, the convergence decision enable signal En_conv follows the following working logic when the Transformer computing circuit 200 performs each round of calculation: it is in an invalid state when the Transformer computing circuit 200 first starts the current round of calculation; it switches to an active state after the Transformer computing circuit 200 has completed the current round of calculation.
[0069] In one example, the convergence decision enable signal En_conv can be configured to be active when it is high (i.e., En_conv=1) and inactive when it is low (i.e., En_conv=0).
[0070] The first AND gate 41114 is configured to output a stop indication level to the multi-input OR gate 4114 when the first digital comparator 41113 outputs a stop indication level and the convergence decision enable signal En_conv is valid.
[0071] For example, when the shutdown indication level is configured to be high, the first AND gate 41114 specifically outputs a shutdown indication level of Cond_valid1=1 when the first digital comparator 41113 outputs a high level and the convergence decision enable signal En_conv is in the valid state of En_conv=1.
[0072] In some embodiments of this disclosure, such as Figure 9 As shown, the FLEQ instruction branch module 4112 may include a sign and zero decision unit 41121, a PC selector 41122, and a stop instruction decoder 41123 connected in sequence.
[0073] The sign-to-zero decision 41121 is configured to compare the branch status flag mem[flag] with zero, and output a valid condition enable signal En_flag when the branch status flag mem[flag] is not greater than zero.
[0074] In one example, the conditional enable signal En_flag can be configured to be active when it is high (i.e., En_flag=1) and inactive when it is low (i.e., En_flag=0).
[0075] PC selector 41122 is configured to: when the condition enable signal En_flag is valid, determine the instruction jump target p_p in the storage location, that is, the jump target p_p of the jump instruction, as the instruction address p_next of the next instruction to be executed; otherwise, determine the program counter count value p_PC as the instruction address p_next of the next instruction to be executed.
[0076] In one example, the PC selector 41122 can be implemented through a MUX: Specifically, the conditional enable signal En_flag can be connected to the control terminal of the MUX, and the memory area used to store the jump target p_p and the program counter can be connected to different input terminals of the MUX, so that the MUX, under the control of the conditional enable signal En_flag, can select either the jump target p_p or the count value p_PC as the instruction address p_next of the next instruction to be executed and route it to the halt instruction decoder 41123.
[0077] In one example, such as Figure 9 As shown, the instruction address p_next selected by PC selector 41122 can also be routed to the temporary storage area of the unified on-chip memory space for subsequent calls.
[0078] The stop instruction decoder 41123 is configured to: determine whether the instruction address p_next matches the stop address HALT_ADDR, and output a stop indication level when the instruction address p_next matches the stop address HALT_ADDR.
[0079] In one example of this disclosure, the output signal of the stop instruction decoder 41123 can be directly used as a logic input signal into the multi-input OR gate 4114 to achieve logic control of the multi-input OR gate 4114.
[0080] In another preferred example of this disclosure, it may also be as follows: Figure 9 As shown, a second AND gate 41124 is further configured within the FLEQ instruction branch module 4112. One input of the second AND gate 41124 is connected to the output of the stop instruction decoder 41123, and the other input is connected to the stop decision enable signal En_fleq. The output signal of the second AND gate 41124 is used as the logic input signal Cond_valid2 within the multi-input OR gate 4114.
[0081] The halt decision enable signal En_fleq is a flag signal used to indicate whether the halt address HALT_ADDR has been written to the register. Specifically, before the halt address HALT_ADDR is written to the register, the halt decision enable signal En_fleq remains in an invalid state; after the halt address HALT_ADDR is written to the register, the halt decision enable signal En_fleq switches to an active state.
[0082] In one example, the shutdown decision enable signal En_fleq can be configured to be active when it is high (i.e., En_fleq=1) and inactive when it is low (i.e., En_fleq=0).
[0083] The second AND gate 41124 is configured to output the stop indication level to the multi-input OR gate 4114 when the stop instruction decoder 41123 outputs a stop indication level and the stop decision enable signal En_fleq is valid.
[0084] For example, when the stop indication level is configured to be high, the second AND gate 41124 specifically outputs a stop indication level of Cond_valid2=1 when the stop instruction decoder 41123 outputs a high level and the stop decision enable signal En_fleq is in the valid state of En_fleq=1.
[0085] In some embodiments of this disclosure, such as Figure 10As shown, the loop step control module 4113 includes a step accumulator counter 41131 and a second digital comparator 41132 connected in sequence.
[0086] The step count accumulator counter 41131 is configured to count the number of iteration rounds n of the current Token based on the number of rising edges of the accessed iterative pulse signal Step_tick.
[0087] The iterative pulse signal Step_tick generates a rising edge each time the Transformer calculation circuit 200 performs a round of calculation.
[0088] In this embodiment, during the computation preparation phase before performing iterative computation on each Token, the step accumulator counter 41131 can be reset using the global reset signal Rst_n to clear its count value. Thus, during the iterative computation of a Token by the cyclic Transformer inference circuit, the step accumulator counter 41131 can increment the count of iterations for that Token under the driving of the rising edge of the iteration pulse signal Step_tick.
[0089] The second digital comparator 41132 is configured to compare the iteration calculation round n with the preset maximum iteration step number N_max, and output a stop indication level when the iteration calculation round n reaches the preset maximum iteration step number N_max.
[0090] In one example, it can be like Figure 10 As shown, the preset maximum iteration step number N_max is configured in the CSR register so that the second digital comparator 41132 reads the preset maximum iteration step number N_max based on the CSR register and compares it with the iteration calculation round n.
[0091] In one example of this disclosure, the output signal of the second digital comparator 41132 can be directly used as a logic input signal into the multi-input OR gate 4114 to achieve logic control of the multi-input OR gate 4114.
[0092] In another preferred example of this disclosure, it may also be as follows: Figure 10 As shown, a third AND gate 41133 is further configured within the loop step control module 4113. One input of the third AND gate 41133 is connected to the output of the second digital comparator 41132, and the other input is connected to the timeout protection enable signal En_timeout. The output signal of the third AND gate 41133 is used as the logic input signal Cond_valid3 within the multi-input OR gate 4114.
[0093] The timeout protection enable signal En_timeout is a flag signal used to indicate whether the iterative calculation time for the current token has expired. Specifically, when the iterative calculation time of the loop Transformer inference circuit for the current token has not reached the preset timeout period, the timeout protection enable signal En_timeout remains in an invalid state; after the iterative calculation time of the loop Transformer inference circuit for the current token reaches the preset timeout period, the timeout protection enable signal En_timeout switches to an active state.
[0094] In one example, the timeout protection enable signal En_timeout can be configured to be active when it is high (i.e., En_timeout=1) and inactive when it is low (i.e., En_timeout=0).
[0095] The third AND gate 41133 is configured to output a stop indication level to the multi-input OR gate 4114 when the second digital comparator 41132 outputs a stop indication level and the timeout protection enable signal En_timeout is valid.
[0096] For example, when the shutdown indication level is configured to be high, the third AND gate 41133 outputs a high level when the second digital comparator 41132 outputs a high level, and when the timeout protection enable signal En_timeout is in the valid state of En_timeout=1, it outputs a shutdown indication level of Cond_valid3=1.
[0097] In one example of this disclosure, the control signals involved in the aforementioned hidden state convergence evaluation module 4111, FLEQ instruction branch module 4112, and loop step control module 4113, such as the convergence decision enable signal En_conv, the stop decision enable signal En_fleq, the global reset signal Rst_n, the iterative pulse signal Step_tick, and the timeout protection enable signal En_timeout, can all be implemented through bus configuration. This disclosure does not impose any specific limitations.
[0098] To facilitate understanding of the foregoing description of the embodiments of this disclosure, Figure 11 A possible timing waveform diagram of a cyclic Transformer inference circuit is provided.
[0099] like Figure 11As shown, during the first iteration for a token, the Transformer computation circuit 200 receives the computation request pulse Req_calc and initiates tensor computation for valid data. After aligning the computation delay T_delay via the delay matching sub-circuit 412, it outputs the computation completion handshake signal Req_ready. If the first iteration does not meet the loop termination condition, the state decision signal Cond_valid remains low (in...). Figure 11 When Cond_valid remains low, it means that Cond_valid1, Cond_valid2, and Cond_valid3 are all low; when Cond_valid is high, it means that at least one of Cond_valid1, Cond_valid2, and Cond_valid3 has switched to a high level. The stop enable signal E_stop also remains low, therefore the asynchronous latch circuit 420 maintains a low-level output control signal V_ACK = 0. (This is in conjunction with the previous explanation and...) Figure 11 As shown in the routing path of the output selector 300, under the control of the low-level control signal V_ACK, the output selector 300 maintains the feedback loop on, and the new state data output by the Transformer calculation circuit 200 flows from the feedback output terminal OUT2 to the input selector 100, so that the Transformer calculation circuit 200 starts the next round of time iteration.
[0100] During the Nth iteration, the Transformer computation circuit 200 performs tensor computation in the same manner as in the first iteration. However, in this Nth iteration, the decision sub-circuit 411 determines that the iteration meets the termination loop condition, so the state decision signal Cond_valid jumps to a high level, and the drive stop enable signal E_stop is also set to a high level. Thus, when the rising edge of the handshake signal Req_ready arrives after the calculation is complete, the asynchronous latch circuit 420 flips its state at the instant the double high input condition is met, outputs a high-level control signal V_ACK=1, and completes asynchronous latching; after the handshake signal Req_ready falls back to a low level, the asynchronous latch circuit 420 continues to latch the control signal V_ACK to a high level state using its internal hysteresis characteristic. Under the control of the high-level control signal V_ACK, the output selector 300 cuts off the feedback loop and switches the routing path to the result output terminal OUT1, so that the final token vector flows to the output buffer unit through the result output terminal OUT1. Then, under the trigger of the global reset signal Rst_n, the control state is cleared, realizing the adaptive closed-loop calculation driven by pure hardware and without global clock blocking.
[0101] The above-described cyclic Transformer inference circuit provided in this disclosure can achieve the following beneficial effects: First, this cyclic Transformer inference circuit can achieve on-chip static resident of parameters and basically eliminate off-chip memory access: by reusing a single shared weight in the time dimension through a single Transformer computing circuit, the weight parameters are completely statically resident in the on-chip SRAM, eliminating the bandwidth bottleneck of off-chip memory access and data migration power consumption in the traditional multi-layer Transformer inference process.
[0102] Secondly, this cyclic Transformer inference circuit can achieve hardware-level token adaptive deep inference: by monitoring the changes in hidden state differences in real time through hardware decision sub-circuit, tokens with different computational complexities can adaptively exit the iteration in advance, eliminating computational redundancy in deep fixed iterations and reducing end-to-end inference latency and computational energy consumption.
[0103] Third, the cyclic Transformer inference circuit can achieve fully asynchronous self-sequencing control and reliable shutdown arbitration: it uses a delay matching sub-circuit and an asynchronous latch circuit to achieve self-timed handshaking and routing latching, eliminating global clock tree overhead and metastability risks; at the same time, it integrates three decision methods: differential convergence, FLEQ instruction jump and iteration step limit, effectively preventing cyclic deadlock and false shutdown.
[0104] Based on the same inventive concept, this disclosure also provides a control method for an inference circuit, which can be applied to the cyclic Transformer inference circuit provided in any of the foregoing embodiments. See also Figure 12 The control method includes the following steps: Step S1201: Obtain the model description information of the target Transformer model, and generate configuration information for the loop Transformer inference circuit based on the model description information.
[0105] The target Transformer model is a specific loop Transformer model implemented in hardware using a loop Transformer inference circuit. For example, the model description information can be the source code of this loop Transformer model or a generic control flow program.
[0106] Configuration information may include model parameter information for the recurrent Transformer model, as well as preset values configured for fixed parameters within the recurrent Transformer inference circuit (such as the convergence threshold ε and the maximum number of iterations N_max).
[0107] Step S1202: Load the configuration information into the loop Transformer inference circuit so that the loop Transformer inference circuit can perform model calculations.
[0108] To facilitate understanding, the following will be combined with Figure 13 The code compilation and configuration loading process involved in steps S1201-S1202 will be further explained in detail. Figure 13 The document illustrates the compilation and mapping process of the Transformer model and the general program, as well as the complete hardware execution based on the configured loop Transformer inference circuit, specifically including the following steps: Step S1301: Obtain the source code of the looping Transformer model.
[0109] Step S1302: Obtain the general control flow procedure.
[0110] Step S1303: Dedicated compiler front-end syntax parsing and control flow graph construction.
[0111] Specifically, the dedicated compiler can receive the model source code of the loop Transformer model in step S1301, or receive the general control flow program of the loop Transformer model in step S1302; and parse the received model source code or general control flow program through the front-end syntax to construct a control flow graph (CFG).
[0112] Step S1304: Unify the sequence memory space partitioning and data structure arrangement.
[0113] For example, the compiler can perform three-zone delimitation and data structure arrangement on the on-chip unified storage space. Specifically, the on-chip unified storage space can be divided into a temporary storage area, a data area, and an instruction area: the temporary storage area is used to store the program counter value p_PC, calculation intermediate variables, and branch status flag mem[flag]; the data area is used to store input feature vectors, embedded tensors, and historical state vectors; and the instruction area is used to store coded FLEQ jump instructions, operator selection pointers, and jump targets p_p.
[0114] Step S1305: Static parameter extraction, quantization, and hardware configuration image generation.
[0115] In step S1305, on the one hand, model parameter binding can be performed, that is, the shared parameter matrix θ of the loop Transformer model is statically compiled and an SRAM loading image is generated. On the other hand, CSR configuration generation can be performed, that is, the convergence threshold ε, the maximum number of steps N_max, and the enable signals (convergence decision enable signal En_conv, shutdown decision enable signal En_fleq, and timeout protection enable signal En_timeout).
[0116] Step S1306: Load on-chip SRAM data and initialize control registers.
[0117] Specifically, static weights can be written into the on-chip shared parameter storage SRAM, and a unified initial sequence can be used. x Write 0 to the on-chip input sequence SRAM, write the CSR configuration parameters, and then release the global reset signal Rst_n.
[0118] Step S1307: The Transformer computation circuit performs tensor computation.
[0119] See the previous description of the Transformer computing circuit.
[0120] Step S1308: The decision sub-circuit performs convergence evaluation, FLEQ decoding, and timeout comparison in parallel.
[0121] See the explanation of the decision circuit above.
[0122] Step S1309: The asynchronous latch circuit asynchronously arbitrates and latches the signal and generates the control signal V_ACK.
[0123] When V_ACK=1, proceed with step S1310 below; when V_ACK=0, proceed with step S1311 below. For details, please refer to the previous explanation.
[0124] Step S1310: The output selector cuts off the feedback loop and routes the calculation result to the output buffer unit.
[0125] Step S1311: The output selector conducts the feedback loop to the input selector, initiating the next round of hardware iteration calculation for the time step.
[0126] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0127] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
[0128] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.
Claims
1. A cyclic Transformer inference circuit, characterized in that, include: Inlet selector, Transformer calculation circuit, outlet selector and loop control circuit; The inlet selector has a data receiving end, a data output end, a feedback receiving end, and a first control end; the data receiving end is connected to an upstream data source, and the data output end is connected to the input end of the Transformer computing circuit. The output selector has a result receiving end, a result output end, a feedback output end, and a second control end; the result receiving end is connected to the output end of the Transformer calculation circuit, and the feedback output end is connected to the feedback receiving end. The output of the Transformer calculation circuit is also connected to the input of the loop control circuit, and the output of the loop control circuit is connected to the first control terminal and the second control terminal respectively. The Transformer computing circuit is configured to perform inference calculations on the data output from the data output terminal and output the calculated results. The loop control circuit is configured as follows: After the Transformer calculation circuit completes each round of calculation, if the preset termination loop condition is not met, it outputs a first control level to the first control terminal and the second control terminal, so that the output selector routes the calculation result of the round to the feedback output terminal, and the input selector routes the calculation result received at the feedback receiving terminal to the data output terminal. If the termination loop condition is met, a second control level is output to the first control terminal and the second control terminal, so that the exit selector routes the calculation result of the first round to the result output terminal, and the entry selector routes the subsequent unprocessed data received at the data receiving terminal to the data output terminal.
2. The looping Transformer inference circuit according to claim 1, characterized in that, The loop control circuit includes a state evaluation circuit and an asynchronous latch circuit connected in sequence. The state evaluation circuit includes a decision sub-circuit and a delay matching sub-circuit. The output of the decision sub-circuit is connected to the first input of the asynchronous latch circuit, and the output of the delay matching sub-circuit is connected to the second input of the asynchronous latch circuit. The decision sub-circuit is configured to output a valid shutdown enable signal to the first input terminal when it is determined that the termination loop condition is met. The delay matching sub-circuit is configured to: in response to the accessed computing circuit ready signal switching to an active state, perform delay time matching between the computing circuit ready signal and the stop enable signal, and output a valid handshake pulse of the computing completion handshake signal to the second input terminal, so that the valid handshake pulse is used to drive the asynchronous latch circuit together with the valid stop enable signal; wherein, the computing circuit ready signal generates an active level pulse when the Transformer computing circuit completes each round of calculation, and returns to an inactive state after the active level pulse ends; The asynchronous latching circuit is configured to: output the first control level to the first control terminal and the second control terminal in the initial state of the cyclic Transformer inference circuit starting the cyclic iteration calculation of the current Token; and switch the control level output to the first control terminal and the second control terminal to the second control level in response to the shutdown enable signal and the calculation completion handshake signal both switching to the valid state, and latch the second control level.
3. The cyclic Transformer inference circuit according to claim 2, characterized in that, The delay matching sub-circuit is specifically configured to: in response to the accessed computing circuit ready signal switching to the valid state, after waiting for a preset delay period, output a valid handshake pulse of the calculation completion handshake signal to the second input terminal, so that the valid handshake pulse is used to drive the asynchronous latch circuit together with the valid stop enable signal.
4. The cyclic Transformer inference circuit according to claim 2, characterized in that, The asynchronous latch circuit also has a reset control terminal, which is connected to a global reset signal. The global reset signal is valid during the calculation preparation phase before performing iterative calculation on each Token, and remains invalid at other times. The asynchronous latch circuit is further configured to: in response to the global reset signal switching to an active state, reset the control level output to the first control terminal and the second control terminal to the first control level.
5. The cyclic Transformer inference circuit according to claim 2, characterized in that, The decision sub-circuit includes a hidden state convergence evaluation module, a FLEQ instruction branch module, a loop step control module, and a multi-input OR gate; the output terminals of the hidden state convergence evaluation module, the FLEQ instruction branch module, and the loop step control module are respectively connected to different input terminals of the multi-input OR gate. The hidden state convergence evaluation module is configured to: compare the calculation result of the current round with the historical calculation result of the previous round to determine whether the preset convergence condition is met; and output a shutdown indication level when the preset convergence condition is met. The FLEQ instruction branch module is configured to: determine the instruction address of the next instruction to be executed; and output a stop indication level when the instruction address matches the stop address. The cycle step control module is configured to: count the iteration calculation rounds of the Transformer calculation circuit for the current Token, and output a stop indication level when the iteration calculation rounds reach the preset maximum number of iteration steps; The multiple-input OR gate is configured to output a valid stop enable signal to the first input terminal when at least one of the hidden state convergence evaluation module, the FLEQ instruction branch module, and the loop step control module outputs the stop indication level.
6. The cyclic Transformer inference circuit according to claim 5, characterized in that, The hidden state convergence evaluation module includes a difference subtractor, an accumulator and a first digital comparator connected in sequence. The differential subtractor is configured to perform element-wise differential calculation on the calculation result of the current round and the historical calculation result of the previous round to obtain the element-wise differential calculation result. The accumulator is configured to perform element-wise modulus accumulation calculation on the difference calculation results to obtain the state change amount of the current round; The first digital comparator is configured to compare the state change with a preset convergence threshold, and output a shutdown indication level when the state change does not exceed the convergence threshold.
7. The cyclic Transformer inference circuit according to claim 6, characterized in that, The hidden state convergence evaluation module also includes a first AND gate, one input of which is connected to the output of the first digital comparator, and the other input is connected to a convergence decision enable signal; the convergence decision enable signal is in an invalid state when the Transformer calculation circuit starts the calculation of the current round, and switches to an valid state after the Transformer calculation circuit completes the calculation of the current round. The first AND gate is configured to output the stop indication level to the multi-input OR gate when the first digital comparator outputs a stop indication level and the convergence decision enable signal is valid.
8. The cyclic Transformer inference circuit according to claim 5, characterized in that, The FLEQ instruction branch module includes a sign and zero decision unit, a PC selector, and a stop instruction decoder connected in sequence. The symbol and zero decision mechanism is configured to compare a branch status flag with zero, the branch status flag being used to indicate whether a jump has occurred at the address of the next instruction to be executed, and output a valid condition enable signal when the branch status flag is not greater than zero. The PC selector is configured to: when the condition enable signal is valid, determine the jump target of the jump instruction as the instruction address of the next instruction to be executed; otherwise, determine the count value of the program counter as the instruction address of the next instruction to be executed. The stop instruction decoder is configured to: determine whether the instruction address matches the stop address, and when the instruction address matches the stop address, output a stop indication level.
9. The cyclic Transformer inference circuit according to claim 8, characterized in that, The FLEQ instruction branch module also includes a second AND gate, one input of which is connected to the output of the stop instruction decoder, and the other input is connected to the stop decision enable signal; the stop decision enable signal remains in an invalid state before the stop address is written to the register, and switches to an valid state after the stop address is written to the register; The second AND gate is configured to output the stop indication level to the multi-input OR gate when the stop instruction decoder outputs a stop indication level and the stop decision enable signal is valid.
10. The looping Transformer inference circuit according to claim 5, characterized in that, The cycle step control module includes a step accumulator counter and a second digital comparator connected in sequence. The step count accumulator counter is configured to count the iteration calculation rounds of the current Token based on the number of rising edges of the accessed iterative pulse signal. The iterative pulse signal generates a rising edge each time the Transformer computing circuit performs a round of calculation; The second digital comparator is configured to compare the iteration calculation round with a preset maximum number of iteration steps, and output a stop indication level when the iteration calculation round reaches the preset maximum number of iteration steps.
11. The looping Transformer inference circuit according to claim 10, characterized in that, The loop step control module also includes a third AND gate, one input of which is connected to the output of the second digital comparator, and the other input is connected to a timeout protection enable signal; the timeout protection enable signal remains in an invalid state until the iteration calculation time for the current Token reaches the preset timeout time, and switches to an active state after the iteration calculation time reaches the preset timeout time. The third AND gate is configured to output the stop indication level to the multi-input OR gate when the second digital comparator outputs a stop indication level and the timeout protection enable signal is valid.
12. A control method for an inference circuit, characterized in that, The method, applied to a loop Transformer inference circuit as described in any one of claims 1-11, comprises: Obtain the model description information of the target Transformer model, and generate configuration information for the loop Transformer inference circuit based on the model description information; The configuration information is loaded into the loop Transformer inference circuit so that the loop Transformer inference circuit performs model calculations.
