Multimodal inference chips and industrial fault diagnosis methods
By using unified instruction scheduling and reusable processing unit clusters in multimodal inference chips, the problems of resource duplication and latency in multimodal neural network computing in existing technologies are solved, achieving efficient multimodal neural network computing and reducing latency and power consumption.
Patent Information
- Application Number
- CN202611593742.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-15
- Publication Date
- 2026-11-10
AI Technical Summary
Existing inference chips struggle to efficiently and uniformly schedule multimodal neural network computations, leading to resource duplication, increased overhead from network weight loading and intermediate data movement, and higher computational latency and power consumption for different network types.
By employing a multimodal inference chip, a unified instruction scheduler and reusable processing unit clusters are used to achieve unified computation of convolution, long short-term memory and liquid neural networks, reducing network type switching and intermediate data transfer. Local weight cache and weight encoding/decoding units are used to optimize network weight loading, and strict priority and weighted round-robin scheduling are used to optimize data flow, thereby reducing latency and power consumption.
It achieves efficient and unified scheduling of multimodal neural network computation, reduces resource duplication and intermediate data movement, improves the utilization of the multiply-accumulate array, reduces latency and power consumption, and improves computational efficiency.
Smart Images

Figure CN123070355A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of chip technology, and in particular to a multimodal inference chip and an industrial fault diagnosis method. Background Technology
[0002] Industrial equipment fault diagnosis typically requires processing multi-source sensor data, including vibration, acoustics, current, temperature, and pressure. Convolutional neural networks are suitable for extracting local spatial features, frequency domain features, or time-frequency plot features; long short-term memory networks are suitable for handling long-term dependencies in discrete-time series; and liquid neural networks are suitable for describing continuous-time state evolution and dynamic processes under irregular sampling conditions. Therefore, the same industrial fault diagnosis task may require sequential or interleaved execution of the above-mentioned network calculations.
[0003] Existing inference chips are mostly designed for a single network type. Accelerators that only support convolutional neural networks cannot directly perform gated state updates for long short-term memory networks or continuous-time state updates for liquid neural networks. Although some general-purpose neural network processors can perform matrix multiplication, different network types often occupy dedicated computing resources or rely on microcontrollers to perform subsequent nonlinear processing and state updates, resulting in increased chip area, power consumption, control overhead, and data transfer overhead.
[0004] Traditional convolutional inference typically executes convolution, activation, and pooling separately. Activation results need to be repeatedly written to and read from the activation memory, resulting in significant on-chip storage bandwidth and dynamic power consumption overhead. When multiple processing unit clusters generate activation results simultaneously, the fixed polling method cannot differentiate scheduling based on the real-time and throughput requirements of each data stream, easily causing latency jitter in critical data streams.
[0005] If the four types of gate projections and element-wise state updates of Long Short-Term Memory (LSTM) networks are executed sequentially, the inference latency is high. State updates in liquid neural networks involve exponential decay factors; performing division and exponentiation operations in real-time at each time step increases hardware complexity and processing cycle time. Repeated loading of network weights from off-chip or on-chip weight memory also limits the utilization of the multiply-accumulate array. Therefore, how to uniformly schedule multimodal neural network computations and reduce network weight loading and intermediate data movement are technical problems that urgently need to be solved by those skilled in the art. Summary of the Invention
[0006] This application provides a multimodal inference chip and an industrial fault diagnosis method, which utilizes a reusable cluster of processing units to perform convolution or matrix projection operations common to three types of computing networks, and the subsequent processing is completed by the unit corresponding to the computing type, thereby reducing the overhead of network type switching, network weight loading and intermediate data transfer.
[0007] In a first aspect, this application provides a multimodal inference chip, comprising: an instruction input unit for receiving an instruction containing an opcode and operation parameters; a scheduler for parsing the instruction, determining the type of computation to be executed based on the opcode, and forming a computation task and start control information corresponding to the computation type based on the computation type and the operation parameters; the computation type includes convolutional neural network computation, long short-term memory network computation, and liquid neural network computation; a cluster of multiple processing units for performing convolution operations or matrix projection operations based on network weights and activation data to obtain computation results corresponding to the computation type, and transmitting the computation results to a streaming processing unit, a long short-term memory network processing unit, or a liquid neural network processing unit based on the computation type; the streaming processing unit for performing streaming arbitration and pooling processing on the computation results corresponding to the convolutional neural network computation after receiving streaming arbitration and pooling mode control information sent by the scheduler; The Long Short-Term Memory (LSTM) network processing unit is configured to perform gate activation and state fusion on the computation result corresponding to the LTM network computation after receiving the gating fusion start information sent by the scheduler; and the Liquid Neural Network (LNN) processing unit is configured to perform neuron state update on the computation result corresponding to the LNN computation after receiving the closed-loop state update start information sent by the scheduler; wherein, the scheduler is further configured to distribute the computation task to at least one of the processing unit clusters, and send the start control information to the streaming processing unit, the LTM network processing unit, or the LNN processing unit corresponding to the computation type.
[0008] In one embodiment of the first aspect, the instruction is a 64-bit fixed-length instruction, the instruction including an opcode field, a flag field, and multiple parameter fields; the scheduler includes an instruction decoding unit, an instruction distribution unit, and a completion signal detection unit; the instruction decoding unit is used to form the computation task and the start control information based on the opcode field, the flag field, and the multiple parameter fields; the instruction distribution unit is used to send the computation task to at least one of the processing unit clusters, and to send the start control information to the streaming processing unit, the long short-term memory network processing unit, or the liquid neural network processing unit corresponding to the computation type; the completion signal detection unit is used to control the acquisition and distribution of subsequent instructions based on the completion signal returned by the unit receiving the computation task or the start control information; wherein, the computation task includes a convolution operation task, a long short-term memory network gate projection task, or a liquid neural network matrix projection task.
[0009] In one embodiment of the first aspect, an on-chip storage subsystem and a weight encoding / decoding unit are further included; the on-chip storage subsystem includes a weight memory for storing the network weights and an activation memory for storing the activation data, the calculation results, and state data; the weight encoding / decoding unit is used to perform differential decoding, zero-value run-length decoding, or a hybrid decoding combining differential decoding and zero-value run-length decoding on the compressed network weights; the weight encoding / decoding unit includes a prefetch cache, which is used to cache the decoded network weights required for subsequent calculations during the calculation of the current network weights by the processing unit cluster.
[0010] In one embodiment of the first aspect, each of the processing unit clusters includes a multiply-accumulate array and a local weight cache; the local weight cache is used to simultaneously cache network weights corresponding to multiple output channels; the multiply-accumulate array is used to obtain the convolution operation results corresponding to each output channel based on the same input activation window and the network weights corresponding to each output channel.
[0011] In one embodiment of the first aspect, each of the processing unit clusters further includes an activation circuit and a streaming output interface; the streaming processing unit includes a streaming arbiter and a pooling unit; the scheduler is configured to configure the processing unit clusters, the activation circuit, the streaming output interface, the streaming arbiter, and the pooling unit to form a convolution, activation, and pooling continuous processing path, so that the activation result obtained by the activation circuit is transmitted to the pooling unit via the streaming output interface and the streaming arbiter without being written to the activation memory, and the pooling unit writes the pooling result to the activation memory.
[0012] In one embodiment of the first aspect, the flow arbitrator has a strict priority scheduling mode and a weighted round-robin scheduling mode; in the strict priority scheduling mode, the flow arbitrator is used to select the highest priority processing unit cluster from the processing unit clusters with valid activation results; in the weighted round-robin scheduling mode, the flow arbitrator is used to select each processing unit cluster based on the scheduling weight corresponding to each processing unit cluster and the relative service quota corresponding to the scheduling weight.
[0013] In one embodiment of the first aspect, the scheduler is used to distribute matrix projection calculations corresponding to the forget gate, input gate, output gate, and candidate cell state to multiple processing unit clusters to obtain the forget gate projection result, input gate projection result, output gate projection result, and candidate cell state projection result in parallel, and send the gating fusion start information to the long short-term memory network processing unit; the long short-term memory network processing unit includes multiple parallel gating calculation channels for performing nonlinear processing and element-wise state fusion on the forget gate projection result, the input gate projection result, the output gate projection result, and the candidate cell state projection result to obtain the cell state and hidden state at the current time step.
[0014] In one embodiment of the first aspect, the liquid neural network processing unit includes an exponential decay factor lookup table and a state update circuit; the exponential decay factor lookup table is used to output a pre-stored exponential decay factor according to a time constant when a preset time interval is 1; the state update circuit is used to obtain the current neuron state based on the closed state update relationship after receiving the closed state update start information sent by the scheduler. This is done based on the neuron state at the previous moment, the input projection result and state projection result obtained by the processing unit cluster, the bias, and the exponential decay factor.
[0015] Secondly, this application provides an industrial fault diagnosis method, comprising: acquiring operating data of industrial equipment; inputting the operating data as activation data into a multimodal inference chip as described in the first aspect, and acquiring inference results using the multimodal inference chip; determining the fault state of the industrial equipment based on the inference results; wherein acquiring the inference results comprises: receiving an instruction containing an opcode and operating parameters; parsing the instruction, determining the type of computation to be executed from convolutional neural network computation, long short-term memory network computation, and liquid neural network computation based on the opcode, and forming a computation task and startup control information corresponding to the computation type based on the computation type and the operating parameters; distributing the computation task to at least one of the processing unit clusters, and sending the startup control information to the streaming processing unit, the long short-term memory network processing unit, or the liquid neural network processing unit corresponding to the computation type, and performing convolution operations or matrix projection operations based on network weights and activation data through the processing unit clusters to acquire computation results; According to the calculation type, the calculation result is transmitted to the streaming processing unit, the long short-term memory network processing unit, or the liquid neural network processing unit that has received the start-up control information, so as to perform streaming arbitration and pooling processing, gate activation and state fusion, or neuron state update to obtain the inference result.
[0016] In one embodiment of the second aspect, the multimodal inference chip further includes an activation memory for storing the activation data and the calculation results; when the calculation type is the convolutional neural network calculation, obtaining the inference results further includes: performing convolution operations through the processing unit clusters and performing activation processing on the convolution operation results, and transmitting the obtained activation results to the streaming processing unit without writing them to the activation memory; through the streaming processing unit, selecting the processing unit cluster with the highest priority and valid activation results based on the priority corresponding to each processing unit cluster, or selecting each processing unit cluster based on the relative service quota corresponding to the scheduling weights, and performing pooling processing on the activation results output by the selected processing unit clusters.
[0017] In summary, the multimodal inference chip and industrial fault diagnosis method provided in this application have the following beneficial effects:
[0018] (1) The same scheduler determines whether to perform convolutional neural network computation, long short-term memory network computation or liquid neural network computation according to unified instructions, and distributes high computational linear operations to reusable processing unit clusters, which can reduce resource duplication caused by setting up basic computing arrays for different network types.
[0019] (2) The local weight cache simultaneously holds the network weights corresponding to multiple output channels. The processing unit cluster can reuse the input activation window, reducing the repeated loading of network weights and improving the utilization of the multiply-accumulate array.
[0020] (3) The weight encoding and decoding unit, in conjunction with the prefetch buffer, can reduce the amount of data for network weights in compressed form and hide at least part of the decoding delay.
[0021] (4) The continuous processing path of convolution, activation and pooling allows the activation result to bypass the activation memory and directly enter the pooling unit, reducing intermediate writes and rereads.
[0022] (5) Strict priority scheduling can prioritize data streams with high real-time requirements, while weighted round-robin scheduling can provide proportional service quotas based on scheduling weights.
[0023] (6) The four types of gate projection and gating state fusion of long short-term memory networks can be executed in parallel to reduce the latency of temporal reasoning.
[0024] (7) The liquid neural network processing unit updates the neuron state based on the closed state update relationship using the pre-obtained exponential decay factor, which can reduce the hardware for real-time division and exponential operation. Attached Figure Description
[0025] Figure 1This is a schematic diagram of the basic architecture of a multimodal inference chip in one embodiment of this application.
[0026] Figure 2 This is a schematic diagram of an extended architecture of a multimodal inference chip in another embodiment of this application.
[0027] Figure 3 This is a schematic diagram of the fields and operation categories of a fixed-length instruction in one embodiment of this application.
[0028] Figure 4 This is a schematic diagram illustrating the internal structure of the scheduler and the distribution of computational tasks and start-up control information in one embodiment of this application.
[0029] Figure 5 This is a schematic diagram of the internal structure of the processing unit cluster in one embodiment of this application.
[0030] Figure 6 This is a schematic diagram illustrating the relationship between the local weight cache and the multiply-accumulate array in one embodiment of this application.
[0031] Figure 7 This is a schematic diagram of network weight encoding / decoding, error correction protection, and two-level caching in one embodiment of this application.
[0032] Figure 8 This is a schematic diagram of a continuous processing path of convolution, activation, and pooling in one embodiment of this application.
[0033] Figure 9 This is a schematic diagram of strict priority scheduling and weighted round-robin scheduling in one embodiment of this application.
[0034] Figure 10 This is a schematic diagram of parallel projection and gating fusion of a long short-term memory network in one embodiment of this application.
[0035] Figure 11 This is a schematic diagram of the closed-loop state update of a liquid neural network in one embodiment of this application.
[0036] Figure 12 This is a flowchart illustrating an industrial fault diagnosis method according to one embodiment of this application.
[0037] Figure 13 This is a schematic diagram of the process for obtaining reasoning results in one embodiment of this application. Detailed Implementation
[0038] The following specific examples illustrate the implementation of this application. Those skilled in the art can understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.
[0039] As used herein, the singular forms “a,” “an,” and “the” also include the plural forms, unless the context clearly indicates otherwise. The terms “comprising” and “including” indicate the presence of the said feature, operation, element, or component, but do not exclude the presence or addition of one or more other features, operations, elements, or components. “Or” and “and / or” should be interpreted as including any one or any combination thereof.
[0040] To ensure consistency in terminology, the reconfigurable hardware units that perform basic convolution and matrix projection operations are uniformly referred to as "processing unit clusters" in the following embodiments of this application; the parameters involved in neural network calculations are uniformly referred to as "network weights," and are defined as "compressed network weights" or "decoded network weights" according to their state; the data read by the current convolution or matrix projection operation is referred to as "activation data," the data generated by the convolution operation is referred to as "convolution operation result," the data generated by the activation process is referred to as "activation result," and the data generated by the pooling process is referred to as "pooling result."
[0041] Furthermore, in the following embodiments of this application, "output channel" refers to the output channel of the convolution result in the feature map channel dimension; "input path" refers to the hardware connection path between the stream arbitrator and different processing unit clusters. "Output channel" and "input path" are different concepts. "Computation type" refers to convolutional neural network computation, long short-term memory network computation, or liquid neural network computation; "long short-term memory network" is abbreviated as "LSTM", "liquid neural network" is abbreviated as "LNN", and "convolutional neural network" is abbreviated as "CNN"; "computation task" refers to the convolution operation task, long short-term memory network gate projection task, or liquid neural network matrix projection task sent by the scheduler to the processing unit cluster; "startup control information" refers to the stream arbitration and pooling mode control information, gate fusion startup information, or closed state update startup information sent by the scheduler to the post-processing unit; "computation result" refers to the data output by the processing unit cluster after performing convolution operation or matrix projection operation.
[0042] Currently, there are three main types of inference chips for industrial equipment fault diagnosis: pure convolutional neural network accelerators, general-purpose neural network processors, and discrete multi-chip solutions. Due to fixed hardware, instruction set limitations, and inter-chip communication bottlenecks, these three architectures cannot efficiently unify the inference of CNN, LSTM, and LNN on a single chip. They also share common defects such as memory walls, low parallelism, and insufficient industrial reliability.
[0043] Pure CNN accelerators employ fixed systolic arrays and hard-wired state machines, supporting only standard convolution operators. When LSTM is needed for time-series prediction or LNN for dynamic system modeling, an external microcontroller unit (MCU) or digital signal processor (DSP) must be connected for software computation, leading to a sharp increase in power consumption and latency. Furthermore, convolution, activation, and pooling operators are executed independently, with intermediate tensors repeatedly written back to static random access memory (SRAM), creating a severe memory wall bottleneck. While general-purpose NPUs provide matrix instructions, LSTM's four-gate projection still executes sequentially, and gating fusion lacks 128-way parallel hardware. The division and exponentiation operations required for LNN's Ordinary Differential Equation (ODE) stepping can only be completed by the Arithmetic Logic Unit (ALU) over multiple cycles. In addition, these chips do not support weight compression and error-correcting codes (ECC), resulting in a lack of reliability in harsh industrial environments.
[0044] Furthermore, the discrete solution uses a dedicated CNN chip paired with an ARM or RISC-V microcontroller, communicating via a Serial Peripheral Interface (SPI) or Inter-Integrated Circuit (I2C) bus. The bandwidth is typically ≤50Mbps, and the power consumption and area of the dual-chip solution exceed the sensor node budget. The fixed polling arbitration lacks Quality of Service (QoS) classification, making the critical path delay uncertain.
[0045] To at least address the aforementioned technical problems, this application provides a multimodal inference chip and an industrial fault diagnosis method. This method can select at least one type of computation, such as convolutional neural network computation, long short-term memory network computation, and liquid neural network computation, based on different industrial equipment operating data and inference tasks through a unified input instruction. It then uses a unified processing unit cluster to complete convolutional computation or matrix projection, and subsequently uses the corresponding type of neural network to complete the inference computation. This reduces the overhead of network type switching, network weight loading, and intermediate data transfer.
[0046] Figure 1 A schematic diagram of the architecture of the multimodal inference chip 1 described in an embodiment of this application is shown. Figure 1 As shown, the multimodal inference chip 1 includes an instruction input unit 10, a scheduler 20, multiple processing unit clusters 30, a streaming processing unit 40, a long short-term memory network processing unit 50, and a liquid neural network processing unit 60.
[0047] like Figure 1 As shown, in some embodiments, the instruction input unit 10 first receives instructions characterizing the computation type and operation parameters, and sends the instructions to the scheduler 20. The scheduler 20 parses the instructions, forms convolution operation tasks, long short-term memory network gate projection tasks, or liquid neural network matrix projection tasks, and distributes the corresponding computation tasks to at least one processing unit cluster 30. The scheduler 20 also sends streaming arbitration and pooling mode control information to the streaming processing unit 40, gate fusion start information to the long short-term memory network processing unit 50, or closed-loop state update start information to the liquid neural network processing unit 60, depending on the computation type. Multiple processing unit clusters 30 reuse their respective multiply-accumulate resources to perform convolution operations or matrix projection operations and output computation results. After receiving the corresponding start control information and computation results, the streaming processing unit 40, the long short-term memory network processing unit 50, or the liquid neural network processing unit 60 performs streaming arbitration and pooling processing, gate activation and state fusion, or closed-loop neuron state update. Figure 1 The dashed lines in the diagram represent the path through which the scheduler 20 sends start control information to the post-processing unit, while the solid lines represent the transmission paths for instructions, computation tasks, or computation results.
[0048] Figure 2 This diagram illustrates an extended architecture of the multimodal inference chip 1 according to another embodiment of this application. Figure 2 As shown, the multimodal inference chip 1 may also include an on-chip storage subsystem 70, a weight encoding and decoding unit 80, a peripheral and input / output subsystem 90, a power management unit 100, an online anomaly detection unit 110, and an activation processing unit 120, and may be connected to an off-chip non-volatile memory 130.
[0049] like Figure 2As shown, after the multimodal inference chip 1 is powered on, the startup loading unit 904 reads the inference program and compressed network weights from the off-chip non-volatile memory 130 through the peripheral input / output subsystem 90, and sends the startup command to the scheduler 20. The weight encoding / decoding unit 80 determines the original, differential, zero-value run, or mixed encoding mode through the encoding mode recognition unit 802, and the decoding unit 803 completes the corresponding decoding; the prefetch cache 801 caches the decoded network weights required for subsequent calculations, and the weight memory 701 and activation memory 702 of the on-chip storage subsystem 70 provide network weights and activation data to the processing unit cluster 30, respectively. The scheduler 20 distributes the convolution operation task, the long short-term memory network gate projection task, or the liquid neural network matrix projection task to the processing unit cluster 30, and sends startup control information to the corresponding streaming processing unit 40, long short-term memory network processing unit 50, or liquid neural network processing unit 60. After completing convolution or matrix projection operations, processing unit cluster 30 transmits the convolution results, long short-term memory network gate projection results, or liquid neural network input projection results and state projection results to streaming processing unit 40, long short-term memory network processing unit 50, or liquid neural network processing unit 60, respectively. Each unit executes corresponding subsequent processing upon receiving the corresponding start control information and calculation results. Figure 2 The dashed lines in the diagram represent the path for startup control information, while the solid lines represent the path for instructions, data, or calculation results.
[0050] Furthermore, the activation processing unit 120 performs independent activation processing, the online anomaly detection unit 110 performs statistical monitoring of the inference output results, and outputs an anomaly interrupt via the peripheral input / output subsystem 90. The power management unit 100 supplies power to the corresponding functional parts and, in conjunction with the scheduler 20, implements clock gating or deep sleep control during idle periods.
[0051] Please continue reading. Figure 1 and Figure 2 As shown, in some embodiments, the instruction input unit 10 is used to receive an instruction containing an opcode and operation parameters. The opcode identifies the type of operation or control to be performed; the operation parameters provide an address, data size, operation size, status position, or configuration parameters.
[0052] In some embodiments, the instruction is a 64-bit fixed-length instruction, including an opcode field, a flag field, and multiple parameter fields. Figure 3 This is a diagram illustrating the fields and operation categories of a 64-bit fixed-length instruction, such as... Figure 3 As shown, the opcode field is located at bits 63 to 56, the flag field is located at bits 55 to 48, and the three parameter fields are located at bits 47 to 32, bits 31 to 16, and bits 15 to 0, respectively. Using a fixed-length format facilitates instruction fetching and decoding within a fixed timing sequence.
[0053] In some embodiments, the opcode field can be used to indicate convolution operations, matrix projection operations, activation processing, pooling processing, long short-term memory network gating processing, liquid neural network state update processing, data transmission processing, or control processing.
[0054] In some embodiments, the flag field may carry fusion enable, streaming output enable, quantization enable, compression mode, pooling mode, or scheduling mode.
[0055] In some embodiments, the parameter field may carry input address, output address, network weight address, data size, convolution kernel size, stride, padding method, pooling mode, time constant, state address, or number of iterations.
[0056] like Figure 3 As shown, the instruction input unit 10 provides instructions to the scheduler 20 in a 64-bit fixed-length format. The scheduler 20 acquires the opcode field, flag field, and parameters A, B, and C in parallel within a fixed instruction fetch cycle. The opcode field is used to determine the operation category, such as convolution or matrix projection, activation or pooling, long short-term memory network, liquid neural network, data transmission, or control. The flag field is used to determine fusion enable, streaming output enable, quantization, compression, pooling, or scheduling mode. Each parameter field carries the address, size, stride, time constant, or other operation parameters based on the interpretation rules corresponding to the opcode. Based on this, the scheduler 20 forms computational tasks for the processing unit cluster 30 and startup control information for the corresponding post-processing units, enabling different computation types to use a unified instruction entry point while maintaining the definite position and fixed decoding timing of each parameter field.
[0057] In some embodiments, the categories of 64-bit fixed-length instructions are shown in Table 1.
[0058] Table 1. Categories of 64-bit fixed-length instructions
[0059] Instruction categories Command Name Opcode Function Description Convolutional classes CONV_1D 0x01 One-dimensional convolution CONV_2D 0x02 Two-dimensional convolution (kernel 1×1 / 3×3 / 5×5, stride / padding optional) DWCONV 0x03 Depthwise separable convolution PWCONV 0x04 Pointwise convolution Fully connected DENSE 0x05 Matrix-vector multiplication Fusion Command FUSED_CAP 0x15 Conv+Act+Pool Single Instruction Full Pipeline Straight Through LSTM LSTM_PROJ_F 0x16 Forget gate linear projection → PE Cluster LSTM_PROJ_I 0x17 Input gate linear projection → PE Cluster LSTM_PROJ_O 0x18 Output gate linear projection → PE Cluster LSTM_PROJ_C 0x19 Cell gate linear projection → PE Cluster LSTM_GATE 0x1A 128-channel parallel gating fusion (3 cycles) LNN LNN_PROJ_WX 0x1B LNN Input projection LNN_PROJ_UH 0x1C LNN State projection LNN_ODE_STEP 0x1D Liquid state update closed-loop step activation RELU / SIGMOID / TANH / LEAKY_RELU 0x08-0x0B LUT Hardware Acceleration Activation Pooling POOL_MAX / AVG / GLOBAL 0x0C-0x0E 2×2 / 3×3 window pooling, stride 1 / 2 element by element ELEM_ADD / MUL 0x0F-0x10 Tensor element-wise operations Data transmission DMA_LOAD / STORE 0x11-0x12 SRAM ↔ External Data Transfer control SYNC / CONFIG / END_PROG 0x13 / 0x14 / 0xFF Synchronization Fence / Configuration / Program End
[0060] Please continue reading. Figure 1 and Figure 2 As shown, in some embodiments, the scheduler 20 is used to parse the instruction, determine the type of computation to be executed according to the opcode, and form a computation task according to the computation type and the operation parameters. The computation type includes convolutional neural network computation, long short-term memory network computation, and liquid neural network computation.
[0061] Figure 4The diagram shows the internal structure of the scheduler 20 and the distribution of computational tasks and control information. In some embodiments, the scheduler 20 includes an instruction decoding unit 201, an instruction distribution unit 202, and a completion signal detection unit 203. The instruction decoding unit 201 is used to form the computational task and the start control information based on the opcode field, the flag field, and the plurality of parameter fields; the instruction distribution unit 202 is used to send the computational task to the processing unit cluster 30 and send the start control information to the corresponding post-processing unit; the completion signal detection unit 203 is used to control the acquisition and distribution of subsequent instructions based on the completion signal returned by the unit that received the computational task or the start control information. Figure 4 In the diagram, solid lines represent computational tasks, while dashed lines represent startup control information.
[0062] For example, for convolutional neural network computation, scheduler 20 forms convolution operation tasks and flow arbitration and pooling mode control information; for long short-term memory network computation, scheduler 20 forms gate projection tasks corresponding to forget gate, input gate, output gate and candidate cell state respectively, and gate fusion start information; for liquid neural network computation, scheduler 20 forms input matrix projection task corresponding to the current input, state matrix projection task corresponding to the neuron state at the previous time step, and closed state update start information.
[0063] In some embodiments, the instruction decoding unit 201 forms a computation task and startup control information based on the opcode field, the flag field, and multiple parameter fields. The computation task sent to the processing unit cluster 30 may include the operation type, network weight address, activation data address, output address, matrix dimension, convolution kernel size, stride, and padding parameters; the startup control information sent to the streaming processing unit 40, the long short-term memory network processing unit 50, or the liquid neural network processing unit 60 may include the scheduling mode, pooling mode, gating fusion startup flag, closed-loop state update startup flag, or time constant.
[0064] In some embodiments, the instruction distribution unit 202 sends convolution operation tasks, long short-term memory network gate projection tasks, or liquid neural network matrix projection tasks to the corresponding processing unit clusters 30 according to the computation type, and sends streaming arbitration and pooling mode control information, gating fusion startup information, or closed-loop state update startup information to the corresponding streaming processing unit 40, long short-term memory network processing unit 50, or liquid neural network processing unit 60. The instruction distribution unit 202 determines the timing of sending computation tasks and corresponding startup control information based on data dependencies and completion signals.
[0065] In some embodiments, the completion signal detection unit 203 receives a calculation completion signal returned by the processing unit cluster 30, and receives a post-processing completion signal returned by the streaming processing unit 40, the long short-term memory network processing unit 50, or the liquid neural network processing unit 60, thereby controlling the acquisition and distribution of subsequent instructions.
[0066] Furthermore, the signal detection unit 203 can also be configured with a timeout counter; for example, an error status can be output when the waiting time exceeds 200 clock cycles.
[0067] Continue reading Figure 4 As shown, in some embodiments, the scheduler 20 may further include an instruction cache unit 204 and an instruction prefetch unit 205. The instruction cache unit 204 may use a first-in-first-out queue to cache instructions to be executed; the instruction prefetch unit 205 may prefetch subsequent instructions during the execution of the current instruction to hide instruction read latency. For example, the instruction cache unit 204 has a depth of 16, and the instruction prefetch unit 205 includes a four-level prefetch buffer; however, those skilled in the art can adjust these values according to actual needs, and this application does not impose any limitations on this.
[0068] Continue reading Figure 1 and Figure 2 As shown, the scheduler 20 is also used to distribute convolution operation tasks, long short-term memory network gate projection tasks, or liquid neural network matrix projection tasks to at least one processing unit cluster 30, and to send stream arbitration and pooling mode control information, gating fusion start information, or closed-loop state update start information to the corresponding streaming processing unit 40, long short-term memory network processing unit 50, or liquid neural network processing unit 60, respectively. After the processing unit cluster 30 completes the convolution operation or matrix projection operation, it transmits the output calculation result to the streaming processing unit 40, long short-term memory network processing unit 50, or liquid neural network processing unit 60 that has received the corresponding start control information.
[0069] For example, when the computation type is a convolutional neural network computation, the scheduler 20 distributes the convolution operation task to at least one processing unit cluster 30 and sends streaming arbitration and pooling mode control information to the streaming processing unit 40; after the processing unit cluster 30 completes the convolution operation, it performs activation processing on the convolution operation result and transmits it to the streaming processing unit 40. When the computation type is a long short-term memory network computation, the scheduler 20 distributes the four types of gate projection tasks to multiple processing unit clusters 30 and sends gating fusion start information to the long short-term memory network processing unit 50; the processing unit cluster 30 transmits the four types of gate projection results to the long short-term memory network processing unit 50. When the computation type is a liquid neural network computation, the scheduler 20 distributes the input matrix projection task and the state matrix projection task to at least one processing unit cluster 30 and sends closed-loop state update start information to the liquid neural network processing unit 60; the processing unit cluster 30 transmits the input projection result and the state projection result to the liquid neural network processing unit 60. Thus, the same group of processing units 30 undertakes the high computational linear operations required by the three networks, and different network units complete the corresponding network-specific subsequent processing under the control of the scheduler 20.
[0070] like Figure 4 As shown, external instructions sequentially enter the instruction cache unit 204 and the instruction prefetch unit 205, while the scheduler 20 prefetches subsequent instructions during the execution of the current instruction. The instruction decoding unit 201 identifies the computation type and dependencies based on the opcode, flag field, and parameter field, and respectively forms computation tasks for the processing unit cluster 30 and start control information for the post-processing unit. The instruction distribution unit 202 sends convolution operation tasks, long short-term memory network gate projection tasks, or liquid neural network matrix projection tasks to the processing unit cluster 30; and according to the computation type, sends streaming arbitration and pooling mode control information to the streaming processing unit 40, sends gating fusion start information to the long short-term memory network processing unit 50, or sends closed-loop state update start information to the liquid neural network processing unit 60. The processing unit cluster 30 and the corresponding post-processing unit return completion signals respectively. The scheduler 20, together with the dependency management unit 206, the loop prediction and repetition counting unit 207, and the clock gating and timeout protection unit 208, determines whether the current instruction is completed. After the dependency and completion conditions are met, the next instruction is fetched, thus forming a closed-loop control of instruction fetching, decoding, distribution, waiting for completion, and continuing instruction fetching.
[0071] Please continue reading. Figure 1 As shown, in some embodiments, multiple processing unit clusters 30 are used to perform convolution operations or matrix projection operations based on network weights and activation data to obtain computation results corresponding to the computation type. It should be noted that the processing unit clusters 30 are basic computing resources reused by all three computation types and do not form a fixed combination with the computation type of a particular network.
[0072] In some embodiments, each processing unit cluster 30 can read data based on the network weight address, activation data address, matrix dimension, convolution parameters, and output position in the computation task, and perform convolution or matrix projection operations through multiply-accumulate processing. It should be noted that the number of processing unit clusters 30, the number of multiply-accumulate units per cluster, and the numerical format can all be configured as needed, and this application does not impose any limitations on these. In some embodiments, four processing unit clusters 30 can be set, and each processing unit cluster 30 includes sixteen multiply-accumulate units.
[0073] Figure 5 This is a schematic diagram of the internal structure of processing unit cluster 30. (See diagram below.) Figure 5 As shown, each processing unit cluster 30 includes a multiply-accumulate array 301 and a local weight cache 302.
[0074] The local weight cache 302 is used to cache network weights corresponding to multiple output channels simultaneously; the multiply-accumulate array 301 is used to obtain the convolution operation results corresponding to each output channel based on the same input activation window and the network weights corresponding to each output channel.
[0075] In some embodiments, the local weight cache 302 includes four cache channels, each storing sixteen 8-bit network weights. After the calculation of the current output channel is completed, the system can switch to another cache channel via a channel selection signal without having to reread the network weights from the weight memory 701.
[0076] like Figure 5 As shown, the processing unit cluster 30 receives the computational tasks and operation parameters sent by the scheduler 20. The main state machine 306 controls the loading of network weights, reading of activation data, multiplication and accumulation calculation, activation processing, and result output according to the task. The network weights are loaded into the local weight cache 302 through the weight memory interface 307, and the activation data enters the row cache 305 through the activation memory read interface 308 and forms a convolution sliding window. The local weight cache 302 and the row cache 305 provide network weights and activation data to the multiplication and accumulation array 301, respectively. The multiplication and accumulation array 301 generates convolution results or matrix projection results. After the results are processed by the activation circuit 303, they can be directly sent to the subsequent processing unit through the streaming output interface 304, or written to the activation memory 702 through the activation memory write interface 309 via the traditional write-back path. The main state machine 306 selects one of the two output methods according to the task configuration and feeds back the completion status to the scheduler 20 after completion.
[0077] In some embodiments, such as Figure 5As shown, the main state machine 306 can sequentially enter the idle, decoding, loading network weights, calculation, activation, streaming output, storage, and completion states. When streaming pass-through is enabled, the main state machine 306 controls the streaming output interface 304 to output the activation result after the activation process; when streaming pass-through is not enabled, the main state machine 306 controls the activation memory write interface 309 to complete the traditional write-back.
[0078] Figure 6 This diagram illustrates the relationship between the local weight cache 302 and the multiply-accumulate array 301 as described in this embodiment. Figure 6 As shown, the weight memory 701 loads network weights corresponding to multiple output channels into the local weight cache 302 via the network weight data bus. The local weight cache 302 stores the network weights for output channels 0 to 3, respectively. During calculation, the channel selection logic 310 reads the network weights from the corresponding cached channels according to the current output channel or parallel output configuration, and provides the network weights of each channel to the corresponding MAC row of the multiply-accumulate array 301. Each MAC row is multiplied and accumulated with the same input activation window to form the calculation result of the corresponding output channel. After the current channel calculation is completed, the channel selection logic 310 switches to the next resident channel without accessing the weight memory 701 again, thereby decoupling the network weight loading from the continuous multi-output channel calculation and increasing the on-chip reuse rate of the network weights.
[0079] In some embodiments, the number of cache channels can be configured to 2, 4, 8, or other numbers. As long as the local weight cache 302 can simultaneously maintain the network weights corresponding to at least two output channels and reuse the network weights in multiple computation stages, it belongs to the multi-output channel network weight residency mode.
[0080] In some embodiments, the same input activation window is maintained or broadcast to the multiply-accumulate array 301, and the convolution operation results of multiple output channels are obtained by sequentially selecting network weights in different cached channels. For example, a 4×4 two-dimensional multiply-accumulate array 301 is used, including sixteen multiply-accumulate units, with 8-bit integers for multiplication inputs and 32-bit integers for accumulation results. The array size and numerical format can be adjusted according to accuracy, area, and throughput requirements.
[0081] Please continue reading. Figure 2 As shown, the on-chip storage subsystem 70 includes a weight memory 701 for storing network weights and an activation memory 702 for storing activation data, calculation results, and state data. The state data may include cell states and hidden states of a long short-term memory network, and neuron states of a liquid neural network.
[0082] In some embodiments, the weighted memory 701 and the activation memory 702 each have a capacity of 256KB and are divided into two memory banks. The activation memory 702 adopts a dual-port structure with interleaved dual memory banks. The above-mentioned capacity can be adjusted according to chip area and throughput requirements.
[0083] Please continue reading. Figure 2 As shown, the weight encoding / decoding unit 80 includes an encoding pattern recognition unit 802 and a decoding unit 803. The encoding pattern recognition unit 802 is used to identify the encoding pattern of the network weights, and the decoding unit 803 is used to perform differential decoding, zero-value run-length decoding, or a hybrid decoding combining differential decoding and zero-value run-length decoding on the compressed network weights.
[0084] Among them, differential decoding recovers the network weights based on the difference between the baseline network weights and the weights of adjacent networks; zero-value run decoding outputs the corresponding number of zero values based on the zero-value flag and the number of consecutive zero values; hybrid decoding can first identify zero-value runs and then perform differential recovery on non-zero network weights.
[0085] In some embodiments, the differential encoding principle is to store the base value in the first byte. Each subsequent byte is encoded with two 4-bit differential values. The differential encoding principle and differential decoding are shown in equation (1).
[0086]
[0087]
[0088] Where i represents the position of the network weight in the decoded sequence, wᵢ represents the i-th network weight recovered by decoding, wᵢ₋1 represents the recovered previous network weight, Δᵢ represents the difference between the i-th network weight in the encoded data and the previous network weight, and sign_extend(·) represents extending the signed difference to the target network weight bit width. The significance of this formula is that the original network weights are reconstructed item by item using the differences between adjacent network weights; when the changes in adjacent network weights are small, Δᵢ can be represented with a smaller bit width, thereby reducing the storage capacity and transmission bandwidth of the network weights, and hardware decoding can be completed by a sign extension circuit and an adder.
[0089] Furthermore, the weight encoding / decoding unit 80 can also support the raw mode, allowing uncompressed network weights to pass directly, in order to be compatible with network weights that are not suitable for differential coding or zero-value run-length coding.
[0090] In some embodiments, the weight encoding / decoding unit 80 includes a prefetch buffer 801. The prefetch buffer 801 is used to pre-decode and cache the decoded network weights required for subsequent calculations during the computation of the processing unit cluster 30 using the current network weights. In some embodiments, eight prefetch channels may be set, each storing 16 8-bit network weights, and a valid flag may be set for each prefetch channel.
[0091] Furthermore, the prefetch cache 801 writes the decoded network weights into the weight memory 701; when the processing unit cluster 30 performs calculations, the weight memory 701 provides the corresponding network weights to the local weight cache 302 of each processing unit cluster 30. Thus, the prefetch cache 801 and the local weight cache 302 constitute a two-level cache structure that combines global prefetching and intra-cluster residency.
[0092] Figure 7 This diagram illustrates network weight encoding / decoding, error correction protection, and a two-level cache. (For example...) Figure 7 As shown, the off-chip non-volatile memory 130 provides compressed network weights to the weight encoding / decoding unit 80. The encoding pattern recognition unit 802 determines the encoding pattern, the decoding unit 803 performs the corresponding decoding, and writes the decoded network weights required for subsequent calculations into the prefetch cache 801. The prefetched network weights are written into the weight memory 701 after the error correction encoding circuit 704 in the single-bit error correction and double-bit error detection unit 703 generates verification information. When reading network weights from the weight memory 701, the error correction decoding circuit 705 completes single-bit error correction or double-bit error detection, and then provides the network weights to the local weight cache 302 of the processing unit cluster 30. The local weight cache 302 then continuously supplies network weights to the multiply-accumulate array 301. Thus, the off-chip reading, pattern recognition, decoding prefetching, error correction protection, intra-cluster residence, and multiply-accumulate supply of network weights form a continuous data path.
[0093] Please continue reading. Figure 7 As shown, the original network weights can be pre-encoded using differential encoding, zero-value run-length encoding, or a DIFF+RLE hybrid encoding, and stored in compressed form in off-chip non-volatile memory 130. During the computation of the processing unit cluster 30 using the current network weights, the weight encoding / decoding unit 80 pre-decodes the network weights required for subsequent computations and writes them into the prefetch cache 801, so that the decoding prefetch process at least partially overlaps with the current multiply-accumulate computation, thereby reducing the time for off-chip memory access and the time for the processing unit cluster 30 to wait for network weights.
[0094] like Figure 7As shown, in some embodiments, the weight memory 701 may also be connected to a single error correction and double error detection unit 703. For example, Hamming (72,64) encoding is used to generate 8-bit check information for each 64-bit network weight, including 7 Hamming parity bits and 1 global even parity bit. When writing to the weight memory 701, the error correction encoding circuit 704 generates check information based on the 64-bit network weight and stores the network weight and check information together.
[0095] When the weight memory 701 is read, the error correction decoding circuit 705 recalculates the check information based on the read data and compares it with the stored check information to obtain the error syndrome. When the error syndrome indicates a single-bit error, the error correction decoding circuit 705 flips the data bit or check bit corresponding to the error position; when the error syndrome indicates a double-bit error, the error correction decoding circuit 705 outputs an uncorrectable error flag. The corrected network weights are sent to the processing unit cluster 30, and the error status can be recorded by the scheduler 20 or output through the peripheral input / output subsystem 90.
[0096] Please continue reading. Figure 1 As shown, the streaming processing unit 40 is used to receive streaming arbitration and pooling mode control information sent by the scheduler 20, and under the control of the control information, to perform streaming arbitration and pooling processing on the calculation results corresponding to the convolutional neural network calculation.
[0097] In some embodiments, please refer to Figure 5 As shown, each processing unit cluster 30 includes an activation circuit 303 and a streaming output interface 304. The activation circuit 303 performs ReLU, Leaky ReLU, or other activation processing on the convolution operation result to obtain the activation result; the streaming output interface 304 transmits the activation result through data, valid, and end flags.
[0098] Figure 8 This diagram illustrates the sequential processing path of convolution, activation, and pooling as described in an embodiment of this application. Figure 8 As shown, in some embodiments, the streaming processing unit 40 includes a streaming arbitrator 401 and a pooling unit 402. The streaming output interfaces 304 of the multiple processing unit clusters 30 are respectively connected to different input paths of the streaming arbitrator 401. The streaming arbitrator 401 selects the activation result to be transmitted from the input path with valid activation results and sends it to the pooling unit 402. The pooling unit 402 performs max pooling, average pooling, or global average pooling according to the pooling mode and pooling window.
[0099] In some embodiments, the scheduler 20 is configured to form a convolution, activation, and pooling continuous processing path by the processing unit cluster 30, activation circuit 303, streaming output interface 304, streaming arbitrator 401, and pooling unit 402, so that the activation result obtained by the activation circuit 303 is transmitted to the pooling unit 402 via the streaming output interface 304 and streaming arbitrator 401 without being written to the activation memory 702, and the pooling unit 402 writes the pooling result to the activation memory 702.
[0100] like Figure 8 As shown, the scheduler 20 first configures the working parameters of the processing unit clusters 30 and the pooling unit 402, and establishes a continuous processing path for convolution, activation, and pooling. Multiple processing unit clusters 30 perform multiply-accumulate and activation processing respectively, generating multiple valid activation results. Each activation result is not written back to the activation memory 702 first, but is directly sent to the stream arbitrator 401 via the corresponding streaming output interface 304. The stream arbitrator 401 selects one of the multiple valid activation results according to the current scheduling mode and outputs it to the pooling unit 402. The pooling unit 402 completes pooling based on max pooling, average pooling, or global average pooling configuration, and writes the pooling result to the activation memory 702 for subsequent layers to read. This logic connects the convolution, activation, and pooling stages with a single streaming transfer, eliminating the intermediate write-back and re-reading of the activation result. In compatibility mode, streaming passthrough can be disabled, allowing the activation result to be written to the activation memory 702 before being read by the pooling unit 402.
[0101] Furthermore, in some embodiments, the stream arbitrator 401 has a strict priority scheduling mode and a weighted round-robin scheduling mode.
[0102] In strict priority scheduling mode, each input path can be configured with a high-priority flag or a low-priority flag. The flow arbitrator 401 first checks the input paths with high-priority flags and valid outputs in each service time slot; when multiple high-priority input paths are valid simultaneously, a round-robin approach is used among the high-priority input paths; only when none of the high-priority input paths have valid data is a low-priority input path polled.
[0103] In the weighted round-robin scheduling mode, the stream arbitrator 401 allocates a number of transmission time slots to each input path according to the scheduling weight corresponding to each processing unit cluster 30. For example, when the scheduling weights of the four processing unit clusters 30 are 1, 3, 1, and 1 respectively, the second processing unit cluster 30 obtains three transmission time slots in a complete scheduling cycle, and the other processing unit clusters 30 each obtain one transmission time slot.
[0104] Figure 9 The diagram shows strict priority scheduling and weighted round-robin scheduling. For example... Figure 9As shown, the stream arbitrator 401 receives valid flags, priority flags, scheduling weights, and activation data output by multiple processing unit clusters 30, and the configuration register 406 selects between strict priority scheduling mode and weighted round-robin scheduling mode. In strict priority scheduling mode, the priority scan logic 403 first checks input paths with high priority flags and valid outputs; when multiple high-priority input paths are valid simultaneously, the polling pointer 405 serves these input paths sequentially; only when all high-priority input paths have no valid data does it backtrack to low-priority input paths. In weighted round-robin scheduling mode, the service quota counter 404 records the service quota of each input path according to its scheduling weight, and advances the polling pointer 405 after the quota of the current input path is exhausted. The selected activation data is sent to the pooling unit 402, and the service quota and the polling pointer 405 are updated simultaneously.
[0105] Furthermore, in some embodiments, the configuration register 406 includes a scheduling mode field, a priority configuration field, and a scheduling weight configuration field. An external controller can write the corresponding configuration via the serial peripheral interface 901 to switch between strict priority scheduling mode and weighted round-robin scheduling mode.
[0106] Please continue reading. Figure 1 and Figure 2 As shown, the Long Short-Term Memory Network (LSTM) processing unit 50 is used to receive the gating fusion start information sent by the scheduler 20, and after receiving the gating fusion start information, performs gate activation and state fusion on the forget gate projection result, input gate projection result, output gate projection result and candidate cell state projection result output by the processing unit cluster 30.
[0107] Figure 10 The diagram shown illustrates the parallel projection and gating fusion of the Long Short-Term Memory (LSTM) network described in this embodiment. Figure 10 As shown, the scheduler 20 distributes the matrix projection calculations corresponding to the forget gate, input gate, output gate, and candidate cell state to multiple processing unit clusters 30, respectively, to obtain the forget gate projection results, input gate projection results, output gate projection results, and candidate cell state projection results in parallel, and sends gating fusion start information to the Long Short-Term Memory Network Processing Unit 50. The Long Short-Term Memory Network Processing Unit 50 includes multiple parallel gating calculation channels 501, which, after receiving the gating fusion start information, perform nonlinear processing and element-by-element state fusion on the four types of gate projection results to obtain the cell state and hidden state at the current time step.
[0108] like Figure 10As shown, the scheduler 20 distributes the four types of gate projection tasks—forget gate, input gate, output gate, and candidate cell state—in parallel to multiple processing unit clusters 30, and sends gating fusion start information to the Long Short-Term Memory (LSTM) network processing unit 50. The processing unit clusters 30 obtain the four types of gate projection results based on the current input and the hidden state of the previous time step, and transmit these results to the LTM network processing unit 50. Upon receiving the gating fusion start information and the four types of gate projection results, the LTM network processing unit 50 executes either the Sigmoid function or the hyperbolic tangent function through multiple parallel gating computation channels 501 to obtain the forget gate value, input gate value, output gate value, and candidate cell state value. These gate values and candidate cell state values enter the second pipeline stage and are fused element-wise with the cell state of the previous time step to obtain the current time step cell state and then the current time step hidden state. The current time step cell state and hidden state are written to the state storage location and used as feedback input in the next time step.
[0109] In some embodiments, the four types of matrix projections described above can be executed by different processing unit clusters 30, or the output dimension of each type of matrix projection can be split into multiple processing unit clusters 30. The four complete gating computation relationships of the Long Short-Term Memory Network are shown in equations (2) to (5), wherein the linear operation results within the parentheses in each equation are calculated by the processing unit cluster 30, and the Long Short-Term Memory Network processing unit 50 then performs corresponding nonlinear processing on the linear operation results.
[0110]
[0111] Where t represents the current time step, x t h represents the input vector at the current time step. t ₋1 represents the hidden state at the previous time step, [h t ₋1, x t ] represents the joint input vector formed by concatenating the two; Wf, Wi, Wo, and Wc represent the network weight matrices corresponding to the forget gate, input gate, output gate, and candidate cell state, respectively; bf, bi, bo, and bc represent the corresponding bias vectors, respectively; σ(·) represents the Sigmoid function, and tanh(·) represents the hyperbolic tangent function; f t i t o t and c̃ t These represent the forgetting gate, input gate, output gate, and candidate cell state value obtained after nonlinear processing, respectively. In equations (2) to (5), Wf·[h t ₋1, x t ]+bf、Wi·[h t₋1, x t ]+bi、Wo·[h t ₋1, x t ]+bo and Wc·[h t ₋1, x t ]+bc correspond to the four types of gate projection results output by the processing unit cluster 30 respectively; the long short-term memory network processing unit 50 performs nonlinear processing on the four types of gate projection results to obtain the corresponding gate values and candidate cell state values.
[0112] Furthermore, such as Figure 10 As shown, the Long Short-Term Memory (LSTM) network processing unit 50 includes a multi-path parallel gated computation channel 501 corresponding to the hidden layer dimension. Each gated computation channel performs a Sigmoid function or its approximation on the projection results of the forget gate, the input gate, and the output gate, and performs a hyperbolic tangent function or its approximation on the projection results of the candidate cell state, as shown in equations (6) and (7).
[0113]
[0114]
[0115] Where x represents the fixed-point input value to be processed nonlinearly, 128 represents the scaling factor corresponding to the fixed-point quantization format used, and 0.5 represents the output center of the Sigmoid function; clamp(y,a,b) means restricting the input y to between the lower limit a and the upper limit b. The first approximation restricts the output of the Sigmoid function to 0 to 1, and the second approximation restricts the output of the hyperbolic tangent function to -1 to 1.
[0116] Furthermore, the multi-parallel gated computing channel 501 obtains the cell state of the current time step in the second pipeline based on the forget gate value, the input gate value, the candidate cell state value and the cell state of the previous time step; and then obtains the hidden state of the current time step based on the output gate value and the cell state of the current time step, as shown in equations (8) and (9).
[0117]
[0118]
[0119] Among them, c t and c t ₋1 represents the cell state at the current time step and the previous time step, respectively, h t Indicates the hidden state at the current time step; f t i t and o t These represent the forget gate, input gate, and output gate, respectively. tThe candidate cell state value is represented by the symbol ⊙, which represents element-wise multiplication. Equation (8) retains the valid information in the previous cell state through the forget gate and writes the candidate cell state through the input gate; Equation (9) controls the output of the current cell state after hyperbolic tangent processing through the output gate. As shown in Equations (8) and (9), the multi-channel parallel gated computing channel 501 completes the retention of historical information, the writing of new information and the generation of the current output through element-wise operations, thereby establishing a time state recursive relationship that can be executed by the parallel gated computing channel 501.
[0120] In some embodiments, 128 parallel gated computation channels can be configured, employing three pipeline stages: activation, state fusion, and result output. The parallelism can also be configured to 32, 64, or other values; when the parallelism is less than the hidden layer dimension, the computation of all hidden units can be completed in multiple data blocks.
[0121] Please continue reading. Figure 1 and Figure 2 As shown, the liquid neural network processing unit 60 is used to receive the closed-loop state update start information sent by the scheduler 20, and after receiving the closed-loop state update start information, performs closed-loop neuron state update on the input projection result corresponding to the current input and the state projection result corresponding to the neuron state at the previous moment output by the processing unit cluster 30.
[0122] Figure 11 This diagram illustrates the closed-loop state update of the liquid neural network described in an embodiment of this application. Figure 11 As shown, the liquid neural network processing unit 60 includes an exponential decay factor lookup table 601 and a state update circuit 602. In a lookup table embodiment with a preset time interval of 1, the exponential decay factor lookup table 601 outputs a pre-stored exponential decay factor based on the time constant; the state update circuit 602 obtains the current neuron state based on the previous neuron state, the input projection result and state projection result obtained from the processing unit cluster 30, the bias, and the exponential decay factor, according to a closed-loop state update relationship.
[0123] like Figure 11 As shown, scheduler 20 first sends the input matrix projection task and the state matrix projection task to processing unit cluster 30, and sends closed-loop state update initiation information to liquid neural network processing unit 60. The current input x and the previous neuron state h are respectively projected by processing unit cluster 30 as input matrix Wx and state matrix Uh; processing unit cluster 30 transmits the two projection results to liquid neural network processing unit 60. After receiving the closed-loop state update initiation information and the two projection results, liquid neural network processing unit 60 adds the two projection results to the bias b and performs hyperbolic tangent processing to obtain the driving term B. Figure 11In the illustrated embodiment, the time interval Δt is preset to 1, and the exponential decay factor lookup table 601 outputs the exponential decay factor based on the time constant τ. The state update circuit 602 multiplies the previous neuron state h(t) by an exponential decay factor, multiplies the driving term B by a value obtained by subtracting the exponential decay factor, and then adds the two terms together to obtain the current neuron state h(t+1). The obtained current neuron state is written to the state storage location and used as the previous neuron state for the next time step.
[0124] In some embodiments, the scheduler 20 controls at least one cluster of processing units 30 to perform matrix projection operations based on the current input x and the input projection network weights W to obtain the input projection result Wx; and performs matrix projection operations based on the neuron state h and the state projection network weights U at the previous time step to obtain the state projection result Uh. The state evolution relationship of the liquid neural network is shown in equation (10).
[0125]
[0126] Where h=h(t) represents the neuron state vector at time t, dh / dt represents the rate of change of the neuron state with respect to time, τ represents the time constant vector corresponding to each neuron, and the symbol ⊙ represents element-wise multiplication; x=x(t) represents the current input vector, W represents the input projection network weight matrix, U represents the state projection network weight matrix, b represents the bias vector, and tanh(·) represents the hyperbolic tangent function.
[0127] Furthermore, the exponential decay factor lookup table 601 pre-stores the exponential decay factors when the time interval Δt is 1 and the time constant τ is 1 to 256. The lookup table 601 is used to find the exponential decay factor based on the time constant τ. In some embodiments, the exponential decay factor lookup table 601 uses a 256×16-bit read-only memory, and the exponential decay factor uses a Q8.8 fixed-point format.
[0128] Furthermore, the state update circuit 602 sums and performs nonlinear processing on the input projection result, the state projection result, and the bias to obtain the driving term B, as shown in equation (11):
[0129] B = tanh(Wx + Uh + b)
[0130] Where B represents the nonlinear driving term in the current update interval, x represents the current input vector, h represents the neuron state vector at the previous time step, W and U represent the input projection network weight matrix and the state projection network weight matrix, respectively, b represents the bias vector, and tanh(·) represents the hyperbolic tangent function.
[0131] Furthermore, the state update circuit 602 multiplies the previous neuron state by an exponential decay factor to obtain a first calculation result, multiplies the driving term by a value obtained by subtracting the exponential decay factor to obtain a second calculation result, and then adds the first calculation result and the second calculation result to obtain the current neuron state, as shown in equation (12):
[0132]
[0133] Where t represents the previous sampling time, Δt represents the time interval between two adjacent state updates, and τ represents the time constant or time constant vector. Δt represents the exponential decay factor within the time interval Δt; h(t) and h(t+Δt) represent the neuron states before and after the update, respectively; and B represents the nonlinear driving term of the current update interval.
[0134] Therefore, in the lookup table embodiment with a time interval Δt of 1, the runtime state update mainly includes one read-only memory lookup, two multiplications, and one addition, without the need for real-time division and exponentiation operations. Thus, the liquid neural network processing unit 60 eliminates the divider and exponentiation unit, reducing the state update per neuron from approximately 16 cycles to 1 cycle, thus minimizing memory usage. This closed-form state update relationship is unconditionally stable and allows for arbitrarily large step sizes. Instead of diverging.
[0135] Please continue reading. Figure 2 As shown, the multimodal inference chip 1 also includes an online anomaly detection unit 110. The online anomaly detection unit 110 receives the inference output results and calculates the mean and variance of the inference output results in real time using a 64-sample sliding window. The variance is accumulated as each sample arrives. and When the window is full, calculate equations (13) and (14):
[0136]
[0137]
[0138] Where x represents the x-th sample in the 64-sample window, μ represents the sample mean in the window, σ² represents the sample variance in the window, Σx represents the sum of sample values in the window, and Σx² represents the sum of sample squares in the window. Equation (13) is used to obtain the mean based on the accumulated sample values, and Equation (14) is used to obtain the variance based on the accumulated sample squares and the square of the mean. Therefore, it is not necessary to save all samples in the window.
[0139] Furthermore, the online anomaly detection unit 110 is configured with a mean threshold and a variance threshold. When the mean or variance of the 64-sample window exceeds the corresponding threshold, the online anomaly detection unit 110 outputs an anomaly interrupt signal, which is then output to the outside via the general-purpose input / output control unit 902. The mean threshold and variance threshold can be set via a configuration register.
[0140] Please continue reading. Figure 2 As shown, the peripheral and input / output subsystem 90 may include a serial peripheral interface 901, a general-purpose input / output control unit 902, a four-wire serial peripheral interface 903, and a startup loading unit 904. The serial peripheral interface 901 is used to receive instructions and configuration parameters and output chip status; the general-purpose input / output control unit 902 is used to connect sensors, external interrupts, and exception interrupt signals; the four-wire serial peripheral interface 903 is used to read the inference program and compressed network weights from the off-chip non-volatile memory 130; and the startup loading unit 904 is used to trigger the autonomous loading of the program and network weights after the chip is powered on.
[0141] In some embodiments, the general purpose input / output control unit 902 includes 16 configurable input / output ports and is used to output abnormal interrupt signals generated by the online abnormality detection unit 110.
[0142] Please continue reading. Figure 2 As shown, in some embodiments, the power management unit 100 includes a low-dropout linear regulator circuit 1001 for providing corresponding operating voltages to the computing, storage, and input / output sections, and for implementing multi-power domain management and deep sleep control.
[0143] In some embodiments, the scheduler 20 generates clock enable signals corresponding to each processing unit cluster 30, streaming processing unit 40, long short-term memory network processing unit 50, and liquid neural network processing unit 60. Each unit outputs an idle indicator in the idle state, and the scheduler 20, combining the target execution unit of the current instruction with the idle indicator, shuts down the clock of the execution unit not used by the current instruction.
[0144] Furthermore, when subsequent instructions require the use of execution units that are in a clock-off state, scheduler 20 first enables the corresponding clock. For processing unit clusters 30, scheduler 20 then sends convolution operation tasks, long short-term memory network gate projection tasks, or liquid neural network matrix projection tasks. For streaming processing units 40, long short-term memory network processing units 50, and liquid neural network processing units 60, scheduler 20 then sends the corresponding start control information. Each processing unit cluster 30 can be clock-gated independently; streaming processing units 40, long short-term memory network processing units 50, and liquid neural network processing units 60 can also be clock-gated separately. For configuration registers and state storage that need to be retained, they can be set in an ungated clock domain or used as holding registers.
[0145] Comprehensive reference Figures 1 to 11 After the chip is powered on, the instruction input unit 10 receives the instructions from the inference program, and the scheduler 20 completes the instruction decoding and determines the calculation type corresponding to the current instruction. The on-chip storage subsystem 70 provides network weights and activation data to the processing unit cluster 30; when the network weights are stored in compressed form, the weight encoding and decoding unit 80 first decodes the network weights and uses the prefetch cache 801 to prepare the decoded network weights required for subsequent calculations in advance.
[0146] The processing unit cluster 30 performs convolution or matrix projection operations according to the computation task and outputs the computation results. The scheduler 20 also sends start control information to the corresponding post-processing units. For convolutional neural network computation, the computation results are activated and then enter the streaming processing unit 40, which has received the stream arbitration and pooling mode control information. The stream arbitrator 401 selects the data stream and the pooling unit 402 completes the pooling process. For long short-term memory network computation, the four types of gate projection results enter the long short-term memory network processing unit 50, which has received the gate fusion start information, to complete the gate activation and state fusion. For liquid neural network computation, the input projection results and state projection results enter the liquid neural network processing unit 60, which has received the closed state update start information, and the neuron state update at the current time is completed by combining the exponential decay factor.
[0147] After completing their respective processing, each unit returns a completion signal to the scheduler 20. The scheduler 20 then acquires and distributes subsequent instructions based on the completion signal, enabling network weight prefetching, processing unit cluster 30 computation, and subsequent processing to overlap while satisfying data dependencies. Thus, the instruction control flow, network weight supply flow, activation data flow, and state feedback flow form a coordinated relationship within the same chip.
[0148] Figure 12 This diagram illustrates a flowchart of an industrial fault diagnosis method according to an embodiment of this application. Figure 12As shown, the industrial fault diagnosis method described in this application includes steps S10 to S30.
[0149] S10: Acquire operating data of industrial equipment.
[0150] In some embodiments, the operating data may be acquired by vibration sensors, acoustic sensors, current sensors, temperature sensors, pressure sensors, or combinations thereof, and may be subjected to sampling alignment, denoising, normalization, framing, spectral transformation, or time-frequency transformation.
[0151] S20: The running data is input as activation data into the multimodal inference chip, and the inference result is obtained using the multimodal inference chip.
[0152] S30, determine the fault status of industrial equipment based on the reasoning results.
[0153] In step S20, Figure 13 A schematic diagram illustrating the process of obtaining inference results as described in an embodiment of this application is shown. Figure 13 As shown, obtaining the reasoning result includes steps S201 to S204.
[0154] S201 receives an instruction containing an opcode and operation parameters.
[0155] S202, Parse instructions, determine the type of computation to be executed from convolutional neural network computation, long short-term memory network computation and liquid neural network computation according to the opcode, and form a computation task and corresponding start control information based on the computation type and operation parameters.
[0156] S203, the computation task is distributed to at least one cluster of processing units, and the start control information is sent to the streaming processing unit, long short-term memory network processing unit or liquid neural network processing unit corresponding to the computation type. The processing unit cluster performs convolution operation or matrix projection operation based on network weights and activation data to obtain the computation result.
[0157] S204, depending on the calculation type, transmit the calculation result to the streaming processing unit, long short-term memory network processing unit, or liquid neural network processing unit that has received the start control information, so as to perform streaming arbitration and pooling processing, gate activation and state fusion, or neuron state update to obtain the inference result.
[0158] Furthermore, in step S20, the multimodal inference chip also includes an activation memory for storing activation data and calculation results. When the calculation type is a convolutional neural network calculation, obtaining the inference result further includes: performing convolution operations through the processing unit clusters and performing activation processing on the convolution operation results; transmitting the obtained activation results to the streaming processing unit without writing them to the activation memory; and through the streaming processing unit, selecting the processing unit cluster with the highest priority and valid activation results based on the priority corresponding to each processing unit cluster, or selecting each processing unit cluster based on the relative service quota corresponding to the scheduling weights, and performing pooling processing on the activation results output by the selected processing unit clusters.
[0159] It should be noted that the structure and inference process of the multimodal inference chip in step S20 can be referred to the content described in the above embodiments, and will not be repeated here.
[0160] In step S30, the fault state of the industrial equipment is determined based on the inference result. In some embodiments, the inference result can be compared with a preset judgment rule, classification threshold, or regression threshold to determine a normal state, a warning state, a fault state, or a state corresponding to a specific component and fault type. The inference result can be a fault category, fault probability, health status, etc.
[0161] In some embodiments, vibration sensors, acoustic sensors, current sensors, temperature sensors, and / or pressure sensors acquire operational data from industrial equipment. The acquired data can be written to the activation memory via a serial peripheral interface or a general-purpose input / output control unit. Convolutional neural network computations perform local feature extraction on the original waveform, spectrum, or time-frequency plot; convolution operations, activation processing, and pooling processing are completed through a continuous processing path.
[0162] The feature sequence output by the convolutional neural network can be used as the input to the current time step of the long short-term memory network. The scheduler distributes four types of gate projection tasks to multiple clusters of processing units, and the long short-term memory network processing units obtain the cell state and hidden state at the current time step to identify the degradation trend accumulated over time. For data with irregular sampling time intervals or continuously changing equipment conditions, the liquid neural network processing units update the neuron state according to the actual time interval and time constant to characterize the continuous dynamic process.
[0163] Furthermore, the online anomaly detection unit performs online mean and variance calculations on classification scores, regression results, hidden states, or neuron states. When the statistics exceed a threshold, an anomaly interruption signal is generated, which the external controller can use to trigger alarms, load reduction, shutdown for inspection, or save diagnostic data. It should be noted that the above embodiments only illustrate one application scenario of the three calculation types in industrial fault diagnosis; this application does not require that all three network calculations be executed sequentially for every inference.
[0164] In summary, this application enables convolutional neural network computation, long short-term memory network computation, and liquid neural network computation to be executed collaboratively within the same chip through unified instructions, reusable processing unit clusters, and subsequent processing paths corresponding to the computation type. Furthermore, it reduces network weight and intermediate data transfer overhead through network weight residency, fusion passthrough, and streaming priority scheduling.
[0165] First, the scheduler determines the computation type based on unified instructions and distributes the convolution or matrix projection operations common to the three types of networks to reusable processing unit clusters, reducing the area and power consumption overhead caused by repeatedly configuring basic computing arrays for different network types.
[0166] Secondly, the network weight encoding and decoding, prefetching cache, local weight cache, and continuous processing paths of convolution, activation, and pooling work together to reduce the storage bandwidth and dynamic power consumption overhead caused by repeated loading of network weights, intermediate write-back of activation results, and rereading.
[0167] Finally, the Long Short-Term Memory Network (LSTM) processing unit and the Liquid Neural Network (LNN) processing unit respectively complete gate activation and state fusion, and closed neuron state update, enabling the chip to select the appropriate computation path for the discrete-time characteristics and continuous-time dynamic characteristics of industrial equipment, thereby improving the adaptability of industrial fault diagnosis deployment.
[0168] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A multimodal inference chip, characterized in that, include: The instruction input unit is used to receive instructions containing opcodes and operation parameters; A scheduler is used to parse the instruction, determine the type of computation to be executed based on the opcode, and form a computation task and corresponding start control information based on the computation type and the operation parameters; the computation type includes convolutional neural network computation, long short-term memory network computation, and liquid neural network computation; Multiple processing unit clusters are used to perform convolution operations or matrix projection operations based on network weights and activation data to obtain calculation results corresponding to the calculation type, and to transmit the calculation results to the corresponding streaming processing unit, long short-term memory network processing unit or liquid neural network processing unit based on the calculation type. The streaming processing unit is configured to perform streaming arbitration and pooling processing on the calculation result corresponding to the calculation of the convolutional neural network after receiving the streaming arbitration and pooling mode control information sent by the scheduler. The Long Short-Term Memory (LSTM) network processing unit is configured to perform gate activation and state fusion on the calculation result corresponding to the LSM network calculation after receiving the gate fusion start information sent by the scheduler. as well as The liquid neural network processing unit is used to perform neuron state update on the calculation result corresponding to the liquid neural network calculation after receiving the closed state update start information sent by the scheduler. The scheduler is further configured to distribute the computing tasks to at least one of the processing unit clusters, and to send the start control information to the streaming processing unit, the long short-term memory network processing unit, or the liquid neural network processing unit corresponding to the computing type.
2. The multimodal inference chip according to claim 1, characterized in that, The instruction is a 64-bit fixed-length instruction, which includes an opcode field, a flag field, and multiple parameter fields. The scheduler includes an instruction decoding unit, an instruction distribution unit, and a completion signal detection unit; The instruction decoding unit is used to form the computation task and the start control information based on the opcode field, the flag field and the plurality of parameter fields; The instruction distribution unit is used to send the computing task to at least one of the processing unit clusters, and to send the startup control information to the streaming processing unit, the long short-term memory network processing unit or the liquid neural network processing unit corresponding to the computing type. The completion signal detection unit is used to control the acquisition and distribution of subsequent instructions based on the completion signal returned by the unit that receives the computation task or the start control information; wherein, the computation task includes a convolution operation task, a long short-term memory network gate projection task, or a liquid neural network matrix projection task.
3. The multimodal inference chip according to claim 1, characterized in that, It also includes an on-chip storage subsystem and a weighted encoding / decoding unit; The on-chip storage subsystem includes a weight memory for storing the network weights and an activation memory for storing the activation data, the calculation results, and the state data. The weight encoding / decoding unit is used to perform differential decoding, zero-value run-length decoding, or a hybrid decoding combining differential decoding and zero-value run-length decoding on the compressed network weights. The weight encoding / decoding unit includes a prefetch cache, which is used to cache the decoded network weights required for subsequent calculations during the calculation of the current network weights by the processing unit cluster.
4. The multimodal inference chip according to claim 3, characterized in that, Each of the aforementioned processing unit clusters includes a multiply-accumulate array and a local weight cache; The local weight cache is used to simultaneously cache network weights corresponding to multiple output channels; The multiply-accumulate array is used to obtain the convolution operation results corresponding to each output channel based on the same input activation window and the network weights corresponding to each output channel.
5. The multimodal inference chip according to claim 3, characterized in that, Each of the processing unit clusters further includes an activation circuit and a streaming output interface; the streaming processing unit includes a streaming arbitrator and a pooling unit; The scheduler is configured to form a continuous processing path of convolution, activation, and pooling among the processing unit cluster, the activation circuit, the streaming output interface, the streaming arbitrator, and the pooling unit, so that the activation result obtained by the activation circuit is transmitted to the pooling unit via the streaming output interface and the streaming arbitrator without being written to the activation memory, and the pooling unit writes the pooling result to the activation memory.
6. The multimodal inference chip according to claim 5, characterized in that, The stream arbitrator has a strict priority scheduling mode and a weighted round-robin scheduling mode; In the strict priority scheduling mode, the stream arbitrator is used to select the highest priority processing unit cluster from the processing unit clusters with valid activation results; In the weighted round-robin scheduling mode, the stream arbitrator is used to select each processing unit cluster based on the scheduling weight corresponding to each processing unit cluster and the relative service quota corresponding to the scheduling weight.
7. The multimodal inference chip according to claim 1, characterized in that, The scheduler is used to distribute the matrix projection calculations corresponding to the forget gate, input gate, output gate and candidate cell state to multiple processing unit clusters, so as to obtain the forget gate projection results, input gate projection results, output gate projection results and candidate cell state projection results in parallel, and send the gated fusion start information to the long short-term memory network processing unit. The long short-term memory network processing unit includes multiple parallel gated computing channels for performing nonlinear processing and element-wise state fusion on the forget gate projection results, the input gate projection results, the output gate projection results, and the candidate cell state projection results to obtain the cell state and hidden state at the current time step.
8. The multimodal inference chip according to claim 1, characterized in that, The liquid neural network processing unit includes an exponential decay factor lookup table and a state update circuit; The exponential decay factor lookup table is used to output the pre-stored exponential decay factor according to the time constant when the preset time interval is 1. The state update circuit is used to obtain the current neuron state based on the closed state update relationship after receiving the closed state update start information sent by the scheduler, according to the neuron state at the previous moment, the input projection result and state projection result obtained by the processing unit cluster, the bias, and the exponential decay factor.
9. An industrial fault diagnosis method, characterized in that, include: Acquire operational data from industrial equipment; The running data is input as activation data into the multimodal inference chip according to any one of claims 1 to 8, and the inference result is obtained using the multimodal inference chip; The fault state of the industrial equipment is determined based on the reasoning result; wherein, obtaining the reasoning result includes: Receive instructions containing opcodes and operation parameters; The instruction is parsed, and the type of computation to be executed is determined from convolutional neural network computation, long short-term memory network computation, and liquid neural network computation based on the opcode. A computation task and startup control information corresponding to the computation type are formed based on the computation type and the operation parameters. The computation task is distributed to at least one of the processing unit clusters, and the start control information is sent to the streaming processing unit, the long short-term memory network processing unit, or the liquid neural network processing unit corresponding to the computation type. The processing unit clusters then perform convolution operations or matrix projection operations based on network weights and activation data to obtain the computation results. According to the calculation type, the calculation result is transmitted to the streaming processing unit, the long short-term memory network processing unit, or the liquid neural network processing unit that has received the start-up control information, so as to perform streaming arbitration and pooling processing, gate activation and state fusion, or neuron state update to obtain the inference result.
10. The industrial fault diagnosis method according to claim 9, characterized in that, The multimodal inference chip also includes an activation memory for storing the activation data and the calculation results; When the computation type is the convolutional neural network computation, obtaining the inference result further includes: The processing unit cluster performs convolution operations and performs activation processing on the convolution operation results, and transmits the obtained activation results to the streaming processing unit without writing them to the activation memory. The streaming processing unit selects the processing unit cluster with the highest priority and effective activation result based on the priority corresponding to each processing unit cluster, or selects each processing unit cluster based on the relative service quota corresponding to the scheduling weight, and performs pooling processing on the activation result output by the selected processing unit cluster.
