A weight switching acceleration architecture and method suitable for streaming matrix operation

Abstract

The application provides a weight switching acceleration architecture and method suitable for stream matrix operation, the architecture comprising: a parameter loading control module configured to load parameter sets from an off-chip memory according to parameter set identifiers; a parameter storage module comprising a plurality of ping-pong buffers, each ping-pong buffer being connected to a corresponding calculation unit and having at least two logical buffers, each logical buffer storing a parameter slice; a parameter set identifier delivery module configured to obtain parameter set identifiers corresponding to input data streams and provide the parameter set identifiers to the ping-pong buffers to enable selection of matching logical buffers as data sources; and a stream calculation module comprising one or more calculation layers arranged in series and configured to receive the input data streams and perform matrix operation. While the current logical buffer supplies data to the calculation unit, the parameter loading control module loads the next parameter set into another logical buffer in the same ping-pong buffer; when the parameter set identifier is switched, the ping-pong buffer switches the data source to the other logical buffer.

Description

Technical Field

[0001] This invention belongs to the fields of computer hardware and artificial intelligence, and in particular to a weight switching acceleration architecture and method suitable for streaming matrix operations. Background Technology

[0002] Matrix multiplication and matrix multiplication accumulation are crucial operators in neural networks, scientific computing, and graph computing. Current hardware accelerators primarily employ two architectures: one is the general-purpose parallel architecture represented by GPUs, which relies on a large number of computing cores and flexible scheduling mechanisms to accelerate matrix computation; the other is the systolic array architecture represented by TPUs, which improves the throughput of matrix multiplication operations by normalizing the data flow. While these approaches perform well in typical high-volume matrix computation scenarios, they still exhibit significant limitations in scenarios with no batch or mini-batch computations, vector-level streaming input, large aspect ratios of matrices, or tight dependencies between multiple computation layers.

[0003] In streaming matrix multiplication scenarios, data often arrives continuously in the form of row vectors, column vectors, or small vector blocks. If a large-scale two-dimensional systolic array is still used directly, it usually requires a long injection and emptying time, resulting in many idle cycles of computing resources during the input and output phases, and a decrease in overall resource utilization. At the same time, when the current layer has not yet completed the computation of the entire matrix, the next layer is in an idle state, which affects inter-layer parallelism.

[0004] On the other hand, many practical tasks do not always use the same set of weight parameters. For example, in scenarios such as multi-type object processing, hybrid expert networks, scientific intelligent computing, switchable layer configuration, and segmented task computing, the computation process requires switching between multiple sets of parameters based on a certain parameter set identifier. If all parameter sets are kept resident in on-chip memory at the same time, the on-chip cache capacity will expand rapidly with the increase in the number of parameter sets; if they are instead read from off-chip memory during switching, the off-chip memory access latency is much greater than the on-chip memory access latency, which will disrupt the continuity of streaming computing and thus reduce computational efficiency.

[0005] Existing solutions typically focus more on streaming data supply under fixed weights or on reusing parameters for extended periods after a single offline load, offering insufficient support for scenarios where parameter sets need to be continuously switched during computation. Especially when each computation unit requires stable and continuous access to parameter data, a critical challenge remains: how to enable simultaneous computation of the current parameter set and loading of the next parameter set within limited on-chip storage resources, thereby concealing weight replacement latency. Summary of the Invention

[0006] To address the problems in existing technologies, such as matrix multiplication calculation pauses caused by parameter switching, excessive on-chip parameter storage overhead, and difficulty in continuous inter-layer pipeline operation, this invention provides a weight switching acceleration architecture and method suitable for streaming matrix operations.

[0007] The purpose of this invention is to enable the calculation process of the current parameter set and the loading process of the next parameter set to be executed in parallel, while maintaining vector granular input and inter-layer pipelined execution mode. This is achieved by setting a ping-pong cache on the parameter supply path, setting parameter loading control logic at the top level, passing parameter set identifiers between layers, and setting a pre-loading and calculation alternating state machine inside the cache. This allows the continuous switching of multiple parameter sets to be supported with a small on-chip storage overhead.

[0008] To achieve the above objectives, a first aspect of the present invention provides a weight switching acceleration architecture suitable for streaming matrix operations, comprising: a parameter loading control module, configured to load a parameter set from off-chip memory according to a parameter set identifier; a parameter storage module, comprising ping-pong cache layers configured for each computation layer, each ping-pong cache layer including multiple ping-pong caches, each ping-pong cache being connected to a corresponding computation unit and having at least two logical buffers, each logical buffer being used to store corresponding parameter fragments after the parameter set has been split; and a parameter set identifier transmission module, configured to obtain the parameter set identifier corresponding to the input data stream and provide the parameter set identifier to the ping-pong cache, so that the ping-pong cache selects a parameter set identifier that matches the parameter set identifier. The logical buffer serves as the data source; and the streaming computing module includes one or more computing layers connected in series, each computing layer including multiple cascaded computing units, the computing layer being used to receive input data streams and perform matrix operations through the multiple computing units, each computing unit being connected to a corresponding ping-pong buffer to obtain parameters from the data source; wherein, while the logical buffer corresponding to the current parameter set supplies data to the corresponding computing unit, the parameter loading control module loads the next parameter set into another logical buffer in the same ping-pong buffer; when the parameter set identifier is switched to the next parameter set identifier, the ping-pong buffer switches the data source to the other logical buffer that has loaded the next parameter set.

[0009] The parameter set identifier transmission module obtains the parameter set identifier through the sideband control path, and the sideband control path is set in parallel with the main data path of the input data stream.

[0010] The parameter set identifier transmission module is used to maintain a correspondence between the parameter set identifier corresponding to each data transaction and the input data of the data transaction or the intermediate calculation result generated by the input data, and to transmit it along the multiple serially connected computing layers. When the data transaction arrives at any computing layer, the parameter set identifier is provided to the ping-pong cache corresponding to the corresponding computing unit in that computing layer, so that the ping-pong cache selects a logical buffer storing parameter fragments that match the parameter set identifier as the data source for the data transaction. Among them, data transactions corresponding to different parameter set identifiers can be located in different computing layers at the same time, and each ping-pong cache independently selects the data source according to the parameter set identifier corresponding to the data transaction arriving at the corresponding computing layer.

[0011] The logical buffer is equipped with a state machine, which includes at least an idle state, a preloaded state, a ready state, and a computation state. The idle state indicates that the logical buffer is free and available for allocation. The preloaded state indicates that the logical buffer is receiving parameters for loading or waiting for writing to complete. The ready state indicates that the logical buffer has completed loading and is waiting to participate in data supply. The computation state indicates that the logical buffer is outputting parameters to the corresponding computation unit.

[0012] When a data transaction with a different parameter set identifier than the parameter fragment stored in the logical buffer currently in the computation state arrives at the corresponding computation layer, and another logical buffer in the preload state has already entered the ready state, the other logical buffer takes over the data supply and enters the computation state, while the original logical buffer that supplied the data exits the computation state.

[0013] The streaming computing module also includes a selection unit, which is connected to multiple computing units and is used to select and forward a portion of the computing results generated by the previous computing layer to the next computing layer without waiting for the entire layer to complete the computing.

[0014] The ping-pong buffer is implemented using a dual-port memory. The read port outputs parameters to the corresponding computing unit, and the write port writes parameters to the parameter loading control module. The at least two logical buffers maintain independent read pointers and write pointers respectively.

[0015] The parameter set includes weight parameters and auxiliary parameters, and the auxiliary parameters include at least one of bias parameters, scaling parameters, lookup table parameters, or gating parameters.

[0016] The parameter loading control module splits the data from the off-chip memory and sends it to multiple ping-pong caches within the same computing layer.

[0017] The parameter set identifier is used to represent at least one of the following: object type, task number, expert number, model version, layer configuration, or operator instance.

[0018] A second aspect of the present invention provides a streaming matrix operation accelerator, including the weight switching acceleration architecture described above.

[0019] A third aspect of the present invention also provides a weight switching acceleration method suitable for streaming matrix operations, employing the aforementioned weight switching acceleration architecture, comprising the following steps: a parameter loading control module loads a parameter set from off-chip memory into a ping-pong cache layer according to a parameter set identifier, each ping-pong cache corresponding to a computing unit, and each ping-pong cache having at least two logical buffers; a parameter set identifier transmission module obtains the parameter set identifier corresponding to the input data stream and provides the parameter set identifier to the ping-pong cache; the ping-pong cache selects a matching logical buffer as a data source according to the parameter set identifier and outputs parameters to the corresponding computing unit; the computing layer receives the input data stream and performs matrix operations through multiple cascaded computing units, each computing unit obtaining parameters from the corresponding ping-pong cache; wherein, while the logical buffer corresponding to the current parameter set is supplying data to the corresponding computing unit, the parameter loading control module loads the next parameter set into another logical buffer in the same ping-pong cache; when the parameter set identifier is switched to the next parameter set identifier, the ping-pong cache switches the data source to the other logical buffer that has already loaded the next parameter set.

[0020] As can be seen from the above technical solutions, the advantages of the present invention are:

[0021] This invention configures a ping-pong cache for each computing unit and sets up at least two logical buffers within the cache. While the current parameter set is being used for calculation, the parameter loading control module preloads the next parameter set into another logical buffer within the same ping-pong cache. When the parameter set identifier changes, the preloaded logical buffer seamlessly takes over the data supply. Therefore, only the current and next parameter sets need to be stored on-chip, reducing storage resource requirements; parameter switching does not require waiting for off-chip memory access, eliminating pipeline disruption caused by memory access latency; and combined with the inter-layer pipeline structure, it ensures continuous execution of matrix operations. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the weight switching acceleration architecture provided in an embodiment of the present invention;

[0023] Figure 2 for Figure 1 A schematic diagram of the structure of the intermediate parameter storage module and the streaming computing module;

[0024] Figure 3 for Figure 2 A schematic diagram of the internal structure of the Ping Pong buffer (I);

[0025] Figure 4 for Figure 2 Schematic diagram of the internal structure of the Ping Pong buffer (II);

[0026] Figure 5 A flowchart for passing parameter set identifiers between ping-pong buffers;

[0027] Figure 6 A flowchart of a weight switching acceleration method provided in another embodiment of the present invention;

[0028] In the attached figures, the following labels are used:

[0029] 1- Weight switching acceleration architecture;

[0030] 10-Parameter loading control module;

[0031] 11-Parameter storage module;

[0032] 110-Ping-Pong Cache Layer;

[0033] 111-Ping Pong Cache;

[0034] 112 - First logical buffer;

[0035] 113 - Second logical buffer;

[0036] 114 - Selector;

[0037] 115 - State Machine;

[0038] 12-Parameter Set Identifier Transmission Module;

[0039] 13-Streaming Computing Module;

[0040] 130 - Computational Layer;

[0041] 131 - Computational Unit;

[0042] 132 - Select Unit;

[0043] 133 - Auxiliary arithmetic unit;

[0044] 2-Off-chip memory;

[0045] 3- Weight switching acceleration methods;

[0046] Steps S1~S4. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0048] In streaming matrix multiplication or neural network computation, different data segments, task types, model configurations, or object categories correspond to different parameter sets. However, the on-chip storage capacity of computing chips is limited, necessitating frequent on-chip parameter replacements. This process can easily disrupt the continuity of the data pipeline, thus impacting computational efficiency. This invention addresses this by adding multiple ping-pong buffer groups to the parameter supply side of a vector-granular input and inter-layer pipelined matrix multiplication computation architecture. Each ping-pong buffer includes at least two logical buffers. Through a coordinated scheduling parameter loading control module, parameter set identifier transmission module, and pre-loading-computation alternation state machine, parallel execution of the current parameter set participating in computation and the next parameter set being pre-loaded in the background is achieved. When a parameter set switch is detected, the pre-loaded logical buffers seamlessly take over the data supply, ensuring uninterrupted and low-latency continuous execution of the computation process. This invention reduces parameter storage resource requirements, hides parameter replacement overhead, and achieves high-throughput continuous execution of streaming matrix multiplication. It can be applied to fully connected layers, matrix multiplication after convolutional transformations, linear transformations in graph computation, hybrid expert networks, and other scenarios requiring weight parameter switching.

[0049] like Figure 1 The diagram shown is a structural schematic of a weight switching acceleration architecture 1 suitable for streaming matrix operations according to an embodiment of the present invention. The weight switching acceleration architecture 1 includes: a parameter loading control module 10, a parameter storage module 11, a parameter set identifier transmission module 12, and a streaming computation module 13.

[0050] like Figure 2 As shown, the streaming computing module 13 includes one or more computing layers 130 connected in series, and each computing layer 130 includes multiple cascaded computing units 131. The computing layer 130 receives the input data stream at the vector granularity and performs matrix operations, such as vector-matrix multiplication and accumulation, through the multiple computing units 131. Each computing unit 131 is connected to the ping-pong buffer 111 corresponding to the parameter storage module 11 to obtain parameters from the data source.

[0051] The streaming computing module 13 also includes a selection unit 132, which is connected to multiple computing units 131 and is used to select and forward some of the calculation results generated by multiple computing units 131 of the previous computing layer 130 to the next computing layer 130 without waiting for the entire layer to complete the calculation.

[0052] Specifically, after completing the current calculation, the current calculation unit 131 forwards the data to the next level calculation unit 131. Once the current calculation layer 130 produces a partial result that can be used by the subsequent calculation layer 130, it is sent to the next calculation layer 130 through the selection unit 132, without having to wait for the entire layer to finish outputting, thus forming an inter-layer pipeline.

[0053] In an optional embodiment, an auxiliary operation unit 133 (such as an addition, activation, or any other calculation unit) is further provided between the selection unit 132 and the next calculation layer 130. The auxiliary operation unit 133 is used to perform auxiliary operations on some calculation results. The auxiliary operations include at least one of bias addition, scaling, activation function operation, or multiplication and addition operation. The result processed by the auxiliary operation unit is then sent to the next calculation layer 130 to participate in the matrix operation of the next calculation layer 130.

[0054] To accommodate differences in matrix dimensions, aspect ratios, and operator stages, each computation layer 130 can be flexibly configured as an input layer, intermediate layer, or output layer. In different application scenarios, the number of computation units 131, the number of weight columns each unit is responsible for, the partial and cumulative depths, and whether auxiliary computational components such as bias addition, scaling, activation, or multiplication-addition are configured can all be adjusted to match the computational pace of each computation layer 130, preventing any single layer from becoming a pipeline bottleneck.

[0055] The parameter storage module 11 includes ping-pong cache layers 110 configured for each computing layer 130. Each ping-pong cache layer 110 includes multiple ping-pong caches 111. Each ping-pong cache 111 is connected to the corresponding computing unit 131 and has at least two logical buffers. Each logical buffer is used to store the parameter set loaded by the parameter loading control module. While the logical buffer corresponding to the current parameter set supplies data to the corresponding computing unit 131, the parameter loading control module 10 loads the next parameter set into another logical buffer within the same ping-pong cache. When the parameter set identifier is switched to the next parameter set identifier, the ping-pong cache 111 switches its data source to the other logical buffer that has already loaded the next parameter set.

[0056] In this embodiment, a complete parameter set (e.g., all weight parameters of a layer of network) is divided into multiple parameter fragments, which are stored in multiple ping-pong caches 111 corresponding to the same computation layer 130. Each ping-pong cache 111 stores one parameter fragment. Each computation unit 131 retrieves its required parameter fragment from the corresponding ping-pong cache 111 and collaboratively completes all matrix operations for that layer. At the logical functional level, each fragment can still be considered as part of the parameter set.

[0057] The parameter storage module 11 works in conjunction with the streaming computation module 13. Each computation unit 131 that requires external parameters is configured with a corresponding ping-pong cache 111. The ping-pong cache 111 stores at least the weight parameters and can further store bias parameters, scaling parameters, lookup table parameters, gating parameters, or other auxiliary parameters related to the current parameter set. For the same parameter set, only the set of parameters currently involved in the computation and the set of parameters to be used in the next switch are retained on-chip, without having to store the entire parameter set on-chip continuously.

[0058] like Figure 3 As shown, preferably, each ping-pong buffer 111 is implemented using a dual-port memory, and its address space is logically divided into a first logical buffer 112 and a second logical buffer 113. For example... Figure 4 As shown, the ping-pong buffer 111 has selectors 114 at its input and output terminals, used to select whether to write or read parameters from the first logical buffer 112 or the second logical buffer 113. The read port continuously outputs parameters to the computing unit 131, while the write port writes parameters to be preloaded to the external interface or the upper-level loading controller. Each logical buffer maintains a read pointer and a write pointer: during the loading phase, parameters are written in a predetermined address order; during the computing phase, parameters are read continuously in the order required for computing.

[0059] Furthermore, each logical buffer is configured with a state machine 115, preferably including at least four states: IDLE, PRELOAD, DONE, and COMPUTE. IDLE indicates that the logical buffer is free and allocable; PRELOAD indicates that the logical buffer is receiving the next parameter set (i.e., receiving parameter set loading) or waiting for the write to complete; DONE indicates that the logical buffer has completed loading and is waiting to participate in data supply; COMPUTE indicates that the logical buffer is outputting parameters to the corresponding computation unit. After system initialization, one of the logical buffers can be put into PRELOAD and loaded with the first parameter set. After loading is complete, it can switch to DONE. When the first batch of input data arrives and the identifier matches, it can switch to COMPUTE.

[0060] When one logical buffer (e.g., the first logical buffer 112) is in the COMPUTE state and continuously supplies data to the computation unit 131, another logical buffer (e.g., the second logical buffer 113) can enter the PRELOAD state in parallel to load the next parameter set. If the currently used logical buffer detects a change in the parameter set identifier of subsequent input data, and the other logical buffer has finished loading, then at the current switching boundary, the logical buffer in the DONE state takes over the data supply and switches to COMPUTE, while the original logical buffer that supplied data returns to IDLE or directly enters the new PRELOAD state. Through the above alternation process, at any given time, one logical buffer is always responsible for the current computation, and the other logical buffer is responsible for preparing for the next switch, thus achieving seamless parameter switching.

[0061] The parameter set identifier transmission module 12 is used to obtain the parameter set identifier corresponding to the input data stream and provide the parameter set identifier to the ping-pong buffer 111 so that the ping-pong buffer 111 selects the logical buffer that matches the parameter set identifier as the data source.

[0062] In this invention, each set of consecutive input data carrying the same parameter set identifier and its intermediate calculation results generated in each computing layer constitutes a data transaction. That is, the input data stream consists of multiple data transactions, and different data transactions correspond to different parameter set identifiers. The parameter set identifier transmission module 12 ensures that the parameter set identifier corresponding to each data transaction maintains a correspondence with the input data or its intermediate calculation results of that data transaction, and transmits it along multiple serially connected computing layers 130. When a data transaction arrives at any computing layer 130, the parameter set identifier is provided to the ping-pong buffer 111 in that computing layer 130 corresponding to the corresponding computing unit 131, so that the ping-pong buffer 111 selects a logical buffer storing parameter fragments matching the parameter set identifier as the data source for that data transaction.

[0063] Since data transactions corresponding to different parameter set identifiers can reside simultaneously in different computation layers 130, each ping-pong cache 111 independently selects its data source based on the parameter set identifier corresponding to the data transaction arriving at the corresponding computation layer 130. Therefore, the system can simultaneously process multiple data transactions corresponding to different parameter sets in the pipeline, achieving continuous and efficient computation in multi-parameter set scenarios.

[0064] like Figure 5 As shown, in addition to the main numerical data path (such as the activation value data path), the system also has a sideband control path (i.e., the parameter set identifier path) for transmitting parameter set identifiers. Figure 4 In this process, the weight data path is used to transfer parameters from the off-chip memory 2 to the corresponding computing unit 131 via the ping-pong cache 111; the activation value data path is used to transfer the input data stream between each computing layer 130; the parameter set identifier path, as a sideband control path, runs in parallel with the activation value data path and is used to transmit the parameter set identifier to control the selection and switching of the ping-pong cache 111.

[0065] The parameter set identifier can represent object type, task number, expert number, model version, layer configuration, operator instance, or other semantic tags that cause parameter switching. When each computation layer 130 receives input data, the parameter set identifier carried in the input data stream is synchronously transmitted to the corresponding ping-pong buffer 111 through the sideband control path; the ping-pong buffer 111 selects the matching logical buffer as the current data source based on the parameter set identifier.

[0066] If the logical buffer currently providing the data is providing parameters for calculation, while another logical buffer is empty or reusable, then the ping-pong buffer 111 sends a preload request to the parameter loading control module 10 in order to reserve a loading slot for the next parameter set to come.

[0067] Through this mechanism, the parameter set identifier does not directly participate in numerical calculations, but it determines where the parameters are read from, when preloading is triggered, and when the data source switch is completed. This allows the parameter switching process to proceed independently via sideband control while the main data stream continues to advance at clock speeds, reducing disturbances to the timing of the main data path.

[0068] The parameter loading control module 10 is used to load the parameter set from the off-chip memory 2 according to the parameter set identifier. Specifically, the parameter loading control module 10 establishes a fixed-width memory access interface with the off-chip memory 2, and performs sequential loading or grouped parallel loading on multiple ping-pong caches 111 according to the interface bit width, parameter arrangement format, computation layer order, and parameter category order. For the computation layer 130 that only contains weights and biases, the weight loading process and the bias loading process can be executed sequentially; for the computation layer 130 that also contains auxiliary parameters, the loading process corresponding to the auxiliary parameters is also executed.

[0069] In a preferred implementation, the top-level parameter loading control module 10 organizes loading in a "parallel loading of caches within a layer and sequential loading between layers" manner. That is, within the same computing layer, a single frame of data from the external interface is split and sent to multiple ping-pong caches. After the required parameter categories for that computing layer have been loaded, the process moves to the next computing layer. For cases where the data bit width and the number of cache write entries are inconsistent, adaptation can be achieved through data alignment, inter-frame concatenation, or segmented writing.

[0070] With the above-mentioned organization method, the present invention can not only adapt to the loading of weight matrices, but also to the loading of different parameters such as bias, gating coefficients and lookup tables; when the parameter set is switched, the top-level loading control logic only needs to reselect the off-chip address range and distribution target cache according to the new parameter set identifier, and the subsequent preloading can be completed in the background.

[0071] During system operation, the parameter loading control module 10 reads the corresponding weight parameters and related auxiliary parameters from the off-chip memory 2 according to the current parameter set identifier, and loads them into a logical buffer (e.g., the first logical buffer 112) in the corresponding ping-pong buffer 111 of each computing layer 130. After loading, the logical buffer enters a data-available state. When the input data stream carrying the parameter set identifier arrives, each computing layer 130 performs multiplication-accumulation or related operations sequentially in an inter-layer pipeline manner, and continues to pass the resulting partial results to subsequent computing layers 130 until the output result data stream is output.

[0072] While the current parameter set is being calculated, the parameter loading control module 10 can write the parameters corresponding to the next parameter set into another logical buffer (e.g., a second logical buffer 113) within the same ping-pong cache 111 in parallel. Since the parameter loading path and the calculation data supply path are independent of each other, the preloading of the next parameter set usually does not interrupt the current calculation process. When the parameter set identifier corresponding to the subsequent input data stream changes, and the next parameter set has already been preloaded, the system uses a new logical buffer to take over the data supply at the corresponding switching boundary. The logical buffer that was originally involved in the calculation is released and can continue to be used for loading subsequent parameter sets, thereby achieving continuous switching between multiple parameter sets.

[0073] Finally, the output results of each computing layer 130 are sent to downstream modules, cache units or other processing units in the form of data streams; when the downstream still uses a streaming interface, it is possible to output while computing, so as to maintain end-to-end pipeline.

[0074] The following section uses a streaming structure consisting of three computing layers as an example to introduce the overall workflow of the system.

[0075] Assume the system performs a three-layer streaming matrix multiplication task, with an input of a 1×2048 vector stream. The first layer weight matrix W0 has a size of 2048×240, the second layer weight matrix W1 has a size of 240×240, and the third layer weight matrix W2 has a size of 240×64. The external parameter interface has a width of 512 bits, and the main data input interface inputs 8 FP32 data per frame.

[0076] The system first reads W0, W1, W2 and their corresponding bias parameters from off-chip memory according to the identifier of parameter set A, and writes them into the three logical buffers currently in use in the computing layer. After writing is completed, each logical buffer enters the ready state.

[0077] When the first input vector carrying the identifier of parameter set A arrives, the first layer begins to perform multiplication and accumulation of the vector with W0; once the first layer forms an output partial result, it sends the partial result to the second layer, which then performs multiplication and accumulation with W1; similarly, the third layer receives the result from the second layer and calculates with W2, finally outputting a 1×64 result vector.

[0078] While the input vectors corresponding to parameter set A are continuously fed into the computation unit and processed, the parameter loading control module writes parameter set B into another logical buffer in each layer via an off-chip 512-bit interface. Since the write and read ends are independent of each other, loading parameter set B will not interrupt the computation process of parameter set A.

[0079] When the parameter set identifier of the subsequent input vector changes from A to B, and all logical buffers used for parameter set B in each layer have been preloaded, each layer is supplied with data by the new logical buffer at its respective switching boundary. Thereafter, the new input vector continues to proceed in the order of "first-layer calculation—second-layer calculation—third-layer calculation—result output," while the logical buffers that previously served parameter set A are released to load the next parameter set C.

[0080] The final layer outputs the calculated results to downstream modules or result caches in vector stream form. If the downstream still uses a streaming interface, the calculation can be performed and transmitted simultaneously without waiting for the entire batch of data to be loaded, thus maintaining continuous end-to-end computation.

[0081] In a scenario with multiple consecutive parameter sets, the system can sequentially operate in a loop following the pattern of "calculating input type A and preloading input type B — switching to input type B and preloading input type C — switching to input type C and preloading subsequent parameter sets". If the number of consecutive inputs corresponding to the same parameter set is large enough, the loading time of the next parameter set can be completely covered by the current calculation stage.

[0082] The weight switching acceleration architecture 1 of this invention only needs to reserve storage space on-chip for the current parameter set and the next parameter set, without storing the entire parameter set on-chip simultaneously, thus significantly reducing on-chip cache resource requirements. Through the parallel organization method of "current computation + next parameter set preloading," the parameter replacement process overlaps with the computation process in time, hiding most or even all parameter replacement delays and reducing pipeline stalls and latency increases caused by off-chip memory access. It is naturally compatible with vector-granular input and inter-layer pipelined computational structures, maintaining continuous progress between computational layers during parameter switching, avoiding the long injection and long emptying problems of traditional large-scale two-dimensional pulsating arrays. A unified control timing boundary is established through parameter set identifier passing and a ready / valid handshake mechanism, making it easy to implement on FPGA, ASIC, or other reconfigurable acceleration platforms. It has a wide range of applications, suitable for neural network layers, scientific computing submodules, and other scenarios requiring parameter set switching by identifier during streaming matrix multiplication; neural network potential function calculation in molecular dynamics is just one preferred application.

[0083] The following are method embodiments corresponding to the above-described architecture embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments remain valid in this embodiment, and will not be repeated here to reduce repetition. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0084] like Figure 6As shown, another embodiment of the present invention provides a weight switching acceleration method 3 suitable for streaming matrix operations. This weight switching acceleration method 3 adopts the weight switching acceleration architecture 1 described above and includes the following steps:

[0085] S1: The parameter loading control module loads the parameter set from the off-chip memory into the ping-pong cache layer according to the parameter set identifier. Each ping-pong cache corresponds to a computing unit and each ping-pong cache has at least two logical buffers.

[0086] Specifically, based on the current parameter set identifier, the parameter loading control module 10 reads the weight parameters and related auxiliary parameters corresponding to the parameter set from the off-chip memory 2. A complete parameter set is split into multiple parameter fragments, which are loaded into multiple ping-pong caches corresponding to the same computing layer, and then loaded into the logical buffer of the corresponding ping-pong cache 111, so that the logical buffer enters the data-available state.

[0087] S2: Obtain the parameter set identifier corresponding to the input data stream through the parameter set identifier transmission module, and provide the parameter set identifier to the ping-pong cache. The parameter set identifier is transmitted along multiple computing layers with the data transaction. When the data transaction reaches any computing layer, the parameter set identifier corresponding to the data transaction is provided to the ping-pong cache corresponding to the corresponding computing unit in that computing layer.

[0088] S3: Ping-pong buffer selects the matching logical buffer as the data source based on the parameter set identifier and outputs the parameters to the corresponding computing unit.

[0089] S4: The computing layer receives the input data stream and performs matrix operations through multiple cascaded computing units. Each computing unit obtains parameters from its corresponding ping-pong buffer.

[0090] When the input data stream carrying the current parameter set identifier arrives, each computation layer 130 reads the parameters from the logical buffer that matches the parameter set identifier, and performs multiply-accumulate operations or related operations in a pipeline manner, and passes the resulting partial results to the subsequent computation layers.

[0091] In this process, while the logical buffer corresponding to the current parameter set supplies data to the corresponding computing unit, the parameter loading control module loads the next parameter set into another logical buffer in the same ping-pong cache; when the parameter set identifier is switched to the next parameter set identifier, the ping-pong cache switches the data supply source to another logical buffer that has already loaded the next parameter set.

[0092] Specifically, while the current parameter set is involved in the calculation, the parameter loading control module 10 writes the parameters corresponding to the next parameter set into another logical buffer in parallel to complete the background preloading of the next parameter set. The parameter loading path is independent of the current calculation data supply path.

[0093] When the parameter set identifier corresponding to the subsequent input data stream changes, and another logical buffer has completed the preloading of the next parameter set, the system controls a new logical buffer to take over the data supply at the corresponding switching boundary, so that the calculation process can smoothly switch from the current parameter set to the next parameter set.

[0094] After the data supply switch is completed, the logical buffer that was originally involved in the calculation of the current parameter set is released and re-enters the idle state or preload state for loading subsequent parameter sets, thereby realizing the cyclic switching between multiple parameter sets.

[0095] The final layer outputs the calculated results to downstream modules, cache units, or other processing units in the form of a data stream. When the downstream uses a streaming interface, the results can be calculated and output simultaneously to maintain end-to-end pipelined execution.

[0096] In a preferred embodiment, at any given time, one logical buffer is maintained for supplying data to the current parameter set, and another logical buffer is maintained for preloading the next parameter set, so as to achieve time overlap between the parameter replacement process and the calculation process, and reduce or hide the pipeline interruption caused by parameter switching.

[0097] Another embodiment of the present invention provides a streaming matrix operation accelerator (not shown in the figure), which includes the weight switching acceleration architecture 1 described above.

[0098] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics in these embodiments are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of the present invention, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed by the present invention should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A weight switching acceleration architecture suitable for streaming matrix operations, characterized in that, include: The parameter loading control module is used to load the parameter set from off-chip memory according to the parameter set identifier; The parameter storage module includes a ping-pong cache layer configured for each computing layer. Each ping-pong cache layer includes multiple ping-pong caches. Each ping-pong cache is connected to a corresponding computing unit and has at least two logical buffers. Each logical buffer is used to store the corresponding parameter fragments after the parameter set is split. The parameter set identifier transmission module is used to obtain the parameter set identifier corresponding to the input data stream and provide the parameter set identifier to the ping-pong buffer so that the ping-pong buffer selects the logical buffer that matches the parameter set identifier as the data source. A streaming computing module includes one or more computing layers connected in series. Each computing layer includes multiple computing units connected in cascade. The computing layer is used to receive an input data stream and perform matrix operations through the multiple computing units. Each computing unit is connected to a corresponding ping-pong buffer to obtain parameters from the data source. While the logical buffer corresponding to the current parameter set supplies data to the corresponding computing unit, the parameter loading control module loads the next parameter set into another logical buffer in the same ping-pong cache; when the parameter set identifier is switched to the next parameter set identifier, the ping-pong cache switches the data supply source to the other logical buffer that has already loaded the next parameter set.

2. The weight switching acceleration architecture according to claim 1, characterized in that, The parameter set identifier transmission module obtains the parameter set identifier through the sideband control path, and the sideband control path is set in parallel with the main data path of the input data stream.

3. The weight switching acceleration architecture according to claim 2, characterized in that, The parameter set identifier transmission module is used to maintain a correspondence between the parameter set identifier corresponding to each data transaction and the input data of the data transaction or the intermediate calculation result generated by the input data, and to transmit it along the multiple serially configured calculation layers; when the data transaction reaches any calculation layer, the parameter set identifier is provided to the ping-pong cache corresponding to the corresponding calculation unit in that calculation layer, so that the ping-pong cache selects a logical buffer that stores parameter fragments that match the parameter set identifier as the data source for the data transaction; Among them, data transactions corresponding to different parameter set identifiers can be located in different computing layers at the same time, and each of the ping-pong caches independently selects the data source according to the parameter set identifier corresponding to the data transaction that arrives at the corresponding computing layer.

4. The weight switching acceleration architecture according to claim 1, characterized in that, The logical buffer is equipped with a state machine, which includes at least an idle state, a preloaded state, a ready state, and a computation state. The idle state indicates that the logical buffer is free and available for allocation; The preloaded state indicates that the logical buffer is receiving parameters for loading or waiting for writing to complete. The ready state indicates that the logical buffer has been loaded and is waiting to participate in data supply. The computation state indicates that the logical buffer is outputting parameters to the corresponding computation unit.

5. The weight switching acceleration architecture according to claim 4, characterized in that, When a data transaction with a different parameter set identifier than the parameter fragment stored in the logical buffer currently in the computation state arrives at the corresponding computation layer, and another logical buffer in the preload state has already entered the ready state, the other logical buffer takes over the data supply and enters the computation state, while the original logical buffer that supplied the data exits the computation state.

6. The weight switching acceleration architecture according to claim 1, characterized in that, The streaming computing module also includes a selection unit, which is connected to multiple computing units and is used to select and forward a portion of the computing results generated by the previous computing layer to the next computing layer without waiting for the entire layer to complete the computing.

7. The weight switching acceleration architecture according to claim 1, characterized in that, The ping-pong buffer is implemented using a dual-port memory. The read port outputs parameters to the corresponding computing unit, and the write port writes parameters to the parameter loading control module. The at least two logical buffers maintain independent read pointers and write pointers respectively.

8. The weight switching acceleration architecture according to claim 1, characterized in that, The parameter set includes weight parameters and auxiliary parameters, and the auxiliary parameters include at least one of bias parameters, scaling parameters, lookup table parameters, or gating parameters.

9. The weight switching acceleration architecture according to claim 1, characterized in that, The parameter loading control module splits the data from the off-chip memory and sends it to multiple ping-pong caches within the same computing layer.

10. The weight switching acceleration architecture according to claim 1, characterized in that, The parameter set identifier is used to represent at least one of the following: object type, task number, expert number, model version, layer configuration, or operator instance.

11. A streaming matrix operation accelerator, characterized in that, Includes the weight switching acceleration architecture as described in any one of claims 1 to 10.

12. A weight switching acceleration method suitable for streaming matrix operations, characterized in that, The weight switching acceleration architecture, as described in any one of claims 1 to 10, includes: The parameter loading control module loads the parameter set from the off-chip memory into the ping-pong cache layer according to the parameter set identifier. Each ping-pong cache corresponds to a computing unit and each ping-pong cache has at least two logical buffers. The parameter set identifier corresponding to the input data stream is obtained through the parameter set identifier transmission module, and the parameter set identifier is provided to the ping-pong buffer. The ping-pong buffer selects a matching logical buffer as a data source based on the parameter set identifier and outputs parameters to the corresponding computing unit. The computing layer receives the input data stream and performs matrix operations through multiple cascaded computing units, with each computing unit obtaining parameters from the corresponding ping-pong buffer. While the logical buffer corresponding to the current parameter set supplies data to the corresponding computing unit, the parameter loading control module loads the next parameter set into another logical buffer in the same ping-pong cache; when the parameter set identifier is switched to the next parameter set identifier, the ping-pong cache switches the data supply source to the other logical buffer that has already loaded the next parameter set.

Images