Model reasoning acceleration device and method based on heterogeneous system-on-chip

Abstract

The invention provides a model reasoning acceleration device and method based on a heterogeneous system-on-chip, and relates to the technical field of artificial intelligence, and the device comprises a processing system end and a programmable logic end; the processing system end comprises a text word segmentation module and an instruction control module; the programmable logic end comprises an instruction decoding module, a scheduling module, a matrix operation module, a sparse control module, a storage control module and an on-chip memory module. The device and the method provided by the invention can adapt to large-scale language models of different scales, support dynamic adjustment of the models, realize acceleration of reasoning of the large-scale language models in embedded equipment with limited computing resources, and improve the computing efficiency of reasoning of the large-scale language models.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model reasoning acceleration device and method based on a heterogeneous system-on-chip. Background Art

[0002] With the rapid development of large language models (LLMs), their application in natural language processing is becoming increasingly widespread. However, the high computational and storage requirements of these models make them difficult to support on most edge devices. Related technologies have primarily focused on accelerating inference for large language models on cloud field-programmable gate array (FPGA) platforms, but these approaches are not suitable for embedded FPGAs such as heterogeneous system-on-chips.

[0003] Therefore, how to achieve inference acceleration of large language models in edge devices with limited computing resources has become a technical problem that needs to be solved urgently in the industry. Summary of the Invention

[0004] The present invention provides a model inference acceleration device and method based on heterogeneous system-on-chip, which is used to solve the technical problem of how to achieve inference acceleration of large language models in edge devices with limited computing resources.

[0005] The present invention provides a model reasoning acceleration device based on a heterogeneous system-on-chip, comprising: Processing system side and programmable logic side; The processing system end includes: a text segmentation module and an instruction control module; The text segmentation module is used to perform text segmentation processing on the input data; The instruction control module is used to determine the control instructions for executing the target model reasoning task; The programmable logic end includes: an instruction decoding module, a scheduling module, a matrix operation module, a sparse control module, a storage control module and an on-chip memory module; The instruction decoding module is used to decode the control instruction and generate operation instructions corresponding to the scheduling module, the matrix operation module, the sparse control module and the storage control module respectively; The scheduling module is used to schedule the matrix operation module, the sparse control module and the storage control module; The sparse control module is used to control the matrix operation module to perform sparse matrix operations; The storage control module is used to read weight parameters and input data from the on-chip memory module, and write output data of the target model to the on-chip memory module.

[0006] In some embodiments, the processing system side and the programmable logic side are connected based on an on-chip bus.

[0007] In some embodiments, the programmable logic terminal further includes a function processing module; The function processing module is connected to the storage control module and is used to perform at least one of activation function operation, normalization processing, custom function operation and quantization processing on input data or intermediate data.

[0008] In some embodiments, the on-chip memory module includes a weight storage unit, an input data storage unit, an intermediate data storage unit, and an output data storage unit; The storage control module is used to control the weight storage unit to store the weight parameters input by the external storage device, control the input data storage unit to store the input data after text segmentation processing, control the intermediate data storage unit to store the intermediate data generated by the target model during the inference process, control the output data storage unit to store the output data generated by the target model, and control the output data storage unit to send the output data to the external storage device.

[0009] In some embodiments, the matrix operation module includes multiple processing units; the processing units are pulse arrays.

[0010] The present invention provides a model reasoning acceleration method based on a heterogeneous system-on-chip, which is applied to the model reasoning acceleration device based on a heterogeneous system-on-chip, comprising: Receive a text segmentation sequence and a control instruction sent by a processing system end; the text segmentation sequence is obtained after the text segmentation module in the processing system end performs text segmentation processing on the input data; In the pre-filling stage of the target model, the sparse control module and the matrix operation module are controlled based on the control instruction to perform sparse matrix operation on the text word segmentation sequence; In the decoding stage of the target model, the storage control module is controlled based on the control instruction to generate a sliding window in the on-chip memory module, and output data is generated based on the sliding window attention mechanism.

[0011] In some embodiments, the method further comprises: quantifying the weight parameters and intermediate data of the target model; The quantization precision of the weight parameter is a 4-bit integer, and the quantization precision of the intermediate data is a 16-bit floating point number.

[0012] The present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the model reasoning acceleration method based on heterogeneous system-on-chip is implemented.

[0013] The present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the model reasoning acceleration method based on heterogeneous system-on-chip is implemented.

[0014] The present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the model reasoning acceleration method based on heterogeneous system-on-chip as described above.

[0015] The model reasoning acceleration device and method based on heterogeneous system-level chips provided by the present invention include a processing system end and a programmable logic end; the processing system end includes a text segmentation module and an instruction control module; the processing system end can perform text segmentation on input data so that the target model can better understand and process the input data; the programmable logic end includes an instruction decoding module, a scheduling module, a matrix operation module, a sparse control module, a storage control module, and an on-chip memory module; matrix operations can be implemented through the programmable logic end, the amount of calculation can be reduced through sparse processing, the data transmission delay can be reduced through on-chip memory, the access to external memory is reduced, and the memory bandwidth requirement is reduced. The above architecture can adapt to large-scale language models of different sizes, support dynamic adjustment of the model, and has the advantages of high efficiency, low power consumption and scalability. It realizes the acceleration of the reasoning of large-scale language models in embedded devices with limited computing resources, and improves the computing efficiency of large-scale language model reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the structural schematic diagrams of the model reasoning acceleration device based on heterogeneous system-level chip provided by the present invention.

[0019] Figure 2This is the second structural diagram of the model reasoning acceleration device based on heterogeneous system-level chip provided by the present invention.

[0020] Figure 3 It is the internal logic diagram of the sparse matrix multiplication provided by the present invention.

[0021] Figure 4 This is a hardware implementation principle diagram of the function processing module provided by the present invention.

[0022] Figure 5 It is a flow chart of the model reasoning acceleration method based on heterogeneous system-on-chip provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," and the like in the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps, units, or modules is not necessarily limited to those steps, units, or modules that are explicitly listed, but may include other steps, units, or modules that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0026] In order to solve the shortcomings of related technologies, Figure 1 This is one of the structural diagrams of the model reasoning acceleration device based on heterogeneous system-level chip provided by the present invention, such as Figure 1 As shown, the model inference acceleration device 100 includes a processing system end 110 and a programmable logic end 120 .

[0027] The processing system end 110 includes a text segmentation module 111 and an instruction control module 112 .

[0028] The text segmentation module is used to perform text segmentation on the input data. The instruction control module is used to determine the control instructions for executing the target model inference task.

[0029] The programmable logic end 120 includes an instruction decoding module 121 , a scheduling module 122 , a matrix operation module 123 , a sparse control module 124 , a storage control module 125 and an on-chip memory module 126 .

[0030] The instruction decoding module is used to decode the control instructions and generate operation instructions corresponding to the scheduling module, matrix operation module, sparse control module and storage control module respectively.

[0031] The scheduling module is used to schedule the matrix operation module, the sparse control module and the storage control module.

[0032] Matrix operation module, used to perform matrix operations.

[0033] The sparse control module is used to control the matrix operation module to perform sparse matrix operations.

[0034] The storage control module is used to read weight parameters and input data from the on-chip memory module and write output data of the target model to the on-chip memory module.

[0035] Specifically, the model inference acceleration device provided by the embodiment of the present invention is suitable for accelerating the inference of various types of large language models and implementing them in embedded FPGAs. The model inference acceleration device is implemented based on a heterogeneous system-on-chip.

[0036] Heterogeneous SoCs integrate multiple different types of processing units (such as central processing units, graphics processing units, field-programmable gate arrays, and digital signal processors) along with other functional modules (such as memory controllers and communication interfaces) to meet diverse computing needs in different application scenarios. These processing units differ in architecture and functionality.

[0037] The heterogeneous system-on-chip in the embodiment of the present invention may be a ZYNQ chip. The ZYNQ chip combines a dual-core processor and a field programmable gate array, and may include a processing system (PS) and a programmable logic (PL).

[0038] In addition to the processor, the processing system also includes a set of related processing resources. From a functional perspective, the processing system is primarily responsible for running the operating system and control logic, which can include a text tokenizer module and an instruction control module.

[0039] The text segmentation module is primarily used to perform word segmentation on input data. It breaks down input text data (such as sentences and paragraphs) into semantic units (also called tokens) to enable the model to better understand and process the text. The processing system can connect to an external network through a communication interface to obtain input data. Input data can be in text format.

[0040] The instruction control module is primarily responsible for determining control instructions for executing target model inference tasks. The target model can be any type of large language model. Configuration parameters refer to the structural parameters of the target model and can include the number of network layers, hidden layer dimensions, number of attention heads, positional embedding method, and so on. Control instructions are instructions for controlling the inference process of the target model. Control instructions can be generated by the processor in the processing system and define the operations to be performed by the programmable logic side, such as initiating a hardware acceleration task or configuring various modules in the programmable logic side.

[0041] The programmable logic side is used to implement hardware acceleration and custom logic. Functionally, it can include an instruction decoding module, a scheduling module, a matrix operation module, a sparse control module, a storage control module, and an on-chip memory module.

[0042] After receiving control instructions from the processing system, the programmable logic needs to decode them to understand their meaning and execute the corresponding operations. This means converting the control instructions into actionable instructions for the programmable logic circuit. These instructions may include an opcode and operands.

[0043] Decoding control instructions can be implemented by an instruction decoding module, which can be implemented using a register bank. The instruction decoding module is connected to the scheduling module, matrix operation module, sparse control module, and storage control module respectively. After decoding the control instructions, it generates the corresponding operation instructions for each module.

[0044] The scheduling module is connected to the matrix operation module, sparse control module and storage control module respectively, and is responsible for scheduling these modules, including the control and pipelining of dense and sparse matrices, as well as the block control of storage.

[0045] The matrix operation module mainly performs matrix operations and is responsible for accelerating computationally intensive tasks such as matrix operations and convolution.

[0046] The sparse control module controls the matrix operation module to perform sparse matrix operations during the target model's inference process. Sparse matrix operations can optimize inference efficiency by reducing computational and storage overhead. They take advantage of the large number of zero values in matrices and skip the calculation of these zero values, significantly improving computational efficiency and reducing memory usage.

[0047] On-chip memory module refers to the memory integrated on the chip, which is used to temporarily store the data and instructions required for the programmable logic side to perform model inference.

[0048] The storage control module is connected to the on-chip memory module and is used to read weight parameters and input data from the on-chip memory module and write the output data of the target model to the on-chip memory module. Weight parameters are the coefficients that connect different neurons (or nodes) in the neural network and determine how the input data affects the output data. The input data stored in the on-chip memory module is the result of processing by the processing system.

[0049] The embodiment of the present invention provides a model reasoning acceleration device based on a heterogeneous system-level chip, which includes a processing system end and a programmable logic end; the processing system end includes a text segmentation module and an instruction control module; the processing system end can perform text segmentation on input data so that the target model can better understand and process the input data; the programmable logic end includes an instruction decoding module, a scheduling module, a matrix operation module, a sparse control module, a storage control module, and an on-chip memory module; matrix operations can be implemented through the programmable logic end, the amount of calculation can be reduced through sparse processing, the delay in data transmission can be reduced through on-chip memory, access to external memory can be reduced, and memory bandwidth requirements can be reduced. The above architecture can adapt to large-scale language models of different sizes, support dynamic adjustment of models, and has the advantages of high efficiency, low power consumption and scalability. It realizes the acceleration of reasoning of large language models in embedded devices with limited computing resources, and improves the computing efficiency of reasoning of large language models.

[0050] In some embodiments, the processing system side and the programmable logic side are connected based on an on-chip bus.

[0051] Specifically, an on-chip bus refers to a bus within a chip used to transmit signals between registers, arithmetic logic units, and control components. In embodiments of the present invention, the processing system and the programmable logic can be connected via an on-chip bus, specifically an Advanced eXtensible Interface (AXI) bus, which features unidirectional channels and burst transmission.

[0052] The model inference acceleration device based on heterogeneous system-level chips provided by an embodiment of the present invention connects the processing system end and the programmable logic end through an on-chip bus, which can achieve efficient data transmission and low-latency communication, and can improve the computational efficiency of large-scale language model reasoning.

[0053] In some embodiments, the programmable logic end also includes a function processing module; the function processing module is connected to the storage control module and is used to perform at least one of activation function operation, normalization processing, custom function operation and quantization processing on the input data or intermediate data.

[0054] Specifically, the programmable logic side can also include a function processing module (SFU) to perform calculations for special functions. This function processing module, connected to the storage control module, can perform special processing on input data or intermediate data, including activation function operations, normalization, custom function operations, and quantization. Special functions in the function processing module can be customized based on actual needs.

[0055] The model inference acceleration device based on heterogeneous system-level chips provided by the embodiment of the present invention can define the processing modules corresponding to special functions through the programmable logic end, and can meet different data processing requirements in large language models.

[0056] In some embodiments, the on-chip memory module includes a weight storage unit, an input data storage unit, an intermediate data storage unit, and an output data storage unit; The storage control module is used to control the weight storage unit to store the weight parameters input by the external storage device, control the input data storage unit to store the input data after text segmentation processing, control the intermediate data storage unit to store the intermediate data generated by the target model during the inference process, control the output data storage unit to store the output data generated by the target model, and control the output data storage unit to send the output data to the external storage device.

[0057] Specifically, the on-chip memory module can be divided according to different types of data, including weight storage units, input data storage units, intermediate data storage units, and output data storage units.

[0058] The storage control module controls each of the aforementioned storage units. It can control the weight storage unit to retrieve and store weight parameters from an external storage device during model inference. The external storage device can be, for example, a double data rate (DDR) memory. Before model inference, the input data storage unit is controlled to store input data sent by the processing system. During model inference, the intermediate data storage unit is controlled to store generated intermediate data. After model inference, the output data storage unit is controlled to store generated output data and to send output data to an external storage device.

[0059] The model inference acceleration device based on heterogeneous system-on-chip provided by the embodiment of the present invention can effectively store input data, intermediate data and output data through reasonable partitioning through on-chip memory modules, thereby improving the computational efficiency of large-scale language model inference.

[0060] In some embodiments, the matrix operation module includes multiple processing units; the processing units are pulse arrays.

[0061] Specifically, the matrix operation module is mainly composed of multiple processing elements (PEs), which are mainly pulse arrays (Systolic Arrays).

[0062] In Zynq, each processing unit forms a two-dimensional array, such as a 16×16 systolic array. Each processing unit is responsible for computing one element in a matrix multiplication, and data is passed between processing units row by row or column by column. This structure is particularly well-suited for matrix multiplication and convolution operations, which are highly regular and parallel.

[0063] An embodiment of the present invention provides a model inference acceleration device based on a heterogeneous system-on-chip. The processing unit in the matrix operation module is a pulse array. Through the orderly flow of data and parallel computing, it provides efficient hardware acceleration support for the matrix operation and model inference tasks in ZYNQ.

[0064] Figure 2 This is the second structural diagram of the model reasoning acceleration device based on heterogeneous system-level chip provided by the present invention, such as Figure 2 As shown in the figure, matrix calculation is mainly composed of multiple processing units (PEs), each of which contains a 16×16 systolic array. The sparse control module is used to perform sparse control on the matrix operation module to support structured sparse and sparse-dense matrix operations in large language models.

[0065] Figure 3 It is the internal logic diagram of the sparse matrix multiplication provided by the present invention, such as Figure 3 As shown, the compression index (indice) can be set to implement multi-way selection of the data in matrix a, thereby achieving sparse or dense operations with the data in matrix b.

[0066] The memory control unit module operates on three memory components: weight memory, input memory, and intermediate memory. The weight and input memories store converter parameters and input data, respectively, from off-chip DDR. Results are also written back to the input memory and transferred to off-chip DDR storage.

[0067] Figure 4 This is a hardware implementation schematic diagram of the function processing module provided by the present invention, such as Figure 4As shown in the figure, the function processing module mainly completes some low-density calculations and some nonlinear operations, including activation function operation (softmax), normalization processing (RMSNORM), custom function operation (Vector_unit) and quantization processing (Quant).

[0068] The hardware of the function processing module mainly includes a lookup table (LUT), an adder (add), a multiply-accumulate unit (mac), a register (reg), an add tree (add tree), and a divider. These basic components can implement digital logic and calculation tasks.

[0069] Figure 5 This is a flow chart of the model reasoning acceleration method based on heterogeneous system-level chips provided by the present invention, such as Figure 5 As shown, the method is applied to the model reasoning acceleration device in the above embodiment, including step 510, step 520 and step 530.

[0070] Step 510: Receive the text segmentation sequence and control instructions sent by the processing system end; the text segmentation sequence is obtained after the text segmentation module in the processing system end performs text segmentation processing on the input data.

[0071] Specifically, the processing system includes a text segmentation module and an instruction control module. The text segmentation module performs text segmentation on input data to generate a text segmentation sequence. The main purpose of text segmentation is to break down continuous text strings into meaningful units (tokens), which can be words, subwords, characters, or other linguistic units. The instruction control module generates control instructions based on user-configured parameters for the target model.

[0072] During the initialization phase, the software program on the processing system sends the configurable parameters of the model structure to the programmable logic side. Before the programmable logic side performs calculations, the input data and weight parameters are preloaded into memory. When the hardware accelerator on the programmable logic side is ready, the software receives the corresponding results from memory. External memory (external storage device) and on-chip memory in the programmable logic side together store all model parameters and input samples.

[0073] Step 520: In the pre-filling stage of the target model, the sparse control module and the matrix operation module are controlled based on the control instructions to perform sparse matrix operations on the text word segmentation sequence.

[0074] Specifically, the inference process of a large language model (LLM) usually consists of two main stages: prefill and decoding, which are responsible for processing input data and gradually generating output data, respectively.

[0075] During inference on a large language model (LLM), the pre-population phase is a critical initial step, primarily responsible for processing input data and generating an initial context representation for the decoding phase. During this phase, the collaborative work of the sparse control module and the scheduling module significantly optimizes computing resource utilization, reducing computational effort and memory usage. Text segmentation is already implemented on the processing system side. The programmable logic side primarily implements embedding, positional encoding, attention masking, initial context calculation, and key-value cache initialization.

[0076] The programmable logic side can flexibly control the calculations in the processing elements (PEs) of the matrix operation module through the sparsity control module based on the structured sparsity ratio configured in the instruction decoding module (register group). This sparsification process can reduce unnecessary calculations, especially when processing long context inputs, and can significantly reduce the latency of the pre-fill phase.

[0077] The scheduling module coordinates the work of various modules during the pre-population phase, including pipelined computation of dense and sparse matrices and control of storage partitioning. Proper scheduling maximizes the utilization of PE computing resources and prevents waste of computing power due to oversizing of the systolic array.

[0078] To further improve computational efficiency, multiple small systolic arrays can be concatenated into a large systolic array. This design allows for flexible invocation of small systolic arrays across multiple PEs, enabling more efficient parallel computation. This optimization is particularly useful for processing long context inputs, significantly reducing computation time and memory usage during the pre-population phase.

[0079] Step 530: During the decoding phase of the target model, the storage control module is controlled based on the control instruction to generate a sliding window in the on-chip memory module, and output data is generated based on the sliding window attention mechanism.

[0080] Specifically, the decoding phase is the dynamic generation phase of model inference, which gradually generates output data.

[0081] During the decoding phase, the storage control module uses a fixed-size first-in, first-out (FIFO) queue to implement a sliding window mechanism. This mechanism uses sliding window attention to limit the attention range of each semantic unit (token) to a fixed number of adjacent semantic units before and after it, forming a diagonal sparsity pattern. This diagonal sparsity pattern reduces unnecessary computation and memory access, improving computational efficiency. This mechanism is particularly suitable for processing long sequence inputs because it can significantly reduce memory usage and computational complexity.

[0082] Input data is held in a buffer after loading, while the necessary computational resources are placed around it. This design reduces the movement of data within on-chip memory, thereby better utilizing on-chip memory bandwidth. This "input-at-rest" dataflow design significantly improves performance by reducing data transfer overhead.

[0083] The embodiment of the present invention provides a model reasoning acceleration method based on a heterogeneous system-level chip, which receives a text segmentation sequence and a control instruction sent by a processing system end; the text segmentation sequence is obtained after the text segmentation module in the processing system end performs text segmentation processing on the input data; in the pre-filling stage of the target model, the sparse control module and the matrix operation module are controlled based on the control instruction to perform sparse matrix operations on the text segmentation sequence; in the decoding stage of the target model, the storage control module is controlled based on the control instruction to generate a sliding window in the on-chip memory module, and output data is generated based on the sliding window attention mechanism; not only is the hardware structure optimized and designed, but also sparse matrix operations are jointly implemented by the sparse control module and the matrix operation module in the pre-filling stage of model reasoning, which not only reduces the delay in the pre-filling stage, but also improves the overall reasoning efficiency, so that the target model can process long context input more quickly; in the decoding stage of the model reasoning, the sliding window attention mechanism is used to generate output data, which improves the performance and efficiency of the model in the decoding stage.

[0084] In some embodiments, the method further comprises: Quantify the weight parameters and intermediate data of the target model; The quantization precision of the weight parameter is 4-bit integer, and the quantization precision of the intermediate data is 16-bit floating point number.

[0085] Specifically, in the inference optimization of the model, quantizing the model's weight parameters and intermediate data can reduce the model's storage space and computing resource requirements while accelerating the inference speed.

[0086] The quantization precision of weight parameters is 4-bit integers (INT4). This quantization method can significantly reduce the storage space and computational complexity of the model. For example, quantizing 32-bit floating-point (FP32) weights to 4-bit integers can reduce the storage space to 1 / 8 of the original.

[0087] Intermediate data is quantized to 16-bit floating-point numbers (FP16), which reduces storage and computing resource requirements while maintaining a certain level of computational accuracy. Compared to FP32, FP16 can reduce storage space by half and generally offers higher computational efficiency on embedded hardware devices.

[0088] The model inference acceleration method based on heterogeneous system-on-chip provided by the embodiment of the present invention quantizes the weight parameters and intermediate data of the target model, which can maintain high inference performance and accuracy while reducing model storage space and computing resources.

[0089] Figure 6 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor (Processor) 610, a communication interface (Communications Interface) 620, a memory (Memory) 630 and a communication bus (Communications Bus) 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic commands in the memory 630 to execute the method described in the above embodiment, for example: The text segmentation sequence and control instructions sent by the receiving processing system are obtained after the text segmentation module in the processing system performs text segmentation on the input data; in the pre-filling stage of the target model, the sparse control module and the matrix operation module are controlled based on the control instructions to perform sparse matrix operations on the text segmentation sequence; in the decoding stage of the target model, the storage control module is controlled based on the control instructions to generate a sliding window in the on-chip memory module, and output data is generated based on the sliding window attention mechanism.

[0090] Furthermore, the logical commands in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several commands for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0091] The processor in the electronic device provided by the embodiment of the present invention can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the above method and can achieve the same beneficial effects, which will not be repeated here.

[0092] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method provided in the above embodiments is implemented.

[0093] Its specific implementation is consistent with the aforementioned method implementation and can achieve the same beneficial effects, so it will not be repeated here.

[0094] An embodiment of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the method described above is implemented.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0096] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A model reasoning acceleration device based on heterogeneous system-on-chip, characterized in that: include: Processing system side and programmable logic side; The processing system end includes: a text segmentation module and an instruction control module; The text segmentation module is used to perform text segmentation processing on the input data; The instruction control module is used to determine the control instructions for executing the target model reasoning task; The programmable logic end includes: an instruction decoding module, a scheduling module, a matrix operation module, a sparse control module, a storage control module and an on-chip memory module; The instruction decoding module is used to decode the control instruction and generate operation instructions corresponding to the scheduling module, the matrix operation module, the sparse control module and the storage control module respectively; The scheduling module is used to schedule the matrix operation module, the sparse control module and the storage control module; The sparse control module is used to control the matrix operation module to perform sparse matrix operations; The storage control module is used to read weight parameters and input data from the on-chip memory module, and write output data of the target model to the on-chip memory module.

2. The model reasoning acceleration device based on heterogeneous system-on-chip according to claim 1, characterized in that: The on-chip memory module includes: a weight storage unit, an input data storage unit, an intermediate data storage unit and an output data storage unit; The storage control module is used to control the weight storage unit to store the weight parameters input by the external storage device, control the input data storage unit to store the input data after text segmentation processing, control the intermediate data storage unit to store the intermediate data generated by the target model during the inference process, control the output data storage unit to store the output data generated by the target model, and control the output data storage unit to send the output data to the external storage device.

3. The model reasoning acceleration device based on heterogeneous system-on-chip according to claim 1, characterized in that: The programmable logic end also includes a function processing module; The function processing module is connected to the storage control module and is used to perform at least one of activation function operation, normalization processing, custom function operation and quantization processing on input data or intermediate data.

4. The model reasoning acceleration device based on heterogeneous system-on-chip according to claim 1, characterized in that: The matrix operation module includes multiple processing units; the processing units are pulse arrays.

5. The model reasoning acceleration device based on heterogeneous system-on-chip according to claim 1, characterized in that: The processing system end and the programmable logic end are connected based on an on-chip bus.

6. A model reasoning acceleration method based on heterogeneous system-level chips, characterized in that: The model reasoning acceleration device based on a heterogeneous system-on-chip as claimed in any one of claims 1 to 5 comprises: Receive a text segmentation sequence and a control instruction sent by a processing system end; the text segmentation sequence is obtained after the text segmentation module in the processing system end performs text segmentation processing on the input data; In the pre-filling stage of the target model, the sparse control module and the matrix operation module are controlled based on the control instruction to perform sparse matrix operation on the text word segmentation sequence; In the decoding stage of the target model, the storage control module is controlled based on the control instruction to generate a sliding window in the on-chip memory module, and output data is generated based on the sliding window attention mechanism.

7. The model reasoning acceleration method based on heterogeneous system-on-chip according to claim 6, characterized in that: The method further comprises: quantifying the weight parameters and intermediate data of the target model; The quantization precision of the weight parameter is a 4-bit integer, and the quantization precision of the intermediate data is a 16-bit floating point number.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the model reasoning acceleration method based on heterogeneous system-on-chip described in claim 6 or 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model reasoning acceleration method based on heterogeneous system-on-chip according to claim 6 or 7 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the model reasoning acceleration method based on heterogeneous system-on-chip as described in claim 6 or 7 is implemented.

Images