Acceleration method and system for model speculation reasoning

Abstract

The embodiment of the invention provides a model speculation reasoning acceleration method and system, and the method comprises the steps: respectively deploying model parameters needed by a verification task in independent storage spaces of a neural network processor side and a calculation module side when a plurality of candidate token sequences outputted by a draft model are verified, the neural network processor only uses the first parameter subset in the first storage space to verify the plurality of candidate token sequences, and the calculation module only uses the second parameter subset in the second storage space to verify the plurality of candidate token sequences, so that the neural network processor and the calculation module can execute verification tasks in parallel; according to the method, the mutual interference of parameter loading is avoided, the data migration overhead in the verification process is remarkably reduced, the overall reasoning throughput is improved, the locality advantage of the near memory computing architecture is effectively utilized, that is, the near data is computed, the energy efficiency ratio of data verification is improved, and therefore the large model reasoning efficiency and the system resource utilization rate of the mobile terminal are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model speculation reasoning acceleration method and system. BACKGROUND

[0002] With the wide application of large language models (LLMs), the efficient inference of LLMs on mobile devices is increasingly in demand. However, due to the computing power, memory bandwidth and power consumption budget of mobile devices, deploying LLMs directly on mobile devices faces severe challenges.

[0003] In order to more efficiently perform large model inference on mobile devices, traditional solutions usually use speculation decoding technology to accelerate inference speed, that is, by introducing a lightweight draft model to generate multiple candidate tokens, and by a target model to verify the candidate tokens in parallel, the serial inference is converted into parallel computation, which significantly improves the inference throughput. When the speculation decoding technology converts the matrix-vector multiplication corresponding to the traditional autoregressive decoding into matrix-matrix multiplication, it also brings greater data access, which exacerbates the pressure on the limited bandwidth of mobile devices. However, mobile devices generally use LPDDR or low-power DRAM, which has limited memory bandwidth and cannot support the batch data access demand brought by speculation decoding, resulting in limited performance improvement.

[0004] In addition, existing processing-in-memory (PIM) technology integrates a computing module inside the DRAM to efficiently perform matrix-vector multiplication operations to reduce data transfer overhead. However, the existing PIM architecture mainly targets matrix-vector multiplication operations in serial inference, and its computing module structure, data scheduling mechanism and parallel granularity are difficult to adapt to matrix-matrix multiplication operations in speculation decoding, resulting in that the performance and energy efficiency advantages of the PIM architecture cannot be fully utilized, which leads to a decrease in data transfer efficiency and a sharp increase in storage miss rate, ultimately offsetting the benefits of computation parallelization.

[0005] Therefore, there is an urgent need for a model speculation reasoning acceleration system for mobile devices to solve the mismatch between the speculation decoding algorithm and the PIM hardware. SUMMARY

[0006] The embodiments of the present application provide a model speculation reasoning acceleration method and system to improve the performance and efficiency of large model speculation reasoning on mobile devices.

[0007] In a first aspect, the embodiments of the present application provide a model speculation reasoning acceleration system, which includes a main processor, a neural network processor and a hybrid processing-in-memory module, wherein the hybrid processing-in-memory module includes a computing module and a storage module.

[0008] The main processor is configured to load a plurality of candidate token sequences output by a current decoding stage of a draft model and model parameters of a target model, and split the model parameters into a first parameter subset and a second parameter subset;

[0009] The main processor is configured to load the first parameter subset into a first storage space corresponding to the storage module, and load the second parameter subset into a second storage space located in the calculation module.

[0010] The neural network processor is configured to verify the plurality of candidate token sequences based on the first parameter subset in the first storage space to obtain a first sub-verification result.

[0011] The calculation module is configured to verify the plurality of candidate token sequences based on the second parameter subset in the second storage space to obtain a second sub-verification result, and transmit the second sub-verification result to the neural network processor.

[0012] The neural network processor is configured to determine a verification result corresponding to the plurality of candidate token sequences of the current decoding stage based on the first sub-verification result and the second sub-verification result.

[0013] In a second aspect, an embodiment of the present application provides an acceleration method of model speculation reasoning, applied to an acceleration system of model speculation reasoning, the system comprising a main processor, a neural network processor, and a hybrid near-memory computing module, the hybrid near-memory computing module comprising a calculation module and a storage module; the method comprising:

[0014] The main processor loads a plurality of candidate token sequences output by a current decoding stage of a draft model and model parameters of a target model, and splits the model parameters into a first parameter subset and a second parameter subset.

[0015] The main processor loads the first parameter subset into a first storage space corresponding to the storage module, so that the neural network processor verifies the plurality of candidate token sequences based on the first parameter subset in the first storage space to obtain a first sub-verification result.

[0016] The main processor loads the second parameter subset into a second storage space located in the calculation module, so that the calculation module verifies the plurality of candidate token sequences based on the second parameter subset in the second storage space to obtain a second sub-verification result, and transmits the second sub-verification result to the neural network processor, so that the neural network processor determines a verification result corresponding to the plurality of candidate token sequences of the current decoding stage based on the first sub-verification result and the second sub-verification result.

[0017] In a third aspect, an embodiment of the present application provides an acceleration device for model speculation reasoning, applied to an acceleration system for model speculation reasoning, the system comprising: a host processor, a neural network processor, a hybrid near-memory computing module, the hybrid near-memory computing module comprising a computing module and a storage module;

[0018] The device is located in the host processor and comprises:

[0019] A first loading module is configured to load a plurality of candidate token sequences output by a current decoding stage of a draft model and model parameters of a target model, and split the model parameters into a first parameter subset and a second parameter subset;

[0020] A second loading module is configured to load the first parameter subset into a first storage space corresponding to the storage module, so that the neural network processor verifies the plurality of candidate token sequences based on the first parameter subset in the first storage space to obtain a first sub-verification result;

[0021] A third loading module is configured to load the second parameter subset into a second storage space located in the computing module, so that the computing module verifies the plurality of candidate token sequences based on the second parameter subset in the second storage space to obtain a second sub-verification result, and transmits the second sub-verification result to the neural network processor, so that the neural network processor determines a verification result corresponding to the plurality of candidate token sequences of the current decoding stage based on the first sub-verification result and the second sub-verification result.

[0022] In a fourth aspect, an embodiment of the present application provides a host processor located in an acceleration system for model speculation reasoning, comprising a processor and a memory, the memory being configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the model speculation reasoning acceleration method in the second aspect. The electronic device can further comprise a communication interface for communicating with other devices or communication systems.

[0023] In a fifth aspect, an embodiment of the present application provides a non-transitory machine-readable storage medium, the non-transitory machine-readable storage medium storing executable code, when the executable code is executed by a processor in a host processor, the processor can at least implement the model speculation reasoning acceleration method in the second aspect.

[0024] In a sixth aspect, an embodiment of the present application provides an acceleration method for model speculation reasoning, applied to an acceleration system for model speculation reasoning, the system comprising: a host processor, a neural network processor, a hybrid near-memory computing module, the hybrid near-memory computing module comprising a computing module and a storage module; the method comprising:

[0025] The neural network processor verifies a plurality of candidate token sequences output by the current decoding stage of the draft model based on a first parameter subset in a first storage space corresponding to the storage module, to obtain a first sub-verification result; wherein the first parameter subset is loaded into the first storage space corresponding to the storage module after being split from model parameters of the target model by the host processor;

[0026] The neural network processor receives the second sub-verification result sent by the computing module, and determines a verification result corresponding to the plurality of candidate token sequences of the current decoding stage based on the first sub-verification result and the second sub-verification result; wherein the second sub-verification result is obtained by the computing module based on a second parameter subset in a second storage space in the computing module; and the second parameter subset is loaded into the second storage space corresponding to the computing module after being split from model parameters of the target model by the host processor.

[0027] In a seventh aspect, an embodiment of the present application provides an acceleration device for model speculation reasoning, applied to an acceleration system for model speculation reasoning, the system comprising: a host processor, a neural network processor, a hybrid near-memory computing module, the hybrid near-memory computing module comprising a computing module and a storage module;

[0028] The device is located in the neural network processor and comprises:

[0029] The verification module is configured to verify a plurality of candidate token sequences output by the current decoding stage of the draft model based on a first parameter subset in a first storage space corresponding to the storage module, to obtain a first sub-verification result; wherein the first parameter subset is loaded into the first storage space corresponding to the storage module after being split from model parameters of the target model by the host processor.

[0030] The receiving module is configured to receive the second sub-verification result sent by the computing module, and determine a verification result corresponding to the plurality of candidate token sequences of the current decoding stage based on the first sub-verification result and the second sub-verification result; wherein the second sub-verification result is obtained by the computing module based on a second parameter subset in a second storage space in the computing module; and the second parameter subset is loaded into the second storage space corresponding to the computing module after being split from model parameters of the target model by the host processor.

[0031] In an eighth aspect, an embodiment of the present application provides a neural network processor located in a model speculative inference acceleration system, comprising a processor and a memory, the memory being configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the model speculative inference acceleration method in the sixth aspect. The electronic device can further comprise a communication interface configured to communicate with other devices or communication systems.

[0032] In a ninth aspect, an embodiment of the present application provides a non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor in a neural network processor, causes the processor to implement at least the model speculative inference acceleration method in the sixth aspect.

[0033] In a tenth aspect, an embodiment of the present application provides a model speculative inference acceleration method applied to a model speculative inference acceleration system, the system comprising a host processor, a neural network processor, and a hybrid near-memory computing module, the hybrid near-memory computing module comprising a computing module and a storage module; the method comprising:

[0034] The computing module verifies a plurality of candidate token sequences output by a current decoding stage of a draft model based on a first parameter subset in a first storage space in the computing module, to obtain a first sub-verification result; the first parameter subset is loaded into the first storage space corresponding to the computing module by the host processor splitting model parameters of a target model;

[0035] The computing module transmits the first sub-verification result to the neural network processor, so that the neural network processor determines a verification result corresponding to the plurality of candidate token sequences of the current decoding stage based on the first sub-verification result and a second sub-verification result;

[0036] The second sub-verification result is obtained by the neural network processor verifying the plurality of candidate token sequences based on a second parameter subset in a second storage space corresponding to the storage module; the second parameter subset is loaded into the second storage space corresponding to the storage module by the host processor splitting model parameters of the target model.

[0037] In an eleventh aspect, an embodiment of the present application provides a model speculative inference acceleration device applied to a model speculative inference acceleration system, the system comprising a host processor, a neural network processor, and a hybrid near-memory computing module, the hybrid near-memory computing module comprising a computing module and a storage module;

[0038] The device is located in the computing module and comprises:

[0039] The verification module is configured to verify the plurality of candidate token sequences output by the current decoding stage of the draft model based on a first parameter subset in a first storage space in the computing module, to obtain a first sub-verification result; the first parameter subset is loaded into the first storage space corresponding to the computing module by the host processor after splitting the model parameters of the target model;

[0040] The transmission module is configured to transmit the first sub-verification result to the neural network processor, so that the neural network processor determines a verification result corresponding to the plurality of candidate token sequences of the current decoding stage based on the first sub-verification result and a second sub-verification result.

[0041] The second sub-verification result is obtained by the neural network processor based on a second parameter subset in a second storage space corresponding to the storage module; and the second parameter subset is loaded into the second storage space corresponding to the storage module by the host processor after splitting the model parameters of the target model.

[0042] In a twelfth aspect, an embodiment of the present application provides a computing module located in an acceleration system of model speculative reasoning, including a processor and a memory, the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the model speculative reasoning acceleration method in the tenth aspect. The electronic device can also include a communication interface for communicating with other devices or communication systems.

[0043] In a thirteenth aspect, an embodiment of the present application provides a non-transitory machine-readable storage medium, the non-transitory machine-readable storage medium stores executable code, when the executable code is executed by the processor in the computing module, the processor can at least implement the model speculative reasoning acceleration method in the tenth aspect.

[0044] In the model speculative reasoning acceleration method provided by the embodiments of the present application, the model parameters of the target model are used to verify the plurality of candidate token sequences output by the current decoding stage of the draft model through the acceleration system of model speculative reasoning. The acceleration system of model speculative reasoning includes a host processor, a neural network processor, a hybrid near-memory computing module, the hybrid near-memory computing module includes a computing module and a storage module, and the acceleration system is provided with a first storage space in the storage module and a second storage space in the computing module. The plurality of candidate token sequences output by the draft model are verified by the neural network processor and the computing module together, so as to improve the resource utilization and the speculative reasoning speed of the target model.

[0045] In the speculative decoding verification process, first, the main processor loads the plurality of candidate token sequences output by the current decoding stage of the draft model and the model parameters of the target model, and splits the model parameters into a first parameter subset and a second parameter subset. Among them, the first parameter subset contains a first target number of column vectors, and the second parameter subset contains a second target number of column vectors. Then, the main processor loads the first parameter subset into the first storage space corresponding to the storage module, and loads the second parameter subset into the second storage space located in the calculation module. Then, the neural network processor reads the first parameter subset from the first storage space, and verifies the plurality of candidate token sequences based on the first parameter subset to obtain a first sub-verification result. The calculation module reads the second parameter subset from the second storage space, and verifies the plurality of candidate token sequences based on the second parameter subset to obtain a second sub-verification result. Then, the calculation module transmits the second sub-verification result to the neural network processor, so that the neural network processor determines the verification result corresponding to the plurality of candidate token sequences of the current decoding stage according to the first sub-verification result and the second sub-verification result.

[0046] In the above scheme, when verifying the plurality of candidate token sequences output by the draft model, the parameters required for the verification task are respectively deployed in the independent storage spaces of the neural network processor side and the calculation module side, and the neural network processor only uses the first parameter subset in the first storage space to verify the plurality of candidate token sequences, and the calculation module only uses the second parameter subset in the second storage space to verify the plurality of candidate token sequences, so that the neural network processor and the calculation module can perform the verification task in parallel, avoid the mutual interference of parameter loading, significantly reduce the data moving overhead in the verification process, improve the overall inference throughput, effectively utilize the locality advantage of the near-memory computing architecture, i.e., calculation close to data, improve the energy efficiency ratio of data verification, realize the collaborative optimization of computing and storage resources, and thus improve the large model inference efficiency and system resource utilization of the mobile terminal. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0048] Figure 1 A schematic diagram of a model speculative inference acceleration system provided by an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a candidate token tree structure provided by an embodiment of the present application;

[0050] Figure 3 A flow chart of a model speculation reasoning loading method provided for an embodiment of the present application;

[0051] Figure 4 A flow chart of another model speculation reasoning loading method provided for an embodiment of the present application;

[0052] Figure 5 A flow chart of still another model speculation reasoning loading method provided for an embodiment of the present application;

[0053] Figure 6 An application schematic diagram of a model speculation decoding acceleration system provided for an embodiment of the present application;

[0054] Figure 7 A data flow schematic diagram of a workload scheduling provided for an embodiment of the present application;

[0055] Figure 8 A schematic diagram of an NPU-PIM with dynamic adjustment and without dynamic adjustment provided for an embodiment of the present application;

[0056] Figure 9 A structural schematic diagram of a model speculation reasoning acceleration device provided for an embodiment of the present application;

[0057] Figure 10 A structural schematic diagram of a main processor device provided for an embodiment of the present application;

[0058] Figure 11 A structural schematic diagram of another model speculation reasoning acceleration device provided for an embodiment of the present application;

[0059] Figure 12 A structural schematic diagram of a neural network processor device provided for an embodiment of the present application;

[0060] Figure 13 A structural schematic diagram of still another model speculation reasoning acceleration device provided for an embodiment of the present application;

[0061] Figure 14 A structural schematic diagram of a computing module device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0063] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0064] First, some concepts involved in the embodiments of the present application are explained.

[0065] Speculative decoding: a parallel inference technique for accelerating the autoregressive generation process of large language models. The core idea is to generate multiple candidate token sequences (Candidate Tokens) speculatively through a lightweight draft model (Draft Model), and then verify these candidate token sequences in parallel by the target large model (Target Model), thereby significantly reducing the high latency problem caused by generating candidate tokens one by one while ensuring output quality.

[0066] In the traditional autoregressive decoding process, each candidate token must wait for the generation of the previous candidate token to start computing, resulting in slow inference speed. Speculative decoding introduces the mechanism of "guessing first and then verifying" to realize batch verification of multiple candidate tokens, effectively improving the generation throughput.

[0067] The specific process of speculative decoding is as follows:

[0068] Draft phase: a draft model with small computational overhead quickly generates multiple candidate token sequences according to the current context;

[0069] Verification phase: the target large model verifies each candidate token in multiple candidate sequences in parallel until the first failed candidate token appears. All verified candidate tokens are "accepted" and output; from the first failed position, the subsequent candidate tokens are discarded, and the last verified candidate token is used to regenerate the above process to generate subsequent content.

[0070] However, when speculative decoding converts the matrix-vector multiplication in the traditional autoregressive decoding scheme to matrix-matrix multiplication, it also brings greater data access volume, exacerbating the pressure on the limited bandwidth of mobile terminals. However, mobile devices generally use LPDDR or low-power DRAM, which has limited memory bandwidth and is difficult to support the batch data access demand brought by speculative decoding, resulting in limited performance improvement.

[0071] In addition, existing processing-in-memory (PIM) technology can efficiently perform matrix-vector multiplication operations by integrating a computing module inside a DRAM to reduce data transfer overhead. However, the existing PIM architecture mainly targets matrix-vector multiplication operations in serial inference, and its computing module structure, data scheduling mechanism, and parallel granularity are difficult to adapt to matrix-matrix multiplication operations in speculative decoding, resulting in a failure to fully exert the performance and energy efficiency advantages, which reduces data transfer efficiency and increases storage miss rate, ultimately offsetting the benefits of computing parallelization.

[0072] Therefore, to solve the above technical problems, the embodiments of the present application propose an acceleration scheme for model speculative inference, which fully considers the characteristics of speculative decoding in parallel verification and large data access volume, uses the storage blocks inside the storage module and the computing module in the hybrid processing-in-memory (PIM) module as storage space for storing model parameters, and jointly uses the neural network processor and the computing module to participate in parallel verification of multiple candidate token sequences, thereby improving the overall inference throughput and resource utilization, significantly reducing the memory access delay and power consumption in the verification process, achieving collaborative optimization of computing and storage resources, and improving the large model inference efficiency and system resource utilization of mobile terminals.

[0073] Some embodiments of the present application can be described in detail below with reference to the accompanying drawings. The embodiments described below and the features, steps in the embodiments can be combined with each other without conflict. In addition, the sequence of steps in each method embodiment described below is only an example and is not strictly limited.

[0074] Figure 1 A schematic diagram of an acceleration system for model speculative inference provided by the embodiments of the present application is shown in FIG. 1, which includes a host processor, a neural network processor, and a hybrid processing-in-memory (PIM) module including a computing module and a storage module. Figure 1

[0075] The host processor is configured to split the model parameters required in the decoding stage and allocate data to the model parameters. For example, when verifying the multiple candidate sequences output by the draft model in the current decoding stage, the host processor adjusts the parameter allocation ratio of the model parameters based on the current neural network model, the device performance of the computing module, and the multiple candidate token sequences, and loads the respective parameter subsets corresponding to the adjusted parameter allocation ratio into the first storage space corresponding to the storage module and the second storage space corresponding to the computing module. In addition, the host processor can be a CPU corresponding to a mobile terminal.

[0076] ​The neural network processor is configured to perform sub-verification tasks corresponding to the plurality of candidate token sequences together with the computing module and aggregate verification results corresponding to the plurality of sub-verification tasks. That is, the neural network processor can perform part of the task of verifying the plurality of candidate token sequences. In addition, in actual application, the neural network processor can adopt an NPU suitable for a mobile terminal, which has a high-parallel multiply-add operation array and a tightly coupled local cache structure, can efficiently perform matrix-matrix multiplication operations frequently occurring in the speculative decoding process, so as to realize fast calculation for the matrix-matrix multiplication operations in the speculative decoding process, improve the execution efficiency of candidate token verification, reduce the delay and energy consumption of the mobile terminal target model in the speculative decoding process, and thus realize efficient and low-power real-time language generation on a mobile device with limited resources. Moreover, the neural network processor and the computing module communicate with each other through a data bus. The computing module is configured to perform sub-verification tasks corresponding to the plurality of candidate token sequences together with the neural network processor. That is, the computing module can perform part of the task of verifying the plurality of candidate token sequences. In addition, in actual application, the computing module can include a plurality of PIM modules, each PIM module including 4 PIM dies, which share a control / address bus (C / A bus) and a chip select signal (CS), but are connected to separate data buses. Alternatively, each PIM module can be an LPDDR-PIM module, which integrates a PIM unit and an LPDDR5 memory array in the same chip.

[0077] Alternatively, each PIM die includes a PIM control circuit, a plurality of matrix processing units, and a plurality of memory blocks. The PIM control circuit is responsible for the read / write and calculation functions of the PIM. Each matrix processing unit is shared by two adjacent memory blocks and can receive their data for calculation or write the calculation results back to the two memory blocks. Moreover, the matrix processing unit includes a control circuit, a set of instruction registers, a matrix accumulation register, a matrix general-purpose register, a matrix scalar register, and four bound 32-channel vector arithmetic units. In actual application, the control circuit can extract instructions from the instruction registers to control the operation of the matrix processing unit, and the data source of the vector arithmetic unit can be the register and the memory block to which it belongs, which can support INT8 arithmetic operations and INT32 accumulation operations.

[0078] The storage module is configured to store model parameters such as weight matrices, KV cache values, and calculation results of the target model. In actual application, the storage module can include a plurality of storage modules. Alternatively, the storage module can be a plurality of LPDDR modules.

[0079] In the speculative decoding process, multiple candidate token sequences are generated by the draft model at each decoding stage. When verifying the multiple candidate token sequences, the host processor first loads the multiple candidate token sequences corresponding to the current decoding stage output by the draft model and the model parameters corresponding to the target model.

[0080] When verifying the multiple candidate token sequences, the model parameters of the target model are used. To efficiently store and access these model parameters, the multiple storage blocks and storage modules inside the computing module can be used to store the model parameters required in the verification process. Specifically, before starting the verification, the model parameters required in the verification process can be stored in the host disk, or the model parameters can be divided in advance to obtain multiple parameter subsets, and the multiple parameter subsets are initially stored in the multiple storage blocks and storage modules inside the computing module. That is, each parameter subset is a part of the model parameters, and the multiple storage blocks inside the computing module are used to store a part of the model parameters, and the storage module is used to store a part of the model parameters.

[0081] When starting the verification, the host processor loads the model parameters of the target model and then splits the model parameters to split the model parameters into a first parameter subset and a second parameter subset. When splitting the model parameters, the model parameters can be split according to the column dimension of the model parameters to obtain the first parameter subset and the second parameter subset, wherein the first parameter subset contains a first target number of column vectors, and the second parameter subset contains a second target number of column vectors. The model parameters can also be split according to the row dimension of the model parameters to obtain the second parameter subset and the second parameter subset, wherein the first parameter subset contains a third target number of row vectors, and the second parameter subset contains a fourth target number of row vectors. The model parameters can also be split according to the number of layers of the model or according to the number of decoding heads, which is not limited in the embodiments of the present application, and the corresponding splitting method can be selected according to actual needs.

[0082] In the embodiments of the present application, the model parameters are preferably split by column. By dividing the weight matrix of the target model by column, the neural network processor and the computing module can independently output complete dimensional hidden states or prediction results when performing candidate token verification, avoiding intermediate result merging operations across computing units, significantly reducing communication overhead between heterogeneous computing systems, and being particularly suitable for bandwidth-limited mobile end inference scenarios.

[0083] Afterwards, the main processor loads the first parameter subset into the first storage space corresponding to the storage module, and loads the second parameter subset into the second storage space located in the calculation module. In actual application, the storage blocks inside the storage module and the calculation module in the hybrid near-memory computing module can be used as the storage space of the model parameters, so as to store the model parameters required by the neural network processor verification task in the independent storage space on the neural network processor side and store the model parameters required by the calculation model verification task in the independent storage space on the calculation module side. In this way, the occupation of the calculation module to the storage module in the traditional PIM architecture can be solved, so as to affect the data reading of the neural network processing.

[0084] It should be noted that: if the model parameters of the target model are completely stored in the host disk at the beginning, then after the main processor splits the model parameters into the first parameter subset and the second parameter subset, the first parameter subset and the second parameter subset are loaded into the corresponding storage space respectively. If the model parameters of the target model have been split and distributed to the first storage space corresponding to the storage module and the second storage space corresponding to the calculation module at the beginning, then the model parameter subsets stored in the first storage space and the second storage space at the beginning can be migrated based on the first parameter subset and the second parameter subset, so that the first parameter subset is stored in the adjusted first storage space, and the second parameter subset is stored in the adjusted second storage space.

[0085] Then, the neural network processor can read the first parameter subset from the first storage space and verify the plurality of candidate token sequences based on the first parameter subset to obtain a first sub-verification result. That is, the neural network processor independently completes the verification task of the plurality of candidate token sequences using part of the model parameters stored in the first storage space to generate a partial verification result.

[0086] At the same time, the calculation module can read the second parameter subset from the second storage space and verify the plurality of candidate token sequences based on the second parameter subset to obtain a second sub-verification result. That is, the calculation module independently completes the verification task of the plurality of candidate token sequences using part of the model parameters stored in the second storage space to generate a partial verification result.

[0087] Then, the calculation module transmits the second sub-verification result to the neural network processor, so that the neural network module receives the second sub-verification result and determines the verification result corresponding to the plurality of candidate token sequences in the current decoding stage based on the first sub-verification result and the second sub-verification result.

[0088] In an optional embodiment, the neural network processor and the computing module can adopt a tensor parallel verification mechanism. The parameters of the target model are deployed in the respective corresponding storage spaces in slices. For the multiple candidate token sequences generated by the draft model, the neural network processor first completes the inference calculation of the front network based on the first parameter subset to generate the first sub-verification result. The PIM module thereof continues to complete the calculation of the subsequent network layer based on the second parameter subset and transmits the intermediate result to the neural network processor to obtain the final verification output. Through this collaborative verification mode of phased, parallelized, and near-data calculation, the verification efficiency of the candidate sequence and the overall throughput of the system are significantly improved.

[0089] For example, it is assumed that the parameters of each layer of the target model are divided into two parts from the column direction, one part as the first parameter subset and the other part as the second parameter subset, both of which are different columns of the same layer. That is, the first parameter subset includes part of the parameters of each layer of the target model, and the second parameter subset includes the remaining part of the parameters of each layer of the target model. The first parameter subset is loaded into the first storage space corresponding to the neural network processor, and the second parameter subset is loaded into the local storage block inside the computing module, i.e., the second storage space.

[0090] The neural network processor and the computing module can perform parallel calculation on the input corresponding to the same decoding stage to verify multiple candidate token sequences in parallel. Specifically, the neural network processor performs verification calculation on multiple candidate tokens in parallel using part of the model parameters in the first storage space to generate a first sub-verification result as the first stage of the overall verification process. The computing module performs the same verification calculation on multiple candidate tokens in parallel using the remaining part of the model parameters in the second storage space to generate a second sub-verification result as the second stage of the overall verification process. After generating the second sub-verification result, the computing module transmits it to the neural network processor, which aggregates the first sub-verification result and the second sub-verification result to obtain the verification result corresponding to the multiple candidate token sequences of the current decoding stage, which is the third stage of the overall verification process.

[0091] Therefore, in the above scheme, since the parameters required for the verification task are respectively deployed in the independent storage spaces of the neural network processor side and the computing module side when verifying the multiple candidate token sequences output by the draft model, and the neural network processor only uses the first parameter subset in the first storage space to verify the multiple candidate token sequences, and the computing module only uses the second parameter subset in the second storage space to verify the multiple candidate token sequences, the neural network processor and the computing module can perform the verification task in parallel, avoiding mutual interference of parameter loading, significantly reducing the data movement overhead in the verification process, improving the overall inference throughput, effectively utilizing the locality advantage of the near-memory computing architecture, i.e., computation close to data, improving the energy efficiency ratio of data verification, realizing the collaborative optimization of computing and storage resources, thereby improving the large model inference efficiency and system resource utilization of the mobile terminal.

[0092] The above description of the loading system composition and working principle shown in Figure 1 During the introduction process, the main emphasis is on the splitting of model parameters and the verification process of verifying multiple candidate sequences in parallel. Actually, the main processor also includes a candidate token pruner and a data allocation unit. The candidate token pruner is used to prune the candidate token tree corresponding to the multiple candidate token sequences output by the draft model to speed up the speculative inference process of the target model. The data allocation unit is used to determine the parameter allocation ratio allocated to the first storage space corresponding to the neural network processor and the second storage space corresponding to the computing module, to perform data relocation on the model parameters in the first storage space and the second storage space, and then dynamically adjust the workloads corresponding to the neural network processor and the computing module, thereby improving the overall hardware utilization and speeding up the speculative inference process.

[0093] The processing processes of the candidate token pruner and the data allocation unit will be described in detail below.

[0094] Specifically, before the main processor performs the model parameter splitting process, the main processor can first generate a candidate token tree according to the multiple candidate token sequences and transmit the candidate token tree to the candidate token pruner. The candidate tokens can be organized through a tree structure, each node in the candidate token tree represents a candidate token, and each edge connecting the parent node and the child node represents the order relationship between the candidate tokens. In this way, each path from the root node to the leaf node represents a candidate token sequence.

[0095] Suppose the draft model includes two secondary prediction heads, such as Figure 2 the secondary prediction head 1 and the secondary prediction head 2 in Figure 2 , and the target model includes a main prediction head, where the secondary prediction head is used to generate a candidate token sequence, and each prediction head can generate multiple candidate tokens, for example, Figure 2 the secondary prediction head 1 generates 2 candidate tokens t1,1 , t 1,2 The prediction head 2 generates 2 candidate tokens t 2,1 , t 2,2 The main prediction head is the verification result of the final output of the target model. Each sub-prediction head in the draft model can generate a plurality of candidate token sequences corresponding to the current time based on the verification result t0 output by the main prediction head at the last time as input. Then when constructing the candidate token tree, the verification result t0 output by the main prediction head at the last time and the plurality of candidate tokens output by each prediction head at the current time can be used to generate a candidate token tree, such as the candidate token tree shown in FIG. 8. Figure 2

[0096] In actual application, most of the candidate tokens in the tree-shaped candidate token set generated by the draft model will be rejected after verification by the target model, which will result in a large amount of wasted computing power. In the verification process, if each candidate token in the tree-shaped candidate token set is verified at the same time, the amount of calculation is very large, which may affect the speculative reasoning efficiency. Therefore, in the embodiments of the present application, a candidate token pruner is introduced to screen the candidate tokens in the tree-shaped candidate token set generated by the draft model, and then the screened candidate tokens are verified. In this way, not only the computing power can be saved, but also the speculative decoding reasoning speed can be accelerated.

[0097] Specifically, the candidate token pruner prunes the generated candidate token tree to obtain a pruned target candidate token tree. When pruning, pruning may affect the coverage range and semantic integrity of the candidate path in the speculative decoding process. If the pruning strategy is too aggressive, it may cause the legal generated path to be truncated prematurely, thereby increasing the number of verification failures and model rollback overhead, which may weaken the acceleration effect. Therefore, when pruning the candidate token tree, the hardware performance (such as the utilization rate of the calculation unit, the memory bandwidth occupancy rate, the power consumption level, etc.) can be combined, and the token tree structure can be dynamically optimized according to the current running state of the system to balance the reasoning speed and energy efficiency.

[0098] In an optional embodiment, the candidate token pruner includes a candidate token accuracy model and a candidate tree structure explorer. The candidate token accuracy model is a model for evaluating the reliability of each branch (candidate sequence) in the candidate token tree. Its core function is to predict which candidate sequence is more likely to be accepted by the target model, thereby helping to screen high-quality candidate tokens and reducing invalid calculations. The score of each branch in the newly generated candidate token tree can be scored by learning the rules of candidate sequences accepted by the large model in historical data. The higher the score, the more likely the branch is to be recognized by the target model.

[0099] ​The candidate tree structure explorer is responsible for efficiently generating the target candidate token tree. Its core role is to balance the quality of candidates and generation efficiency by ensuring the diversity of candidate sequences while avoiding the generation of excessive redundant branches and reducing invalid exploration.

[0100] In practical applications, the candidate token pruner operates in a closed loop during the pruning process: the verification results of the previous iteration update the accuracy statistics of the secondary prediction head. These statistics, combined with the current device resources and optimization goals, drive the optimization of the candidate token tree. In specific implementation, the candidate token accuracy model is used to obtain historical verification results corresponding to a plurality of candidate token sequences output by the draft model, and according to the historical verification results, the expected acceptance length corresponding to each candidate token sequence output by each decoding head in the draft model is determined. Finally, based on the expected acceptance length corresponding to each candidate token sequence, the expected acceptance length corresponding to the candidate token tree is determined.

[0101] The expected acceptance length corresponding to a candidate token sequence refers to the mathematical expectation value of the number of consecutive word pieces successfully accepted by the target model from the starting position in the sequence, which is used to represent the probability of the target candidate token sequence being accepted by the target model. The expected acceptance length corresponding to the candidate token tree refers to the weighted average value of the length of each path accepted by the target model among all possible paths contained in the tree structure, which is used to represent the expected number of reasoning steps that can be accelerated in the speculative decoding process of the candidate token tree as a whole, reflecting the comprehensive performance of the verification efficiency and generation quality of the tree structure.

[0102] The specific determination process of the expected acceptance length corresponding to each candidate token sequence is roughly the same, and here we only give a detailed description of the determination of the expected acceptance length by taking the target candidate token sequence as an example. Among them, the target candidate token sequence is any one of the plurality of candidate token sequences.

[0103] For the target candidate token sequence, the expected accuracy of each candidate token in the target candidate token sequence is determined according to the historical verification results. The product value between the expected accuracy of each candidate token is obtained, and the product value is determined as the expected acceptance length of the target candidate token sequence. The expected acceptance length of the target candidate token sequence is used to represent the probability of the target candidate token sequence being accepted by the target model. The sum value between the expected acceptance lengths of the plurality of candidate token sequences is obtained, and the sum value is determined as the expected acceptance length corresponding to the candidate token tree.

[0104] That is, the expected accuracy of each prediction result in each secondary prediction in the draft model can be tracked after each decoding, for example, the expected accuracy of the kth prediction result in the ith secondary prediction head is tracked after each decoding (the main prediction head is always correct, i.e. ). And based on the previous validation results, the prediction accuracy of each prediction result in each secondary prediction head is updated after each decoding, so as to determine the prediction accuracy of each candidate token corresponding to the current decoding stage. By recording the historical accuracy of each prediction result in each secondary prediction head, the model can remember which guesses are more likely to be correct, so as to make the final generated target candidate token tree more accurate and cut off useless branches, which does not affect the accuracy of the inference result of the target model, but also reduces the waste of computing resources.

[0105] For example, when writing a sentence: the next word verified by the primary prediction head is "weather", which is assumed to be correct with a probability of 0.9, and the accuracy is recorded as The secondary prediction head 1 (auxiliary idea 1) generates two prediction results: the first prediction result is "sunny", and the second prediction result is "rainy". According to the previous validation results, "sunny" has been accepted 3 times and rejected 1 time, so its prediction accuracy is "Rainy" has been accepted 1 time and rejected 2 times, and the prediction accuracy is The secondary prediction head 2 (auxiliary idea 2) generates two prediction results: "warm" and "cold". Similarly, the prediction accuracy of each is calculated according to the historical validation results.

[0106] After determining the prediction accuracy of each candidate token, the product value between the prediction accuracy of each candidate token in the target candidate token sequence is obtained, and the product value is determined as the expected acceptance length of the target candidate token sequence. For example, for the target candidate sequence, the expected acceptance length is the product of the prediction accuracy of each candidate token in the target candidate token sequence, that is, where t i represents a candidate token in the target candidate token sequence, for example, For the entire candidate token tree, the expected acceptance length is the sum of the prediction accuracy of all candidate tokens.

[0107] When the candidate token accuracy model determines the expected acceptance length corresponding to the candidate token tree, the candidate tree structure explorer can perform pruning on the candidate token tree according to the expected acceptance length corresponding to the candidate token tree, the total transmission delay corresponding to the current decoding stage, and the total energy consumption corresponding to the current decoding stage, to obtain the pruned target candidate token tree.

[0108] where the total transmission delay is the total delay of data transmission of the neural network processor and the computing module in the current decoding stage, and the total energy consumption is the total energy consumption of data processing and the total energy consumption of data transmission of the neural network processor and the computing module in the current decoding stage.

[0109] In an optional embodiment, the candidate tree structure explorer is specifically configured to: generate a set of candidate tokens to be expanded, the set of candidate tokens to be expanded comprising a plurality of candidate tokens in the candidate token tree; select, according to the greedy sampling, a target candidate token with a highest prediction accuracy in the current iteration from the set of candidate tokens to be expanded, and add the target candidate token to the candidate token tree being currently constructed as a temporary node; determine a total transmission delay corresponding to the verification of the candidate token tree being currently constructed and a total energy consumption required for the verification of the candidate token tree being currently constructed; perform energy consumption and performance verification on the candidate token tree being currently constructed according to the total transmission delay corresponding to the verification of the candidate token tree being currently constructed and the total energy consumption required for the verification of the candidate token tree being currently constructed; if the verification is passed, retain the target candidate token in the candidate token tree being currently constructed as a valid node, remove the target candidate token from the set of candidate tokens to be expanded, and continue to repeatedly perform the above selection and verification operations on the remaining candidate tokens; and if the verification is not passed, remove the target candidate token from the candidate token tree being currently constructed, and determine the candidate token tree after the removal as the target candidate token tree.

[0110] The hardware evaluation model can be used to determine the total transmission delay corresponding to the verification of the candidate token tree being currently constructed and the total energy consumption required for the verification of the candidate token tree being currently constructed. The hardware evaluation model can include a performance model and a power consumption model. The performance model can quantify the limitation of the parallel execution of the neural network processor-computing module. The power consumption model can quantify the computing power consumption of the neural network processor and the computing module and the power consumption overhead of the internal / external data transmission when the candidate token tree being currently constructed is calculated. The performance model can be used to determine the total transmission delay corresponding to the verification of the candidate token tree being currently constructed in each iteration. Specifically, the power consumption model can be used to determine the total energy consumption required for the verification of the candidate token tree being currently constructed in each iteration.

[0111] That is, in the pruning process, the candidate tree structure explorer adopts a hardware-based greedy strategy to construct the optimized target candidate token tree from the root node to the leaf node. In each sampling process, the candidate token tree explorer adds the candidate token with the highest prediction accuracy in the current sub-prediction header to the current candidate token tree. Then, the prediction reasoning performance and energy consumption are evaluated according to the expected acceptance length of the current candidate token tree, and it is determined whether to accept the candidate token according to the optimization target. If accepted, the candidate token with the highest prediction accuracy in the current sub-prediction header is selected from the remaining candidate tokens and added to the current candidate token tree to update the current candidate token tree. The method is repeatedly used to verify whether to accept the candidate token until the currently added candidate token is no longer accepted. This dynamic pruning process generates an optimized token tree for the subsequent decoding step.

[0112] In the method, the candidate tree structure explorer determines a first parameter access amount required by the neural network processor when verifying the current constructed candidate token tree and a second parameter access amount required by the calculation module when verifying the current constructed candidate token tree; determines a first transmission delay corresponding to the neural network processor when verifying the current constructed candidate token tree according to the first parameter access amount and a first data transmission rate between the neural network processor and the calculation module; determines a second transmission delay corresponding to the calculation module when verifying the current constructed candidate token tree according to the second parameter access amount and a second data transmission rate within the calculation module; and determines the total transmission delay corresponding to the current constructed candidate token tree as a larger value between the first transmission delay and the second transmission delay.

[0113] An optional implementation of the candidate tree structure explorer determining the first transmission delay corresponding to the neural network processor when verifying the current constructed candidate token tree according to the first parameter access amount and the first data transmission rate between the neural network processor and the calculation module can be: determining a first cache capacity occupied when caching the model parameters corresponding to the first parameter access amount; obtaining a first ratio of the first cache capacity to the first data transmission rate, and determining the first ratio as the first transmission delay corresponding to the neural network processor when verifying the current constructed candidate token tree.

[0114] An optional implementation of the candidate tree structure explorer determining the second transmission delay corresponding to the calculation module when verifying the current constructed candidate token tree according to the second parameter access amount and the second data transmission rate within the calculation module can be: determining a second cache capacity occupied when caching the model parameters corresponding to the second parameter access amount; determining the number of arithmetic units in each matrix processing unit in the calculation module; determining the number of target candidate tokens contained in the current constructed candidate token tree; obtaining a second ratio of the second cache capacity to the second data transmission rate; obtaining a third ratio of the number of target candidate tokens to the number of arithmetic units; determining a product value of the second ratio and the third ratio; and determining the product value as the second transmission delay corresponding to the calculation module when verifying the current constructed candidate token tree.

[0115] For example, the following formula can be used to calculate the total transmission delay corresponding to the neural network processor and the calculation module when verifying the current constructed candidate token tree. Ttotal = min(T NPU , T PIM ). In the formula, the neural network processor can be modeled using a rooline roof model, in which the calculation throughput is limited by the off-chip memory bandwidth. For the calculation module, the internal bandwidth matches its calculation performance. Therefore, the execution delay of the neural network processor can be modeled as follows:

[0116] where N params,DRAM represents the parameter weight / key-value cache capacity required to be used by the neural network processor when performing the verification computation on the current constructed candidate token tree. BW Off-chip represents the external bandwidth of the hybrid near-memory computation module.

[0117] The execution latency of the computation module can be modeled as follows: where N params,PIM represents the parameter weight / key-value cache capacity required to be used by the computation module when performing the verification computation on the current constructed candidate token tree. BW PIM represents the internal bandwidth of the hybrid near-memory computation module. N ALU represents the number of arithmetic units in each matrix processing unit within the computation module. L spec represents the number of candidate tokens in the current constructed candidate token tree.

[0118] The specific implementation process of determining the total energy consumption required by the candidate tree structure explorer when verifying the current constructed candidate token tree is introduced as follows.

[0119] The candidate tree structure explorer first determines the first computation amount corresponding to the neural network processor when verifying the current constructed candidate token tree and the second computation amount corresponding to the computation module when verifying the current constructed candidate token tree. Then, according to the first computation amount corresponding to the neural network processor when verifying the current constructed candidate token tree and the second computation amount corresponding to the computation module when verifying the current constructed candidate token tree, the total computation consumption required when verifying the current constructed candidate token tree is determined. Further, the data transmission amount corresponding to verifying the current constructed candidate token tree is determined, and according to the data transmission amount corresponding to verifying the current constructed candidate token tree and the data transmission power consumption corresponding to unit data, the total data transmission power consumption required when verifying the current constructed candidate token tree is determined.

[0120] For example, the total data transmission power consumption required when verifying the current constructed candidate token tree = the first computation amount corresponding to the neural network processor when verifying the current constructed candidate token tree x the computation power consumption of the neural network processor per time + the second computation amount corresponding to the computation module when verifying the current constructed candidate token tree x the computation power consumption of the computation module per time + the data transmission amount corresponding to verifying the current constructed candidate token tree x the data transmission power consumption per time.

[0121] In addition, the average transmission delay borne by the unit expected acceptance length can be further calculated, which can reflect the communication resources consumed by each unit of speculative acceleration effect, so as to realize the fair evaluation and efficient screening of different candidate token tree structures. When the transmission delay corresponding to the unit acceptance length exceeds the preset threshold, it indicates that the communication overhead of the currently constructed candidate token tree is too high relative to its acceleration benefit, at which time the pruning mechanism is triggered to improve the overall energy efficiency ratio and reasoning efficiency of the system.

[0122] Specifically, in order to accurately evaluate the communication overhead of the currently constructed candidate token tree relative to its acceleration benefit, in the embodiments of the present application, the candidate tree structure explorer is specifically configured to: determine the transmission delay corresponding to the unit acceptance length of the current decoding stage according to the total transmission delay and the expected acceptance length corresponding to the candidate token tree; determine the energy consumption corresponding to the unit acceptance length of the current decoding stage according to the total energy consumption and the expected acceptance length corresponding to the candidate token tree; determine whether the transmission delay corresponding to the unit acceptance length of the current decoding stage and the energy consumption corresponding to the unit acceptance length of the current decoding stage meet the preset condition; if the preset condition is met, it is confirmed that the energy consumption and performance verification of the currently constructed candidate token tree is passed. Wherein, the preset condition is a preset optimization target.

[0123] From the above description, it can be known that the candidate token accuracy model is used to screen out the most likely accepted branch, the candidate tree structure explorer generates a reasonable target candidate token tree structure, and finally reduces the verification cost of the target model and the computing power consumption, that is, by combining the candidate token accuracy model and the candidate tree structure explorer, the efficiency of speculative decoding can be significantly improved.

[0124] In addition, the speculation length corresponding to the target candidate token tree determined by each decoding stage during system operation is different, and a workload scheduler combined with data relocation can be introduced to fully utilize the neural network processor-computing module parallel execution and maximize hardware utilization.

[0125] Specifically, after the candidate token pruner generates the optimized target candidate token tree, the data allocation unit determines the parameter allocation ratio of the neural network processor and the computing module corresponding to the current decoding stage according to the target candidate token tree. The main processor re-divides the model parameters based on the determined parameter allocation ratio of the neural network processor and the computing module corresponding to the current decoding stage, to divide the model parameters into a first parameter subset and a second parameter subset, to relocate the data, migrate the model parameters stored in the original first storage space and second storage space, and also use the workload scheduler to schedule the neural network processor-computing module parallel execution verification subtasks.

[0126] In an optional implementation, the data allocation unit determines the target token quantity contained in the target candidate token tree, acquires a model parameter allocation table, in which a correspondence between candidate token quantities and preset parameter allocation ratios is stored, queries the preset parameter allocation ratio matching the target token quantity from the model parameter allocation table, and determines the parameter allocation ratio corresponding to the current decoding stage of the neural network processor and the computing module according to the preset parameter allocation ratio.

[0127] In addition, if the model parameters in the first storage space and the second storage space are relocated and data migration is performed at each decoding stage, a large amount of data migration overhead will be caused, and the execution efficiency of the parallel verification task of the neural network processor and the computing module will be affected. In an optional embodiment, the data allocation unit further includes a counter, and the counter can be used to record whether the parameter allocation ratio corresponding to each decoding stage is the same as the parameter allocation ratio corresponding to the previous decoding stage. When the parameter allocation ratio is not the same for a plurality of consecutive times, the model parameters in the first storage space and the second storage space are relocated.

[0128] Therefore, after the data allocation unit determines the parameter allocation ratio corresponding to the current decoding stage each time, whether to perform data relocation can be determined in combination with the count value corresponding to the counter. Specifically, the current count value corresponding to the counter is acquired. If the count value meets a preset value, the preset parameter allocation ratio is determined as the parameter allocation ratio corresponding to the current decoding stage of the neural network processor and the computing module, and data migration is performed. If the count value does not meet the preset value, the parameter allocation ratio corresponding to the previous decoding stage of the neural network processor and the computing module is determined as the parameter allocation ratio corresponding to the current decoding stage of the neural network processor and the computing module, and data migration is not performed.

[0129] That is, the number of mismatches can be recorded using a counter. Only when the mismatch occurs for a plurality of consecutive times and it is confirmed that the mismatch is not accidental, the allocation mode is actually switched. In this way, the resources can be allocated reasonably, the trouble caused by frequent switching can be avoided, and the efficiency can be improved.

[0130] The model speculation reasoning loading system is described above. The following describes an optional model speculation reasoning acceleration execution process in combination with the following embodiments.

[0131] Figure 3A flowchart of a model speculative reasoning loading method provided by an embodiment of the present application is provided. The method is applied to a model speculative reasoning acceleration system, which includes a host processor, a neural network processor, and a hybrid near-memory computing module. The hybrid near-memory computing module includes a computing module and a storage module. The model speculative reasoning loading method provided by the embodiment corresponds to the host processor as an execution subject. Specifically, the method can include the following steps:

[0132] 301. Load the multiple candidate token sequences output by the current decoding stage of the draft model and the model parameters of the target model, and split the model parameters into a first parameter subset and a second parameter subset.

[0133] 302. Load the first parameter subset into a first storage space corresponding to the storage module, so that the neural network processor verifies the multiple candidate token sequences based on the first parameter subset in the first storage space to obtain a first sub-verification result.

[0134] 303. Load the second parameter subset into a second storage space located in the computing module, so that the computing module verifies the multiple candidate token sequences based on the second parameter subset in the second storage space to obtain a second sub-verification result, and transmits the second sub-verification result to the neural network processor, so that the neural network processor determines the verification result corresponding to the multiple candidate token sequences of the current decoding stage based on the first sub-verification result and the second sub-verification result.

[0135] The host processor loads the multiple candidate token sequences output by the current decoding stage of the draft model and the model parameters of the target model, and splits the model parameters into a first parameter subset and a second parameter subset. Then, the host processor loads the first parameter subset into a first storage space corresponding to the storage module, so that the neural network processor verifies the multiple candidate token sequences based on the first parameter subset in the first storage space to obtain a first sub-verification result. At the same time, the host processor loads the second parameter subset into a second storage space located in the computing module, so that the computing module verifies the multiple candidate token sequences based on the second parameter subset in the second storage space to obtain a second sub-verification result, and transmits the second sub-verification result to the neural network processor, so that the neural network processor determines the verification result corresponding to the multiple candidate token sequences of the current decoding stage based on the first sub-verification result and the second sub-verification result.

[0136] The method further comprises, before the main processor splits the decoding layer parameter matrix into the first parameter subset and the second parameter subset, the main processor generating a candidate token tree according to a plurality of candidate token sequences, and transmitting the candidate token tree to a candidate token pruner. The candidate token pruner prunes the candidate token tree to obtain a pruned target candidate token tree. The data allocation unit determines a parameter allocation ratio corresponding to a current decoding stage of the neural network processor and the second processing unit according to the target candidate token tree. Then, the main processor splits the decoding layer parameter matrix into the first parameter subset and the second parameter subset based on the parameter allocation ratio corresponding to the current decoding stage.

[0137] The candidate token pruner comprises a candidate token accuracy model and a candidate tree structure explorer. An optional specific implementation manner in which the candidate token pruner prunes the candidate token tree to obtain the pruned target candidate token tree can be: obtaining, by the candidate token accuracy model, historical verification results corresponding to the plurality of candidate token sequences output by the draft model; determining, by the candidate token accuracy model according to the historical verification results, an expected acceptance length corresponding to each candidate token sequence output by each decoding head in the draft model, and determining an expected acceptance length corresponding to the candidate token tree based on the expected acceptance lengths corresponding to the plurality of candidate token sequences; and pruning, by the candidate tree structure explorer according to the expected acceptance length corresponding to the candidate token tree, a total transmission delay corresponding to the current decoding stage, and a total energy consumption corresponding to the current decoding stage, the candidate token tree to obtain the pruned target candidate token tree. The total transmission delay is a total delay corresponding to data transmission of the neural network processor and the computing module at the current decoding stage, and the total energy consumption is a total energy consumption corresponding to data processing of the neural network processor and the computing module at the current decoding stage and a total energy consumption consumed during data transmission.

[0138] An optional specific implementation manner in which the candidate token accuracy model determines the expected acceptance length corresponding to each candidate token sequence output by each decoding head in the draft model according to the historical verification results, and determines the expected acceptance length corresponding to the candidate token tree based on the expected acceptance lengths corresponding to the plurality of candidate token sequences can be: for a target candidate token sequence, determining, according to the historical verification results, a speculation accuracy corresponding to each candidate token in the target candidate token sequence, the target candidate token sequence being any one of the plurality of candidate token sequences; obtaining a product value between the speculation accuracies corresponding to the plurality of candidate tokens, and determining the product value as an expected acceptance length of the target candidate token sequence, the expected acceptance length of the target candidate token sequence being used to represent a probability that the target candidate token sequence is accepted by the target model; and obtaining a sum value between the expected acceptance lengths of the plurality of candidate token sequences, and determining the sum value as the expected acceptance length corresponding to the candidate token tree.

[0139] In an optional specific implementation manner, the candidate token tree is pruned according to an expected acceptance length corresponding to the candidate token tree, a total transmission delay corresponding to a current decoding stage, and a total energy consumption corresponding to the current decoding stage, to obtain the pruned target candidate token tree. The optional specific implementation manner can be: generating a set of candidate tokens to be expanded, the set of candidate tokens to be expanded including a plurality of candidate tokens in the candidate token tree; selecting, according to greedy sampling, a target candidate token with a highest current prediction accuracy from the set of candidate tokens to be expanded, and adding the target candidate token as a temporary node to the candidate token tree currently constructed; determining a total transmission delay corresponding to verification of the candidate token tree currently constructed and a total energy consumption required for verification of the candidate token tree currently constructed; performing energy consumption and performance verification on the candidate token tree currently constructed according to the total transmission delay and the total energy consumption; if the verification is passed, retaining the target candidate token as a valid node in the candidate token tree currently constructed, removing the target candidate token from the set of candidate tokens to be expanded, and continuing to repeatedly perform the selecting and verifying operations on the remaining candidate tokens; and if the verification is not passed, removing the target candidate token from the candidate token tree currently constructed, and determining the candidate token tree after the removal as the target candidate token tree.

[0140] In an optional specific implementation manner of determining the total transmission delay corresponding to the verification of the candidate token tree currently constructed, a first parameter access amount required for the neural network processor to verify the candidate token tree currently constructed and a second parameter access amount required for the computing module to verify the candidate token tree currently constructed are determined; a first transmission delay corresponding to the verification of the candidate token tree currently constructed by the neural network processor is determined according to the first parameter access amount and a first data transmission rate between the neural network processor and the computing module; a second transmission delay corresponding to the verification of the candidate token tree currently constructed by the computing module is determined according to the second parameter access amount and a second data transmission rate within the computing module; and the greater value between the first transmission delay and the second transmission delay is determined as the total transmission delay corresponding to the verification of the candidate token tree currently constructed.

[0141] In an optional specific implementation manner of determining the first transmission delay corresponding to the verification of the candidate token tree currently constructed by the neural network processor according to the first parameter access amount and the first data transmission rate between the neural network processor and the computing module, a first cache capacity occupied when the model parameters corresponding to the first parameter access amount are cached is determined; a first ratio of the first cache capacity to the first data transmission rate is obtained, and the first ratio is determined as the first transmission delay corresponding to the verification of the candidate token tree currently constructed by the neural network processor.

[0142] The specific implementation process of determining the second transmission delay corresponding to the verification of the candidate token tree currently constructed by the calculation module according to the second data transmission rate between the second parameter access amount and the calculation module can be: determining the second cache capacity occupied when the model parameters corresponding to the second parameter access amount are determined; determining the number of arithmetic units in each matrix processing unit in the calculation module; determining the number of target candidate tokens contained in the candidate token tree currently constructed; obtaining a second ratio of the second cache capacity and the second data transmission rate; obtaining a third ratio of the number of target candidate tokens and the number of arithmetic units; determining a product value of the second ratio and the third ratio; and determining the product value as the second transmission delay corresponding to the verification of the candidate token tree currently constructed by the calculation module.

[0143] An optional specific implementation of determining the total energy consumption required for verifying the candidate token tree currently constructed can be: determining a first calculation amount corresponding to the neural network processor when verifying the candidate token tree currently constructed and a second calculation amount corresponding to the calculation module when verifying the candidate token tree currently constructed; determining the total calculation consumption required for verifying the candidate token tree currently constructed according to the first calculation amount and the second calculation amount; determining the data transmission amount corresponding to the verification of the candidate token tree currently constructed; determining the total data transmission power required for verifying the candidate token tree currently constructed according to the data transmission amount and the data transmission power corresponding to the unit data; and determining the total energy consumption required for verifying the candidate token tree currently constructed according to the total calculation consumption and the total data transmission power.

[0144] An optional specific implementation of performing energy consumption and performance verification on the candidate token tree currently constructed according to the total transmission delay and the total energy consumption can be: determining the transmission delay corresponding to the unit acceptance length of the current decoding stage according to the total transmission delay and the expected acceptance length corresponding to the candidate token tree; determining the energy consumption corresponding to the unit acceptance length of the current decoding stage according to the total energy consumption and the expected acceptance length corresponding to the candidate token tree; judging whether the transmission delay corresponding to the unit acceptance length of the current decoding stage and the energy consumption corresponding to the unit acceptance length of the current decoding stage meet a preset condition; and if the preset condition is met, confirming that the energy consumption and performance verification on the candidate token tree currently constructed is passed.

[0145] An optional specific implementation of determining the parameter allocation ratio corresponding to the current decoding stage of the neural network processor and the calculation module according to the target candidate token tree by the data allocation unit can be: determining the number of target tokens contained in the target candidate token tree; obtaining a model parameter allocation table, the model parameter allocation table storing a corresponding relationship between the number of candidate tokens and a preset parameter allocation ratio; querying the preset parameter allocation ratio matching the number of target tokens from the model parameter allocation table; and determining the parameter allocation ratio corresponding to the current decoding stage of the neural network processor and the calculation module according to the preset parameter allocation ratio.

[0146] In the data distribution unit, the counter is further included. In an optional specific implementation, the parameter distribution ratio corresponding to the current decoding stage of the neural network processor and the computing module is determined as follows: a current count value corresponding to the counter is obtained; if the count value meets a preset value, the preset parameter distribution ratio is determined as the parameter distribution ratio corresponding to the current decoding stage of the neural network processor and the computing module; if the count value does not meet the preset value, the parameter distribution ratio corresponding to the neural network processor and the computing module in the previous decoding stage is determined as the parameter distribution ratio corresponding to the current decoding stage of the neural network processor and the computing module.

[0147] The related content not described in the embodiment can be referred to the related description in the foregoing embodiments, which will not be described herein.

[0148] To sum up, by using the storage blocks in the storage module and the computing module in the hybrid near-memory computing module as the storage space of the model parameter, and jointly using the neural network processor and the computing module to participate in the parallel verification of the multiple candidate token sequences, the overall inference throughput and resource utilization are improved, the memory access delay and power consumption in the verification process are significantly reduced, the collaborative optimization of the computing and storage resources is realized, and the large model inference efficiency of the mobile terminal and the system resource utilization are improved.

[0149] Figure 4 Another flowchart of a model speculative inference loading method provided by the embodiment of the application is provided. The method is applied to an acceleration system of model speculative inference. The system includes a host processor, a neural network processor, and a hybrid near-memory computing module. The hybrid near-memory computing module includes a computing module and a storage module. The model speculative inference loading method provided by the embodiment of the application corresponds to the neural network processor as an execution subject. Specifically, the method can include the following steps:

[0150] 401. Verify the multiple candidate token sequences output by the current decoding stage of the draft model based on the first parameter subset in the first storage space corresponding to the storage module, to obtain a first sub-verification result. The first parameter subset is loaded into the first storage space corresponding to the storage module by the host processor splitting the model parameters of the target model.

[0151] 402. Receive the second sub-verification result sent by the computing module, and determine the verification result corresponding to the multiple candidate token sequences in the current decoding stage based on the first sub-verification result and the second sub-verification result. The second sub-verification result is obtained by the computing module verifying the multiple candidate token sequences based on the second parameter subset in the second storage space in the computing module. The second parameter subset is loaded into the second storage space corresponding to the computing module by the host processor splitting the model parameters of the target model.

[0152] The neural network processor can read the first parameter subset in the first storage space corresponding to the storage module to verify the plurality of candidate token sequences output by the current decoding stage of the draft model, to obtain a first sub-verification result. The second sub-verification result sent by the computing module is received, and based on the first sub-verification result and the second sub-verification result, a verification result corresponding to the plurality of candidate token sequences of the current decoding stage is determined.

[0153] The related content not described in the embodiment is described in the foregoing embodiments, and will not be described here.

[0154] Figure 5 A flowchart of another model speculation reasoning loading method provided by the embodiment of the application is provided. The method is applied to a model speculation reasoning acceleration system, and the system includes a host processor, a neural network processor, and a hybrid near-memory computing module. The hybrid near-memory computing module includes a computing module and a storage module. The model speculation reasoning loading method provided by the embodiment has a corresponding execution subject, which is the computing module. Specifically, the method can include the following steps:

[0155] 501. Verify the plurality of candidate token sequences output by the current decoding stage of the draft model based on the first parameter subset in the first storage space located in the computing module, to obtain a first sub-verification result. The first parameter subset is loaded into the first storage space corresponding to the computing module by the host processor splitting the model parameters of the target model.

[0156] 502. Transmit the first sub-verification result to the neural network processor, so that the neural network processor determines a verification result corresponding to the plurality of candidate token sequences of the current decoding stage based on the first sub-verification result and a second sub-verification result.

[0157] The computing module can verify the plurality of candidate token sequences output by the current decoding stage of the draft model based on the first parameter subset in the first storage space located in the computing module, to obtain a first sub-verification result. The first sub-verification result is transmitted to the neural network processor, so that the neural network processor determines a verification result corresponding to the plurality of candidate token sequences of the current decoding stage based on the first sub-verification result and a second sub-verification result.

[0158] The second sub-verification result is obtained by the neural network processor verifying the plurality of candidate token sequences based on the second parameter subset in the second storage space corresponding to the storage module. The second parameter subset is loaded into the second storage space corresponding to the storage module by the host processor splitting the model parameters of the target model.

[0159] It should be noted that the first storage space and the second storage space are named for the convenience of description, and are not the first storage space and the second storage space in the above-mentioned embodiments. Similarly, the first parameter set and the second parameter set are named for the convenience of description, and are not the first parameter set and the second parameter set in the above-mentioned embodiments.

[0160] The related content not described in the embodiment can refer to the related description in the foregoing embodiments, and will not be described here.

[0161] In order to facilitate understanding of the acceleration scheme of the model speculation reasoning provided above, the specific implementation process of the scheme will be illustrated below by combining the specific application scenarios shown in the accompanying drawings. Figures 6-8 The specific implementation process of the scheme will be illustrated below by combining the specific application scenarios shown in the accompanying drawings.

[0162] The model speculation reasoning acceleration system combines a mobile SoC (System on Chip) with a hybrid LPDDR5-PIM (Low Power Double Data Rate, fifth generation, processing in memory) module. The SoC includes a mobile NPU (Neural Network Processor) and a host CPU, and the communication between the two is through a 64-bit data bus. The NPU architecture is referenced from the design of the mobile NPU and has a temporary storage area memory, a vector unit, and 16 computing units, each equipped with a matrix unit and a local buffer. To mitigate the impact of PIM devices on DRAM capacity, the hybrid LPDDR5-PIM module integrates multiple DRAM modules and multiple PIM modules, with a total capacity of 16 GB, thus achieving a balance between cost and performance. This memory capacity makes it possible to deploy edge large language models (LLMs) at INT8 data precision.

[0163] Each DRAM module or PIM module is composed of four chips, which operate synchronously through the same command and address (C / A) signals and chip selection (CS) signals, while using different data lines, following the JEDEC LPDDR5 specification. In addition, the hybrid LPDDR5-PIM module also contains a near data memory controller (NMC) that can dynamically reallocate data between DRAM and PIM ranks at low cost.

[0164] The host integrates an LP-Spec scheduler, which is composed of a draft token pruner (DTP) and a data allocation unit (DAU). The DTP generates an optimized token tree after each decoding iteration, resulting in different speculation lengths. The DAU dynamically reallocates model weights between DRAM and PIM ranks through the NMC, ensuring the synchronized parallel execution of PIM and NPU devices.

[0165] Figure 6The left side details the design of the LPDDR5-PIM chip. Based on the existing LPDDR5-PIM microarchitecture and instruction set architecture (ISA), the PIM function is extended and enhanced to accelerate speculative reasoning while addressing the constraints of mobile devices. Each PIM chip integrates PIM control logic, 16 memory blocks, and 8 matrix processing units (MPUs), each of which shares between two memory blocks.

[0166] An MPU is composed of three key components: (1) four 32-channel SIMD arithmetic logic units (ALUs), (2) command registers, general-purpose registers, accumulation registers, scalar registers (CRF, GRF, ARF, and SRF), and a controller. Given the popularity of quantized LLM models in edge deployments, the 32-wide ALU has 32 INT8 multipliers and adders. Since an INT8 MAC unit only occupies 26.5% of the area and consumes 63.6% of the energy of an FP16 MAC in a 20nm DRAM process, four 32-wide ALUs are integrated while maintaining an acceptable hardware cost.

[0167] The registers include 32 32-bit CRFs, 16 4x256-bit matrix GRFs, 16 4x8-bit matrix SRFs, and 8 4x1024-bit matrix ARFs. The ARF extends the precision from INT8 to INT32 for accumulation. The controller fetches PIM instructions from the CRF and schedules the MPU accordingly. Similar to Samsung's LPDDR5-PIM, the MPU supports source operands from registers and DRAM memory modules, with data from memory modules broadcast to all four ALUs for parallel computation.

[0168] Since most of the power consumption during PIM execution comes from DRAM access (>90%), compared to HBM-PIM, the MPU power consumption only increases by 23.2%, which is still lower than the DRAM power consumption budget. For the LPDDR5-PIM architecture, broadcast operations can be easily implemented by enabling all memory modules and activating all CS signals. The NMC has two key functions: providing parallel PIM computation and standard DRAM access for the SoC, and efficiently redistributing data between PIM and DRAM levels. To support simultaneous PIM computation and DRAM access, the NMC receives independent C / A signals for the DRAM level and the PIM level, enabling parallel operation.

[0169] Since the SoC can only access either the DRAM memory space or the PIM memory space at any given time, the PIM and DRAM can share the same DQ lines. However, larger input matrices in speculative inference can require reloading of partial sums of inputs and outputs, causing data bus contention with the NPU. To solve this problem, the NMC integrates a dedicated 4KB PIM global buffer for PIM read and write operations. Unused C / A pins are repurposed as the address for this global buffer. When the execution time of the NPU and the PIM is unbalanced, the model parameters need to be reallocated between the DRAM storage space and the PIM storage space. A simple approach is to read the model parameters to the host and then write them back to the DRAM storage space or the PIM storage space, which introduces additional latency and IO energy consumption. In contrast, the NMC introduces a feed-forward path from the read data buffer to the write data arbiter, so that the data read from the DRAM / PIM can be directly copied to the PIM / DRAM. To support the function of the NMC, additional tag bits are needed for guidance, and the timing constraints of the DRAM must be carefully considered.

[0170] The model speculative decoding acceleration process is described in detail below. First, the host processor loads the multiple candidate token sequences output by the current decoding stage of the draft model and the model parameters of the target model, and splits the model parameters into a first parameter subset and a second parameter subset. Specifically, the model parameters (weight matrix) of the target model are divided by column, and when the matrix multiplication involved in the verification process is performed, the neural network processor and the computing module perform verification calculation based on the respective partial model parameters, respectively, to obtain partial verification results, and finally obtain the verification results through aggregation.

[0171] Note that the NPU and the PIM are not verifying different candidate tokens, but are jointly verifying different calculation stages of the same candidate token sequence.

[0172] When the computing module performs verification processing in parallel with the neural network processor, the NPU and the PIM form a heterogeneous collaborative verification architecture, which jointly completes efficient verification of multiple candidate tokens through tensor parallelism and parameter slicing, accelerating the speculative decoding inference process of the model.

[0173] In addition, before splitting the model parameters, the corresponding parameter allocation ratio in the DRAM storage space and the PIM storage space can be determined. Specifically, Figure 7 The overall data flow of workload scheduling is shown, and the design of the candidate token pruner and the data allocation unit in the host is also shown. The candidate token pruner includes a candidate token tree accuracy model and a candidate tree structure explorer. For the candidate token tree accuracy model, the present application determines the accuracy of the tree by estimating the expected length of the candidate token tree. For example, Figure 7As shown, the present application tracks the accuracy of the k-th prediction in the i-th secondary prediction head after each decoding based on the previous validation results (the primary prediction head is always correct, i.e. ). For a token ti in the candidate sequence, its expected acceptance length is the product of the accuracy of the token in the sequence and the expected acceptance length of the sequence, For example, For the entire tree, the expected acceptance length is the sum of all token nodes.

[0174] For the hardware evaluation model in the candidate token tree explorer, it combines the performance model and the power consumption model to perform the overhead-aware token tree optimization. The performance model quantifies the limitations of NPU-PIM parallel execution, and the overall delay can be represented as Ttotal = min(T NPU ,T PIM ). Among them, the NPU can be modeled by the roof model, because its computing throughput is limited by the off-chip bandwidth. For PIM, its internal bandwidth matches its computing performance. Therefore, their delays can be represented as where N params,DRAM and N params,PIM represent the weight / key value cache capacity used in NPU and PIM computation, respectively. BW Off-chip and BW PIM are the external and internal bandwidths of LPDDR5-PIM, respectively, N ALU is the number of ALUs in each matrix processing unit in PIM, which is 4 in the present application. L spec represents the number of candidate tokens. For the energy model, it covers the power consumption overhead of PIM and NPU computation and internal / external data transmission.

[0175] In the pruning process, the candidate token pruner runs in a closed loop: the verification results of the previous iteration update the accuracy statistics of the secondary prediction head. These statistics, combined with the current device resources and optimization goals, drive the optimization of the candidate token tree. The candidate token tree explorer (TTE) adopts a hardware-based greedy strategy to build the optimized token tree from the root node to the leaf node. In each sampling step, TTE adds the node with the highest prediction accuracy to the tree. Subsequently, the hardware evaluation model evaluates the performance and energy consumption of the speculative inference based on the expected acceptance length of the current candidate token tree, and decides whether to accept the node according to the optimization goal. This dynamic pruning process generates an optimized token tree for subsequent decoding steps.

[0176] For candidate trees with different numbers of candidate tokens, fixed workload allocation can reduce hardware utilization and performance, so the application designs a data allocation unit to dynamically allocate workload. The data allocation unit receives the optimized candidate token tree and dynamically reallocates workload according to the number of candidate tokens to improve hardware resource utilization. It formulates different optimal allocation ratios for different numbers of candidates according to hardware characteristics. In order to prevent the data migration overhead caused by frequent switching, the data allocation unit also maintains a 2-bit saturation counter. Only when the number of times that the group ID to which the current candidate token belongs is different from the currently determined allocation group ID exceeds a certain number, the data DRAM and PIM memory space are reallocated.

[0177] Figure 8 Two examples of dynamic workload scheduling are shown. When the data allocation unit is activated, the specific model parameters are reallocated between DRAM and PIM storage space. During the reallocation, the NPU reads the weights from the corresponding DRAM / PIM storage space for calculation, while migrating data to PIM / DRAM storage space, thereby effectively overlapping the latency of NPU computation and parameter reallocation. Compared with parallel execution without dynamic workload scheduling, both NPU and PIM can be underutilized as the number of candidate tokens changes. Although the parallel execution during the decoding iteration involving data reallocation can be unbalanced, the 2-bit saturation counter ensures that there is no large fluctuation in the subsequent rounds, thereby maintaining synchronization between PIM and NPU.

[0178] The following will describe in detail the acceleration device of model speculative reasoning of one or more embodiments of the application. Those skilled in the art can understand that these devices can be configured using commercially available hardware components through the steps taught by the present application.

[0179] Figure 9 A structural schematic diagram of an acceleration device of model speculative reasoning provided by an embodiment of the application is shown. The device is located in the main processor of the model speculative reasoning acceleration system, as shown in Figure 9 The device comprises a first loading module 11, a second loading module 12, and a third loading module 13.

[0180] The first loading module 11 is configured to load a plurality of candidate token sequences output by the current decoding stage of the draft model and model parameters of the target model, and split the model parameters into a first parameter subset and a second parameter subset.

[0181] The second loading module 12 is configured to load the first parameter subset into the first storage space corresponding to the storage module, so that the neural network processor verifies the plurality of candidate token sequences based on the first parameter subset in the first storage space to obtain a first sub-verification result.

[0182] The third loading module 13 is configured to load the second parameter subset into a second storage space located in the computing module, so that the computing module verifies the plurality of candidate token sequences based on the second parameter subset in the second storage space to obtain a second sub-verification result, and transmits the second sub-verification result to the neural network processor, so that the neural network processor determines a verification result corresponding to the plurality of candidate token sequences in the current decoding stage based on the first sub-verification result and the second sub-verification result.

[0183] Optionally, the main processor further comprises a candidate token pruner and a data allocation unit. The first loading module 11 is further configured to: generate a candidate token tree according to the plurality of candidate token sequences, and transmit the candidate token tree to the candidate token pruner; the candidate token pruner prunes the candidate token tree to obtain a pruned target candidate token tree; and the data allocation unit determines a parameter allocation ratio corresponding to the neural network processor and the current decoding stage of the computing module according to the target candidate token tree.

[0184] Figure 9 The apparatus can perform the steps performed by the main processor in the foregoing embodiments, and the detailed execution process and technical effects are described in the foregoing embodiments, which will not be described here.

[0185] In one possible design, the above Figure 9 The structure of the apparatus for accelerating model-based speculation inference can be implemented as a main processor device, such as Figure 10 The main processor device can include a first processor 21, a first memory 22, and a first communication interface 23. The first memory 22 stores executable code. When the executable code is executed by the first processor 21, the first processor 21 can at least implement the steps performed by the main processor in the foregoing embodiments.

[0186] Figure 11 Another structure of an apparatus for accelerating model-based speculation inference is provided in the embodiments of this application, and the apparatus is located in a neural network processor in a system for accelerating model-based speculation inference, such as Figure 11 The apparatus includes a verification module 31 and a receiving module 32.

[0187] The verification module 31 is configured to verify a plurality of candidate token sequences output by a current decoding stage of a draft model based on a first parameter subset in a first storage space corresponding to the storage module to obtain a first sub-verification result. The first parameter subset is loaded into the first storage space corresponding to the storage module by the main processor splitting model parameters of a target model.

[0188] The receiving module 32 is configured to receive the second sub-verification result sent by the computing module, and determine a verification result corresponding to the plurality of candidate token sequences in the current decoding stage based on the first sub-verification result and the second sub-verification result; the second sub-verification result is obtained by verifying the plurality of candidate token sequences based on a second parameter subset in a second storage space in the computing module; and the second parameter subset is loaded into the second storage space corresponding to the computing module by the host processor after splitting the model parameters of the target model.

[0189] Figure 11 The apparatus can perform the steps performed by the neural network processor in the foregoing embodiments, and the detailed execution process and technical effects are described in the foregoing embodiments, which will not be described here.

[0190] In one possible design, the above Figure 11 The structure of the apparatus for accelerating model-based inference can be implemented as a neural network processor device, such as Figure 12 As shown in the figure, the device can include a second processor 41, a second memory 42, and a second communication interface 43. The second memory 42 stores executable code, and when the executable code is executed by the second processor 41, the second processor 41 can at least implement the steps performed by the neural network processor in the foregoing embodiments.

[0191] Figure 13 Another structure of an apparatus for accelerating model-based inference is provided in the embodiments of this application, and the apparatus is located in a computing module in a system for accelerating model-based inference, such as Figure 13 As shown in the figure, the apparatus includes a verification module 51 and a transmission module 52.

[0192] The verification module is configured to verify the plurality of candidate token sequences output by the draft model in the current decoding stage based on a first parameter subset in a first storage space in the computing module, to obtain a first sub-verification result; and the first parameter subset is loaded into the first storage space corresponding to the computing module by the host processor after splitting the model parameters of the target model.

[0193] The transmission module is configured to transmit the first sub-verification result to the neural network processor, so that the neural network processor determines a verification result corresponding to the plurality of candidate token sequences in the current decoding stage based on the first sub-verification result and a second sub-verification result.

[0194] The second sub-verification result is obtained by the neural network processor based on a second parameter subset in a second storage space corresponding to the storage module, and the second parameter subset is loaded into the second storage space corresponding to the storage module by the host processor splitting model parameters of a target model.

[0195] Figure 13 The device can perform the steps executed by the computing module in the foregoing embodiments, and the detailed execution process and technical effects are described in the foregoing embodiments, which will not be described here.

[0196] In one possible design, the above Figure 13 The structure of the model speculation reasoning acceleration device can be implemented as a computing module device, such as a server. Figure 14 The device can include a third processor 61, a third memory 62, and a third communication interface 63. The third memory 62 stores executable code, and when the executable code is executed by the third processor 61, the third processor 61 can at least implement the steps executed by the computing module in the foregoing embodiments.

[0197] In addition, the embodiments of the present application provide a non-transitory machine readable storage medium, which stores executable code, and when the executable code is executed by the processor of the electronic device, the processor can at least implement the model speculation reasoning acceleration method provided in the foregoing embodiments.

[0198] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0199] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a general hardware platform, and of course can also be implemented by means of a combination of hardware and software. Based on such understanding, the above technical solutions can be embodied in the form of a computer product, and the present application can be implemented in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0200] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An acceleration system for model speculation reasoning, characterized in that, include: The system comprises a main processor, a neural network processor, and a hybrid near-memory computing module, wherein the hybrid near-memory computing module includes a computing module and a storage module. The main processor is used to load multiple candidate token sequences output by the current decoding stage of the draft model and the model parameters of the target model, and to split the model parameters into a first parameter subset and a second parameter subset. The main processor is used to load the first subset of parameters into the first storage space corresponding to the storage module, and to load the second subset of parameters into the second storage space located within the computing module. The neural network processor is used to verify the plurality of candidate token sequences based on a first parameter subset within the first storage space to obtain a first sub-verification result; The computing module is used to verify the plurality of candidate token sequences based on a second parameter subset in the second storage space, obtain a second sub-verification result, and transmit the second sub-verification result to the neural network processor; The neural network processor is used to determine the verification result corresponding to the plurality of candidate token sequences in the current decoding stage based on the first sub-verification result and the second sub-verification result.

2. The system of claim 1, wherein, The main processor also includes a candidate token pruner and a data allocation unit; Before the main processor splits the model parameters into a first parameter subset and a second parameter subset, the main processor is further configured to: generate a candidate token tree based on the plurality of candidate token sequences, and transmit the candidate token tree to the candidate token pruner; The candidate token pruner is used to: prune the candidate token tree to obtain the pruned target candidate token tree; The data allocation unit is used to: determine the parameter allocation ratio between the neural network processor and the computing module at the current decoding stage based on the target candidate token tree.

3. The system of claim 2, wherein, The main processor is specifically used to: split the model parameters into a first parameter subset and a second parameter subset based on the parameter allocation ratio corresponding to the current decoding stage.

4. The system of claim 2, wherein, The candidate token pruner includes a candidate token precision model and a candidate tree structure explorer; The candidate token precision model is used to: obtain the historical verification results of multiple candidate token sequences output by the target model for the draft model; The candidate token precision model is used to: determine the expected acceptance length corresponding to each of the multiple candidate token sequences output by each decoder head in the draft model based on the historical verification results, and determine the expected acceptance length corresponding to the candidate token tree based on the expected acceptance length corresponding to each of the multiple candidate token sequences. The candidate tree structure explorer is used to: prune the candidate token tree according to the expected acceptance length corresponding to the candidate token tree, the total transmission delay corresponding to the current decoding stage, and the total energy consumption corresponding to the current decoding stage, to obtain the pruned target candidate token tree; The total transmission delay is a total delay corresponding to data transmission of the neural network processor and the computing module at the current decoding stage, and the total energy consumption is total energy consumption corresponding to data processing of the neural network processor and the computing module at the current decoding stage and total energy consumption consumed during data transmission.

5. The system of claim 4, wherein, The candidate token precision model is specifically used for: For a target candidate token sequence, determining a respective presumed accuracy of each candidate token in the target candidate token sequence according to the historical verification result, the target candidate token sequence being any one of the plurality of candidate token sequences; Obtaining a product value between the respective presumed accuracies of the plurality of candidate tokens, and determining the product value as an expected acceptance length of the target candidate token sequence, the expected acceptance length of the target candidate token sequence being used to represent a probability that the target candidate token sequence is accepted by the target model; Obtaining a sum value between the expected acceptance lengths of the plurality of candidate token sequences, and determining the sum value as an expected acceptance length corresponding to the candidate token tree.

6. The system of claim 4, wherein, The candidate tree structure explorer is specifically used for: generating a set of candidate tokens to be expanded, the set of candidate tokens to be expanded including a plurality of candidate tokens in the candidate token tree; selecting, according to greedy sampling, a target candidate token with the highest current presumed accuracy from the set of candidate tokens to be expanded, and adding the target candidate token as a temporary node to the candidate token tree being currently constructed; determining a total transmission delay corresponding to verification of the candidate token tree being currently constructed and total energy consumption required for verification of the candidate token tree being currently constructed; performing energy consumption and performance verification on the candidate token tree being currently constructed according to the total transmission delay and the total energy consumption; if the verification passes, retaining the target candidate token as a valid node in the candidate token tree being currently constructed, removing the target candidate token from the set of candidate tokens to be expanded, and continuing to repeatedly perform the above operations on the remaining candidate tokens; if the verification fails, removing the target candidate token from the candidate token tree being currently constructed, and determining the candidate token tree after removal as a target candidate token tree.

7. The system of claim 6, wherein, The candidate tree structure explorer is specifically used for: determining a first parameter access amount required for the neural network processor to verify the candidate token tree being currently constructed and a second parameter access amount required for the computing module to verify the candidate token tree being currently constructed; determining a first transmission delay corresponding to verification of the candidate token tree being currently constructed by the neural network processor according to the first parameter access amount and a first data transmission rate between the neural network processor and the computing module; determining a second transmission delay corresponding to verification of the candidate token tree being currently constructed by the computing module according to the second parameter access amount and a second data transmission rate between the computing module and the inside of the computing module; determining a total transmission delay corresponding to verification of the candidate token tree being currently constructed as a larger value between the first transmission delay and the second transmission delay.

8. The system of claim 7, wherein, The candidate tree structure explorer is specifically used for: determining a first storage capacity occupied when storing the model parameters corresponding to the first parameter access amount; obtaining a first ratio of the first storage capacity and a first data transmission rate, and determining the first ratio as a first transmission delay corresponding to the verification of the current constructed candidate token tree by the neural network processor; determining a second storage capacity occupied when storing the model parameters corresponding to the second parameter access amount; determining the number of arithmetic units in each matrix processing unit in the calculation module; determining the number of target candidate tokens contained in the current constructed candidate token tree; obtaining a second ratio of the second storage capacity and a second data transmission rate; obtaining a third ratio of the number of target candidate tokens and the number of arithmetic units; determining a product value of the second ratio and the third ratio; determining the product value as a second transmission delay corresponding to the verification of the current constructed candidate token tree by the calculation module.

9. The system of claim 6, wherein, The candidate tree structure explorer is specifically configured to: determine a first calculation amount corresponding to the neural network processor when verifying the current constructed candidate token tree, and a second calculation amount corresponding to the calculation module when verifying the current constructed candidate token tree; determine a total calculation consumption required when verifying the current constructed candidate token tree according to the first calculation amount and the second calculation amount; determine a data transmission amount corresponding to the verification of the current constructed candidate token tree; determine a total data transmission power consumption required when verifying the current constructed candidate token tree according to the data transmission amount and a data transmission power consumption corresponding to a unit of data; determine a total energy consumption required when verifying the current constructed candidate token tree according to the total calculation consumption and the total data transmission power consumption.

10. The system of claim 6, wherein, The candidate tree structure explorer is specifically configured to: determine a transmission delay corresponding to a unit of acceptance length of the current decoding stage according to the total transmission delay and an expected acceptance length corresponding to the candidate token tree; determine an energy consumption corresponding to a unit of acceptance length of the current decoding stage according to the total energy consumption and an expected acceptance length corresponding to the candidate token tree; determine whether the transmission delay corresponding to a unit of acceptance length of the current decoding stage and the energy consumption corresponding to a unit of acceptance length of the current decoding stage meet a preset condition; if the preset condition is met, it is determined that the energy consumption and performance verification of the current constructed candidate token tree is passed.

11. The system of claim 2, wherein, The data allocation unit is specifically configured to: determine the number of target tokens contained in the target candidate token tree; obtain a model parameter allocation table, in which a corresponding relationship between a candidate token number and a preset parameter allocation ratio is stored; query a preset parameter allocation ratio matching the number of target tokens from the model parameter allocation table; determine a parameter allocation ratio corresponding to the current decoding stage of the neural network processor and the calculation module according to the preset parameter allocation ratio.

12. The system of claim 11, wherein, The data allocation unit further comprises a counter, and the data allocation unit is specifically configured to: obtain a current count value corresponding to the counter; If the count value meets the preset value, a parameter allocation ratio of the preset parameter is determined as a parameter allocation ratio of the neural network processor and the calculation module corresponding to a current decoding stage; If the count value does not meet the preset value, a parameter allocation ratio of the neural network processor and the calculation module corresponding to a previous decoding stage is determined as a parameter allocation ratio of the neural network processor and the calculation module corresponding to the current decoding stage.

13. A method of accelerating model-based speculation reasoning, the method comprising: An acceleration system applied to model speculative reasoning, the system comprising: a host processor, a neural network processor, a hybrid near-memory calculation module, the hybrid near-memory calculation module comprising a calculation module and a storage module; the method comprising: The host processor loads a plurality of candidate token sequences output by a current decoding stage of a draft model and model parameters of a target model, and splits the model parameters into a first parameter subset and a second parameter subset; The host processor loads the first parameter subset into a first storage space corresponding to the storage module, so that the neural network processor verifies the plurality of candidate token sequences based on the first parameter subset in the first storage space to obtain a first sub-verification result; The host processor loads the second parameter subset into a second storage space located in the calculation module, so that the calculation module verifies the plurality of candidate token sequences based on the second parameter subset in the second storage space to obtain a second sub-verification result, and transmits the second sub-verification result to the neural network processor, so that the neural network processor determines a verification result corresponding to the plurality of candidate token sequences of the current decoding stage based on the first sub-verification result and the second sub-verification result.

14. The method of claim 13, wherein, The host processor further comprises a candidate token pruner and a data allocation unit; the method further comprises: The host processor generates a candidate token tree according to the plurality of candidate token sequences, and transmits the candidate token tree to the candidate token pruner; The candidate token pruner prunes the candidate token tree to obtain a pruned target candidate token tree; The data allocation unit determines a parameter allocation ratio of the neural network processor and the calculation module corresponding to the current decoding stage according to the target candidate token tree.

15. A method of accelerating model-based speculation reasoning, the method comprising: An acceleration system applied to model speculative reasoning, the system comprising: a host processor, a neural network processor, a hybrid near-memory calculation module, the hybrid near-memory calculation module comprising a calculation module and a storage module; the method comprising: The neural network processor verifies a plurality of candidate token sequences output by a current decoding stage of a draft model based on a first parameter subset in a first storage space corresponding to the storage module to obtain a first sub-verification result; wherein the first parameter subset is split by the host processor from model parameters of a target model and loaded into the first storage space corresponding to the storage module; The neural network processor receives the second sub-verification result sent by the computing module, and determines a verification result corresponding to the plurality of candidate token sequences in the current decoding stage based on the first sub-verification result and the second sub-verification result; wherein the second sub-verification result is obtained by the computing module based on a second parameter subset in a second storage space in the computing module; and the second parameter subset is loaded into the second storage space corresponding to the computing module by the host processor after splitting the model parameters of the target model.

16. A method of accelerating model-based speculation reasoning, the method comprising: An acceleration system applied to model speculation reasoning, the system comprising: a host processor, a neural network processor, a hybrid near-memory computing module, the hybrid near-memory computing module comprising a computing module and a storage module; the method comprising: The computing module verifies the plurality of candidate token sequences output by the draft model in the current decoding stage based on a first parameter subset in a first storage space in the computing module, to obtain a first sub-verification result; the first parameter subset is loaded into the first storage space corresponding to the computing module by the host processor after splitting the model parameters of the target model; The computing module transmits the first sub-verification result to the neural network processor, so that the neural network processor determines a verification result corresponding to the plurality of candidate token sequences in the current decoding stage based on the first sub-verification result and a second sub-verification result; Wherein, the second sub-verification result is obtained by the neural network processor based on a second parameter subset in a second storage space corresponding to the storage module; and the second parameter subset is loaded into the second storage space corresponding to the storage module by the host processor after splitting the model parameters of the target model.

Images