Inference method, device, equipment, medium and product of large model

Abstract

The application discloses a large model reasoning method, device, equipment, medium and product, the method comprises the following steps: acquiring a target kernel function execution sequence matched with a current input large model reasoning task; obtaining all required kernel function description information according to a target start descriptor package complete delivery logic matched with the target kernel function execution sequence; generating a logic task graph matched with the large model reasoning task according to kernel function description information corresponding to each target kernel function; obtaining an executable task graph after instantiation processing; in the large model reasoning process, each target start descriptor package obtained from the executable task graph is delivered to a device end, and each word element matched with the large model reasoning task is generated and output. The technical scheme effectively reduces reasoning task execution resource consumption and improves reasoning task scheduling efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a reasoning method, apparatus, device, medium, and product for large models. Background Technology

[0002] With the rapid iteration and large-scale deployment of large-scale model technology, high-performance, low-latency, and high-throughput model inference deployment has become a core research direction in the field of artificial intelligence computing power. Currently, the industrial deployment of large models generally adopts a heterogeneous acceleration architecture that coordinates the host and device sides. Relying on the parallel computing advantages of heterogeneous computing power, it efficiently carries the massive matrix operations, feature calculations and other inference tasks of large models, which is the mainstream technical solution for achieving efficient inference of large models.

[0003] Heterogeneous acceleration systems for large language model deployments typically consist of two parts: a host side and a device side. The host side is responsible for scheduling, verifying, and submitting kernel functions (also known as kernels) to the device, while the device side is responsible for executing the actual computation of the kernel functions. A typical large model inference architecture on the host side includes several main software modules such as an inference engine (or service scheduling layer), a deep learning framework layer, an operator library layer, and a system runtime and driver layer. A single inference process often contains multiple iteration steps (referred to as Steps), and each iteration step contains several kernel function computation processes (referred to as Launches). The computation of each kernel function on the host side usually involves a call from the framework layer to the underlying runtime and driver layer.

[0004] In the process of developing this invention, the inventors discovered that when performing large model inference in mainstream lightweight deployment scenarios such as small batch sizes, long text sequences, and low bit quantization, the host-side scheduling overhead is directly exposed in the critical path of inference performance because the device can quickly complete the computation task and enter an idle state. This technical defect causes problems such as idle device computing power, significantly worsened inference first-word latency and character-by-character decoding latency, and significantly limits inference throughput and service stability in multi-stream, multi-concurrent request scenarios. Summary of the Invention

[0005] This invention provides a method, apparatus, device, medium, and product for inference of large models, in order to solve the technical problems in heterogeneous acceleration systems where the fixed scheduling and operation overhead of the host cannot be masked by the device's computation time in scenarios involving small batches, long sequences, and low bit quantization of large models, resulting in increased inference latency and limited concurrent throughput and service stability.

[0006] According to one aspect of the present invention, a large-model inference method is provided, executed by a host side of a heterogeneous acceleration system. The heterogeneous acceleration system includes a host side and a device side. The host side deploys a multi-level framework layer of the large model, and the device side deploys heterogeneous computing hardware for executing the kernel functions of the large model. The method includes: Obtain the target kernel function execution sequence that matches the current input large model inference task. Here, the host encapsulates a kernel function and sends a startup descriptor package to the device to trigger the device to execute the kernel function. Before large model inference, based on the complete delivery logic of the target startup descriptor package that matches the target kernel function execution sequence, all kernel function description information required before executing the delivery operation of each target startup descriptor package is obtained. The kernel function description information includes all operations to be executed and all kernel function parameters to be used. Based on the kernel function description information corresponding to each target kernel function, a logical task graph matching the large model inference task is generated, and the logical task graph is instantiated to obtain an executable task graph. During the large model inference process, the target startup descriptor packages obtained from the executable task graph are sent to the device to generate and output each lexical that matches the large model inference task.

[0007] According to another aspect of the present invention, a large-scale model inference apparatus is provided, comprising: The kernel function execution sequence acquisition module is used to acquire the target kernel function execution sequence that matches the current input large model inference task. The host encapsulates a kernel function and sends a startup descriptor package to the device to trigger the device to execute the kernel function. The kernel function description information acquisition module is used to obtain all kernel function description information required before executing the delivery operation of each target startup descriptor package, based on the complete delivery logic of the target startup descriptor package that matches the target kernel function execution sequence before large model inference. The kernel function description information includes all operations to be executed and all kernel function parameters to be used. The task graph generation module is used to generate a logical task graph that matches the large model inference task based on the kernel function description information corresponding to each target kernel function, and to instantiate the logical task graph to obtain an executable task graph. The startup descriptor package delivery module is used to deliver the startup descriptor packages of each target obtained from the executable task graph to the device during the large model inference process, and generate and output each lexical that matches the large model inference task.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the large model inference method described in any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the large model inference method described in any embodiment of the present invention.

[0010] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the method as described in any embodiment of the present invention.

[0011] The technical solution of this invention is executed on the host side of a heterogeneous acceleration system. The host side encapsulates a startup descriptor package for each kernel function and sends it to the device side to trigger the execution of that kernel function. Before large model inference, the execution sequence of the target kernel function matching the current input large model inference task is obtained. Based on the complete sending logic of the target startup descriptor package matching the target kernel function execution sequence, the host side obtains all operations required by the multi-level framework layer of the large model before the target startup descriptor package sending operation, as well as the kernel function description information consisting of all corresponding kernel function parameters. Based on the kernel function description information corresponding to each target kernel function required by the current large model inference task, a matching logical task graph is generated. After instantiating the logical task graph, an executable task graph is obtained. During large model inference, each target startup descriptor package obtained from the executable task graph is sent to the device side, and each target kernel function is executed to generate each term of the large model inference task. This technical solution integrates the preprocessing stage operations of the large model inference task kernel function to construct an executable task graph, which is then uniformly sent to the device side to execute the inference task operations, effectively reducing the resource overhead of inference task execution and improving the scheduling efficiency of inference tasks.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating a large-model inference method in the existing technology; Figure 2 This is a diagram illustrating how the host-side execution time of the large model inference process is hidden in existing technologies; Figure 3 This is a diagram illustrating that the host-side execution time of the large model inference process in existing technologies is not hidden. Figure 4 This is a flowchart of a large-scale model inference method provided by an embodiment of the present invention; Figure 5 This is a flowchart of another reasoning method for a large model provided according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the traditional execution mode process in the inference process of a large model; Figure 7 This is a schematic diagram of a large model inference process flow capture mode provided by an embodiment of the present invention; Figure 8 This is a schematic diagram of the execution flow of a large model inference process according to an embodiment of the present invention; Figure 9 This is a flowchart of an optimized large-scale model inference method provided according to an embodiment of the present invention; Figure 10 This is a flowchart of another optimized large model inference method provided according to an embodiment of the present invention; Figure 11 This is a schematic diagram illustrating the step-by-step optimization of the execution flow of a large model inference process according to an embodiment of the present invention; Figure 12 This is a schematic diagram of the structure of a large-scale inference device according to an embodiment of the present invention; Figure 13 This is a schematic diagram of the structure of an electronic device that implements the large-model inference method of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] To facilitate the description of the technical solutions of the various embodiments of the present invention, the technical problems existing in the existing large model inference process will be analyzed in detail first.

[0018] Specifically, in Figure 1 The diagram illustrates a flowchart of a large model inference method in the prior art. For example... Figure 1 As shown, the host side constructs a multi-level software scheduling architecture to preprocess the kernel functions required for large model inference, such as... Figure 1 In the `launch0` section, the preprocessing steps of the kernel function `kernel0` involve various operations and parameter configurations from the inference engine / service scheduling layer to the driver layer. `launch0` through `launchN` correspond to the kernel functions `kernel0` through `kernelN` required to complete the current large model inference task. The process of generating each lexical in the inference task begins with the first kernel function call `step0` and continues until the final kernel function call `stepN` completes the inference task. For example... Figure 2 The diagram illustrates a scenario where the preparation time of the host-side kernel function is hidden during a single inference iteration of a large model. This means that the execution time of a single kernel function on the device side is greater than the preparation time of a single kernel function on the host side. For example, the execution time of kernel function kernel0 is greater than the execution time of the launch1 preparation phase corresponding to kernel function kernel1. This ensures that the execution of each kernel function on the device side is continuous and without time gaps. Figure 3The diagram illustrates the case where the preparation time of the host-side kernel function is not hidden during a single inference iteration of a large model. That is, the execution time of a single kernel function on the device side is less than the preparation time of a single kernel function on the host side. For example, the execution time of kernel function kernel0 is less than the execution time of the corresponding preparation phase launch1 of kernel function kernel1. Similarly, time gaps (GAPS) are generated during the execution of each kernel function on the device side, resulting in low scheduling efficiency of each sum function during the execution of the large model inference task.

[0019] Figure 4 This is a flowchart of a large model inference method provided in Embodiment 1 of the present invention. This embodiment is applicable to the optimization of host-side kernel function processing in heterogeneous acceleration systems performing large model inference. The method can be executed by a large model inference device, which can be implemented in hardware and / or software, and is generally configured on the host side of the heterogeneous acceleration system. The heterogeneous acceleration system includes a host side and a device side. The host side deploys a multi-level framework layer of the large model, and the device side deploys heterogeneous computing hardware for executing the large model kernel function, such as... Figure 4 As shown, the method includes: S410. Obtain the target kernel function execution sequence that matches the current input large model inference task.

[0020] Specifically, the host encapsulates a kernel function and sends a boot descriptor package to the device to trigger the device to execute the kernel function.

[0021] Here, a kernel function can refer to a dedicated mapping function used to perform nonlinear similarity matching and feature association calculations on the input vector and hidden layer feature vectors of a large model. It can efficiently complete semantic association, attention matching, and feature space fitting operations without explicit dimensionality increase. A target kernel function can refer to the kernel function corresponding to the current inference task execution stage when performing a large model inference task. A target kernel function execution sequence can refer to an execution sequence formed by arranging multiple target kernel functions in a large model inference task according to a preset fixed execution order. This sequence defines the order in which each target kernel function is executed and carries the configuration parameters and computational input parameters required for the execution process of each kernel function. A startup descriptor package can refer to a dedicated data package in which the host end organizes and encapsulates various parameters and configuration information required for kernel function execution, distributes it to shared hardware memory, and allows the device end to read from this shared hardware memory and schedule the execution of kernel function operations.

[0022] Understandably, when processing large model inference tasks, the heterogeneous acceleration system can identify all kernel functions required to execute the current large model inference task. It can also prepare the kernel function corresponding to the current inference stage based on the inference task. After encapsulating the matching parameter information for each stage of the kernel function's operation into a corresponding startup descriptor package, it sends it to the corresponding hardware shared memory for the device to retrieve and execute, thus completing the inference task. When the large model inference task is determined, the execution order of all required kernel functions is also determined. Therefore, based on the current input large model inference task, the execution order of all matching target kernel functions and the matching parameter information for each stage of the target kernel function's operation can be determined, thereby determining the corresponding target kernel function execution sequence.

[0023] S420. Before large model inference, based on the complete target startup descriptor package delivery logic that matches the target kernel function execution sequence, obtain all kernel function description information required before executing the delivery operation of each target startup descriptor package.

[0024] The kernel function description information includes all the operations to be performed and all the kernel function parameters to be used.

[0025] The target startup descriptor package can refer to the startup descriptor package corresponding to each kernel function in the target kernel function execution sequence when executing the current large model inference task.

[0026] Understandably, after determining the matching target kernel function execution sequence based on the large model inference task, it is possible to further determine all operations performed during the preparation phase of each target startup descriptor package before it is sent to the device, as well as the kernel function parameters required for each operation. Therefore, before the large model inference task is officially executed, all kernel function description information of the target startup descriptor packages corresponding to each target kernel function in the target kernel function execution sequence can be obtained.

[0027] S430. Based on the kernel function description information corresponding to each target kernel function, generate a logical task graph that matches the large model inference task, and instantiate the logical task graph to obtain an executable task graph.

[0028] The logical task graph refers to a data set that records the various operations and corresponding parameters of each kernel function during its execution according to the execution sequence of the target kernel function, and represents the execution order and dependencies of each kernel function. The executable task graph refers to a data set generated by merging and integrating the descriptors of each target kernel function obtained from the logical task graph after centralized verification of all kernel function parameters and preprocessing of required runtime resources. This data set supports the device in scheduling and executing kernel functions according to predetermined logic.

[0029] Understandably, the kernel function description information corresponding to each target kernel function includes all operations required in executing large model inference tasks and all corresponding kernel function parameters. Based on the execution order and dependencies of each kernel function in the target kernel function execution sequence, logical task graph components can be constructed with each kernel function as the index target. Corresponding operations and kernel function parameter information are added to each kernel function within these logical task graph components, ultimately forming a directed acyclic graph (DAG) that can represent the execution relationships between target kernel functions and the corresponding description information of all kernel functions. Instantiating the logical task graph can include verifying the accuracy of all kernel function parameters in the logical task graph and centrally processing the resources required for the operations of each target kernel function. The startup descriptor packages corresponding to each target kernel function obtained after instantiation of the logical task graph are then merged to obtain an executable task graph that can be used by the device to perform large model inference tasks.

[0030] S440. During the large model inference process, the target startup descriptor packages obtained from the executable task graph are sent to the device, and each lexical unit that matches the large model inference task is generated and output.

[0031] Among them, a lexical unit can refer to the basic semantic unit that is successively decoded and output by the large model through the reasoning and calculation process. It carries the semantic information of the text and the sequence association relationship, and constitutes the smallest expressive carrier of the model's reasoning output sequence.

[0032] Understandably, in a heterogeneous acceleration system used to process large model inference tasks, after the host side generates an executable task graph, it can parse the executable task graph to obtain the target startup descriptor packages corresponding to each target kernel function contained therein. Based on the generation requirements of each lexical in the large model inference task, the corresponding target startup descriptor packages can be sent to the device side for generating the corresponding lexical computation tasks.

[0033] The technical solution of this invention is executed on the host side of a heterogeneous acceleration system. The host side encapsulates a startup descriptor package for each kernel function and sends it to the device side to trigger the execution of that kernel function. Before large model inference, the execution sequence of the target kernel function matching the current input large model inference task is obtained. Based on the complete sending logic of the target startup descriptor package matching the target kernel function execution sequence, the host side obtains all operations required by the multi-level framework layer of the large model before the target startup descriptor package sending operation, as well as the kernel function description information consisting of all corresponding kernel function parameters. Based on the kernel function description information corresponding to each target kernel function required by the current large model inference task, a matching logical task graph is generated. After instantiating the logical task graph, an executable task graph is obtained. During large model inference, each target startup descriptor package obtained from the executable task graph is sent to the device side, and each target kernel function is executed to generate each term of the large model inference task. This technical solution integrates the preprocessing stage operations of the large model inference task kernel function to construct an executable task graph, which is then uniformly sent to the device side to execute the inference task operations, effectively reducing the resource overhead of inference task execution and improving the scheduling efficiency of inference tasks.

[0034] Figure 5 This is a flowchart illustrating another large-scale model inference method provided by an embodiment of the present invention. This embodiment is a further specification based on the above embodiments, including: specific methods for constructing the logical task graph and executable task graph of the large-scale model inference process, and specific methods for running the executable task graph. Figure 5 As shown, the method includes: S510. Obtain the target kernel function execution sequence that matches the current input large model inference task.

[0035] Specifically, the host encapsulates a kernel function and sends a boot descriptor package to the device to trigger the device to execute the kernel function.

[0036] S520. In the multi-level framework layer of the large model, simulate the execution of the complete target startup descriptor package delivery logic once, and when the complete target startup descriptor package delivery logic is executed to the target framework layer, intercept the delivery logic executed for the target framework layer and obtain all intercepted kernel function description information.

[0037] The multi-level framework of the large model includes: a top-down inference service scheduling layer, a deep learning framework layer, a computation implementation layer, a system interface layer, and a driver layer, with the target framework layer being the system interface layer.

[0038] Specifically, before formally executing the large model inference task, a complete target startup descriptor package distribution logic can be simulated based on the large model inference task. This includes executing the operations of each layer in the large model multi-level framework layer and the corresponding input parameter information. Therefore, all distribution logic from the beginning to the system interface layer, which serves as the target framework layer, can be obtained during the execution of the large model multi-level framework layer. This allows us to obtain all kernel function description information of the inference service scheduling layer, deep learning framework layer, computation implementation layer, and system interface layer in the execution process of the large model multi-level framework layer.

[0039] Optionally, within the multi-level framework layers of the large model, simulate the execution of the complete target startup descriptor package delivery logic once. When the complete target startup descriptor package delivery logic reaches the target framework layer, intercept the delivery logic executed for that target framework layer and obtain all intercepted kernel function description information, including: In the delivery logic of the multi-level framework layer of the target startup descriptor packet complete delivery logic, a start capture node and an end capture node are inserted respectively to enable the stream capture mode for the target startup descriptor packet complete delivery logic. In stream capture mode, the complete delivery logic of the target startup descriptor package is simulated and executed. Each time the delivery logic of the target framework layer is executed, the delivery logic executed for the target framework layer is intercepted and all kernel function description information intercepted is obtained.

[0040] Among them, the flow capture mode can refer to the process acquisition mechanism that inserts start and end nodes in a multi-level framework layer of a large model, defines the process capture range based on the two nodes, and then captures the execution operations and corresponding parameters of each framework layer.

[0041] Specifically, during the execution of large model inference tasks, in situations such as Figure 6 In the traditional execution mode shown, the preprocessing of each target kernel function requires re-traversing the multi-level framework of the large model, i.e., from the top-level inference service scheduling layer down to the driver layer, where the startup descriptor packages of each target kernel function are sent to the device. Therefore, a stream capture mode is introduced, inserting a start capture node before the top-level inference service scheduling layer of the multi-level framework of the large model, taking the system interface layer as the target framework layer, and inserting an end capture node after the target framework layer to complete the enable operation of the stream capture mode. This allows the execution operations and corresponding parameters of each level to be recorded and saved during the simulated execution of the complete target startup descriptor package sending logic. Furthermore, after reaching the end capture node set after the target framework layer, the sending logic executed for the target framework layer is intercepted, pausing the execution of subsequent level operations, thus completing the capture of kernel function description information from the start capture node to the end capture node of the multi-level framework of the large model. Figure 7As shown, based on the execution sequence of the target kernel function, after simulating the complete delivery logic of the target startup descriptor package corresponding to all target kernel functions, the description information of all intercepted kernel functions can be obtained.

[0042] S530. Treat each target kernel function as a graph node, and bind all operations to be performed for each target kernel function and all kernel function parameters to be used to the matching graph node respectively.

[0043] Specifically, when constructing the logical task graph, the kernel functions required for the current large model inference task can be selected as graph nodes. After capturing all the execution operations and corresponding kernel function parameters of each kernel function from the start capture node to the end capture node in the multi-level framework layer of the large model, they are bound to the corresponding kernel functions in the graph nodes. This allows the cross-language boundary overhead and kernel function distribution and scheduling overhead generated when calling each target kernel function to be extracted and centrally organized into the logical task graph in the form of graph nodes.

[0044] S540. Based on the execution dependencies of each target kernel function in the target kernel function execution sequence, and the execution order between the required operations of each target kernel function, organize each node to obtain a logical task graph.

[0045] Specifically, the obtained graph nodes form a logical task graph. Besides containing all operations and corresponding kernel function parameters for each target kernel function in the graph nodes, it also needs to confirm the dependencies and execution order of each target kernel function based on the execution sequence of the target kernel functions corresponding to the current large-scale model inference task. Then, the graph nodes are organized into a logical task graph according to the execution logic, so that the execution logic of each target kernel function and the description information of all kernel functions required to execute the current large-scale model inference task can be obtained through the logical task. Figure 1 All can be obtained at once.

[0046] S550. Instantiate the logical task graph to obtain the executable task graph.

[0047] Optionally, the logical task graph is instantiated to obtain an executable task graph, including: The parameters of each target kernel function in the logical task graph are centrally verified. After the parameter verification is completed, the parameter memory of each kernel function parameter included in the logical task graph is prepared, and the target address of each kernel function parameter in shared memory is determined to obtain the executable task graph.

[0048] Specifically, the generated logical task graph contains all the kernel function parameters required for the operations of each target kernel function. It can perform centralized parameter verification on each target kernel function and centrally complete the allocation of memory required for the corresponding kernel function parameters of each target kernel function. This allows us to determine the target address of each kernel function parameter in shared memory, as well as the address offset corresponding to the allocated memory space size. After binding the target address and address offset of each target kernel function to the corresponding graph nodes in the logical task graph, we obtain the executable task graph.

[0049] S560. In the single-term inference process of the large model, the parameter configuration and memory filling operations of each target kernel function in the target kernel function execution sequence are completed according to the executable task graph, as well as the preparation operation of the target startup descriptor package and the submission operation of the target startup descriptor package, so as to generate and output the matching single term.

[0050] Specifically, during the formal execution of single-word inference in the large model inference task, based on the kernel function parameters recorded in the graph nodes corresponding to each target kernel function in the executable task graph and the allocated memory address information, the parameter configuration of the target kernel function corresponding to the current execution stage in the target kernel function execution sequence and the filling of parameters into the corresponding memory space are completed. Then, the preparation operation of the target startup descriptor package in the kernel function startup package is completed, that is, the graph node information corresponding to each target kernel function in the executable task graph is organized into the corresponding target startup descriptor package, and stored in the buffer after allocating a matching space. Then, the target startup descriptor package submission operation in the kernel function startup package submission is executed, so that the device can extract the required target startup descriptor package information from the target memory buffer, execute the corresponding target kernel function to generate and output the matching words. The complete kernel function delivery and execution flow logic is as follows: Figure 8 As shown, the graph construction phase completes the capture of description information for the preparation operations launch0 to launchN corresponding to the target kernel functions required for simulating the large model inference task. It starts from the simulation start command capture on and ends with the simulation stop command captureoff. In the instantiation phase, the parameter verification is completed through the graphExecinit operation based on the graph node information, and the splicing and distribution of the target kernel function startup descriptor package is completed. In the graph running phase, when the device uses all target kernel functions to perform different rounds of step0 to stepN operations, the launch graph command completes the one-time acquisition of all target kernel function startup descriptor package information, thus completing the entire inference task.

[0051] The technical solution of this invention obtains the execution sequence of target kernel functions matching the current input large model inference task before large model inference, simulates the execution of the complete target startup descriptor package delivery logic once, and intercepts the delivery logic executed for the target framework layer to obtain all kernel function description information. Each target kernel function is treated as a graph node and bound to all required operations and kernel function parameters. A matching logical task graph is generated based on the execution dependencies and execution order of each target kernel function. The logical task graph is then uniformly allocated with the cache space required for each target kernel function parameter to obtain an executable task graph. During the formal large model inference process, the parameter configuration and memory filling operations for each target kernel function, the preparation operation of the target startup descriptor package, and the submission operation of the target startup descriptor package are completed according to the executable task graph, generating and outputting a matching single token. This technical solution improves the execution efficiency of memory allocation for kernel functions and shortens the host-side kernel function preparation time latency by allocating the entire target startup descriptor package cache area at once.

[0052] Figure 9 This is a flowchart illustrating an optimized large-model inference method provided by an embodiment of the present invention. This embodiment optimizes the above embodiments and includes: an optimized method for running the executable task graph of the large-model inference process. For example... Figure 9 As shown, the method includes: S910. Obtain the target kernel function execution sequence that matches the current input large model inference task.

[0053] S920. In the multi-level framework layer of the large model, simulate the execution of the complete target startup descriptor package delivery logic once, and when the complete target startup descriptor package delivery logic is executed to the target framework layer, intercept the delivery logic executed for the target framework layer and obtain all intercepted kernel function description information.

[0054] S930. Treat each target kernel function as a graph node, and bind all operations to be performed for each target kernel function and all kernel function parameters to be used to the matching graph node respectively.

[0055] S940. Based on the execution dependencies of each target kernel function in the target kernel function execution sequence, and the execution order between the operations required by each target kernel function, organize each node to obtain a logical task graph.

[0056] S950. Perform centralized parameter verification on each target kernel function in the logical task graph. After completing the parameter verification, prepare the parameter memory for each kernel function parameter included in the logical task graph, determine the target address of each kernel function parameter in the shared memory, configure the matching kernel function parameters for each target kernel function, and write each kernel function parameter to the matching target address to obtain the executable task graph.

[0057] Specifically, the generated logical task graph contains all the kernel function parameters required for the operations of each target kernel function. It can perform centralized parameter verification on each target kernel function and centrally complete the allocation of memory required for the corresponding kernel function parameters of each target kernel function. This allows us to determine the target address of each kernel function parameter in shared memory, as well as the address offset corresponding to the allocated memory space size. After binding the obtained target addresses and address offsets of each target kernel function to the corresponding graph nodes in the logical task graph, the parameter configuration of each target kernel function and the operation of filling the parameters into the corresponding memory space are completed centrally, resulting in an executable task graph.

[0058] S960. In the single-term inference process of the large model, the preparation operation of the target start descriptor package for each target kernel function in the target kernel function execution sequence and the target start descriptor package submission operation are completed according to the executable task graph to generate and output the matching single term.

[0059] Specifically, during the formal execution of single-word inference in the large model inference task, the graph node information corresponding to each target kernel function in the executable task graph is organized into corresponding target start descriptor packages. After allocating a buffer of matching space size, the packages are stored in the buffer to complete the preparation operation of the target start descriptor packages in the kernel function start package. Then, the target start descriptor package submission operation in the kernel function start package submission is executed so that the device can extract the required target start descriptor package information from the target memory buffer, execute the corresponding target kernel function to generate and output the matching words.

[0060] The technical solution of this invention obtains the execution sequence of target kernel functions matching the current input large model inference task before large model inference, simulates the execution of the complete target startup descriptor package delivery logic once, and intercepts the delivery logic executed for the target framework layer to obtain all kernel function description information. Each target kernel function is treated as a graph node and bound to all required operations and kernel function parameters. A matching logical task graph is generated based on the execution dependencies and execution order of each target kernel function. The logical task graph undergoes centralized parameter verification and unified cache allocation. After completing the parameter configuration and memory filling operations for each target kernel function, an executable task graph is obtained. During the formal large model inference process, the preparation and submission operations for the target startup descriptor package are completed according to the executable task graph, generating and outputting a matching single token. This technical solution shortens the kernel function parameter processing time and improves the host-side kernel function configuration efficiency by allocating all target startup descriptor package caches at once and centrally completing the memory filling operation.

[0061] Figure 10This is a flowchart of another optimized large-model inference method provided by an embodiment of the present invention. This embodiment optimizes the above embodiments and includes: an optimized method for running the executable task graph of the large-model inference process. For example... Figure 10 As shown, the method includes: S1010: Obtain the target kernel function execution sequence that matches the current input large model inference task.

[0062] S1020. In the multi-level framework layer of the large model, simulate the execution of the complete target startup descriptor package delivery logic once, and when the complete target startup descriptor package delivery logic is executed to the target framework layer, intercept the delivery logic executed for the target framework layer and obtain all intercepted kernel function description information.

[0063] S1030. Treat each target kernel function as a graph node, and bind all operations to be performed for each target kernel function and all kernel function parameters to be used to the matching graph node respectively.

[0064] S1040. Based on the execution dependencies of each target kernel function in the target kernel function execution sequence, and the execution order between the required operations of each target kernel function, organize each node to obtain a logical task graph.

[0065] S1050. Perform centralized parameter verification on each target kernel function in the logical task graph. After completing the parameter verification, prepare the parameter memory for each kernel function parameter included in the logical task graph, determine the target address of each kernel function parameter in the shared memory, configure the matching kernel function parameters for each target kernel function, and write the kernel function parameters to the matching target address. Then, construct the target startup descriptor package that matches each target kernel function to obtain the executable task graph.

[0066] Specifically, the generated logical task graph contains all the kernel function parameters required for the operations of each target kernel function. It allows for centralized parameter verification of each target kernel function and centralized allocation of memory required for its corresponding kernel function parameters. This determines the target address of each kernel function parameter in shared memory, as well as the address offset corresponding to the allocated memory space. After binding the obtained target addresses and address offsets of each target kernel function to the corresponding graph nodes in the logical task graph, it centrally completes the parameter configuration and parameter filling operations for each target kernel function into the corresponding memory space. The graph node information corresponding to each target kernel function in the executable task graph is organized into corresponding target startup descriptor packages, which are then stored in a cache area of ​​matching size after allocation. The target startup descriptor package information is then extracted from the cache area corresponding to each target startup descriptor package. Based on the target kernel function execution sequence, the target startup descriptor package information is concatenated and stored in a contiguous overall cache area to obtain the executable task graph. This completes the preparation operation of the target startup descriptor packages in the kernel function startup packages, resulting in the executable task graph.

[0067] S1060. At the start of large model inference, each target startup descriptor package contained in the executable task graph is sent to the target memory buffer shared with the device, so that the device can directly obtain each target startup descriptor package from the target memory buffer during each single lexical inference process.

[0068] Specifically, during the formal execution of single-word inference in the large model inference task, the graph node information corresponding to each target kernel function in the executable task graph is organized into corresponding target start descriptor packages. After allocating a buffer of matching space size, the packages are stored in the buffer to complete the preparation operation of the target start descriptor packages in the kernel function start package. Then, the target start descriptor package submission operation in the kernel function start package submission is executed so that the device can extract the required target start descriptor package information from the target memory buffer, execute the corresponding target kernel function to generate and output the matching words.

[0069] Optionally, after constructing the target startup descriptor package that matches each target kernel function, the following is also included: Based on the execution continuity between the target kernel functions in the logical task diagram and the event dependencies of the target kernel functions, merge the target start descriptor packages that meet the merging conditions to obtain at least one merged descriptor package. Accordingly, the distribution of each target startup descriptor package contained in the executable task graph to the target memory buffer shared with the device includes: At least one target startup descriptor package and / or at least one merge descriptor package are sent to the target memory buffer shared with the device.

[0070] Specifically, during the formal execution of single-word inference in the large model inference task, after organizing the graph node information corresponding to each target kernel function in the executable task graph into corresponding target startup descriptor packages and storing them in the allocated independent cache areas, the target startup descriptor package information can be extracted from the cache areas corresponding to each target startup descriptor package. The target startup descriptor package information is then concatenated according to the target kernel function execution sequence and stored in a unified contiguous overall cache area. Finally, all target startup descriptor package information in the overall cache area is stored in the target memory buffer shared by the host and the device, completing the target startup descriptor package submission operation.

[0071] The technical solution of this invention involves obtaining the execution sequence of target kernel functions matching the current input large model inference task before large model inference, simulating the execution of the complete target startup descriptor package delivery logic once, intercepting the delivery logic executed for the target framework layer, and obtaining all kernel function description information. Each target kernel function is treated as a graph node and bound to all required operations and kernel function parameters. A matching logical task graph is generated based on the execution dependencies and execution order of each target kernel function. The logical task graph undergoes centralized parameter verification and unified cache allocation processing to complete the parameter configuration and memory filling operations for each target kernel function. After organizing the information of each graph node into kernel function startup descriptor packages and concatenating and storing them, an executable task graph is obtained. During the formal large model inference process, each target startup descriptor package contained in the executable task graph is delivered to a target memory buffer shared with the device, allowing the device to obtain, generate, and output matching single terms. This technical solution unifies the processing of memory resources required by each target kernel function and uniformly distributes the concatenated startup descriptor packages of each target kernel function to the target memory buffer, enabling the preprocessing operations of each kernel function required for large model inference tasks to be completed at once, shortening the scheduling time of the computation kernel function in large model inference tasks and improving the efficiency of lexical generation.

[0072] Figure 11 This invention provides a schematic diagram illustrating the step-by-step optimization of the execution flow of a large model inference process, as shown in the embodiment of the invention. Figure 11 As shown, each layer from top to bottom represents the kernel function processing steps of the host-side inference task based on the traditional large model, corresponding to the step-by-step optimization of the operation process of the above embodiments.

[0073] Figure 11 In the traditional operation mode of the first layer, the preparation phase of each target kernel function (launch0 to launchN) includes parameter verification, parameter memory preparation, parameter configuration and memory filling, kernel function startup package preparation, and kernel function startup package submission. After the initial optimization, as follows... Figure 11As shown in the second-layer graph execution mode scenario, parameter validation and memory preparation for all target kernel functions are completed during the graph instantiation phase. During the graph execution phase, in the preparation processes launch0 to launchN corresponding to each target kernel function, parameter configuration and memory filling, kernel function startup package preparation, and kernel function startup package submission are executed respectively. Further optimization is as follows: Figure 11 As shown in the third-layer graph execution mode optimization parameter memory preparation scenario, during the graph instantiation phase, parameter validation, parameter memory preparation operations, parameter configuration, and memory filling of all target kernel functions are completed. During the graph execution phase, in the preparation operation processes launch0 to launchN corresponding to each target kernel function, kernel function startup package preparation and kernel function startup package submission are executed respectively. Further optimization is as follows: Figure 11 As shown in the fourth layer graph execution mode scenario with added startup package caching, during the graph instantiation phase, all target kernel functions undergo parameter validation, parameter memory preparation, parameter configuration and memory filling, and startup package cache allocation based on corresponding memory during kernel function startup package preparation. During the graph execution phase, the kernel function startup packages are submitted in launch0 to launchN, respectively, during the preparation process for each target kernel function. The final optimization scheme is as follows: Figure 11 The graph execution mode in the fifth layer illustrates the construction of kernel function blocks and the merging of startup packages. During the graph instantiation phase, all target kernel functions undergo parameter validation, parameter memory preparation, parameter configuration and memory filling, and the allocation of corresponding memory for the startup package cache during kernel function startup package preparation. Additionally, the kernel function block construction operation involves concatenating the startup descriptor packages of each target kernel function. During the graph execution phase, in the preparation processes launch0 to launchN for each target kernel function, the merged kernel function startup descriptor package is sent to the target memory buffer, and the corresponding kernel function startup package submission is executed in one go. In actual large-scale model inference task execution, the above optimization schemes can be selected as needed to complete the host-side kernel function preparation work.

[0074] Figure 12 This is a schematic diagram of a large-model inference device provided in an embodiment of the present invention. The device is configured on the host side of a heterogeneous acceleration system. The heterogeneous acceleration system includes a host side and a device side. The host side deploys a multi-level framework layer for the large model, and the device side deploys heterogeneous computing hardware for executing the kernel functions of the large model. Figure 12 As shown, the device includes: a kernel function execution sequence acquisition module 1210, a kernel function description information acquisition module 1220, a task graph generation module 1230, and a startup descriptor packet distribution module 1240.

[0075] The kernel function execution sequence acquisition module 1210 is used to acquire the target kernel function execution sequence that matches the current input large model inference task. The host encapsulates a kernel function and sends a startup descriptor package to the device to trigger the device to execute the kernel function.

[0076] The kernel function description information acquisition module 1220 is used to obtain all kernel function description information required before executing the delivery operation of each target startup descriptor package, based on the complete delivery logic of the target startup descriptor package that matches the target kernel function execution sequence before large model inference. The kernel function description information includes all operations to be executed and all kernel function parameters to be used.

[0077] The task graph generation module 1230 is used to generate a logical task graph that matches the large model inference task based on the kernel function description information corresponding to each target kernel function, and to instantiate the logical task graph to obtain an executable task graph.

[0078] The startup descriptor package distribution module 1240 is used to distribute the startup descriptor packages of each target obtained from the executable task graph to the device during the large model inference process, and generate and output each word that matches the large model inference task.

[0079] The technical solution of this invention is executed on the host side of a heterogeneous acceleration system. The host side encapsulates a startup descriptor package for each kernel function and sends it to the device side to trigger the execution of that kernel function. Before large model inference, the execution sequence of the target kernel function matching the current input large model inference task is obtained. Based on the complete sending logic of the target startup descriptor package matching the target kernel function execution sequence, the host side obtains all operations required by the multi-level framework layer of the large model before the target startup descriptor package sending operation, as well as the kernel function description information consisting of all corresponding kernel function parameters. Based on the kernel function description information corresponding to each target kernel function required by the current large model inference task, a matching logical task graph is generated. After instantiating the logical task graph, an executable task graph is obtained. During large model inference, each target startup descriptor package obtained from the executable task graph is sent to the device side, and each target kernel function is executed to generate each term of the large model inference task. This technical solution integrates the preprocessing stage operations of the large model inference task kernel function to construct an executable task graph, which is then uniformly sent to the device side to execute the inference task operations, effectively reducing the resource overhead of inference task execution and improving the scheduling efficiency of inference tasks.

[0080] Optionally, the kernel function description information acquisition module 1220 can be specifically used to: simulate the execution of the complete target startup descriptor package delivery logic once in the multi-level framework layer of a large model, and when the complete target startup descriptor package delivery logic is executed to the target framework layer, intercept the delivery logic executed for the target framework layer and acquire all intercepted kernel function description information.

[0081] Optionally, the kernel function description information acquisition module 1220 can also be specifically used to: insert start capture nodes and end capture nodes respectively into the delivery logic of the multi-level framework layer of the target startup descriptor package complete delivery logic, so as to enable the stream capture mode for the target startup descriptor package complete delivery logic. In the stream capture mode, the target startup descriptor package complete delivery logic is simulated and executed, and each time the delivery logic of the target framework layer is executed, the delivery logic executed for the target framework layer is intercepted, and all intercepted kernel function description information is obtained.

[0082] Optionally, the task graph generation module 1230 can be specifically used to: treat each target kernel function as a graph node, and bind all operations required for each target kernel function and all kernel function parameters required to be used to the matching graph nodes respectively. Based on the execution dependencies of each target kernel function in the target kernel function execution sequence, and the execution order among the operations required to be executed by each target kernel function, the nodes are organized to obtain a logical task graph.

[0083] Optionally, the task graph generation module 1230 can also be specifically used to: perform centralized parameter verification on each target kernel function in the logical task graph, and after completing the parameter verification, prepare the parameter memory for each kernel function parameter included in the logical task graph, and determine the target address of each kernel function parameter in shared memory to obtain an executable task graph. Correspondingly, the startup descriptor package delivery module 1240 can be specifically used in the single-term inference process of a large model to complete the parameter configuration and memory filling operations for each target kernel function in the target kernel function execution sequence, the preparation operation of the target startup descriptor package, and the submission operation of the target startup descriptor package according to the executable task graph, so as to generate and output the matching single term.

[0084] Optionally, the task graph generation module 1230 can also be specifically used for: performing centralized parameter verification on each target kernel function in the logical task graph, and after completing the parameter verification, preparing parameter memory for each kernel function parameter included in the logical task graph, and determining the target address of each kernel function parameter in shared memory. It can also configure the matching kernel function parameters for each target kernel function and write each kernel function parameter to the matching target address to obtain an executable task graph. Correspondingly, the startup descriptor package delivery module 1240 can be specifically used in the single-term inference process of a large model to complete the preparation operation of the target startup descriptor package for each target kernel function in the target kernel function execution sequence and the target startup descriptor package submission operation based on the executable task graph, so as to generate and output the matching single term.

[0085] Optionally, the task graph generation module 1230 can also be specifically used for: performing centralized parameter verification on each target kernel function in the logical task graph, and after completing the parameter verification, preparing parameter memory for each kernel function parameter included in the logical task graph, and determining the target address of each kernel function parameter in shared memory. After configuring the matching kernel function parameters for each target kernel function and writing each kernel function parameter to the matching target address, a target startup descriptor package matching each target kernel function is constructed to obtain the executable task graph. Correspondingly, the startup descriptor package distribution module 1240 can be specifically used to distribute each target startup descriptor package contained in the executable task graph to the target memory buffer shared with the device at the start of large model inference, so that the device can directly obtain each target startup descriptor package from the target memory buffer during each single lexical inference process.

[0086] Optionally, the task graph generation module 1230 can also be specifically used to: merge at least two target startup descriptor packages that meet the merging conditions based on the execution continuity between the target kernel functions in the logical task graph and the event dependencies of the target kernel functions, to obtain at least one merged descriptor package. Correspondingly, the startup descriptor package distribution module 1240 can be specifically used to distribute at least one target startup descriptor package and / or at least one merged descriptor package to a target memory buffer shared with the device.

[0087] The large model inference device provided in the embodiments of the present invention can execute the large model inference method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0088] Figure 13A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0089] like Figure 13 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0090] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0091] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as inference methods for large models.

[0092] In some embodiments, the large model inference method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the large model inference method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the large model inference method by any other suitable means (e.g., by means of firmware).

[0093] That is, to obtain the target kernel function execution sequence that matches the current input large model inference task, wherein the host encapsulates a kernel function and sends a startup descriptor package to the device to trigger the device to execute the kernel function; Before large model inference, based on the complete delivery logic of the target startup descriptor package that matches the target kernel function execution sequence, we obtain all the kernel function description information required before executing the delivery operation of each target startup descriptor package. The kernel function description information includes all the operations to be executed and all the kernel function parameters to be used. Based on the kernel function description information corresponding to each target kernel function, a logical task graph matching the large model inference task is generated, and the logical task graph is instantiated to obtain an executable task graph. During the large model inference process, the target startup descriptor packages obtained from the executable task graph are sent to the device to generate and output each lexical that matches the large model inference task.

[0094] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0095] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0096] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0097] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0098] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0099] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0100] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0101] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A reasoning method for large models, characterized in that, Executed by the host side of a heterogeneous acceleration system, which includes a host side and a device side, the method includes: The host side deploys a multi-level framework layer for the large model, and the device side deploys heterogeneous computing hardware for executing the kernel functions of the large model. Obtain the target kernel function execution sequence that matches the current input large model inference task. Here, the host encapsulates a kernel function and sends a startup descriptor package to the device to trigger the device to execute the kernel function. Before large model inference, based on the complete delivery logic of the target startup descriptor package that matches the target kernel function execution sequence, all kernel function description information required before executing the delivery operation of each target startup descriptor package is obtained. The kernel function description information includes all operations to be executed and all kernel function parameters to be used. Based on the kernel function description information corresponding to each target kernel function, a logical task graph matching the large model inference task is generated, and the logical task graph is instantiated to obtain an executable task graph. During the large model inference process, the target startup descriptor packages obtained from the executable task graph are sent to the device to generate and output each lexical that matches the large model inference task.

2. The method according to claim 1, characterized in that, Before large model inference, based on the complete target startup descriptor package delivery logic matching the target kernel function execution sequence, we obtain all the kernel function description information required before executing the delivery operation of each target startup descriptor package, including: In the multi-level framework layer of the large model, the complete delivery logic of the target startup descriptor package is simulated once. When the complete delivery logic of the target startup descriptor package reaches the target framework layer, the delivery logic executed for that target framework layer is intercepted, and all intercepted kernel function description information is obtained.

3. The method according to claim 2, characterized in that, In the multi-level framework layer of the large model, the complete delivery logic of the target startup descriptor package is simulated once. When the complete delivery logic of the target startup descriptor package reaches the target framework layer, the delivery logic executed for that target framework layer is intercepted, and all intercepted kernel function description information is obtained, including: In the delivery logic of the multi-level framework layer of the target startup descriptor packet complete delivery logic, start capture node and end capture node are respectively inserted to enable the stream capture mode for the target startup descriptor packet complete delivery logic. In stream capture mode, the complete delivery logic of the target startup descriptor package is simulated and executed. Each time the delivery logic of the target framework layer is executed, the delivery logic executed for the target framework layer is intercepted and all kernel function description information intercepted is obtained.

4. The method according to claim 2, characterized in that, The multi-level framework of the large model includes: a top-down inference service scheduling layer, a deep learning framework layer, a computation implementation layer, a system interface layer, and a driver layer, wherein the target framework layer is the system interface layer.

5. The method according to any one of claims 1-4, characterized in that, Based on the kernel function description information corresponding to each target kernel function, a logical task graph matching the large model inference task is generated, including: Each target kernel function is treated as a graph node, and all operations to be performed and all kernel function parameters to be used for each target kernel function are bound to the matching graph node respectively; Based on the execution dependencies of each target kernel function in the target kernel function execution sequence, and the execution order between the operations required by each target kernel function, the nodes are organized to obtain a logical task graph.

6. The method according to any one of claims 1-4, characterized in that, The logical task graph is instantiated to obtain the executable task graph, including: The parameters of each target kernel function in the logical task graph are centrally verified. After the parameter verification is completed, the parameter memory of each kernel function parameter included in the logical task graph is prepared, and the target address of each kernel function parameter in the shared memory is determined to obtain the executable task graph. Accordingly, during the large model inference process, the target startup descriptor packages obtained from the executable task graph are sent to the device, generating and outputting various lexical units that match the large model inference task, including: In the single-term inference process of the large model, the parameter configuration and memory filling operations of each target kernel function in the target kernel function execution sequence are completed according to the executable task graph, as well as the preparation operation of the target startup descriptor package and the submission operation of the target startup descriptor package, so as to generate and output the matching single term.

7. The method according to any one of claims 1-4, characterized in that, The logical task graph is instantiated to obtain the executable task graph, including: A centralized parameter verification is performed on each target kernel function in the logical task graph. After the parameter verification is completed, parameter memory preparation is performed for each kernel function parameter included in the logical task graph, and the target address of each kernel function parameter in shared memory is determined. Configure the matching kernel function parameters for each target kernel function, and write each kernel function parameter into the matching target address to obtain the executable task graph; Accordingly, during the large model inference process, the target startup descriptor packages obtained from the executable task graph are sent to the device, generating and outputting various lexical units that match the large model inference task, including: During the single-term inference process of the large model, the preparation operation of the target start descriptor package for each target kernel function in the target kernel function execution sequence and the target start descriptor package submission operation are completed according to the executable task graph to generate and output the matching single term.

8. The method according to any one of claims 1-4, characterized in that, The logical task graph is instantiated to obtain the executable task graph, including: A centralized parameter verification is performed on each target kernel function in the logical task graph. After the parameter verification is completed, parameter memory preparation is performed for each kernel function parameter included in the logical task graph, and the target address of each kernel function parameter in shared memory is determined. Configure the matching kernel function parameters for each target kernel function, and write the kernel function parameters to the matching target address. Then, construct the target startup descriptor package that matches each target kernel function to obtain the executable task graph. Accordingly, during the large model inference process, the target startup descriptor packages obtained from the executable task graph are sent to the device, generating and outputting various lexical units that match the large model inference task, including: At the start of large model inference, the target startup descriptor packages contained in the executable task graph are distributed to the target memory buffer shared with the device, so that the device can directly obtain the target startup descriptor packages from the target memory buffer during each single lexical inference process.

9. The method according to claim 8, characterized in that, After constructing the target startup descriptor package that matches each target kernel function, the following is also included: Based on the execution continuity between the target kernel functions in the logical task diagram and the event dependency of each target kernel function, a merging operation is performed on at least two target start descriptor packages that meet the merging conditions to obtain at least one merged descriptor package. Accordingly, the distribution of each target startup descriptor package contained in the executable task graph to the target memory buffer shared with the device includes: At least one target startup descriptor package and / or at least one merge descriptor package are sent to the target memory buffer shared with the device.

10. A large-scale model inference device, characterized in that, Configured on the host side of a heterogeneous acceleration system, the heterogeneous acceleration system includes a host side and a device side. The host side deploys a multi-level framework layer for large models, and the device side deploys heterogeneous computing hardware for executing the kernel functions of large models. This device includes: The kernel function execution sequence acquisition module is used to acquire the target kernel function execution sequence that matches the current input large model inference task. The host encapsulates a kernel function and sends a startup descriptor package to the device to trigger the device to execute the kernel function. The kernel function description information acquisition module is used to obtain all kernel function description information required before executing the delivery operation of each target startup descriptor package, based on the complete delivery logic of the target startup descriptor package that matches the target kernel function execution sequence before large model inference. The kernel function description information includes all operations to be executed and all kernel function parameters to be used. The task graph generation module is used to generate a logical task graph that matches the large model inference task based on the kernel function description information corresponding to each target kernel function, and to instantiate the logical task graph to obtain an executable task graph. The startup descriptor package delivery module is used to deliver the startup descriptor packages of each target obtained from the executable task graph to the device during the large model inference process, and generate and output each lexical that matches the large model inference task.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the reasoning method for the large model according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the reasoning method of the large model according to any one of claims 1-9.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the reasoning method for the large model according to any one of claims 1-9.

Images