A heterogeneous computing execution scheduling method, system and application

Abstract

This invention discloses a heterogeneous computing execution scheduling method for cold-start inference, comprising: during the model cold-start phase, dividing the model into a prefix part and a suffix part; scheduling a first computing unit to perform model prefix computation, and scheduling a second computing unit to perform asynchronous loading of model suffix weights, generating the first token after suffix computation; during the generation of the first token, loading the model prefix weights into the second computing unit in parallel; after the model weights are loaded, constructing a cache state for subsequent inference on the second computing unit, so that the inference computation of subsequent tokens is executed continuously on the second computing unit. This invention also discloses a system and application for implementing the above method.

Description

Technical Field

[0001] This invention belongs to the field of large model technology and relates to a heterogeneous computing execution scheduling method, system and application. Background Technology

[0002] With the rapid development of large model technology, the scale of model parameters is constantly increasing, and the amount of weight data that needs to be loaded during model inference has increased significantly. In actual deployment environments, model weights are usually stored in host memory or external storage media and need to be loaded into GPU memory through the transmission channel from host memory to GPU memory. Since the bandwidth between host memory and GPU memory is usually significantly lower than the internal bandwidth of GPU memory, the model weight loading process often becomes the main performance bottleneck in the cold start phase of inference, directly affecting the generation latency of the first output token.

[0003] To reduce the impact of weight loading on inference performance, existing technologies generally adopt an overlapping loading and computation approach, layering or dividing weights into layers or blocks, gradually loading the required weights during model inference, and simultaneously performing corresponding computation operations based on the already loaded weights. Through pipelined operations, the weight loading delay is hidden to some extent.

[0004] For example, during the computation of the k-th layer of the model, the system can preload the weight data of the (k+1)-th layer or subsequent layers, allowing the weight loading process to partially overlap with the model computation in time. This approach reduces the idle waiting time of the acceleration device during inference, improving overall resource utilization efficiency.

[0005] However, the above technical solutions are based on the premise that the computation of a certain layer in the model can only be executed after the weights of that layer have been loaded. Therefore, during the cold start phase of model inference, the generation of the first output token is still subject to the rigid constraint of the completion of weight loading for the first layer and the preceding few layers of the model. When the model weight loading time is significantly greater than the single-layer computation time, even if layered loading or overlapping loading and computation are adopted, the generation latency of the first output token is still mainly determined by the weight loading process, making it difficult to achieve a substantial reduction. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a heterogeneous computing execution scheduling method, system, and application. This invention is applied to the model cold start phase, reducing the generation time of the first token by reconstructing the critical inference path without altering the model structure or inference results.

[0007] In a first aspect, the present invention provides a heterogeneous computing execution scheduling method, the method being geared towards cold-start inference, comprising:

[0008] During the cold start phase of the model, the model is divided into a prefix part and a suffix part;

[0009] If the model weights are not fully loaded, the first computing unit is scheduled to perform the inference calculation of the prefix part, while the second computing unit is scheduled to perform the loading of the weights of the suffix part.

[0010] After the prefix part is calculated and the suffix part weight is loaded, the inference calculation of the suffix part is performed to generate the first output token;

[0011] During the generation of the first output token, the model prefix weights are loaded into the second computing unit in parallel.

[0012] The remaining weights of the model are loaded, and the inference of subsequent tokens is executed continuously on the second computing unit.

[0013] This invention divides the model into a prefix and a suffix during the cold start phase and enables the prefix calculation and suffix weight loading to be performed in parallel. This makes the generation of the first output token no longer dependent on the completion of loading all model weights, thereby shortening the first token generation time during the cold start phase.

[0014] In one possible implementation, the first computing unit is a CPU and the second computing unit is a GPU. Prefix computation and postfix inference are deployed on the CPU and GPU respectively, allowing different computing resources to undertake their respective computational tasks, thus improving the utilization efficiency of heterogeneous computing resources.

[0015] The prefix portion includes layers 1 to K of the model, and the suffix portion includes layers K+1 to the last layer, where K is a positive integer less than the total number of layers in the model. Dividing the model into a prefix and a suffix portion by layer separates prefix calculation from suffix weight loading, providing a basis for parallel execution during the cold start phase.

[0016] The first computing unit performs inference calculations for the prefix portion to generate intermediate activation data. After the weights for the suffix portion are loaded, the intermediate activation data is submitted to the second computing unit to perform inference calculations for the suffix portion. Submitting intermediate activation data to the suffix inference stage after the prefix calculation is completed allows the suffix inference calculation to be started directly based on the prefix calculation result, thereby reducing the waiting time during the initial token generation process.

[0017] The inference computation of the prefix part is a true forward computation, which is performed layer by layer according to the model's standard inference computation graph and parameter configuration, consistent with the complete model. The prefix part performs true forward computation and maintains a computation path consistent with the complete model, so that the intermediate activation results are consistent with the inference results of the complete model in terms of numerical semantics, thereby ensuring the correctness of the inference results.

[0018] The loading process of the suffix weight does not block the inference calculation of the prefix, allowing the two to be executed in parallel in time.

[0019] In the specific implementation process, when the first output token is generated, the weights of the prefix portion have not yet been fully loaded into the second computing unit. This invention generates the first output token before all prefix weights are loaded into the second computing unit, thus reducing cold start latency because the first token generation path no longer depends on the full weight loading.

[0020] After generating the first output token, the second computing unit performs supplementary calculations on the prefix part based on the loaded prefix part weights to build a consistent cache state for subsequent token inference, thereby ensuring that the subsequent token inference process is consistent with the regular inference process.

[0021] The number of layers K in the prefix portion is dynamically determined based on the operating environment, including:

[0022] Obtain the number of prompts, the computing power of the first computing unit, and the data transfer bandwidth from the host memory to the storage unit of the second computing unit;

[0023] The number of prefix layers K is determined based on the parameters to match the prefix calculation time with the suffix weight loading time.

[0024] The number of prefix layers K is dynamically determined based on the operating environment, so that the prefix calculation time matches the suffix weight loading time, thereby improving the overlap of parallel execution and further shortening the latency of first token generation.

[0025] Secondly, the present invention also provides a heterogeneous computing execution scheduling system for cold-start inference, comprising:

[0026] The prefix calculation module is used to perform inference calculations on the prefix part of the model during the cold start phase;

[0027] The inference execution module is used to perform inference calculations on the model suffix to generate the first output token, and to perform continuous inference on subsequent tokens after the full weights of the model are prepared.

[0028] The storage and data transmission module is used to store model weights and support asynchronous loading of model weights between the host memory and the storage unit of the second computing unit.

[0029] The inference scheduling module is used to schedule the prefix calculation module and the inference execution module to work together to generate the first output token when the model weights have not been fully loaded.

[0030] This invention establishes a collaborative structure comprising a prefix calculation module, an inference execution module, a storage and data transmission module, and an inference scheduling module, enabling the system to perform parallel execution of calculation and weight loading during the cold start phase, thereby reducing the delay in first token generation.

[0031] Thirdly, the present invention also provides the application of the above-described method or system in heterogeneous computing execution scheduling for cold-start inference.

[0032] Applying the method or system to large model cold start inference scenarios enables the model to generate the first output token faster during the first inference, thereby improving the user interaction response experience.

[0033] The beneficial effects of this invention include: During the model cold start phase, by dividing the model into a prefix and a suffix, the first computation unit performs the true forward computation of the prefix, while the second computation unit asynchronously loads the weights of the suffix. Immediately after the prefix computation and suffix weight loading are complete, the suffix computation is executed to generate the first output token, thus eliminating the dependency on the loading of all model weights. By reconstructing the critical path of inference during the cold start phase, the prefix computation path and the suffix weight loading path are executed in parallel, thereby shortening the latency of the first token generation. After the first token is generated, by performing supplementary computation on the prefix on the second computation unit side to build a consistent cache state, subsequent token inference remains consistent with the conventional second computation unit inference process, achieving improved cold start performance without changing the model structure and inference results. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This diagram illustrates the system composition in one possible implementation of the present invention.

[0036] Figure 2The timing diagram for loading and computation of the pipeline.

[0037] Figure 3 This is a timing diagram illustrating the loading and calculation of the present invention.

[0038] Figure 4 This is a flowchart illustrating the cold start inference of the first token in Embodiment 1 of the present invention.

[0039] Figure 5 This is a flowchart illustrating the reasoning process for subsequent tokens in Embodiment 1 of the present invention.

[0040] Figure 6 A flowchart illustrating the calculation of the prefix layer number in this invention. Detailed Implementation

[0041] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0042] This invention provides a method and system for accelerating cold-start inference of large models. The method reconstructs the dependency relationship between weight loading and inference execution during the model cold-start phase, so that the generation of the first output token no longer depends on the completion of loading all model weights. When generating the first output token, only the suffix weights of the model are loaded to complete the inference calculation. Simultaneously, the prefix weights of the model are loaded during inference execution, and this loading process does not block the generation of the first output token. Before subsequent token inference begins, the model prefix layer is recalculated based on the loaded prefix weights to build a cached state, thereby enabling continuous execution of subsequent tokens on the accelerated device. This method effectively reduces the latency of the first output by shortening the critical path of cold start.

[0043] In one possible implementation, the first computing unit in this invention can be a CPU, the second computing unit can be a GPU, and the storage unit of the second computing unit is GPU video memory.

[0044] This invention provides an inference system for accelerating cold-start inference of large models. During the cold-start phase, the model is logically divided into a prefix and a suffix. The prefix includes layers 1 to K, and the suffix includes layers K+1 to N. The inference computation of the prefix layer is separated from the GPU execution path. While loading the suffix layer weights into GPU memory, the CPU uses weights in host memory to complete the prefix computation first and provides the calculated activation values ​​to the GPU. This allows the GPU to start computation once the suffix weights are loaded. The number of prefix layers is determined based on the number of prompts obtained during the inference startup phase, the CPU's computing power parameters, and the data transfer bandwidth between host memory and GPU memory, according to the actual scenario.

[0045] like Figure 1 The diagram illustrates the heterogeneous computing inference system architecture in one possible implementation of the present invention. The system comprises a CPU side, a GPU side, and a storage and data transfer section between them. The CPU side includes an inference scheduling module and a prefix computation module, responsible for scheduling and controlling the model execution during the cold start phase and using prefix weights in host memory to complete the inference computation of the prefix layer. The GPU side includes an inference execution module, primarily used to perform inference computation of the model's suffix portion and continuous inference of subsequent tokens after the suffix weights are loaded. Host memory and GPU memory are connected via a PCIe channel, forming a storage and data transfer module. This module enables asynchronous transmission of model weights and intermediate activation data between host memory and GPU memory, thereby supporting the parallel execution of prefix computation and suffix weight loading in time, and ultimately completing the inference task during the cold start phase.

[0046] Specifically, each module is described below:

[0047] 1. Prefix Calculation Module:

[0048] The prefix computation module is executed on the CPU, directly using the weights in the host memory to complete the true forward computation of the first K layers of the model and generate the corresponding intermediate activation tensors.

[0049] In this invention, the true forward computation refers to performing forward propagation operations on the corresponding layers of the input data sequentially according to the standard computation graph and parameter definitions of the target model in the inference stage. The network structure, weight parameters, calculation formulas and numerical precision used in the computation are consistent with the forward computation of the corresponding layers in the complete model inference process, and the intermediate output results obtained are consistent with the results obtained by performing the same layer computation in the complete model inference in terms of numerical semantics.

[0050] When the prefix calculation is performed, it is necessary to control the calculation to only calculate the first K layers and obtain the intermediate activation tensors, rather than calculating the entire model.

[0051] 2. Inference Execution Module:

[0052] Used to perform the complete inference computation of the model suffix part and subsequent tokens after the suffix weights have been loaded.

[0053] The inference execution module is the core computing unit responsible for accelerating the actual inference task in the system. During the cold start phase, when the inference scheduling module detects that the suffix weights have been loaded and the intermediate activation data generated by the prefix calculation is ready, the inference execution module uses the intermediate activation data as input and executes the inference calculations of each suffix layer sequentially on the GPU according to the model's standard forward computation path, thereby generating the first output token. In this phase, the inference execution module only needs the weight data of the suffix part to complete the calculation of the first token, therefore it does not need to wait for all the prefix weights to be loaded before starting the inference process.

[0054] After the first token is generated, the inference execution module continues to work with the inference scheduling module to perform prefix recalculation. This involves loading the prefix weights into GPU memory and then re-compiling the prefix portion to build a complete and consistent key-value cache on the GPU, ensuring the system's inference state remains consistent with regular GPU inference. Once the recalculation is complete and all model weights and the key-value cache are ready, the inference execution module enters stable inference mode. At this point, all subsequent token generation is performed continuously on the GPU using the complete model, guaranteeing high throughput and stable performance in subsequent inference phases.

[0055] Throughout the execution process, the inference execution module performs forward operations at each layer according to the model's standard computation graph and parameter configuration. The network structure, weight parameters, calculation formulas, and numerical precision used in its calculations are consistent with those of the complete model inference, thereby ensuring that the inference results of the first token and subsequent tokens are consistent with the conventional inference process in terms of numerical semantics. This achieves cold start performance optimization without changing the model structure and inference results.

[0056] 3. Storage and data transmission module:

[0057] Used to store model weights and support asynchronous loading, including:

[0058] The host memory is used to store model prefix weights and data required for CPU inference during the cold start phase; it is accessed by the first computing unit (CPU) and used to store model weights and intermediate data.

[0059] GPU memory is used to store model suffix weights and the final full model weights after loading.

[0060] The asynchronous data transfer unit is used to load model weights from storage media (such as host memory) to GPU memory without blocking computation, and to transfer intermediate activation data from host memory to GPU memory when needed.

[0061] During the cold start phase, the CPU performs prefix computation on the host side while simultaneously performing asynchronous loading of suffix weights. This gradually fills the GPU memory with the weight set required for suffix computation, creating a state where weight loading and computation are executed in parallel. Once the suffix weights are loaded, the module transfers the intermediate activation data generated on the CPU side to the GPU memory for direct use by the inference execution module. Subsequently, after the first token is generated, the module continues loading the prefix weights, forming a complete model weight layout in the GPU memory. This provides the necessary data foundation for prefix completion computation and subsequent stable inference. Through this method, the storage and data transfer module undertakes the functions of weight transport and intermediate data transfer throughout the cold start process, enabling the inference execution path to start ahead of time even before the weights are fully loaded.

[0062] 4. Inference Scheduling Module:

[0063] The inference scheduling module, used to uniformly control the execution order and parallel relationships of each stage during the cold start inference process, is a key component of this invention. This module is configured as follows:

[0064] Based on the model structure, the model is divided into a prefix part and a suffix part;

[0065] During the cold start phase, the CPU is scheduled to perform model prefix calculations, while the GPU is scheduled to perform asynchronous loading of model suffix weights.

[0066] Manage and cache intermediate activation data generated by prefix computation, and submit it to the GPU execution unit when the execution conditions are met;

[0067] Once the prefix calculation is completed and the suffix weights are loaded, the GPU is triggered to perform suffix calculation to generate the first token.

[0068] During the generation of the first token, the loading of the model prefix weights into the GPU memory continues to be scheduled, and after the first token is generated, the GPU is controlled to perform supplementary calculations on the prefix part to generate a consistent KV cache;

[0069] Before inference begins for the second token, confirm that all model weights and the KV cache are ready, and schedule the inference of subsequent tokens entirely to the GPU.

[0070] In this invention, since the model calculation to generate the second token relies on the KV cache calculated in the first calculation, the first K layers are calculated in the CPU, and the generated KV cache is stored in the host memory. Therefore, it is necessary to perform supplementary calculations to ensure that all subsequent calculations are performed on the GPU.

[0071] By decoupling the inference computation of the model's prefix layer from the GPU execution path and having the CPU perform the prefix computation using weights in the host memory, the GPU only needs to wait for the suffix weights to finish loading before it can begin inference computation. In the cold start phase, the generation of the first output token no longer depends on the completion of loading the complete weights, thereby reducing the latency of generating the first token.

[0072] The timing of existing pipelined loading methods is as follows: Figure 2 As shown (assuming all weights are currently stored in the host memory), model inference proceeds layer by layer. After loading the weights for the first layer, the first layer computation begins, while the weights for the second layer are loaded in parallel. Subsequently, after the first layer computation is complete, the second layer computation is performed using the already loaded weights, and the weights for the third layer are loaded in parallel, and so on, forming a pipelined overlapping execution structure of "compute the current layer, load the next layer". In this way, the weight loading process partially overlaps with the computation of adjacent layers in time, reducing the idle time of computational units. However, the computation of each layer still depends on the completion of the weight loading for that layer; therefore, the generation of the first token still requires waiting for the weights of the previous few layers to be loaded.

[0073] Specifically, the existing loading method processes the computation of layer K and the loading of layer K+1 in parallel, with loading and computation overlapping between adjacent layers, thus hiding some loading time. However, since the computation of each layer must wait for the weights of that layer to be loaded, let T_load(i) be the loading time of the weights of layer i, and T_compute(i) be the computation time of layer i, then the generation time of the first token is:

[0074]

[0075] In cold start scenarios, weight loading time is often much longer than computation time. Taking a consumer-grade RTX 4090 graphics card as an example, the weights are loaded into the GPU memory via PCIe 4.0 x16, with a one-way bandwidth of approximately 16GB / s. Considering system overhead, the actual bandwidth is approximately 12GB / s. For the llama3 8B model, the weights are approximately 15GB, and the weight loading time is approximately 1.25s. In actual testing using llama.cpp with a prompt of 5 tokens, the first token takes approximately 25ms to load, and the computation time for the last layer is less than 1ms.

[0076] Therefore, the generation time of the first token is approximately:

[0077]

[0078] like Figure 3 The diagram illustrates the timing process of parallel execution of prefix computation and suffix weight loading during the cold start phase of this invention. The CPU is responsible for performing layer-by-layer computation of the model's prefix portion, while the GPU simultaneously loads the model's suffix weights from back to front. After system startup, the CPU first computes the prefix layer of layer 1, while the GPU loads the suffix weights of layers N, N-1, etc., in parallel. As the CPU's prefix computation progresses to layer K, the GPU also gradually completes the loading of weights for layer K+1 and subsequent layers. Once the CPU completes the computation of layer K and the GPU completes the suffix weight loading, the GPU can directly execute the computation of layers K+1 to N based on the intermediate activations generated by the CPU, thereby generating the first token. This achieves time overlap between the prefix computation path and the suffix loading path, shortening the generation latency of the first token during the cold start phase.

[0079] When prefix computation and suffix weight loading are executed in parallel, as long as the prefix computation and intermediate activation transfer time do not exceed the suffix weight loading time, the model's first output token can be generated before the model weights are fully loaded into the GPU memory, thus significantly reducing the generation time of the first token during the model's cold start phase. Let T_compute_CPU(i) be the CPU time for computing the prefix layer, T_compute_GPU(i) be the GPU time for computing the suffix layer, and T_transfer(activation) be the activation value transfer time.

[0080]

[0081] Since the prefix calculation and suffix weight loading of the CPU are executed in parallel, the amount of activation data transferred is much smaller than the model weight size. By appropriately selecting the K value, it is possible to... and They are approximately equal.

[0082]

[0083] This inequality holds as long as the time for the GPU to compute layers K+1 to N is greater than the loading time for the first K layers.

[0084] In implementing this invention, the model structure and inference results remain unchanged. The only change is the output time of the first token. Instead of waiting for all weights to be loaded into GPU memory before outputting the first token, the process now allows for outputting the first token without requiring all weights to be loaded, thus shortening the time required for outputting the first token. In fact, because the first K layers need to be recalculated, the computation time for the first two tokens is actually slightly longer. Subsequent calculations are identical to conventional inference, but the significantly shorter time for the first token improves the user experience.

[0085] Example 1: A method to accelerate cold start inference for large models

[0086] This example demonstrates the first token process in cold start inference, such as... Figure 4 As shown.

[0087] 1. Receive input and initialize cold start inference state

[0088] After receiving a user input request, the inference system enters a cold start inference state. At this time, the model weights have not yet been loaded into the GPU memory, and only a minimal runtime environment initialization is completed, including the initialization of CPU-side inference operators, memory management modules, and asynchronous loading and scheduling modules.

[0089] 2. The CPU performs model prefix calculation, while the GPU asynchronously loads the model suffix weights.

[0090] The system divides the model into a prefix part (layer 1 to layer K) and a suffix part (layer K+1 to the last layer).

[0091] If the GPU weights are not fully loaded, the CPU uses the weights corresponding to the prefix to perform prefix inference calculations on the user input in the host memory to obtain intermediate activation results.

[0092] Meanwhile, the GPU side uses a DMA data transfer mechanism to load the weight data of the model suffix from the storage medium (such as host memory) to the GPU memory, so that the prefix calculation and suffix weight loading can be performed in parallel in time.

[0093] 3. Trigger the suffix calculation of the first token.

[0094] When the CPU completes the prefix calculation and the GPU has finished loading the suffix weights, the system passes the intermediate activation result obtained from the prefix calculation to the GPU, immediately executes the calculation of the suffix part of the model on the GPU and continues to load the prefix weights, at which point the output result of the first token is generated.

[0095] In this step, the weights of the model prefix are not yet fully loaded onto the GPU, but this does not affect the generation of the first token.

[0096] 4. After the first token is generated, continue loading the prefix weight and recalculating the prefix KV cache.

[0097] After the first token is output, if the model prefix part has not been fully loaded, the system continues to load the weights of the model prefix part (layers 1 to K) into the GPU memory.

[0098] After the prefix weights are loaded, the GPU re-executes the prefix calculation on the prefix part corresponding to the first token to generate a complete and consistent prefix KV cache for inference calculation of subsequent tokens.

[0099] 5. Subsequent tokens will be continuously inferred on the GPU using the full model.

[0100] Before the second token begins calculation, all model weights and the key-value cache are ready. Afterward, the system uses the complete model on the GPU to perform continuous inference on subsequent tokens, without involving the CPU in computation, thus ensuring high throughput and stable performance in subsequent inference stages. Figure 5 As shown.

[0101] Example 2: A Cold Start Inference Acceleration Method with Dynamically Adjustable Prefix Layer Number

[0102] Based on Example 1, this example provides a cold start inference acceleration method with dynamically adjustable prefix layer number.

[0103] Unlike Example 1, the number of layers included in the model prefix in this example is not a fixed value, but is dynamically determined by the inference scheduling module based on the operating environment.

[0104] During the inference startup phase, the inference scheduling module obtains the number of prompts, the CPU's computing power parameters, and the data transfer bandwidth status from host memory to GPU memory, and determines the number of layers K of the prefix part based on the parameters.

[0105] When the number of prompts is small, CPU computing power is high, or the bandwidth from host memory to GPU memory is limited, the inference scheduling module selects a larger number of prefix layers to enable more layers of inference computation to be completed on the CPU; when the load on the first computing unit is high or the suffix weights are loaded quickly, the inference scheduling module selects a smaller number of prefix layers to reduce prefix computation time.

[0106] Regardless of how the number of prefix layers is adjusted, the inference scheduling module ensures that the calculation of the prefix part and the loading of the weights of the suffix part are executed in parallel in time, and triggers the inference calculation of the suffix part to generate the first output token after the execution conditions are met.

[0107] Through the above method, this embodiment can adaptively shorten the generation latency of the first output token under different hardware environments and load conditions, thereby further improving cold start inference performance.

[0108] like Figure 6 As shown, in this embodiment, the process for determining the number of prefix layers N is as follows:

[0109] First, obtain the model structure parameters and system operating environment parameters, including the total number of layers in the model, the calculation type and parameter scale of each layer, the bandwidth from host memory to GPU memory, and the CPU's computing power.

[0110] The loading time of each layer's weights is calculated based on the weight scale of each layer in the model and the bandwidth from the host to the acceleration device.

[0111] The time taken for each layer of the model to perform forward computation on the host side is calculated based on the number of tokens in the prompt and the previously obtained model structure parameters and system operating environment parameters.

[0112] The prefix layer number is incremented starting from 1, and the corresponding prefix calculation time and suffix weight loading time are calculated.

[0113] Determine whether the two meet the time matching condition. If they do, then determine that the current layer is the prefix layer.

[0114] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0115] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0116] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0118] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0119] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0120] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A heterogeneous computing execution scheduling method, characterized in that, The method is geared towards cold-start inference and includes: During the cold start phase of the model, the model is divided into a prefix part and a suffix part; The first computing unit is scheduled to perform model prefix calculation, and the second computing unit is scheduled to perform asynchronous loading of model suffix weights. After performing suffix calculation, the first token is generated. During the generation of the first token, the model prefix weights are loaded into the second computing unit in parallel. After the model weights are loaded, a cache state for subsequent inference is built on the second computing unit, so that the inference calculation of subsequent tokens is executed continuously on the second computing unit.

2. The method as described in claim 1, characterized in that, The first computing unit is a CPU, and the second computing unit is a GPU.

3. The method as described in claim 1, characterized in that, The prefix part includes the first to the Kth layers of the model, and the suffix part includes the (K+1)th to the last layer of the model, where K is a positive integer less than the total number of layers in the model.

4. The method as described in claim 1, characterized in that, The first computing unit performs inference calculations on the prefix portion to generate intermediate activation data, and after the weights of the suffix portion are loaded, submits the intermediate activation data to the second computing unit to perform inference calculations on the suffix portion. The inference calculation of the prefix part is a true forward calculation, which is performed layer by layer according to the model standard inference calculation graph and parameter configuration, consistent with the complete model.

5. The method as described in claim 1, characterized in that, The loading process of the suffix weight does not block the inference calculation of the prefix, allowing the two to be executed in parallel in time.

6. The method as described in claim 1, characterized in that, When the first output token is generated, the weights of the prefix portion have not yet been fully loaded into the second calculation unit.

7. The method as described in claim 1, characterized in that, After generating the first output token, the second computing unit performs supplementary calculations on the prefix part based on the loaded prefix part weights to construct a consistent cache state for subsequent token inference.

8. The method as described in claim 1, characterized in that, The number of layers K in the prefix portion is dynamically determined based on the operating environment, including: Obtain the number of prompts, the computing power of the first computing unit, and the data transfer bandwidth from the host memory to the storage unit of the second computing unit; The number of prefix layers K is determined based on the parameters to match the prefix calculation time with the suffix weight loading time.

9. A heterogeneous computing execution scheduling system, characterized in that, The system includes: The prefix calculation module is used to perform inference calculations on the prefix part of the model during the cold start phase; The inference execution module is used to perform inference calculations on the model suffix and subsequent tokens on the second computing unit; The storage and data transmission module is used to store model weights and support asynchronous loading of weights between the host memory and the storage unit of the second computing unit. The inference scheduling module is used to schedule the prefix calculation module and the inference execution module to work together to generate the first output token when the model weights have not been fully loaded.

10. The method of any one of claims 1-8, or the system of claim 9, in the scheduling of heterogeneous computing execution for cold-start inference.

Images