A method and system for unloading tensors from large model inference in resource-constrained scenarios

Abstract

This invention discloses a method and system for unloading tensors from large model inference in resource-constrained scenarios. The method includes: loading operator semantic knowledge onto target operators; calculating the latency optimization benefit obtainable per unit GPU memory resource for the target operator and normalizing it as GPU affinity, then sorting the GPU affinity in descending order to generate a preliminary priority queue G; introducing cross-device tensor loading latency for the target operators in the preliminary priority queue G to dynamically adjust the operator deployment order, thereby obtaining a final priority queue G'; determining whether the final priority queue G' equals the preliminary priority queue G, and if not, proceeding to step S102 to continue iteration; otherwise, generating a placement scheme for the operator's tensors between the GPU and CPU based on the final priority queue G'. This invention aims to improve the speed of large model inference in resource-constrained scenarios.

Description

Technical Field

[0001] This invention belongs to the field of edge computing, specifically relating to a method and system for unloading large model inference tensors in resource-constrained scenarios. Background Technology

[0002] In recent years, generative large language models (LLMs) have become a core driving force for the development of artificial intelligence. With their superior language understanding and generation capabilities, large language models have performed exceptionally well in language-related fields and are widely used in areas such as automated programming, image generation, and personalized assistants. Large language models based on the Transformer architecture, such as the GPT series, LlaMA series, as well as MiniCPM and DeepSeek, have further reshaped the landscape of machine learning (ML) and natural language processing (NLP). However, these models are typically large in scale and can only be deployed in data centers equipped with numerous expensive server-grade GPUs. Meanwhile, with the increasing demand for enhanced data privacy protection, model customization, and reduced inference costs, local deployment of large language models (such as on resource-constrained terminals like personal computers equipped with consumer-grade GPUs) has become a new trend. Unlike the high throughput requirements of data center deployments, local deployments focus more on the low latency requirements when processing small batches of data.

[0003] Meanwhile, due to the scale, complexity, and high memory and computing resource requirements of large language models, deployment on consumer-grade GPUs still faces significant challenges. In particular, large language models generate text using an autoregressive iterative approach, requiring access to the entire model containing hundreds of billions of parameters for each generation. This severely limits the inference process to GPU memory, especially in resource-constrained consumer-grade GPU scenarios. Currently, model offloading technology is a typical solution in resource-constrained scenarios. Model offloading refers to offloading the computational and memory resources of the model portion from the main device to other storage or computing devices to reduce memory consumption. This technology provides a good idea and solution for running large language models in resource-constrained environments; however, limitations in PCIe interconnect speed and CPU computing power result in low inference efficiency and high latency.

[0004] In the field of deep learning, operators and tensors are key concepts for understanding the operational mechanism of large language models. A tensor is a generalized array; it can be a scalar (zero-dimensional tensor), a vector (one-dimensional tensor), a matrix (two-dimensional tensor), or even a higher-dimensional data structure. In large language models, it is used to represent the model's input, output, and intermediate computation results. Operators, on the other hand, are functions applied to tensors, responsible for performing specific mathematical operations, such as matrix multiplication and activation function calculations. They are the basic operational units for model inference and training. Because operators in each layer of a large language model have different characteristics—some operators have low memory usage but high computational demands, while others are the opposite—simply allocating the model by layer cannot fully utilize the powerful computing capabilities of GPUs. Ideally, operators should be placed on CPUs or GPUs according to their memory requirements and computational characteristics to fully leverage hardware advantages. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a method and system for unloading tensors from large model inference in resource-constrained scenarios, thereby improving the inference speed of large models in resource-constrained scenarios.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A method for unloading large model inference tensors in resource-constrained scenarios includes the following steps:

[0008] S101, load operator semantic knowledge for the target operator in the large model;

[0009] S102 calculates the latency optimization benefit obtainable per unit GPU memory resource for the target operator and normalizes it as GPU affinity, and generates an initial priority queue G by sorting the GPU affinity in descending order;

[0010] S103, introduce cross-device tensor loading delay for the target operators in the initial priority queue G to dynamically adjust the operator deployment order, thereby obtaining the final priority queue G';

[0011] S104, determine whether the final priority queue G' is equal to the initial priority queue G. If not, jump to step S102 to continue iterating; otherwise, generate the distribution scheme of the operator tensor between the GPU and CPU based on the final priority queue G'.

[0012] Optionally, the operator semantic knowledge in step S101 includes the execution latency of the target operator on the CPU and GPU, and the memory usage of the target operator's weight tensor. The target operator is an operator that requires loading model weights for tensor computation. The calculation function expression for the latency optimization benefit obtainable per unit GPU memory resource in step S102 is:

[0013] ;

[0014] in, Let $i$ be the latency optimization benefit achievable per unit of GPU memory resources for the $i$-th objective operator. and Let be the execution latency of the i-th target operator on the CPU and GPU, respectively. Let be the memory usage of the weight tensor of the i-th target operator.

[0015] Optionally, the function expression for the normalization process in step S102 is:

[0016] ;

[0017] in, Let i be the GPU affinity of the i-th target operator. Let $i$ be the latency optimization benefit achievable per unit of GPU memory resources for the $i$-th objective operator. and These represent the minimum and maximum values ​​of the delay optimization benefit, respectively.

[0018] Optionally, the expression for the calculation function of the execution delay is:

[0019] ;

[0020] in, To delay execution, The time required to load the tensors needed by the operator into the corresponding memory on the CPU or GPU. The time for CPU or GPU to compute tensors of operators.

[0021] Optionally, step S103 includes:

[0022] S201, traverse the initial priority queue G to select a target operator as the current target operator. If the traversal is successful, jump to step S202; otherwise, jump to step S204.

[0023] S202 introduces cross-device tensor loading latency to the current target operator to update GPU affinity, including: first, calculating the latency optimization benefit per unit GPU memory resource according to the following formula:

[0024] ;

[0025] in, The cross-device tensor loading latency is assigned to the i-th target operator; then, the latency optimization benefit obtainable per unit GPU memory resource is normalized to obtain the updated GPU affinity of the current target operator.

[0026] S203, Generate a preliminary priority queue G by sorting the GPU affinity of the current target operator in descending order and the GPU affinity of the remaining operators in the preliminary priority queue G; jump to step S201;

[0027] S204, set the initial priority queue G as the final priority queue G', and jump to step S104.

[0028] Optionally, the placement scheme for the tensors of the operators generated based on the final priority queue G' in step S104, which are distributed between the GPU and the CPU, includes: selecting a specified proportion of the GPU memory capacity as the resource constraint boundary, selecting the subset of operators with the highest priority within the resource constraint boundary in the final priority queue G' and deploying them on the GPU device, and allocating the remaining operators to the CPU.

[0029] Optionally, after generating the allocation scheme of the operator tensor between the GPU and CPU based on the final priority queue G' in step S104, the method further includes using an operator backend mapping table to record the placement method of each operator, and allocating memory corresponding to the GPU and CPU for each operator according to the placement method recorded in the operator backend mapping table to place the tensor required by the operator.

[0030] The present invention also provides a large model inference tensor offloading system for resource-constrained scenarios, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the large model inference tensor offloading method for resource-constrained scenarios.

[0031] The present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the large model inference tensor offloading method for resource-constrained scenarios by a processor.

[0032] The present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the large model inference tensor offloading method in the resource-constrained scenario via a processor.

[0033] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: To achieve the goal of efficiently deploying large models on resource-constrained devices, the method of the present invention includes loading operator semantic knowledge for target operators; calculating the latency optimization benefits obtainable per unit GPU memory resource for the target operators and normalizing the calculation as GPU affinity, and generating an initial priority queue G by sorting the GPU affinity in descending order; introducing cross-device tensor loading latency for the target operators in the initial priority queue G to dynamically adjust the operator deployment order, thereby obtaining the final priority queue G'; determining whether the final priority queue G' is equal to the initial priority queue G, and if not, jumping to step S102 to continue iterating; otherwise, generating a distribution scheme for the operator tensors between the GPU and CPU based on the final priority queue G'. Since the higher the operator GPU affinity, the more significant the improvement in inference speed after the operator is unloaded to the GPU for execution, the distribution scheme for the operator tensors between the GPU and CPU generated in the above manner ensures that the tensors of operators with higher GPU affinity are preferentially placed on the GPU, thereby improving the inference speed of large models in resource-constrained scenarios. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the process of generating operator placement schemes using the tensor unloading method in an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of the reasoning process based on the tensor unloading method in an embodiment of the present invention. Detailed Implementation

[0036] To achieve efficient deployment of large models on resource-constrained devices, this invention proposes a large model inference tensor offloading method (named OATO, or Operator-Aware TensorOffloading) for resource-constrained scenarios. This method enables intelligent operator placement based on requests and model architecture within limited memory. To enable those skilled in the art to better understand the technical solution of this invention, the following detailed description, in conjunction with the accompanying drawings of the embodiments of this invention, will further illustrate the technical solution.

[0037] like Figure 1 As shown, the method for unloading large model inference tensors in resource-constrained scenarios in this embodiment includes the following steps:

[0038] S101, load operator semantic knowledge for the target operator in the large model;

[0039] S102 calculates the latency optimization benefit obtainable per unit GPU memory resource for the target operator and normalizes it as GPU affinity, and generates an initial priority queue G by sorting the GPU affinity in descending order;

[0040] S103, introduce cross-device tensor loading delay for the target operators in the initial priority queue G to dynamically adjust the operator deployment order, thereby obtaining the final priority queue G';

[0041] S104, determine whether the final priority queue G' is equal to the initial priority queue G. If not, jump to step S102 to continue iterating; otherwise, generate the distribution scheme of the operator tensor between the GPU and CPU based on the final priority queue G'.

[0042] It should be noted that the large model in this embodiment can adopt the required large model type as needed, and the large model supporting the input data modality can be selected as needed. For example, as an optional implementation, the large model input is text, and the output is the output text corresponding to the input text. Or the large model input is an image, and the output is the output image corresponding to the input image. Or the large model input is audio, and the output is the output audio corresponding to the input audio. Or the large model input is part or all of text, image, and audio, and the output is the output corresponding to the input (part or all of text, image, and audio). The large model inference tensor unloading method (OATO) in resource-constrained scenarios in this embodiment consists of three modules: an operator parser, used to extract the semantic knowledge of operators, i.e., executing step S101; an operator scheduler, which generates an operator placement scheme based on the semantic knowledge, i.e., executing steps S102 to S104; and an operator placer, which simplifies operator management and realizes automated placement, i.e., executing the placement scheme according to the generated placement scheme.

[0043] In resource-constrained large language model inference systems, there are three operator connection methods (CPU-CPU, GPU-GPU, CPU-GPU). These connection methods must be considered, and the additional latency caused by intermediate result transmission (such as GPU→CPU data migration) must be quantified. In this embodiment, operators that require loading model weights for tensor computation are called target operators. Operators that only load intermediate tensors for computation, and operators that can perform operations without requiring other tensors, are classified as non-target operators. The method in this embodiment only analyzes and processes target operators. During the inference preloading stage, the operator parser first classifies operators based on whether they are target operators in the large language model inference iteration (i.e., one round of forward propagation).

[0044] The operator semantic knowledge in step S101 of this embodiment includes the execution latency of the target operator on the CPU and GPU, as well as the memory usage of the target operator's weight tensor. The target operator is the operator that requires loading model weights for tensor computation. Furthermore, the operator semantic knowledge in step S101 of this embodiment also includes the time required to load the tensor required by the operator into the corresponding memory on the CPU or GPU, and the CPU or GPU's tensor computation time for the operator. These parameters constitute the optimization basis for the operator scheduler.

[0045] In resource-constrained scenarios, the limited availability of memory resources becomes a key bottleneck restricting the efficient inference of large language models. To achieve globally optimized allocation of memory resources and rational deployment of operators, we need to regenerate the operator scheduling scheme. After receiving the operator semantic information collected by the operator parser, the operator scheduler uses this data to reschedule operators. The intelligent operator scheduling algorithm proposed in this embodiment focuses on two core objectives: first, to maximize the use of limited GPU memory resources and prioritize storing operators with the highest affinity; second, to fully consider the impact of data transmission on operator GPU affinity and effectively balance computational and data transmission overhead.

[0046] First, through step S102, the latency optimization benefits obtainable per unit of GPU memory resources for each operator can be quantitatively evaluated based on the computational latency parameters obtained by the operator parser. The calculation function expression for the latency optimization benefits obtainable per unit of GPU memory resources in step S102 of this embodiment is:

[0047] ;

[0048] in, Let $i$ be the latency optimization benefit achievable per unit of GPU memory resources for the $i$-th objective operator. and Let be the execution latency of the i-th target operator on the CPU and GPU, respectively. Let be the memory usage of the weight tensor for the i-th target operator. In this embodiment, the function expression for calculating the execution latency is:

[0049] ;

[0050] in, For execution latency (execution latency on CPU or GPU). The time required to load the tensors needed by the operator into the corresponding memory on the CPU or GPU. The time for CPU or GPU to compute tensors of operators.

[0051] In step S102, the normalization process can adopt the desired normalization method / standardization method as needed. For example, as an optional implementation, in this embodiment, the normalization process adopts max-min normalization, and the function expression for the normalization process is:

[0052] ;

[0053] in, Let i be the GPU affinity of the i-th target operator. Let $i$ be the latency optimization benefit achievable per unit of GPU memory resources for the $i$-th objective operator. and These are the minimum and maximum values ​​of the delay optimization benefit, respectively, thus mapping the GPU affinity of the target operator to a value in the range of 0-1. This value is used to characterize the GPU affinity of the operator, and a marginal benefit evaluation model of memory-efficiency is established.

[0054] To eliminate the negative impact of transmission delay on computational gain, an iterative affinity correction mechanism is constructed. In step S103, for each candidate target operator, a cross-device tensor loading delay is introduced to update the delay optimization benefit (transmission delay compensation), and the original calculation result is corrected (priority reordering). By iteratively executing the process of "computational gain evaluation → transmission delay compensation → priority reordering," the operator deployment order is dynamically adjusted to ensure that the priority ranking truly reflects the comprehensive optimization benefit under network transmission constraints. Specifically, step S103 in this embodiment includes:

[0055] S201, traverse the initial priority queue G to select a target operator as the current target operator. If the traversal is successful, jump to step S202; otherwise, jump to step S204.

[0056] S202 introduces cross-device tensor loading latency to the current target operator to update GPU affinity, including: first, calculating the latency optimization benefit per unit GPU memory resource according to the following formula:

[0057] ;

[0058] in, The cross-device tensor loading latency is assigned to the i-th target operator; then, the latency optimization benefit obtainable per unit GPU memory resource is normalized to obtain the updated GPU affinity of the current target operator.

[0059] S203, Generate a preliminary priority queue G by sorting the GPU affinity of the current target operator in descending order and the GPU affinity of the remaining operators in the preliminary priority queue G; jump to step S201;

[0060] S204, set the initial priority queue G as the final priority queue G', and jump to step S104.

[0061] In step S104 of this embodiment, the placement scheme for the tensors generated based on the final priority queue G' and their operators between the GPU and CPU includes: selecting a specified proportion of the GPU memory capacity as a resource constraint boundary, selecting the subset of operators with the highest priority within the resource constraint boundary in the final priority queue G' and deploying them on the GPU device, while allocating the remaining operators to the CPU. For example, as an optional implementation, this embodiment uses 90% of the GPU memory capacity as the resource constraint boundary, selecting the subset of operators with the highest priority within 90% of the GPU memory capacity in the final priority queue G' and deploying them on the GPU device, while allocating the remaining operators to the CPU. This threshold setting ensures both a safe margin for memory and maximizes resource utilization. The final placement scheme achieves Pareto optimality in three dimensions: computational efficiency, transmission overhead, and memory usage, effectively solving the local optima problem that traditional methods easily encounter in heterogeneous hardware environments.

[0062] For the target operator, the operator placer focuses on three key pieces of information: its weight tensor, intermediate tensor, and operation type. The weight tensor, as a crucial carrier of model parameters, possesses attributes such as backend device type and memory size, and is directly mapped to the actual physical device. Therefore, studying the placement of the target operator essentially involves exploring how to place the weight tensor. In the actual operator deployment process, the operator placer first obtains the globally optimal operator placement scheme through the operator parser, and then constructs and maintains an operator-backend device mapping table. This mapping table clearly presents the correspondence between the weight tensor and the backend device, becoming the key basis for the system to identify the physical location of the weight tensor. By dynamically modifying this mapping table, the system can flexibly determine the actual deployment device of the weight tensor and accurately load the tensor on the corresponding backend device, successfully completing the computation task. Figure 2 As shown, in step S104 of this embodiment, after generating the allocation and placement scheme of the operator's tensor between the GPU and CPU based on the final priority queue G', it further includes using an operator backend mapping table to record the placement method of each operator, and allocating the corresponding GPU and CPU memory for each operator according to the placement method recorded in the operator backend mapping table to place the tensor required by the operator. The large model inference process includes: extracting operator semantic knowledge through the operator parser, generating a deployment scheme through the operator scheduler, modifying the operator backend mapping table through the operator placer, requesting CPU / GPU memory from the system and completing the placement, repeating the above operations until the placement of the target operator in the large model inference process is completed, and finally completing the large model inference.

[0063] In summary, the large model inference tensor offloading method in this embodiment, under resource constraints, iteratively sorts the GPU affinity of operators to generate the optimal placement scheme until the best GPU affinity sequence for the operators is found. Throughout the search process, the algorithm primarily tracks whether data transfer exists between the CPU and GPU for the operators. In each iteration, it updates the GPU affinity of the operators accordingly. Ultimately, it generates a deployment scheme that ensures sufficient memory margin while maximizing resource utilization under GPU memory constraints. This scheme achieves Pareto optimality in three dimensions: computational efficiency, transmission overhead, and memory usage, effectively solving the local optima problem that traditional methods easily encounter in heterogeneous hardware environments. Given a large language model (large model), the operator parser extracts the semantic knowledge of each layer of operators in the model, i.e., quantifies the computational cost of the operators, the size of the required tensors (memory usage), and their correlations. Using this semantic knowledge, the operator scheduler generates the GPU affinity of each operator and sorts them to plan the optimal deployment scheme for the operators. Afterwards, the operator placer will maintain a table to distinguish the backend of the operator, and complete the automatic unloading of the operator on the CPU and GPU backends. Through the above method, a placement scheme for the distribution of operator tensors between the GPU and CPU is generated, so that the tensors of operators with higher GPU affinity are preferentially placed on the GPU, thereby improving the inference speed of large models in resource-constrained scenarios.

[0064] Furthermore, this embodiment also provides a large model inference tensor offloading system for resource-constrained scenarios, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the large model inference tensor offloading method for resource-constrained scenarios.

[0065] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the large model inference tensor offloading method in the resource-constrained scenario via a processor.

[0066] Furthermore, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the large model inference tensor offloading method in the resource-constrained scenario via a processor.

[0067] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0068] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for unloading large model inference tensors in resource-constrained scenarios, characterized in that, Includes the following steps: S101, Load operator semantic knowledge for the target operator in the large model. The operator semantic knowledge includes the execution latency of the target operator on the CPU and GPU and the memory usage of the target operator's weight tensor. The target operator is the operator that needs to load model weights for tensor calculation. S102 calculates the latency optimization benefit obtainable per unit GPU memory resource for the target operator and normalizes it as GPU affinity, and generates an initial priority queue G by sorting the GPU affinity in descending order; S103, introduce cross-device tensor loading delay for the target operators in the initial priority queue G to dynamically adjust the operator deployment order, thereby obtaining the final priority queue G'; S104, determine whether the final priority queue G' is equal to the initial priority queue G. If not, jump to step S102 to continue iterating. Otherwise, generate the distribution scheme of the operator tensor between the GPU and CPU based on the final priority queue G'. The calculation function expression for the latency optimization benefit obtainable per unit GPU memory resource in step S102 is: ； in, Let $i$ be the latency optimization benefit achievable per unit of GPU memory resources for the $i$-th objective operator. and Let be the execution latency of the i-th target operator on the CPU and GPU, respectively. Let be the memory usage of the weight tensor of the i-th target operator.

2. The method for unloading large model inference tensors in resource-constrained scenarios according to claim 1, characterized in that, The function expression for the normalization process in step S102 is: ； in, Let i be the GPU affinity of the i-th target operator. Let $i$ be the latency optimization benefit achievable per unit of GPU memory resources for the $i$-th objective operator. and These represent the minimum and maximum values ​​of the delay optimization benefit, respectively.

3. The method for unloading large model inference tensors in resource-constrained scenarios according to claim 1, characterized in that, The function expression for calculating the execution latency is: ； in, To delay execution, The time required to load the tensors needed by the operator into the corresponding memory on the CPU or GPU. The time for CPU or GPU to compute tensors of operators.

4. The method for unloading large model inference tensors in resource-constrained scenarios according to claim 2, characterized in that, Step S103 includes: S201, traverse the initial priority queue G to select a target operator as the current target operator. If the traversal is successful, jump to step S202; otherwise, jump to step S204. S202 introduces cross-device tensor loading latency to the current target operator to update GPU affinity, including: first, calculating the latency optimization benefit per unit GPU memory resource according to the following formula: ； in, The cross-device tensor loading latency is assigned to the i-th target operator; then, the latency optimization benefit obtainable per unit GPU memory resource is normalized to obtain the updated GPU affinity of the current target operator. S203, Generate a preliminary priority queue G by sorting the GPU affinity of the current target operator in descending order and the GPU affinity of the remaining operators in the preliminary priority queue G; jump to step S201; S204, set the initial priority queue G as the final priority queue G', and jump to step S104.

5. The method for unloading large model inference tensors in resource-constrained scenarios according to claim 1, characterized in that, The placement scheme for the tensors of operators generated based on the final priority queue G' in step S104, which are distributed between the GPU and the CPU, includes: selecting a specified proportion of the GPU memory capacity as the resource constraint boundary, selecting the subset of operators with the highest priority within the resource constraint boundary in the final priority queue G' and deploying them on the GPU device, and allocating the remaining operators to the CPU.

6. The method for unloading large model inference tensors in resource-constrained scenarios according to claim 1, characterized in that, After generating the allocation scheme of operator tensors between GPU and CPU based on the final priority queue G' in step S104, the method further includes using an operator backend mapping table to record the placement method of each operator, and allocating memory corresponding to GPU and CPU for each operator according to the placement method recorded in the operator backend mapping table to place the tensor required by the operator.

7. A large model inference tensor offloading system for resource-constrained scenarios, comprising interconnected microprocessors and memory, characterized in that, The microprocessor is programmed or configured to execute the large model inference tensor offloading method for resource-constrained scenarios as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the large model inference tensor offloading method for resource-constrained scenarios as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the large model inference tensor offloading method for resource-constrained scenarios as described in any one of claims 1 to 6.

Images