Task processing method and device based on large model, intelligent agent, equipment, medium and product

Abstract

The invention provides a task processing method and device based on a large model, an intelligent agent, equipment, a medium and a product, and relates to the technical field of computers, in particular to the technical field of artificial intelligence such as natural language processing and deep learning. The task processing method based on the large model comprises the following steps: in response to receiving a to-be-processed task, determining a target batch processing size for the to-be-processed task; determining a storage space for the to-be-processed task according to the target batch processing size; and based on a mapping relationship between the candidate batch processing size and the candidate calculation graph, calling a target calculation graph corresponding to the target batch processing size, and executing large model reasoning on the task data acquired from the storage space to obtain a task processing result.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of artificial intelligence technology such as natural language processing and deep learning, and more specifically to a task processing method, apparatus, intelligent agent, equipment, medium and product based on a large model. Background Art

[0002] As hardware computing power gradually improves, the execution speed of operators gradually increases. In scenarios with low model computational complexity and a large number of small operators, operator scheduling overhead gradually replaces operator execution time and becomes the performance bottleneck for large model inference. Summary of the Invention

[0003] The present disclosure provides a task processing method, apparatus, intelligent agent, equipment, medium and product based on a large model.

[0004] According to one aspect of the present disclosure, a large-model-based task processing method is provided, comprising: in response to receiving a task to be processed, determining a target batch size for the task to be processed; determining a storage space for the task to be processed based on the target batch size; and based on a mapping relationship between candidate batch sizes and candidate computation graphs, calling a target computation graph corresponding to the target batch size, performing large-model inference on the task data obtained from the storage space, and obtaining a task processing result.

[0005] According to another aspect of the present disclosure, a large-model-based task processing device is provided, including: a first determination module for determining a target batch size for the task to be processed in response to receiving the task to be processed; a second determination module for determining a storage space for the task to be processed based on the target batch size; and a task processing module for calling a target computation graph corresponding to the target batch size based on a mapping relationship between candidate batch sizes and candidate computation graphs, performing large-model inference on task data obtained from the storage space, and obtaining a task processing result.

[0006] According to another aspect of the present disclosure, an artificial intelligence agent is provided, which is configured to execute the above-mentioned large model-based task processing method.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0008] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program or instructions are stored. When the computer program or instructions are executed by a processor, the steps of the above method are implemented.

[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0010] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0012] Figure 1 Schematically illustrates a system architecture to which a large model-based task processing method according to an embodiment of the present disclosure can be applied;

[0013] Figure 2 Schematically shows a flow chart of a task processing method based on a large model according to an embodiment of the present disclosure;

[0014] Figure 3 Schematically shows a core architecture diagram according to an embodiment of the present disclosure;

[0015] Figure 4 An example schematic diagram of a scheduling process in the initialization phase according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 5 An example schematic diagram of an overall access architecture according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 6 An example schematic diagram schematically illustrates a scheduling process in the execution phase according to an embodiment of the present disclosure;

[0018] Figure 7 Schematically shows a block diagram of a task processing device based on a large model according to an embodiment of the present disclosure;

[0019] Figure 8 Schematically shows a structural block diagram of an intelligent agent of a large model according to an embodiment of the present disclosure; and

[0020] Figure 9 A block diagram of an electronic device suitable for implementing a large model-based task processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0022] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0024] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0025] Taking GPU as an example, computational graph optimization technologies include CudaGraph acceleration technology, dynamic graph conversion to static graph, and computational graph compilation optimization technology.

[0026] In one example, the Cuda Graph execution process is divided into two phases. The Capture phase records all device operations (memory operations and kernel execution) associated with the captured stream into a graph. The Replay phase re-executes the recorded graph. While CudaGraph acceleration technology offers significant performance improvements, it has a number of limitations, complex adaptation, and challenging troubleshooting.

[0027] Cuda Graph can be accessed using two methods based on its scope: full-graph capture and sub-graph capture. The sub-graph capture method slices the graph after static state, capturing only modules outside of the attention module, giving the attention module greater freedom. While vLLM supports both full-graph and sub-graph capture, CudaGraph is tied to static graphs and cannot be used in dynamic graph scenarios. SGLang, while supporting dynamic graph capture, does not support full-graph capture.

[0028] To this end, embodiments of the present disclosure propose a large-model-based task processing solution. For example, in response to receiving a pending task, a target batch size for the pending task is determined; based on the target batch size, a storage space for the pending task is determined; and based on the mapping relationship between candidate batch sizes and candidate computation graphs, a target computation graph corresponding to the target batch size is called, and large-model inference is performed on the task data obtained from the storage space to obtain the task processing result.

[0029] According to the embodiments of the present disclosure, by dynamically determining the target batch size according to the tasks to be processed, allocating storage space accordingly and calling the corresponding target calculation graph to perform large-model reasoning, on the one hand, by reasonably determining the target batch size and allocating the corresponding storage space, it is avoided to allocate the maximum possible video memory space for each task, thereby reducing video memory occupancy; on the other hand, by using the pre-captured calculation graph for reasoning, it is avoided that complex graph capture operations are required for each reasoning, which greatly improves the reasoning efficiency. While ensuring that the reasoning task can be executed efficiently, it effectively reduces the video memory overhead and computing power waste, enabling large models to run better in resource-constrained environments, while also improving the overall performance and resource utilization of the system, providing strong support for the efficient execution of large-model reasoning tasks.

[0030] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0031] In the technical solution of the present invention, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0032] Figure 1 The system architecture of the task processing method based on the large model according to the embodiment of the present disclosure is schematically shown. It should be noted that Figure 1The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure may not be used in other devices, systems, environments or scenarios.

[0033] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used as a medium for providing a communication link between different devices.

[0034] It should be noted that the large model-based task processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the large model-based task processing device provided in the embodiment of the present disclosure can generally be set in the server 105.

[0035] Alternatively, the large model-based task processing method provided in the embodiment of the present disclosure may also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the large model-based task processing apparatus provided in the embodiment of the present disclosure may also be provided in the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0037] It should be noted that the sequence numbers of the operations in the following method are only used to indicate the operation for the purpose of description, and should not be regarded as indicating the order in which the operations should be performed. Unless explicitly stated, the method does not need to be performed in the order shown.

[0038] Figure 2 The flowchart of the task processing method based on the large model according to the embodiment of the present disclosure is schematically shown.

[0039] like Figure 2 As shown, the large model-based task processing method 200 includes operations S210 to S230.

[0040] In operation S210 , in response to receiving a task to be processed, a target batch size for the task to be processed is determined.

[0041] In operation S220 , storage space for tasks to be processed is determined according to the target batch size.

[0042] In operation S230, based on the mapping relationship between the candidate batch size and the candidate computation graph, the target computation graph corresponding to the target batch size is called, and large model inference is performed on the task data obtained from the storage space to obtain the task processing result.

[0043] In an embodiment of the present disclosure, before obtaining the user's information, the user's consent or authorization may be obtained. For example, before operation S210, a request to obtain the user's information may be issued to the user. If the user agrees or authorizes the acquisition of the user's information, operation S210 is performed.

[0044] A pending task consists of M object requests received at the same time. These M object requests constitute multiple specific request instances of the pending task, where M is a positive integer greater than 1. For example, a chatbot system might receive five different user inquiries at the same time. Each of these inquiries is an object request, such as "Have you eaten today?" These five inquiries together constitute the current pending task.

[0045] When receiving pending tasks, such as questions or requests from multiple users over the network, a target batch size can be determined based on the number of user requests in the pending tasks and the availability of hardware resources. The target batch size is determined based on the number of object requests in the actual pending tasks and the hardware resources available. For example, if the maximum supported batch size is 10 and there are currently 5 object requests to process, the target batch size is determined to be 8 after comprehensively considering factors such as hardware resource utilization.

[0046] Storage space refers to the space used to store data related to pending tasks, i.e., the area allocated in video memory for each candidate batch size. The required storage space can be calculated based on the video memory requirements of the large model for input data and the candidate batch size, and a corresponding contiguous space of the required size can be allocated in video memory. For example, when the candidate batch size is 8, a sufficiently large and fixed video memory area can be allocated to store the data of the pending tasks with a candidate batch size of 8, based on the video memory requirements of the model for input data.

[0047] A mapping relationship is a pre-established correspondence between different candidate batch sizes and their corresponding candidate computation graphs. For example, candidate batch sizes of 1, 2, 4, and 8 are preset, and each candidate batch size corresponds to a candidate computation graph pre-captured using CudaGraph technology. Once the target batch size is determined, this mapping relationship can be used to quickly find the target computation graph corresponding to the target batch size. This target computation graph can then be used to efficiently execute large model inference tasks at the target batch size.

[0048] After determining the storage space for the tasks to be processed and the target computation graph corresponding to the target batch size, the target computation graph can be called to perform inference calculations on the task data stored in the allocated storage space to obtain the task processing results in natural language. For example, for a task to be processed with a target batch size of 8, the corresponding target computation graph is found through a mapping relationship. The target computation graph records all GPU kernel call sequences and operations in the inference process for a target batch size of 8. During execution, the target computation graph is directly used for inference to quickly obtain the task processing results. The task processing result refers to the result obtained after inference on a large model. For example, if the object request can be "Have you eaten today?", the task processing result can be "I haven't eaten yet. Thank you for your concern. Have you eaten?"

[0049] According to the embodiments of the present disclosure, by dynamically determining the target batch size according to the tasks to be processed, allocating storage space accordingly and calling the corresponding target calculation graph to perform large-model reasoning, on the one hand, by reasonably determining the target batch size and allocating the corresponding storage space, it is avoided to allocate the maximum possible video memory space for each task, thereby reducing video memory occupancy; on the other hand, by using the pre-captured calculation graph for reasoning, it is avoided that complex graph capture operations are required for each reasoning, which greatly improves the reasoning efficiency. While ensuring that the reasoning task can be executed efficiently, it effectively reduces the video memory overhead and computing power waste, enabling large models to run better in resource-constrained environments, while also improving the overall performance and resource utilization of the system, providing strong support for the efficient execution of large-model reasoning tasks.

[0050] Figure 3 The core architecture diagram according to an embodiment of the present disclosure is schematically shown.

[0051] like Figure 3 As shown, in 300, in response to receiving the task to be processed 301, the large model reasoning engine control center 302 obtains the output task processing result 312 by managing task scheduling, resource scheduling, object request / response, etc.

[0052] In one embodiment, the large model inference engine control hub 302 includes a first converter 303, a message queue 304, a worker_proc 305, and a second converter 311. The first converter 303 (i.e., InputProcessor) is used to convert object requests in the pending task 301 into feature vectors. The message queue 304 is used to dispatch object requests to workers.

[0053] Worker_proc 305 is used to select a worker from multiple workers 306 to process the current pending task 301. Multiple workers 306 may include, for example, worker 306_1, worker 306_2, worker 306_3, ..., worker 306_N, where N is a positive integer. Each worker 306 corresponds to a set of model executors 307, computation graph modules 308, model modules 309, and attention backend technology modules 310, which are used to manage device resources, initialize models and cache engines, and perform warmup operations.

[0054] The model executor 307 encapsulates the model execution logic, supports dynamic and static graph modes, and manages CudaGraph capture and replay. The attention backend technology module 310 is used to uniformly encapsulate various types of AttentionLayer, initialize them within the ModelRunner, and call the forward propagation of the AttentionLayer during model execution.

[0055] In the embodiment of the present disclosure, the focus is on three modules: multiple workers 306, model executor 307 and calculation graph module 308, while taking into account the adaptation and optimization of video memory management, attention backend technology module 310 and CustomOp.

[0056] After any worker among the multiple workers 306 obtains the execution result by calling its own model executor 307, computational graph module 308, model module 309 and attention back-end technology module 310, etc., it can return the execution result to the large model inference engine control center 302, and use the second converter 311 in the large model inference engine control center 302 to convert the execution result in the form of feature vector to obtain the task processing result in the form of natural language.

[0057] Figure 4 An example diagram of a scheduling process in the initialization phase according to an embodiment of the present disclosure is schematically shown.

[0058] like Figure 4 As shown, in 400, the decorator is a CudaGraph decorator, which is used to provide the whole graph capture capability / subgraph capture capability, and is connected through the decorator to support dynamic graph and static graph modes.

[0059] After applying the CudaGraph decorator to the model networking code, during the initialization phase, the Worker initializes the distributed training environment by calling the init_dist_env() function. The model executor then creates the specific model class by calling the load_model() method. During this process, the CudaGraph decorator is automatically called, modifying the inheritance list of the original model class to inherit from the CudaGraphWrapper class. The original model's __init__() and __call__() methods are then replaced with CudaGraphWrapper's __init__() and CudaGraphBackend's __call__() methods, respectively. This adds CudaGraph inference acceleration without intruding on the original networking code.

[0060] The decorator initializes the computation graph backend by calling the init_cuda_graph_backend() function. The model executor initializes the key-value cache by calling the init_kvcache() function. The worker preheats the model by calling the warm_up_model() function. The model executor captures the model state by calling the capture_model() function. The model generates simulated data for each candidate batch size by calling the _dummy_run() function. The decorator and model executor perform forward computation on the simulated data for each candidate batch size through the target call process, generating candidate computation graphs corresponding to each candidate batch size.

[0061] After initializing the CudaGraphWrapper, the Worker calls the warm_up_model() method to execute forward inference for each model at different batch sizes. This involves calling the CudaGraphBackend's __call__() method. The CudaGraphBackend maintains a dictionary of batch sizes to CudaGraphs. During the warmup process, the CudaGraph capture function for each batch is triggered. After the capture is complete, the captured CudaGraph is used in subsequent inference processes to replace the model's forward computation, reducing kernel startup overhead.

[0062] Figure 5 An example schematic diagram of an overall access architecture according to an embodiment of the present disclosure is schematically shown.

[0063] like Figure 5As shown in 500, @support_cuda_graph represents a Python decorator that can receive a callable class, modify its functionality, and return another callable class. @support_cuda_graph uses the decorator to decorate the __init__() and __call__() methods of the model class and manually configures the model class to inherit the decorator.

[0064] The CUDA graph backend is initialized during the model initialization phase, and the CUDA graph backend is directly called instead of model.forward() during the execution phase. The capture and replay of the computational graph are both implemented in CudaGraphBackend.__call__(). In addition, the CUDA graph backend reserves interfaces for static conversion and graph segmentation and reassembly, and finally the segmented CUDA graph backend completes the subgraph. Figure 1 ,son Figure 2 , ..., subgraph Q, where Q is a positive integer or the capture and replay of the entire graph. For each subgraph / full graph, there can be multiple candidate computation graphs with different candidate batch sizes.

[0065] For example, the candidate computation corresponding to candidate batch size 1 Figure 1 , candidate computation corresponding to candidate batch size 2 Figure 2 , candidate computation corresponding to candidate batch size 3 Figure 3 It should be noted that the three candidate batch sizes and the corresponding three candidate computation graphs are only examples and are not limited here.

[0066] In one embodiment, by calling a decorator, the original initialization process of the model class for the large model is replaced with a target initialization process, so as to complete the acquisition and replay of multiple candidate computation graphs by executing the target initialization process and completing the initialization; by calling a decorator, the original calling process of the model class for the large model is replaced with a target calling process, and a modified model class is obtained, so that multiple candidate computation graphs are used to perform large model reasoning when calling the model class.

[0067] Decorators can be used to add additional functionality without modifying existing functions or classes. In the disclosed embodiment, the decorator can be @support_cuda_graph, which is used to modify the initialization and calling procedures of the model class. The model class defines the structure and behavior of the model, including the original initialization procedure (i.e., _init_()) and the original calling procedure (i.e., _call_()).

[0068] The original initialization process refers to the logical process executed by the original initialization method of the model class, that is, the initialization operations before the decorator's modification. The target initialization process refers to the model class initialization process modified by the decorator. Compared to the original initialization process, it adds operations such as initializing the CudaGraphBackend and completing the acquisition of multiple candidate computation graphs. For example, during the initialization phase, the initialization logic of the CudaGraphWrapper completes the creation of the CudaGraphBackend and captures the computation graphs for each batch size during the warmup process.

[0069] In CudaGraph, replay refers to re-executing the operations in a pre-captured candidate graph in the same order as the previous one. For example, after capturing a candidate graph, during inference, each time the model's forward computation needs to be executed, the captured candidate graph can be replayed, improving execution efficiency.

[0070] By decorating the model class with the @support_cuda_graph decorator, the decorator is automatically called during model class initialization, modifying the original model class's inheritance list to inherit from the CudaGraphWrapper class and replacing the original model's _init__() method with CudaGraphWrapper's __init__() method. During target initialization, CudaGraphBackend is initialized. When the worker calls the warm_up_model() method, forward inference is executed for each model with different batch sizes, triggering the CudaGraph graph capture function for each batch, thereby completing the acquisition and replay preparation of multiple candidate computational graphs.

[0071] The original call process refers to the logical process executed by the original call method of the model class, that is, the call operation before modification by the decorator, which usually directly executes the model's forward calculation logic. The target call process refers to the call process of the model class modified by the decorator, which calls the CudaGraphBackend's computation graph for inference. For example, when calling the _call_() method of the modified model class, it actually executes the replay operation of the computation graph corresponding to the target batch size in the CudaGraphBackend, rather than directly executing the original model's forward calculation logic.

[0072] The original initialization process of the modified model class has been replaced with the target initialization process, and the original call process has been replaced with the target initialization process and the target call process. For example, after the original model class is processed by the @support_cuda_graph decorator, it becomes a modified model class with the ability to use CudaGraph for inference.

[0073] By decorating the model class with the @support_cuda_graph decorator, the original model's __call__() method is replaced with the CudaGraphBackend's __call__() method. When the modified model class's __call__() method is called, it uses multiple pre-obtained candidate computation graphs based on the batch size of the input data to perform large model inference, rather than executing the original model's forward computation logic.

[0074] According to the embodiments of the present disclosure, by calling a decorator to non-invasively modify the model class initialization and calling process, multiple candidate computation graphs are acquired and replayed during the initialization phase. These candidate computation graphs can then be used to efficiently perform large-model inference when the model class is called. This approach eliminates the need to modify the existing model networking code, reducing development difficulty and the risk of errors. Furthermore, by leveraging CudaGraph technology, it effectively reduces kernel startup overhead, improves inference efficiency, accelerates large-model inference, and enhances performance and resource utilization.

[0075] In one embodiment, the mapping relationship includes a correspondence between multiple candidate batch sizes and multiple candidate computation graphs. The mapping relationship can be obtained by: using simulated data for each candidate batch size, invoking the large model through a target call process to perform forward computation to obtain a candidate computation graph corresponding to each candidate batch size; and establishing a mapping relationship based on the multiple candidate batch sizes and the candidate computation graphs corresponding to each candidate batch size.

[0076] A mapping relationship refers to the corresponding association between candidate batch sizes and candidate computation graphs. That is, each candidate batch size corresponds to a specific candidate computation graph. For example, the candidate batch sizes are preset as 1, 2, 4, 8, and so on. Each size corresponds to a candidate computation graph pre-captured using CudaGraph technology. The corresponding relationship between these candidate batch sizes and candidate computation graphs constitutes the mapping relationship.

[0077] Candidate batch sizes are pre-defined batch size options that can be selected based on actual needs during model inference. For example, the maximum supported batch size is set to 10, and based on the common request size distribution, multiple candidate batch sizes, such as 1, 2, 4, and 8, are pre-defined to cover processing needs of varying scales.

[0078] A candidate computation graph is a pre-captured and optimized computation graph for each candidate batch size, used to perform large model inference tasks at that batch size. For example, for a candidate batch size of 8, the corresponding candidate computation graph is a computation graph captured and optimized specifically for inference with a batch size of 8. This graph records the order of all GPU kernel calls and operations for the model's forward computation at that batch size.

[0079] Simulated data refers to virtual data with the same characteristics as real data, generated during the model initialization phase to capture the computational graph. For example, during the warmup process, to obtain the computational graph for different candidate batch sizes, virtual simulated data is constructed. However, the tensor pointer addresses, shapes, and data types of these data are consistent with the input data during real execution.

[0080] During the initialization phase, the warm_up_model() method is called to generate simulated data for each candidate batch size. The target call process (the _call__() method of the CudaGraphBackend) is then used to call the large model for forward computation. During this process, the CudaGraphBackend captures the computation graphs for each candidate batch size and associates these graphs with the corresponding batch sizes, establishing a mapping relationship.

[0081] According to the embodiments of the present disclosure, by establishing a mapping relationship between candidate batch sizes and candidate computation graphs, there is no need to manually capture and manage computation graphs for each possible batch size, which reduces operational complexity and improves the flexibility and adaptability of the system. At the same time, during the inference process, the pre-captured computation graph can be directly used for replay, effectively reducing kernel startup and video memory operation overhead, improving inference efficiency, and reducing video memory usage, thereby improving the overall performance and resource utilization of the system. This allows for the rapid selection of appropriate computation graphs for inference based on the batch size of the actual task during large-model inference, providing strong support for efficient inference of large models at different task scales.

[0082] Figure 6 An example schematic diagram of a scheduling process in the execution phase according to an embodiment of the present disclosure is schematically shown.

[0083] like Figure 6As shown in 600, the worker calls the distributed.barrier() function to synchronize various processes in the distributed training environment. The worker calls the get_tasks() function to obtain a list of tasks to be executed from the message queue. The model executor triggers the model's inference process by calling the execute_model() function. The model executor calls the _prepare_inputs() function to perform necessary preprocessing on the input before model execution, such as normalization, reshaping, batch splitting, or data augmentation, to ensure that it meets the model input requirements.

[0084] The attention backend records and propagates metadata (such as input shape and data type) by calling the forward_metadata() function. The model executor triggers the model's inference process by calling the execute_model() function. The model executor executes a recorded graph in the computational graph framework by calling the graph._call_() function. The model calls compute_logits() on the model's output layer to compute unnormalized predictions, which are typically used for the final prediction in classification tasks. The model calls the sample() function to sample from the model output for task generation or policy sampling in reinforcement learning, selecting subsequent actions or predictions. The model executor calls the post_process() function to further process the model output. The model executor calls the spec_decode() function to decode the output according to specific task or model requirements.

[0085] In one embodiment, a task to be processed includes M object requests received at the same time, where M is a positive integer greater than 1. Determining a target batch size for the task to be processed may include the following operations: dividing the M object requests into multiple subtasks, where each subtask includes at least one object request; and determining a target batch size for each of the multiple subtasks.

[0086] A pending task consists of M object requests received at the same time. These M object requests constitute multiple specific request instances of the pending task. The server can receive multiple object requests from users through mechanisms such as communication queues and simultaneously process these requests as a whole, namely, a pending task. For example, if a communication queue receives 10 different user inquiries, M is 10, and these 10 object requests constitute the pending task at the current moment.

[0087] It's important to note that during execution, the number of user requests in the decode phase is dynamic, so the size of the model's input Tensors is always dynamic. Furthermore, to ensure that the memory address of the CudaGraph input Tensors remains constant, ModelRunner must always maintain a large memory pool capable of handling the maximum batch size and maximum decode length. However, this approach presents the following issues: The use of an input buffer incurs additional memory overhead; the model's forward computation is always calculated at full capacity, wasting significant computing power; and CudaGraph uses dedicated memory, requiring each graph to maintain an additional active memory pool, which also incurs additional memory overhead.

[0088] Based on this, this application reduces resource overhead by bucketing the runtime BatchSize according to the interval set by the user, and uniformly using a graph corresponding to the Padding BatchSize to perform forward reasoning within each interval.

[0089] After receiving the pending task, the M object requests are divided into multiple subtasks. A subtask is a smaller task unit obtained by dividing the pending task, and each subtask contains at least one object request. For example, continuing with the pending task that includes 10 object requests, the 10 object requests can be divided into 4 subtasks, two of which contain 3 object requests each, and the other two subtasks contain 2 object requests each.

[0090] The specific division method can be configured according to actual business needs and is not limited here. For example, the division can be based on the type of object request; alternatively, the division can be based on the urgency of the object request; alternatively, the division can be based on the priority of model processing, etc.

[0091] After dividing into multiple subtasks, the target batch size for each subtask can be determined based on factors such as the number of object requests it contains, the model's input data processing requirements, and hardware resource availability. For example, for a subtask containing three object requests, the target batch size can be set to 4 based on the model's processing capabilities and resource allocation; for a subtask containing two object requests, the target batch size can be set to 2.

[0092] According to the embodiments of the present disclosure, by dividing multiple object requests received at the same time, decomposing the to-be-processed tasks into multiple subtasks, and determining a suitable target batch size for each subtask, it is possible to fully consider the characteristics of different object requests and the processing capabilities of the model, avoid the resource waste and inefficiency problems caused by the one-size-fits-all use of a fixed batch size, reasonably allocate hardware resources, realize refined management and efficient processing of tasks, improve flexibility and resource utilization, and better adapt to dynamically changing request loads.

[0093] In one embodiment, determining the target batch size of each of the multiple subtasks may include the following operations: mapping the multiple subtasks to the minimum upper limit step value in the step interval according to a preset batch size step interval to obtain the target batch size of each of the multiple subtasks, wherein the step interval is determined based on at least one of the video memory limit of the large model and the number of object requests.

[0094] The preset batch size intervals are pre-set based on factors such as the memory limitations of large models and the number of object requests. Each interval has a minimum and maximum step value. For example, if the intervals are (0, 2], (2, 4], (4, 8], and (8, 10], the corresponding minimum and maximum step values ​​are 2, 4, 8, and 10, respectively.

[0095] After the task to be processed is divided into multiple subtasks, the target batch size of each of the multiple subtasks can be determined based on the number of object requests and the step interval in the subtask. In one example, the step interval to which the subtask belongs can be found based on the number of object requests in the subtask, and the target batch size of the subtask can be determined as the minimum upper limit step value of the step interval. For example, for a subtask containing 5 object requests, if the preset step interval is (4,8] and the corresponding minimum upper limit step value is 8, then the target batch size of the subtask is determined to be 8.

[0096] According to the embodiments of the present disclosure, by mapping subtasks to the minimum upper limit step value according to the preset batch size step interval, the target batch size is determined, which can fully utilize video memory, avoid memory waste, and ensure the efficiency of model inference. During large-model inference, reasonable batch size planning can effectively reduce the number of kernel launches, improve inference efficiency, and reduce video memory usage. This allows for a better balance between resource utilization and inference performance when processing subtasks of different sizes, improving overall stability and adaptability.

[0097] In one embodiment, the maximum step value of the step interval is determined based on the available video memory that can be processed by the large model and the required video memory requested by a single object.

[0098] The maximum step value of a step interval refers to the maximum step value among the minimum and upper limit step values. For example, if the step intervals are (0, 2], (2, 4], (4, 8], and (8, 10], and the corresponding minimum and upper limit step values ​​are 2, 4, 8, and 10, respectively, the maximum step value is 10. The available video memory that a large model can process refers to the video memory capacity used to run the large model and is the upper limit of the video memory that the model can use. For example, if the system is configured with 16GB of video memory for large model inference, then the available video memory that the large model can process is 16GB. The required video memory for a single object request refers to the video memory capacity required to process one object request. For example, after model evaluation and testing, an average of 2GB of video memory is required to process one object request.

[0099] According to the embodiments of the present disclosure, by rationally determining the maximum step value of the step interval based on hardware resources and task requirements, video memory resources can be effectively managed. During large-model inference, this avoids video memory waste or insufficient memory caused by improper batch size settings. By rationally planning video memory usage, the system's resource utilization and inference efficiency are improved, enabling better adaptation to tasks of varying scales and enhancing overall performance and stability.

[0100] In one embodiment, the storage space includes candidate sub-storage spaces corresponding to a plurality of candidate computation kernels, each corresponding to a candidate batch size. The large-model-based task processing method may further include: extracting input tensor tuples from each of the plurality of candidate computation kernels, wherein the tensor tuples include at least one of a pointer address of the input tensor corresponding to the sub-storage space, a shape of the input tensor, and a data type; and processing the input tensor tuples using a hash algorithm to obtain a candidate hash, wherein the candidate hash is used as an index to the candidate computation graph.

[0101] A storage space is a memory area used to store data required for model inference. It includes multiple candidate sub-storage spaces corresponding to candidate compute kernels. For example, a contiguous memory area allocated in GPU memory is used to store model input data for different batch sizes. Candidate compute kernels (i.e., kernels) are pre-prepared kernel functions for different candidate batch sizes, with each kernel function corresponding to a candidate batch size. For example, in GPU memory, a candidate compute kernel with a batch size of 2 is allocated a sub-storage space to store model input data for that batch size.

[0102] An input tensor tuple is a combined data structure containing information about the input tensor. An input tensor tuple can include information such as a pointer address, shape, and data type. A pointer address is the address of the data storage location in memory. The shape describes the dimensions of the tensor. For example, a tensor with a shape of [3,3] is a 3x3 matrix. The data type defines the format of the data and the amount of storage space it occupies. For example, the data type can be float32 (32-bit floating point number) or int8 (8-bit integer). For example, an input tensor tuple might contain a pointer address of 0x1000, a shape of [2,2], and a data type of float32.

[0103] A hashing algorithm maps data of arbitrary length to a fixed-length hash value. By processing the input tensor tuple with the hashing algorithm, a candidate hash is obtained. A candidate computation graph is a computation graph corresponding to a candidate computation kernel, representing the computational steps and data flow during model inference. After obtaining a candidate hash, the candidate hash can be used as an index into the candidate computation graph to quickly find and locate the corresponding candidate computation graph, thereby mapping the dynamic input space of the candidate computation kernel to the candidate computation graph.

[0104] The usage restrictions of CudaGraph can be divided into outside the kernel and inside the kernel according to the scope of action. By extracting the input tensor tuples of multiple candidate computing kernels, including at least one of the pointer address of the input tensor corresponding to the sub-storage space, the shape of the input tensor, and the data type, automatic detection can be achieved to solve the dynamic input problem of the kernel.

[0105] According to the embodiments of the present disclosure, by allocating corresponding sub-storage spaces for different candidate batch sizes, and using a hash algorithm to process the input tensor tuple to generate a candidate hash as an index of the computational graph, different computational graphs can be efficiently managed and quickly located, thereby avoiding repeated calculations and waste of resources, improving the efficiency and response speed of model reasoning, and ensuring that preset computing resources can be flexibly switched and utilized when processing inputs of different batch sizes, thereby improving overall performance and resource utilization.

[0106] In one embodiment, determining the storage space for tasks to be processed based on the target batch size may include the following operations: determining a target sub-storage space for each sub-task from a plurality of candidate sub-storage spaces based on the target batch size of each sub-task; and storing the task data corresponding to the sub-task in the target sub-storage space for the sub-task through an asynchronous storage mechanism according to the pointer address of the target sub-storage space.

[0107] A candidate sub-storage space is an area within the storage space used to store input data associated with a specific candidate compute kernel. For example, a sub-area within the storage space is allocated for a candidate compute kernel with a batch size of 2 to store the corresponding input data. A target sub-storage space is a candidate sub-storage space determined based on the target batch size of the subtask and used to store the task data corresponding to that subtask. For example, for a subtask with a target batch size of 4, the corresponding target sub-storage space is the pre-allocated candidate sub-storage space for a batch size of 4.

[0108] After determining the target substorage space for each subtask, the task data corresponding to the subtask can be stored in the target substorage space according to the pointer address of the target substorage space. The pointer address refers to the address of the data location in memory. Task data refers to the data related to the subtask and required for model inference. For example, for a subtask containing three object requests, the task data is the input data corresponding to these three object requests, such as text content, image pixels, etc.

[0109] An asynchronous storage mechanism allows data storage operations to occur in the background without blocking the main thread. For example, after receiving task data, the main thread can continue processing other tasks while the storage operation proceeds in the background thread until the data is stored. For example, if the pointer address of the target sub-storage space is 0x2000, the task data can be written to the storage area at address 0x2000 using the asynchronous storage mechanism. In one example, this can be achieved using CUDA's asynchronous memory copy function, thereby improving data transfer efficiency and reducing the main thread's waiting time.

[0110] According to the embodiments of the present disclosure, by determining the target sub-storage space based on the target batch size of each subtask and using an asynchronous storage mechanism to store task data in the target sub-storage space, it is possible to effectively improve video memory utilization and data transmission efficiency. During large-model inference, frequent video memory allocation and release operations are avoided, reducing memory fragmentation. At the same time, the asynchronous storage mechanism enables data transmission and model calculation to proceed in parallel, improving overall performance and throughput.

[0111] In one embodiment, the task processing method based on the large model may also include the following operations: in response to changes in the input tensor tuple, extracting a new input tensor tuple of the candidate computing kernel; processing the new tensor tuple through a hash algorithm to obtain a new candidate hash.

[0112] During model inference, whenever a change in the input tensor's features (such as pointer address, shape, and data type) is detected, a new input tensor tuple for the candidate computation kernel can be extracted. For example, when the input image size changes from [3, 224, 224] to [3, 256, 256], the system extracts a new input tensor tuple, including the new pointer address, shape, and data type.

[0113] After obtaining a new input tensor tuple, the new input tensor tuple can be processed by a hash algorithm to obtain a new candidate hash, which can be used to find or generate a new candidate computation graph. For example, applying the SHA1 hash algorithm to the new input tensor tuple (pointer address 0x2000, shape [3,256,256], data type float32) will produce a new 160-bit hash value.

[0114] When using CudaGraph to support new networking code, this application can re-examine the operations in the code that cause CudaGraph capture and replay failures, such as synchronization operations between the device and host during capture, pointer changes to model input tensors during replay, array out-of-bounds errors, data staleness, and other issues.

[0115] According to the embodiments of the present disclosure, by promptly extracting new input tensor tuples and generating new candidate hashes when the input tensor tuples change, it is possible to dynamically adapt to different input conditions and ensure the accuracy and effectiveness of the computation graph. During large-model inference, this mechanism can quickly respond to input changes, avoiding errors or performance degradation caused by input mismatches, and improving flexibility and adaptability.

[0116] In one embodiment, the task processing method based on the big model may also include the following operations: when the calculation graph corresponding to the new candidate hash is not included in multiple candidate calculation graphs, the simulation data for the new candidate hash is used to call the big model through the target calling process to perform forward calculation to obtain the candidate calculation graph corresponding to the new candidate hash; and the mapping relationship is updated using the new candidate hash and the candidate calculation graph corresponding to the new candidate hash.

[0117] After obtaining a new candidate hash, it can be determined whether the multiple candidate computation graphs include a computation graph corresponding to the new candidate hash. If the multiple candidate computation graphs do not include a computation graph corresponding to the new candidate hash, the capture of the new computation graph can be triggered. In one example, simulated data for the new candidate hash can be generated. The simulated data for the new candidate hash refers to virtual data generated based on the characteristics of the new input tensor tuple.

[0118] After obtaining simulated data for a new candidate hash, the target call process can be used to call the large model for forward computation, thereby capturing a new computation graph and obtaining a candidate computation graph corresponding to the new candidate hash. After obtaining the candidate computation graph corresponding to the new candidate hash, the new candidate hash and the corresponding computation graph can be added to the mapping relationship. For example, the new hash value and computation graph object can be inserted as a key-value pair into a dictionary storing the mapping relationship, so that the computation graph can be quickly found and used based on the hash value.

[0119] According to the embodiments of the present disclosure, when new input conditions arise, the new computational graph can be automatically captured and the mapping relationship updated, thereby ensuring that a variety of different input scenarios can be processed, improving the flexibility and adaptability of model reasoning, avoiding reasoning errors or performance degradation caused by input changes, locking the problem within the kernel, ensuring the efficient operation of large models in dynamic environments, and improving overall performance and reliability. By timely updating the computational graph and mapping relationship, the optimization advantages of the new computational graph can be fully utilized, further improving reasoning efficiency and resource utilization, while reducing the difficulty of adapting the computational graph.

[0120] In one embodiment, the storage space includes candidate sub-storage spaces corresponding to multiple candidate batch sizes; operation S230 may include the following operations: according to the target batch size, determining a target sub-storage space corresponding to the target batch size from multiple candidate sub-storage spaces, and determining a target calculation graph corresponding to the target batch size from multiple candidate calculation graphs; using the target calculation graph corresponding to the target batch size, performing large model inference on the sub-task data obtained from the target sub-storage space to generate corresponding sub-task processing results, wherein the sub-task processing results include the feature vector output by the large model.

[0121] Candidate batch sizes refer to multiple pre-defined possible batch size options, used to select the appropriate batch size based on different requirements during model inference. For example, candidate batch sizes may include 2, 4, 8, 16, etc. The target batch size refers to the batch size used to process the current subtask, determined based on the current task requirements. For example, for a subtask containing 10 object requests, the target batch size might be set to 16.

[0122] During the execution phase, after determining the target batch size for each subtask, a target substorage space corresponding to the target batch size is identified from multiple candidate substorage spaces. A candidate substorage space is a region within the storage space specifically dedicated to storing data corresponding to a specific candidate batch size. For example, a subregion within the storage space is allocated for data with a hash batch size of 8 to store input data for model inference at that size. The target substorage space is the candidate substorage space corresponding to the target batch size and is used to store the data for the current subtask.

[0123] During the execution phase, after determining the target batch size for each subtask, a target computation graph corresponding to the target batch size can be determined from multiple candidate computation graphs. Candidate computation graphs are computation graphs that have been pre-captured and saved for different candidate batch sizes. Each computation graph corresponds to a specific batch size and is used to efficiently perform model inference at that size. For example, there are corresponding computation graphs for batch sizes of 2, 4, 8, etc. The target computation graph refers to the candidate computation graph corresponding to the target batch size and is used to perform model inference for the current subtask. For example, for a subtask with a target batch size of 16, the target computation graph is the pre-captured computation graph for a batch size of 16.

[0124] For each subtask, after determining the target substorage space and target computation graph, the subtask data for that subtask can be retrieved from the target substorage space. Using the target computation graph, large-scale model inference is performed on the subtask data to generate the corresponding subtask processing result. Large-scale model inference refers to the process of using the large model to perform calculations and analysis to generate feature vectors or other output results. The model inference process is executed using the computation steps defined by the target computation graph, ultimately outputting a feature vector as the subtask processing result. For example, executing the computation graph on the GPU performs inference on the image data in the substorage space to generate a feature vector.

[0125] According to the embodiments of the present disclosure, by allocating sub-storage spaces and computation graphs for different candidate batch sizes, different batch sizes of data can be efficiently processed. During large-model inference, video memory fragmentation can be reduced, improving memory utilization. Dedicated computation graphs can also accelerate the inference process, enabling flexible processing of tasks of various sizes and improving overall performance and adaptability.

[0126] In one embodiment, the support of vLLM, SGLang and the large model-based task processing method provided in this application for static graphs / dynamic graphs, sub-graphs / whole graphs is shown in Table 1 below.

[0127] Table 1

[0128]

[0129] Based on Table 1, it can be seen that in vLLM, although full-graph capture and sub-graph capture are supported, the use of CudaGraph is bound to static graphs, and CudaGraph cannot be used in dynamic graph scenarios. In SGLang, although dynamic graph capture is supported, full-graph capture is not supported. Compared with vLLM and SGLang, the large-model-based task processing method provided in this application simplifies the adaptation work, improves the success rate of CudaGraph use, and can support CudaGraph subgraph / whole capture in static graph scenarios as well as CudaGraph subgraph / whole capture in dynamic graph scenarios.

[0130] Since this application is different from vLLM and SGLang, which strongly rely on dynamic graphs being converted into static graphs before using CudaGraph acceleration, the capture and replay operations on CudaGraph can be performed directly on the dynamic graph, which has higher flexibility.

[0131] Figure 7 The block diagram of a task processing device based on a large model according to an embodiment of the present disclosure is schematically shown.

[0132] like Figure 7 As shown, the large model-based task processing device 700 may include a first determination module 710 , a second determination module 720 and a task processing module 730 .

[0133] The first determining module 710 is configured to determine a target batch size for the task to be processed in response to receiving the task to be processed.

[0134] The second determining module 720 is configured to determine the storage space for the tasks to be processed according to the target batch size.

[0135] The task processing module 730 is used to call the target computation graph corresponding to the target batch size based on the mapping relationship between the candidate batch size and the candidate computation graph, perform large model inference on the task data obtained from the storage space, and obtain the task processing result.

[0136] According to an embodiment of the present disclosure, the task to be processed includes M object requests received at the same time, where M is a positive integer greater than 1.

[0137] According to an embodiment of the present disclosure, the first determining module 710 may include a dividing unit and a first determining unit.

[0138] The dividing unit is configured to divide the M object requests into a plurality of subtasks, wherein each subtask includes at least one object request.

[0139] The first determining unit is configured to determine a target batch size for each of the plurality of subtasks.

[0140] According to an embodiment of the present disclosure, the determining unit may include a mapping subunit.

[0141] A mapping sub-unit is used to map the multiple subtasks to the minimum upper limit step value in the step interval according to a preset batch size step interval, so as to obtain the target batch size of each of the multiple subtasks, wherein the step interval is determined based on at least one of the video memory limit of the large model and the number of object requests.

[0142] According to an embodiment of the present disclosure, the maximum step value of the step interval is determined based on the available video memory that can be processed by the large model and the required video memory requested by a single object.

[0143] According to an embodiment of the present disclosure, the storage space includes candidate sub-storage spaces corresponding to respective candidate computing kernels, and the candidate computing kernels correspond to candidate batch sizes.

[0144] According to an embodiment of the present disclosure, the large model-based task processing apparatus 700 may further include a first extraction module and a first processing module.

[0145] The first extraction module is used to extract the input tensor tuples of each of the multiple candidate computing kernels, wherein the tensor tuple includes at least one of the pointer address of the input tensor corresponding to the sub-storage space, the shape of the input tensor, and the data type.

[0146] The first processing module is used to process the input tensor tuple through a hash algorithm to obtain a candidate hash, wherein the candidate hash is used as an index of a candidate computation graph.

[0147] According to an embodiment of the present disclosure, the second determining module 720 may include a second determining unit and a storage unit.

[0148] The second determining unit is configured to determine a target sub-storage space for each sub-task from among a plurality of candidate sub-storage spaces according to a target batch size of each sub-task.

[0149] The storage unit is used to store the task data corresponding to the subtask into the target substorage space for the subtask through an asynchronous storage mechanism according to the pointer address of the target substorage space.

[0150] According to an embodiment of the present disclosure, the large model-based task processing apparatus 700 may further include a second extraction module and a second processing module.

[0151] The second extraction module is configured to extract a new input tensor tuple of the candidate computing kernel in response to a change in the input tensor tuple.

[0152] The second processing module is used to process the new tensor tuple through a hash algorithm to obtain a new candidate hash.

[0153] According to an embodiment of the present disclosure, the large model-based task processing apparatus 700 may further include a calculation module and an update module.

[0154] The calculation module is used to call the large model for forward calculation through a target calling process using simulated data for the new candidate hash when the calculation graph corresponding to the new candidate hash is not included in multiple candidate calculation graphs, so as to obtain a candidate calculation graph corresponding to the new candidate hash.

[0155] The updating module is used to update the mapping relationship by using the new candidate hash and the candidate computation graph corresponding to the new candidate hash.

[0156] According to an embodiment of the present disclosure, the large model-based task processing apparatus 700 may further include a first replacement module and a second replacement module.

[0157] The first replacement module is used to replace the original initialization process of the model class for the large model with the target initialization process by calling the decorator, so as to complete the acquisition and replay of multiple candidate calculation graphs by executing the target initialization process and completing the initialization.

[0158] The second replacement module is used to replace the original calling process of the model class for the large model with the target calling process by calling the decorator, thereby obtaining a modified model class so that multiple candidate computation graphs can be used for large model reasoning when calling the model class.

[0159] According to an embodiment of the present disclosure, a mapping relationship includes a correspondence between multiple candidate batch sizes and multiple candidate computation graphs; the mapping relationship is obtained in the following manner: using simulated data for each candidate batch size, calling a large model through a target calling process to perform forward calculations to obtain candidate computation graphs corresponding to each candidate batch size; and establishing a mapping relationship based on multiple candidate batch sizes and the candidate computation graphs corresponding to each candidate batch size.

[0160] According to an embodiment of the present disclosure, the storage space includes candidate sub-storage spaces corresponding to respective ones of a plurality of candidate batch sizes.

[0161] According to an embodiment of the present disclosure, the task processing module 730 may include a third determination unit and an inference unit.

[0162] The third determining unit is configured to determine, according to the target batch size, a target sub-storage space corresponding to the target batch size from among the multiple candidate sub-storage spaces, and to determine a target computation graph corresponding to the target batch size from among the multiple candidate computation graphs.

[0163] An inference unit is configured to perform large-model inference on subtask data obtained from a target substorage space using a target computation graph corresponding to a target batch size, and generate corresponding subtask processing results, wherein the subtask processing results include a feature vector output by the large model.

[0164] Figure 8 The structural block diagram of the intelligent agent of the large model according to the embodiment of the present disclosure is schematically shown.

[0165] In the embodiments of the present disclosure, inspired by the von Neumann structure in modern computer theory, such as Figure 8 As shown, the AI ​​agent 800 may include five core modules: an input module 810 , a control module 820 , a storage module 830 , an operation module 840 and an output module 850 .

[0166] Input module 810 is responsible for receiving or perceiving information such as queries, requests, instructions, signals, or data from the outside world (e.g., users or the external environment) and converting it into a format that AI agent 800 can understand and process. Input module 810 is the primary link for AI agent 800 to interact with the outside world. It enables AI agent 800 to efficiently and accurately obtain necessary "sensory" information from the outside world and respond to this information.

[0167] In an example, the input module 810 may input the object request and tasks to be processed described above.

[0168] Control module 820 is the core support for AI agent 800's ability to handle complex tasks. During the initialization phase, control module 820 can execute the previously described methods of replacing the original initialization process of the model class for the large model with the target initialization process by calling a decorator, and replacing the original call process of the model class for the large model with the target call process by calling the decorator. During the model inference phase, control module 820 can execute the previously described method for processing tasks based on the large model.

[0169] In the example, the control module 820 will continuously interact with the storage module 830, the computing module 840, and / or the output module 850 during operation. However, it should be noted that in the embodiment of the present disclosure, the control module 820 acts as a single initiator to initiate communication with the storage module 830, the computing module 840, and / or the output module 850, and there is no communication coupling between the storage module 830, the computing module 840, and the output module 850.

[0170] In this example, the performance of control module 820 may be closely related to the large model underlying AI agent 800. To fully leverage the capabilities of the large language model, the internal structure of control module 820 may be designed to be highly configurable and extensible to handle a variety of different tasks and requirements in real-world scenarios.

[0171] The storage module 830 may be responsible for memorizing the mapping relationship between candidate batch sizes and candidate computation graphs. The candidate computation kernels, candidate sub-storage spaces, and candidate batch sizes mentioned above may also be included in the storage module 830.

[0172] In this example, after receiving a task to be processed, the AI ​​agent 800 may trigger the task processing process, obtain the mapping relationship between the candidate batch size and the candidate computation graph from the storage module 830, and feed it back to the control module 820. The control module 820 may then pass the fed-back mapping relationship between the candidate batch size and the candidate computation graph to the output module 850.

[0173] The operation module 840 can be viewed as a predefined tool library. The tools for performing hash calculations as described above can be included in the operation module 840.

[0174] In this example, when the AI ​​agent 800 needs to process data, it can call the relevant tools from the operation module 840 and feed it back to the control module 820. The control module 820 can then use the fed-back tools to process the input tensor tuples to obtain candidate hashes. It is understandable that although the large language model has excellent language understanding and generation capabilities, it is just like a human, and the tasks it can solve without any tools are very limited. When the AI ​​agent 800 is given the ability to call tools, it can achieve tasks such as completing candidate hashes with the help of tools for hash calculation.

[0175] During the model training phase, the output module 850 may output the task processing results described above.

[0176] The AI ​​agent 800 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and enhance flexibility and versatility.

[0177] Figure 9A block diagram of an electronic device suitable for implementing a large model-based task processing method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0178] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. Computing unit 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to bus 904.

[0179] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0180] The computing unit 901 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the large model-based task processing method. For example, in some embodiments, the large model-based task processing method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 908.

[0181] In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the large model-based task processing method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to execute the large model-based task processing method in any other appropriate manner (e.g., via firmware).

[0182] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0183] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0184] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0185] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0186] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0187] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0188] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0189] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A task processing method based on a large model, comprising: In response to receiving a task to be processed, determining a target batch size for the task to be processed; Determining storage space for the tasks to be processed based on the target batch size; as well as Based on the mapping relationship between the candidate batch processing size and the candidate calculation graph, the target calculation graph corresponding to the target batch processing size is called, and large model reasoning is performed on the task data obtained from the storage space to obtain the task processing result.

2. The method according to claim 1, wherein The pending tasks include M object requests received at the same time, where M is a positive integer greater than 1; Determining a target batch size for the task to be processed includes: Dividing the M object requests into multiple subtasks, wherein each subtask includes at least one object request; and The target batch size of each of the plurality of subtasks is determined.

3. The method according to claim 2, wherein: Determining the target batch size of each of the plurality of subtasks includes: According to a preset step interval of a batch size, the plurality of subtasks are respectively mapped to the minimum upper limit step value in the step interval to obtain the target batch size of each of the plurality of subtasks, wherein the step interval is determined based on at least one of the video memory limit of the large model and the number of object requests.

4. The method according to claim 3, wherein: The maximum step value of the step interval is determined according to the available video memory that can be processed by the large model and the required video memory requested by a single object.

5. The method according to any one of claims 1 to 4, wherein The storage space includes candidate sub-storage spaces corresponding to respective candidate computing kernels, the candidate computing kernels corresponding to the candidate batch sizes; The method further comprises: Extracting input tensor tuples of each of the plurality of candidate computing kernels, wherein the tensor tuples include at least one of a pointer address of the input tensor corresponding to the sub-storage space, a shape of the input tensor, and a data type; and The input tensor tuple is processed by a hash algorithm to obtain a candidate hash, wherein the candidate hash is used as an index of the candidate computation graph.

6. The method according to claim 5, wherein: The determining, based on the target batch size, a storage space for the task to be processed, includes: Determining a target sub-storage space for each of the sub-tasks from among the plurality of candidate sub-storage spaces according to the target batch size of each of the sub-tasks; and According to the pointer address of the target sub-storage space, the task data corresponding to the sub-task is stored in the target sub-storage space for the sub-task through an asynchronous storage mechanism.

7. The method according to claim 5, further comprising: In response to a change in the input tensor tuple, extracting a new input tensor tuple of the candidate computing kernel; as well as The new tensor tuple is processed by a hash algorithm to obtain a new candidate hash.

8. The method according to claim 7, further comprising: When the computation graph corresponding to the new candidate hash is not included in the plurality of candidate computation graphs, using the simulated data for the new candidate hash, calling the large model through the target calling process to perform forward computation to obtain a candidate computation graph corresponding to the new candidate hash; as well as The mapping relationship is updated using the new candidate hash and the candidate computation graph corresponding to the new candidate hash.

9. The method according to any one of claims 1 to 4, further comprising: By calling the decorator, the original initialization process of the model class for the large model is replaced with the target initialization process, so as to complete the acquisition and replay of multiple candidate calculation graphs during initialization by executing the target initialization process; as well as By calling the decorator, the original calling process of the model class for the large model is replaced with the target calling process, and a modified model class is obtained, so that the large model reasoning is performed using the multiple candidate computation graphs when calling the model class.

10. The method according to claim 9, wherein: The mapping relationship includes a correspondence between multiple candidate batch sizes and multiple candidate computation graphs; The mapping relationship is obtained in the following way: Using the simulated data for each candidate batch size, calling the large model to perform forward computation through the target calling process to obtain a candidate computation graph corresponding to each candidate batch size; as well as The mapping relationship is established according to the multiple candidate batch sizes and the candidate computation graphs corresponding to the candidate batch sizes.

11. The method according to claim 9, wherein The storage space includes candidate sub-storage spaces corresponding to each of the plurality of candidate batch sizes; The method of calling a target computation graph corresponding to the target batch size based on a mapping relationship between the candidate batch size and the candidate computation graph, performing large model inference on the task data obtained from the storage space, and obtaining a task processing result includes: According to the target batch size, determining a target sub-storage space corresponding to the target batch size from among the plurality of candidate sub-storage spaces, and determining a target computation graph corresponding to the target batch size from among the plurality of candidate computation graphs; and Using a target computation graph corresponding to the target batch size, large model inference is performed on the subtask data obtained from the target sub-storage space to generate corresponding subtask processing results, wherein the subtask processing results include feature vectors output by the large model.

12. A task processing device based on a large model, comprising: a first determining module configured to determine, in response to receiving a task to be processed, a target batch size for the task to be processed; A second determining module is used to determine the storage space for the task to be processed according to the target batch size; as well as The task processing module is used to call the target computation graph corresponding to the target batch size based on the mapping relationship between the candidate batch size and the candidate computation graph, perform large model inference on the task data obtained from the storage space, and obtain the task processing result.

13. An artificial intelligence agent configured to execute the method according to any one of claims 1 to 11.

14. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 11.

15. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

16. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images