Inference task processing method, distributed system, computing device and computer readable storage medium

Abstract

The embodiment of the invention provides an inference task processing method, a distributed system, computing equipment and a computer readable storage medium, the inference task processing method is applied to a first node of the distributed system, the first node is any one of at least one node of the distributed system, and a parallel computing unit is deployed on the first node. Comprising the following steps: in response to an execution instruction of a current reasoning request on a first node, unloading a task processing model corresponding to a previous reasoning request from a parallel computing unit under the condition that the types of the previous reasoning request and a task processing model corresponding to the current reasoning request are inconsistent, and installing the task processing model corresponding to the current reasoning request; and running the task processing model on the parallel computing unit, and executing the current reasoning request. The task reasoning model instance is intelligently scheduled and operated, and resource allocation is optimized to improve the reasoning task processing efficiency.

Description

Technical Field

[0001] The embodiments of this specification relate to the technical field of artificial intelligence, and in particular, to a method for processing reasoning tasks, a distributed system, a computing device, and a computer-readable storage medium. Background Art

[0002] With the development of artificial intelligence and machine learning technologies, the use of task processing models to perform reasoning tasks has been widely used in many fields, such as image recognition, natural language processing, and intelligent recommendation systems, due to its advantages of efficient processing of complex data, rapid response, and adaptability to diverse application scenarios.

[0003] At present, with the expansion of the user base and the diversification of application scenarios, deploying task reasoning models on distributed systems and utilizing multiple nodes and parallel computing units on the nodes can achieve large-scale concurrent processing and improve overall performance and reliability.

[0004] However, in response to the growing and dynamically changing inference requests, the resources of the parallel computing units running the task inference model have become relatively scarce. At the same time, inference requests often correspond to different inference tasks and application scenarios. Using a single or mismatched task inference model can easily lead to resource waste and reduce the efficiency of inference task processing. How to intelligently schedule the running of task inference model instances and optimize resource allocation to improve the efficiency of inference task processing is an urgent problem to be solved. Summary of the invention

[0005] In view of this, an embodiment of this specification provides a method for processing a reasoning task. One or more embodiments of this specification also relate to another method for processing a reasoning task, a distributed system, a computing device, another computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a method for processing an inference task is provided, which is applied to a first node of a distributed system, where the first node is any one of at least one node of the distributed system, and a parallel computing unit is deployed on the first node, including: In response to an execution instruction of the current reasoning request on the first node, when the types of task processing models corresponding to the previous reasoning request and the current reasoning request are inconsistent, the task processing model corresponding to the previous reasoning request is uninstalled from the parallel computing unit, and the task processing model corresponding to the current reasoning request is installed; Run the task processing model on the parallel computing unit to execute the current inference request.

[0007] According to a second aspect of an embodiment of this specification, another reasoning task processing method is provided, which is applied to a management module of a distributed system, wherein the distributed system further includes at least one node, including: Acquire multiple inference requests for different types of task processing models, and distribute the multiple inference requests to at least one node, wherein the first node is any one of the at least one node, and perform the following steps on the first node: In response to an execution instruction of the current reasoning request on the first node, when the types of task processing models corresponding to the previous reasoning request and the current reasoning request are inconsistent, the task processing model corresponding to the previous reasoning request is uninstalled from the parallel computing unit, and the task processing model corresponding to the current reasoning request is installed; Run the task processing model on the parallel computing unit to execute the current inference request.

[0008] According to a third aspect of an embodiment of this specification, there is provided a distributed system, including a management module and at least one node, any node having a parallel computing unit deployed thereon, and the first node being any one of the at least one node; A management module, configured to obtain multiple inference requests for different types of task processing models and distribute the multiple inference requests to at least one node; The first node is used to respond to the execution instruction of the current reasoning request on the first node. When the types of task processing models corresponding to the previous reasoning request and the current reasoning request are inconsistent, the task processing model corresponding to the previous reasoning request is uninstalled from the parallel computing unit, and the task processing model corresponding to the current reasoning request is installed, and the task processing model on the parallel computing unit is run to execute the current reasoning request.

[0009] According to a fourth aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the reasoning task processing method of the first aspect are implemented.

[0010] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the reasoning task processing method of the second aspect are implemented.

[0011] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned reasoning task processing method are implemented.

[0012] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which implements the steps of the above-mentioned reasoning task processing method when executed by a processor.

[0013] In one embodiment of the present specification, on the first node, the task processing model type of the task processing model type corresponding to the previous reasoning request of the current reasoning request is dynamically identified, and when different types of task processing models are detected, the task reasoning model that is no longer needed is automatically uninstalled, and the task reasoning model suitable for the current reasoning request is installed at the same time, thereby ensuring efficient utilization of resources and rapid response, effectively solving the inefficiency problem caused by resource shortages and the use of mismatched task reasoning models, optimizing resource allocation, and improving the efficiency of reasoning task processing of a single node, thereby improving the overall performance and reliability of the distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of a method for processing an inference task provided by an embodiment of this specification; Figure 2 is a flowchart of another reasoning task processing method provided by an embodiment of this specification; Figure 3 is a processing flow chart of a method for processing a reasoning task applied to reasoning of a large language model provided by an embodiment of this specification; Figure 4 It is a schematic diagram of the structure of a distributed system provided by an embodiment of this specification; Figure 5 This is a schematic diagram of the architecture of a distributed system provided by an embodiment of this specification; Figure 6 is a structural block diagram of a computing device provided by one embodiment of this specification; Figure 7 It is a structural block diagram of another computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0015] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0016] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0017] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0018] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0019] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM), a multi-modal pre-training model, etc.

[0020] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0021] First, the terms involved in one or more embodiments of this specification are explained.

[0022] Reasoning tasks: In the field of machine learning and artificial intelligence, reasoning tasks refer to the process of using a trained model to predict or classify new data. Unlike model training, reasoning focuses on applying models to make fast and accurate judgments, and is widely used in areas such as image recognition and natural language processing. Efficient reasoning can reduce latency and improve decision quality, and is a key link in realizing intelligent services.

[0023] Distributed system: A system consisting of multiple independent computers connected through a network that work together to achieve a common goal. Distributed systems are designed to increase processing speed, system reliability and scalability. They can allocate computing and storage resources on multiple nodes, effectively cope with large-scale data processing and high-concurrency requests, and are suitable for cloud computing, big data analysis and other scenarios.

[0024] Convolutional Neural Network (CNN): A feed-forward neural network whose artificial neurons can respond to surrounding units within a certain coverage area, and performs well in large image processing. Convolutional neural networks are particularly suitable for recognizing visual patterns because they can effectively reduce the dimensionality of image data while retaining key information. This type of network usually contains one or more convolutional layers and pooling layers, followed by fully connected layers (similar to traditional neural networks). Convolutional neural networks are widely used in image classification, object detection and other fields.

[0025] Recurrent Neural Network (RNN): A type of neural network for processing sequential data, where the output of the network depends not only on the current input, but also on the previous state or previous input. RNNs are unique in that they have an internal state (memory) that allows information to persist, which makes them well suited for tasks such as time series analysis, speech recognition, and natural language processing.

[0026] Generative Adversarial Network (GAN): Generative adversarial networks consist of two parts - the generator and the discriminator, both of which are deep learning models. The generator tries to create fake data that looks real, while the discriminator tries to distinguish between real data and fake data produced by the generator. Through this adversarial process, the two networks train and improve each other, and eventually the generator is able to create very realistic synthetic data. Generative adversarial networks are used to generate content such as images, videos, audio, etc.

[0027] Transformer architecture model: A deep learning model based on the attention mechanism, designed to process sequence data without relying on recurrent neural networks (RNNs). It was originally designed to solve natural language processing (NLP) tasks, but its application has expanded to multiple fields including image processing and speech recognition. The core of the Transformer model is the self-attention mechanism, which allows the model to take into account other elements in the entire sequence when processing an element at one position. This feature makes the Transformer particularly suitable for tasks that require understanding of context, such as machine translation, text summarization, and question-answering systems. In addition, the Transformer significantly improves computational efficiency by parallelizing the training process.

[0028] Bidirectional Encoder Representations from Transformers (BERT) model: A pre-trained language representation model based on the Transformer architecture, designed to improve the performance of various natural language processing tasks. Description: Unlike traditional unidirectional language models, BERT uses a bidirectional training approach, which means it can consider contextual information on both the left and right sides of a word. This makes BERT perform well on a variety of natural language processing tasks such as sentiment analysis, named entity recognition, and question-answering systems.

[0029] Graphics Processing Unit (GPU): A processor designed to accelerate graphics rendering and processing, especially good at performing parallel computing tasks. In addition to graphics processing, GPUs are widely used in scientific computing, machine learning and other fields because they can efficiently handle large amounts of data in parallel. Through its numerous cores, GPUs can execute multiple computing threads simultaneously, thus performing well in tasks that require a lot of computing resources, such as deep learning.

[0030] Tensor Processing Unit (TPU): A processor designed specifically to accelerate machine learning workloads, especially tensor operations in deep learning. It optimizes the execution efficiency of neural network calculations and can more efficiently process large-scale data sets and complex model training tasks.

[0031] Neural Processing Unit (NPU): A type of processor designed specifically for performing neural network calculations, designed to accelerate the execution of deep learning algorithms. The NPU optimizes the training and inference speed of machine learning models, and is particularly suitable for processing complex image recognition, natural language processing and other tasks, improving efficiency and reducing energy consumption.

[0032] In this specification, a method for processing an inference task is provided. This specification also involves another method for processing an inference task, a distributed system, a computing device, another computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0033] See also Figure 1 , Figure 1 A flowchart of a method for processing an inference task provided by an embodiment of the present specification is shown. The method is applied to a first node of a distributed system. The first node is any one of at least one node of the distributed system. A parallel computing unit is deployed on the first node, and includes the following specific steps: Step 102: In response to the execution instruction of the current reasoning request on the first node, if the types of task processing models corresponding to the previous reasoning request and the current reasoning request are inconsistent, the task processing model corresponding to the previous reasoning request is uninstalled from the parallel computing unit, and the task processing model corresponding to the current reasoning request is installed.

[0034] A distributed system is a system composed of multiple independent computers connected through a network, and each computer works together to provide reasoning services based on a neural network model. A distributed system includes multiple nodes, and independent resources are deployed on each node, including parallel computing units, host system memory, and persistent storage media. Distributed systems can effectively handle large-scale data processing and multiple high-concurrency reasoning requests, improve the speed of reasoning task processing, and increase system reliability and scalability. For example, a distributed system such as a cloud computing platform includes multiple cloud server nodes, and any cloud server node runs on a corresponding host machine, and any host machine is deployed with a parallel computing unit.

[0035] The task processing model is a neural network model used to perform reasoning tasks, including but not limited to: convolutional neural network, recurrent neural network, generative adversarial network, Transformer architecture model, BERT model, large language model, etc. For example, a large language model applied to natural language reasoning tasks.

[0036] Different reasoning tasks require different types of task processing models to execute. The types of task processing models can be classified according to at least one of the application scenario, data type, model specification, etc. For example, image recognition may require a convolutional neural network model, while text analysis is more suitable for using a recurrent neural network or a large language model. For another example, real-time sentiment analysis in a resource-constrained environment may require the use of a lightweight version of the BERT large language model, while large-scale text summary generation in the cloud is more suitable for using a large language model with a larger Transformer architecture.

[0037] The first node is any node that distributes multiple inference requests. The node has independent resources, such as parallel computing units, host system memory, and persistent storage media, and is used to perform inference tasks based on neural network models. In a distributed system, the first node is the basic unit for processing inference requests. Each first node can work independently.

[0038] The current inference request is the inference task request that needs to be processed currently. It has been distributed to the first node but has not yet started execution. The task processing model for executing the current inference request has not yet been loaded into the parallel computing unit. It is necessary to determine whether the task processing model installed on the parallel computing unit needs to be dynamically switched based on the model type. For example, node A has just received a new inference request for sentiment analysis of text.

[0039] The previous inference request is the most recent inference task that was distributed to the first node before the current inference request. The previous inference request has been completed, and the task processing model that executed the previous inference request is installed on the parallel computing unit. For example, the task of object recognition for an image that node A has just completed is the previous inference request.

[0040] Parallel computing units are model hardware accelerators deployed on nodes that can perform parallel computing, including but not limited to graphics processing units, tensor processing units, and neural network processing units. Parallel computing units are the core components for executing inference requests. Parallel computing units reduce latency and increase throughput by efficiently processing data in parallel. Different parallel computing units are optimized for specific types of computing tasks, so choosing the right computing unit plays a role in improving resource utilization. The efficient operation of task reasoning models depends on the acceleration support of parallel computing units. However, due to factors such as cost, power consumption, and physical space constraints, the parallel computing units deployed on distributed systems are relatively limited, and it is necessary to reasonably plan resource allocation strategies and scheduling algorithms to maximize performance and efficiency using limited hardware resources.

[0041] Optionally, the current inference request can be dynamically determined based on a hybrid scheduling strategy (preemptive mode and first-in-first-out queue mode), or it can be determined based on any mode (preemptive mode or first-in-first-out queue mode). Among them, the preemptive mode is that high-priority requests can interrupt low-priority tasks and load their models first. The first-in-first-out queue mode is to strictly queue according to time order to avoid resource competition. Optionally, the configuration interface allows the definition of mode switching rules through policy files or application programming interfaces (APIs), such as enabling preemptive mode when the node load is below a threshold.

[0042] Identify that the types of the task processing models corresponding to the previous reasoning request and the current reasoning request are inconsistent. One optional way is to identify that the model identifiers of the task processing models corresponding to the previous reasoning request and the current reasoning request are inconsistent. For example, if the model hash of the previous reasoning request is a1b2c3 and the model hash of the current reasoning request is d4e5f6, then the types are determined to be inconsistent. Another optional way is to identify that the request metadata corresponding to the previous reasoning request and the current reasoning request are inconsistent. For example, if the X-Model-Type field (such as model_type=resnet30) extracted from the Hypertext Transfer Protocol (HTTP) header of the current reasoning request is inconsistent with the X-Model-Type field (such as model_type=resnet50) extracted from the HTTP header of the most recent reasoning request recorded by the node, then the types are determined to be inconsistent. Another optional way is to identify that the containers corresponding to the previous reasoning request and the current reasoning request are inconsistent. For example, if the previous request is processed by container pod-1 and the current request needs to be processed by pod-2, then the types are determined to be inconsistent. This is not limited here.

[0043] Unloading the task processing model corresponding to the previous reasoning request from the parallel computing unit of the first node. One optional method is: releasing the unit storage medium of the parallel computing unit of the first node and unloading the task processing model corresponding to the previous reasoning request. Another optional method is: asynchronously migrating the task processing model corresponding to the previous reasoning request of the parallel computing unit of the first node to a backup storage medium, which is not limited here.

[0044] Install the task processing model corresponding to the current reasoning request. One optional method is to load the task processing model corresponding to the current reasoning request to the unit storage medium of the parallel computing unit of the first node. Another optional method is to asynchronously migrate the task processing model corresponding to the current reasoning request in the backup storage medium to the unit storage medium of the parallel computing unit of the first node. This is not limited here.

[0045] For example, on node A, the current reasoning request is reasoning request 1 (real-time chatbot interaction request), and the previous reasoning request is: Reasoning request 4: Multimodal image batch processing request submitted by user D, which requires support from the enterprise version model (large-scale parallel processing is preferred).

[0046] The type of task processing model corresponding to the former is the large language model of the basic version (7B parameters), and the type of task processing model corresponding to the latter is the large language model of the enterprise version (175B parameters). The model identifiers of the two are inconsistent. Release the video memory of the GPU of node A, unload the large language model of the enterprise version of reasoning request 4, and load the large language model of the basic version of reasoning request 1 into the video memory of the GPU of node A.

[0047] When different types of task processing models are detected, the task reasoning model that is no longer needed is automatically uninstalled, and the task reasoning model suitable for the current reasoning request is installed, thereby ensuring efficient utilization of resources and rapid response, effectively solving the inefficiency problems caused by resource shortages and the use of mismatched task reasoning models, optimizing resource allocation, and at the same time, providing an environmental configuration basis for computing resources and models for the subsequent execution of the current reasoning request.

[0048] Step 104: Run the task processing model on the parallel computing unit to execute the current reasoning request.

[0049] Run the task processing model on the parallel computing unit to execute the current reasoning request. One optional way is to run the task processing model on the parallel computing unit and directly execute the current reasoning request. Another optional way is to run the task processing model on the parallel computing unit and execute the current reasoning request in a containerized manner. Another optional way is to run the task processing model on the parallel computing unit and call the hardware acceleration library to execute the current reasoning request. There is no limitation here.

[0050] It should be noted that the task processing model corresponding to the current reasoning request can be installed in step 102 to complete the cold start process of the model. The cold start here refers to the process of loading the model for the first time when no relevant task processing model is preloaded on the parallel computing unit. This process may involve operations such as reading model files from persistent storage media, initializing model parameters, and preparing necessary operating environments. By optimizing this process, such as pre-caching commonly used models and using efficient serialization technology to reduce loading time, the cold start time can be significantly reduced, thereby directly executing reasoning requests and improving the system's response speed and user experience.

[0051] It is also possible to perform hot updates or dynamic adjustments after the model is loaded, which means that the model in use can be updated or adjusted according to actual needs without interrupting the service. For example, in a high-concurrency scenario, the system can monitor real-time performance indicators to decide whether to add more resources to the currently running task processing model, or seamlessly switch to a new model version when a better version of the model is detected to improve reasoning accuracy or efficiency.

[0052] Exemplarily, the basic version of the large language model loaded on the graphics processing unit is run to directly execute the inference request 1 to complete the real-time chatbot interaction.

[0053] In an embodiment of the present specification, on the first node, the task processing model type corresponding to the previous reasoning request of the task processing model type of the current reasoning request is dynamically identified, and when different types of task processing models are detected, the task reasoning model that is no longer needed is automatically uninstalled, and the task reasoning model suitable for the current reasoning request is installed at the same time, thereby ensuring efficient utilization of resources and rapid response, effectively solving the inefficiency problems caused by resource shortages and the use of mismatched task reasoning models, optimizing resource allocation, and improving the efficiency of reasoning task processing of a single node, thereby improving the overall performance and reliability of the distributed system.

[0054] In an optional embodiment of this specification, the following specific steps are also included: In response to the execution instruction of the current reasoning request on the first node, when the types of task processing models corresponding to the previous reasoning request and the current reasoning request are consistent, the task processing model on the parallel computing unit is run to execute the current reasoning request.

[0055] Run the task processing model on the parallel computing unit to execute the current reasoning request. One optional way is to run the task processing model on the parallel computing unit and directly execute the current reasoning request. Another optional way is to run the task processing model on the parallel computing unit and execute the current reasoning request in a containerized manner. Another optional way is to run the task processing model on the parallel computing unit and call the hardware acceleration library to execute the current reasoning request. There is no limitation here.

[0056] For example, on node A, the current inference request is inference request 3 (multi-language translation batch request), and the previous inference request is: Reasoning request 4: Multimodal image batch processing request submitted by user D, which requires support from the enterprise version model (large-scale parallel processing is preferred).

[0057] The type of task processing model corresponding to the former is the large language model of the enterprise version (175B parameters), and the type of task processing model corresponding to the latter is the large language model of the enterprise version (175B parameters), and the model identifiers of the two are consistent.

[0058] Run the enterprise version of the large language model on the GPU and directly execute inference request 3 to complete the language translation batch.

[0059] In the embodiments of this specification, efficient resource utilization and rapid response are achieved by dynamically identifying and switching task processing models in a distributed system. When it is detected that the current inference request uses the same type of model as the previous request, the existing model on the parallel computing unit is directly run for processing, avoiding unnecessary unloading and reloading processes, reducing delays, and improving processing efficiency, which not only optimizes resource allocation, but also ensures the high availability and scalability of the distributed system.

[0060] In an optional embodiment of the present specification, a container corresponding to an inference service of at least one task processing model is running on the first node, and any container processes an inference request of a specific type of task processing model; a method for identifying whether the types of task processing models corresponding to a previous inference request and a current inference request are consistent includes: It is identified whether the previous inference request and the current inference request come from the same container on the first node.

[0061] The inference service of the task processing model is a service that provides inference capabilities based on a specific neural network model, such as image recognition, text analysis, or speech processing. The inference service is deployed in the node container of the distributed system to cope with large-scale concurrent requests and efficient processing requirements, and to achieve isolation and flexible scheduling.

[0062] Containers are a lightweight operating system virtualization technology that allows developers to package applications and their dependencies into a portable package that can run consistently in any environment. Containers provide features such as process isolation and resource restrictions, but share the kernel of the host operating system. Container technology (such as Docker) allows applications to be seamlessly migrated and executed in different environments, greatly improving development efficiency and deployment flexibility. Each container is an independent execution environment with its own file system, memory space, process table, etc., which is lighter than traditional virtual machines.

[0063] The container corresponding to the inference service of the task processing model is used to host the inference service of a specific task processing model. The container contains all the dependencies required to run the model (such as library files, configuration files, etc.) and provides an isolated execution environment. Each containerized inference service can focus on processing inference requests for a specific type of task processing model, thereby achieving efficient resource utilization and service isolation. Through container orchestration tools (such as Kubernetes), the management and scheduling of containers can be further optimized.

[0064] An optional method for identifying whether the previous inference request and the current inference request come from the same container on the first node is: based on the container identifier of the container where the previous inference request and the current inference request are located, identifying whether the previous inference request and the current inference request come from the same container on the first node.

[0065] For example, two containers are running on node A: Container pod-1: deploys the basic version of the large language model (7B parameters), which is dedicated to processing low-latency requests such as real-time chatbots; Container pod-2: deploys the enterprise version of the large language model (175B parameters), which is dedicated to processing large-scale batch processing tasks.

[0066] The last inference request is inference request 4: a multimodal image batch processing request submitted by user D (requires the enterprise edition model), which is processed by container pod-2. The current inference request is inference request 1: a real-time chatbot interaction request submitted by user A (a basic version model is required), which needs to be processed by container pod-1.

[0067] Compare the container IDs "pod-2" and "pod-1" to determine that they are from different containers.

[0068] In the embodiments of this specification, by identifying whether the inference request comes from the same container, accurate scheduling and resource optimization configuration of the task processing model are achieved, unnecessary model switching and resource reallocation are avoided, processing delays are reduced, and response speed and service stability are improved. At the same time, based on the isolation characteristics of the container, the independence between different types of tasks is ensured, the flexibility and scalability of the system are enhanced, and the distributed system can process large-scale concurrent inference requests more efficiently.

[0069] In an optional embodiment of the present specification, in step 102, unloading the task processing model corresponding to the previous reasoning request from the parallel computing unit of the first node includes the following specific steps: Migrating the parameters of the task processing model corresponding to the previous inference request from the unit storage medium of the parallel computing unit of the first node to the host system memory or persistent storage medium of the first node; In step 102, the task processing model corresponding to the current reasoning request is installed, including the following specific steps: The parameters of the task processing model corresponding to the current reasoning request are migrated from the host system memory or persistent storage medium of the first node to the unit storage medium of the parallel computing unit of the first node.

[0070] The parameters of a task processing model are the weights and bias values ​​learned by the task processing model during training, as well as any other configuration parameters or hyperparameters, which together determine the performance of the model. The number and structure of parameters may vary greatly for different task processing models. During the inference phase, these parameters are loaded into the computing unit to perform specific tasks.

[0071] The unit storage medium of the parallel computing unit is a high-speed storage area directly connected to the parallel computing unit, which is used to store the task processing model and its parameters currently in use for fast access and processing. This storage medium usually has high read and write speeds and low latency, allowing the parallel computing unit to efficiently perform computing tasks. The unit storage medium of the parallel computing unit is the video memory or similar fast storage resources on a dedicated hardware accelerator (such as a graphics processing unit, a tensor processing unit, and a neural network processing unit, etc.).

[0072] The host system memory is a high-speed storage area for running the host operating system. It provides temporary data storage space to support the computing needs of the central processing unit (CPU). The host system memory is used to store currently active application code and data, as well as temporary information required by the operating system. Compared with persistent storage media, it has faster access speed, but data will be lost after power failure.

[0073] Persistent storage media are storage devices that can save data even after the power is turned off, including hard disk drives (HDDs), solid state drives (SSDs), etc. Persistent storage media are used to save data for a long time. Persistent storage media have large capacity but relatively slow access speed.

[0074] An optional method is to migrate the parameters of the task processing model corresponding to the previous inference request from the unit storage medium of the parallel computing unit of the first node to the system memory or persistent storage medium of the host machine of the first node using the device-to-host (D2H) access mode to migrate the parameters of the task processing model corresponding to the previous inference request from the unit storage medium of the parallel computing unit of the first node to the system memory or persistent storage medium of the host machine of the first node. Specifically, direct memory access or asynchronous data transmission is performed through the PCIe or NVLink hardware interface from the unit storage medium of the parallel computing unit to the host machine memory.

[0075] An optional method is to migrate the parameters of the task processing model corresponding to the current reasoning request from the host system memory or persistent storage medium of the first node to the unit storage medium of the parallel computing unit of the first node: adopt the host-to-device (H2D) access mode to migrate the parameters of the task processing model corresponding to the current reasoning request from the host system memory or persistent storage medium of the first node to the unit storage medium of the parallel computing unit of the first node. Specifically, direct memory access or asynchronous data transmission is performed through the PCIe or NVLink hardware interface to load the model parameters from the host system memory to the unit storage medium of the parallel computing unit.

[0076] Exemplarily, the device-to-host access mode is adopted to start asynchronous data transmission from the device to the host through the PCIe 4.0 bus and NVLink 3.0 hybrid interface: 17.5 billion parameters of the enterprise version model are read in blocks from the graphics processing unit's video memory, and the model parameters are migrated to the cache area of ​​the host's memory. The 700 million parameters of the basic version model are loaded from the cache area of ​​the host's memory, and the parameters are pipelined through the host-to-device access mode. The parameters are loaded into the graphics processing unit's video memory in batches, and the pre-initialization of the computing core is started at the same time. The dedicated hardware acceleration library is used to optimize the PCIe bandwidth utilization. After loading is completed, the deep learning inference optimization engine is called to compile the model in real time.

[0077] In an embodiment of the present specification, by migrating the task processing model parameters of the previous reasoning request from the unit storage medium of the parallel computing unit to the host system memory or persistent storage medium, and migrating the task processing model parameters corresponding to the current reasoning request to the unit storage medium of the parallel computing unit of the first node, the hardware resources of the parallel computing unit are dynamically scheduled, so that the new task processing model can be quickly loaded into the parallel computing unit for processing, which not only optimizes the efficiency of computing resource utilization, reduces model switching time, and improves the response speed and throughput of the overall distributed system, but also is suitable for application scenarios that require frequent model changes, greatly improving the efficiency and flexibility of reasoning task processing.

[0078] In an optional embodiment of the present specification, the distributed system further includes a management module; Before step 102, the following specific steps are also included: receiving at least one inference request sent by the management module; At least one inference request is placed in a node task queue of the first node.

[0079] The node task queue corresponding to the node is a request buffer structure pre-allocated by the node, and the sequential processing of requests is completed in a first-in-first-out mode. Optionally, on the basis of first-in-first-out, a hybrid scheduling strategy can be implemented in combination with a dynamic priority queue-jumping mechanism. Optionally, the capacity of the node task queue is dynamically adjusted according to the node hardware resources. For example, for nodes equipped with high-computing power parallel computing units, the queue length threshold is set to 50-100 inference requests. For edge computing nodes, the queue length threshold is limited to 5-10 inference requests.

[0080] For example, in the inference task queue of node 1, there are the following requests: Reasoning request 5: User D's real-time video analysis task (priority P1, requires a professional version of the large language model) Node 1 receives two inference requests sent by the management module and puts the two inference requests into the inference task queue of the first node. At this time, the requests in the inference task queue sorted by reception time are as follows: Reasoning request 5: User D's real-time video analysis task (priority P1, requires a professional version of the large language model) Reasoning request 1: User A’s real-time chatbot interaction (priority P3, requires a basic version of the large language model) Reasoning request 2: Multi-language translation batch processing for user C (priority P0, requires large language model of enterprise edition) In the embodiments of this specification, multiple inference requests are placed in the node task queue corresponding to the node, and a first-in-first-out queue mechanism is adopted to achieve efficient task scheduling and resource management, thereby optimizing service response time and system throughput.

[0081] See also Figure 2 , Figure 2 A flowchart of another reasoning task processing method provided by an embodiment of the present specification is shown. The method is applied to a management module of a distributed system. The distributed system also includes at least one node, and includes the following specific steps: Step 202: Obtain multiple inference requests for different types of task processing models, and distribute the multiple inference requests to at least one node, wherein the first node is any one of the at least one node, and perform the following steps on the first node: in response to the execution instruction of the current inference request on the first node, when the types of task processing models corresponding to the previous inference request and the current inference request are inconsistent, uninstall the task processing model corresponding to the previous inference request from the parallel computing unit, and install the task processing model corresponding to the current inference request; run the task processing model on the parallel computing unit to execute the current inference request.

[0082] The inference request for the task processing model is a task request that needs to use the task processing model to perform inference tasks. The inference request is sent to the distributed system, which determines which node to execute on based on the type, priority, resource requirements, etc. of the inference request. For example, an inference request that requires object recognition for an uploaded image will be assigned to a node that is deployed with an efficient graphics processing unit to accelerate the calculation process.

[0083] To obtain multiple inference requests for different types of task processing models, one optional method is to obtain multiple inference requests for different types of task processing models in batches according to time sequence. Another optional method is to obtain multiple inference requests for different types of task processing models in groups according to model type. Another optional method is to obtain multiple inference requests for different types of task processing models in batches according to the priority of the inference request. There is no limitation here.

[0084] Distribute multiple inference requests to at least one node. One optional method is to distribute multiple inference requests to at least one node in a round-robin manner. For example, there are three nodes A, B, and C in a distributed system. When the distributed system receives multiple inference requests, it sequentially distributes the inference requests to nodes A, B, and C, and then starts the cycle from A again. Another optional method is to distribute multiple inference requests to at least one node based on the node status parameters of at least one node. For example, there are three nodes A, B, and C in a distributed system. When the distributed system receives multiple inference requests, node A currently has a low load and has more available resources, while nodes B and C are close to full load. , multiple inference requests are preferentially sent to node A. Another optional method is: based on the request parameters of multiple inference requests, multiple inference requests are distributed to at least one node. For example, there are three nodes A, B and C in the distributed system. When the distributed system receives an inference request G1 for performing an image recognition task and an inference request G2 for text analysis, the graphics processing unit of node A is currently loaded with a visual processing model, and the graphics processing unit of node B is currently loaded with a large-scale Transformer architecture model suitable for natural language processing. Inference request G1 will be sent to node A, and inference request G2 will be sent to node B, without limitation here.

[0085] For example, on a cloud computing platform for a large language model, three specifications of large language model instances are provided to the outside world: basic version (7B parameters), professional version (13B parameters), and enterprise version (175B parameters). Users can deploy on-demand elastically scalable reasoning services on this cloud computing platform according to project requirements. The distributed system divides the corresponding resources (such as GPU video memory, number of CPU cores, etc.) on a host cluster equipped with a high-performance graphics processing unit according to the instance specifications subscribed by the user and real-time load monitoring data, builds cloud server nodes, and isolates and runs reasoning services of different model versions in the form of containers on this cloud server. The distributed system obtains three reasoning requests for different types of task processing models in batches according to the time sequence: Reasoning request 1: A real-time chatbot interaction request submitted by user A, requiring the use of the basic version of the model (low latency first); Reasoning request 2: User B submits a request to generate an academic literature abstract, which requires calling the professional version model (high accuracy is preferred); Reasoning request 3: A multi-language translation batch processing request submitted by user C, which requires support from the enterprise version model (large-scale parallel processing is preferred).

[0086] The distributed system distributes these three inference requests to two nodes (node ​​1 and node 2) in a round-robin manner: inference request 1 and inference request 3 go to node 1, and inference request 2 goes to node 2.

[0087] In the embodiments of the present specification, in a distributed system, multiple inference requests for different types of task processing models are distributed to at least one node, thereby achieving large-scale concurrent processing and allowing different nodes to independently execute their respective inference tasks, thereby reducing waiting time and improving processing speed. At the same time, it lays a solid foundation for subsequent dynamic resource adjustment and intelligent scheduling strategies.

[0088] In an optional embodiment of the present specification, distributing multiple inference requests to at least one node in step 202 includes the following specific steps: Traversing the current node state of at least one node recorded in the polling table, and assigning the first reasoning request to the next available node, wherein the polling table is a pre-built sequential circular list that records the node state of at least one node in real time, and the first reasoning request is any one of the multiple reasoning requests; The first inference request is updated, and the step of traversing the current node state of at least one node recorded in the polling table is returned to be executed until the multiple inference requests are distributed to at least one node.

[0089] The polling table is a pre-built sequential circular list that records the node status of at least one node in real time. It is used to record the node status of at least one node in a distributed system in real time. The polling table is dynamically updated and can be updated at a preset frequency or triggered by a node status change, which is not limited here. The polling table is used to guide how inference requests are distributed among the nodes to ensure load balancing and efficient use of resources. For example, in a distributed system with three nodes A, B, and C, the polling table is initialized to [A, B, C]. When the first inference request arrives, it is assigned to node A; the second request is assigned to node B; the third request is assigned to node C; the fourth request returns to node A again, and so on.

[0090] The current node status of a node is the resource usage status of the node at the current moment, reflecting the current working status, resource usage, and processing capacity of the node. The current node status of a node includes but is not limited to CPU utilization, memory utilization, storage space occupancy, current number of tasks, network bandwidth usage, etc. For example, if the current CPU utilization of a node is 80%, the memory utilization is 60%, and 5 inference requests are being executed, then the current node status of the node is "high load, unavailable". On the contrary, if the CPU utilization of another node is only 20%, the memory utilization is 30%, and there are no inference requests in progress, then the current node status of the node is "low load, available".

[0091] Available nodes are nodes that receive and process new inference requests. Available nodes have sufficient resources (such as computing power, memory, and storage) to ensure that new tasks can be executed smoothly.

[0092] The next available node is the first available node in the sequential loop recorded in the polling table that meets the resource conditions for executing the inference request. For example, the inference request needs to process a large number of matrix operations, requiring at least 4 idle CPU cores and a high-performance GPU. If the polling process finds that the current CPU utilization of a node has reached 80%, only 2 cores are available, and its GPU is occupied by other tasks, then the node does not meet the resource conditions of this inference request, and the system will continue to check the next node. For another example, a certain inference request requires at least 16GB of free RAM to ensure smooth operation because it involves large-scale data set operations. If the polling process finds that a node has sufficient CPU and GPU resources, but its memory utilization has reached 90%, leaving only 8GB unused, then this node does not meet the resource conditions of the inference request, and the system will continue to look for the next node. For another example, for an inference request that needs to load a large model file (such as the enterprise version of the large language model), at least 50GB of storage space may be required for temporary data storage. If the polling process finds that a node has insufficient hard disk space, even if its computing and memory resources are sufficient, it cannot undertake this task and will be skipped. For example, some inference tasks with high real-time requirements (such as real-time video stream analysis) also have specific requirements for network bandwidth, requiring a minimum of 1Gbps of available bandwidth to ensure data transmission efficiency. If during the polling process, it is found that the current network bandwidth of a node has been mostly occupied by other tasks, leaving only 500Mbps of available bandwidth, then it will not be considered as an available node that meets the resource conditions of the inference request.

[0093] For example, in a distributed system that deploys a large language model, the polling table includes a sequential circular list: node 1, node 2, node 3. The current state of each node is marked as "available". When the system receives the following inference request sequence: Inference request 1 (real-time translation task, requiring the basic version model) is assigned to node 1 in round-robin order; Reasoning request 2 (document summary generation, requiring the professional version model) is assigned to node 2 in round-robin order; Inference request 3 (image batch processing, requiring Enterprise Edition model) is assigned to node 3 in polling order.

[0094] In the embodiments of the present specification, the node status is updated in real time through a polling table to complete the uniform distribution of reasoning requests to at least one node, and different reasoning requests are broken up to reduce resource conflicts caused by simultaneous access to the same node. On this basis, since at least one node receives the reasoning request according to the order of available status, load balancing between nodes is achieved, avoiding the accumulation of reasoning requests on the same node, realizing intelligent elastic scheduling, improving resource utilization, reducing request processing delays, and improving the efficiency of reasoning task processing.

[0095] In an optional embodiment of the present specification, after step 202, the following specific steps are also included: When the queue length of the node task queue of the first node exceeds a preset threshold, a capacity expansion strategy for the first node is triggered to be executed, wherein the first node is any one of the at least one node.

[0096] The preset threshold is the upper limit of the queue length that is dynamically adjusted according to the hardware resources and / or historical load data of the first node. For example, for nodes equipped with high-performance graphics processing units, the preset threshold is 50 inference requests; for edge computing nodes, the preset threshold is 10 inference requests; when the real-time delay (such as average waiting time) of the node task queue exceeds 200ms, the threshold is automatically lowered by 20% to prioritize service quality.

[0097] The expansion strategy for nodes is a strategy for automatically adjusting the allocation of computing resources based on real-time monitoring data. Expansion strategies include, but are not limited to: parallel computing unit expansion strategy, node expansion strategy, etc. For example, the automatic horizontal container expansion and contraction (Horizontal Pod Autoscaler, referred to as HPA) rule is used for dynamic expansion. When it is detected that the length of the node task queue of the first node exceeds the preset threshold and the real-time delay (such as the average waiting time) exceeds the acceptable range, the distributed system will trigger the expansion operation. For another example, through the elastic scaling service provided by the cloud computing platform, additional virtual machines or container instances can be quickly deployed to share the load to ensure that all inference requests can be processed within a reasonable time. For another example, the resources of the first node are upgraded, the node specifications are upgraded from 4-core central processing unit + 16GB memory to 8-core central processing unit + 32GB memory, the allocation strategy of the parallel computing unit is dynamically adjusted, and the single graphics processing unit exclusive mode is switched to multi-task sharing mode.

[0098] For example, the task queue threshold of node A is 50 requests. When the real-time monitoring shows that the queue length reaches 55 and the memory occupancy rate of the graphics processing unit exceeds 85%, the following expansion strategy is triggered: According to the type of requests in the queue (70% for the basic version model and 30% for the enterprise version model), the type and number of nodes that need to be expanded are calculated, and the application programming interface of the cloud computing platform is called to start a new cloud server instance. The container image of the basic version of the large language model is preloaded on the new node, and the model parameters in the persistent storage medium are mounted. The new container on node A is registered with the load balancer through the service discovery mechanism so that it can receive subsequent inference requests.

[0099] In the embodiment of this specification, when the length of the task queue of the first node exceeds the preset threshold, the expansion strategy is automatically triggered to ensure that the distributed system can handle more reasoning requests without affecting the service quality. Through real-time monitoring and elastic scaling, not only the increase in delays caused by overload is avoided, but also the stability and response speed of the system are improved, which can effectively balance resource utilization and service efficiency, ensuring user experience while optimizing costs.

[0100] In an optional embodiment of the present specification, when the queue length of the node task queue of the first node exceeds a preset threshold, triggering execution of a capacity expansion strategy for the first node includes the following specific steps: When the queue length of the node task queue of the first node exceeds a preset threshold, an idle candidate node with a parallel computing unit deployed thereon is obtained from the node pool, and the candidate node is added to the distributed system.

[0101] The node pool is a pre-configured collection of spare computing nodes that can be dynamically added to the distributed system. These nodes are usually idle or underloaded, have consistent basic software and hardware configurations, and can quickly respond to system expansion needs to ensure service continuity and efficiency.

[0102] Candidate nodes are spare computing nodes pre-deployed in the node pool and in an idle or low-load state, and their hardware configuration matches the current task requirements of the distributed system. Optionally, the candidate nodes meet the following conditions: they are in an idle state, have installed the host operating system, container environment, hardware driver and basic reasoning service framework, and have deployed parallel computing units to ensure that the candidate nodes can quickly access the distributed system.

[0103] Exemplarily, the task queue length of node B reaches 60 (threshold 50), of which 80% are batch translation requests that require the enterprise version of the large language model, and 20% are real-time summary requests that require the professional version of the model. Select 3 idle A100 nodes (nodes D, E, and F) from the node pool, all of which are pre-installed with the enterprise version model container image. At the same time, start the container service of nodes D and E, load the enterprise version model with 175B parameters, and deploy the professional version model container on node F to share real-time requests. The load balancer distributes the newly arrived batch inference requests to nodes D and E in a round-robin manner, and the real-time inference requests are directed to node F. The 40 batch inference tasks in the node B queue can also be migrated to nodes D and E.

[0104] In the embodiment of this specification, when the queue length of the node task queue of the first node exceeds the preset threshold, a candidate node that is idle and deployed with a parallel computing unit is obtained from the node pool, and the candidate node is added to the distributed system, which significantly improves the scalability and flexibility of the system. By pre-configuring candidate nodes that match the task requirements, the processing capacity can be quickly increased during peak loads, effectively avoiding service delays and performance degradation caused by overload, which not only ensures that high-priority and real-time tasks can be processed in a timely manner, but also optimizes resource utilization efficiency and reduces user waiting time. In addition, the way of dynamically adjusting and allocating computing resources allows the system to respond quickly to load changes and maintain efficient and stable operation of services, thereby improving user experience and service quality.

[0105] In an optional embodiment of the present specification, after step 202, the following specific steps are also included: When the number of nodes whose node task queues corresponding to at least one node are empty queues exceeds a preset threshold, a second node is released from the distributed system, wherein the second node is a node whose node task queues are empty queues among at least one node.

[0106] The preset threshold is a preset upper limit of the number of free nodes in the distributed system. For example, if the preset threshold is set to 5 free nodes and the task queues of 8 nodes are empty, the distributed system will select 3 nodes from the 8 nodes to release, thereby reducing unnecessary resource consumption and saving costs.

[0107] The second node is any node whose node queue is an empty queue. The node has independent resources, such as parallel computing units, host system memory, and persistent storage media, and is used to perform reasoning tasks based on neural network models. In a distributed system, the second node is the basic unit for processing reasoning requests. Each second node can work independently.

[0108] For example, the cloud computing platform currently includes 10 cloud server nodes: Node 1-3: Deploy the enterprise version model; Node 4-7: Deploy the professional version model; Node 8-10: Deploy the basic version model.

[0109] Monitoring shows that the task queues of nodes 1, 2, 4, 5, and 8 are empty and have been idle for more than 15 minutes. The preset threshold is 3 idle nodes. Release nodes 1 and 2, return them to the node pool of the cloud computing platform, and cancel their service registration information. The load balancer will no longer distribute new inference requests to them.

[0110] In the embodiments of this specification, when the number of nodes whose node task queues corresponding to at least one node are empty queues exceeds a preset threshold, the second node is released from the distributed system, effectively reducing unnecessary resource consumption and lowering operating costs. This mechanism dynamically adjusts the number of active nodes to ensure that the system only maintains a reasonable number of working nodes, thereby improving resource utilization and economic benefits. This avoids resource waste without affecting the overall performance and service quality of the system.

[0111] The following combination Figure 3 Taking the application of the reasoning task processing method provided in this specification in large language model reasoning as an example, the reasoning task processing method is further explained. Figure 3 A flowchart of a processing process of a method for processing an inference task applied to large language model inference provided by an embodiment of the present specification is shown. The method is applied to a cloud computing platform, and the cloud computing platform includes multiple nodes, and a graphics processing unit is deployed on any node, including the following specific steps: Step 302: Obtain M inference requests for different types of large language models, traverse the current node states of multiple nodes recorded in the polling table, and assign the M inference requests to the next available node.

[0112] Step 304: put the M inference requests into the node task queues corresponding to the N available nodes.

[0113] Step 306: Identify whether the last inference request of the first node and the current inference request are from the same container on the first node, where the first node is any one of the N available nodes.

[0114] Step 308: If yes, run the large language model on the GPU to execute the current reasoning request.

[0115] Step 310: If not, migrate the parameters of the large language model corresponding to the previous inference request from the video memory of the graphics processing unit of the first node to the host system memory of the first node, and migrate the parameters of the target large language model from the host system memory of the first node to the video memory of the graphics processing unit of the first node, run the large language model on the graphics processing unit, and execute the current inference request.

[0116] Step 312: When the queue length of the node task queue of the first node exceeds a preset threshold, obtain an idle candidate node with a graphics processing unit deployed from the node pool, and add the candidate node to the cloud computing platform to obtain N+1 available nodes.

[0117] Step 314: when the number of nodes whose node task queues corresponding to the N available nodes are empty queues exceeds a preset threshold, the second node is released from the cloud computing platform to obtain N-1 available nodes.

[0118] In the embodiments of this specification, a reasoning architecture that combines elastic reasoning and dynamic unloading is proposed, aiming to significantly improve the efficiency of GPU resource utilization and system flexibility. By dynamically adjusting the size of the GPU instance, the capacity is automatically expanded to meet the demand when the number of reasoning requests increases; the capacity is reduced to release resources when the load is low, effectively reducing the cost. In addition, the dynamic parameter migration mechanism between the GPU and the host system memory is used to support time-sharing sharing of GPU resources by different large language models, thereby improving hardware utilization. Specifically, when it is detected that the first node needs to switch to process tasks of different models, the model parameters of the previous task can be quickly migrated to the host memory, and the new model parameters can be loaded into the video memory of the GPU to achieve efficient task switching. At the same time, for nodes with excessive load, the load is dispersed by adding idle nodes from the node pool to ensure system stability and response speed. For nodes that have been idle for a long time, resources are released to further optimize resource allocation, thereby improving the elasticity and efficiency of the reasoning service as a whole and ensuring a high-quality service experience.

[0119] Corresponding to the above method embodiment, this specification also provides a distributed system embodiment. Figure 4 FIG. 1 shows a schematic diagram of a distributed system provided by an embodiment of this specification. Figure 4 As shown, the distributed system 400 includes a management module 410 and at least one node, any node is deployed with a parallel computing unit, and the first node 420 is any one of the at least one node; A management module 410, configured to obtain multiple reasoning requests for different types of task processing models and distribute the multiple reasoning requests to at least one node; The first node 420 is used to respond to the execution instruction of the current reasoning request on the first node 420. When the types of task processing models corresponding to the previous reasoning request and the current reasoning request are inconsistent, the task processing model corresponding to the previous reasoning request is uninstalled from the parallel computing unit, and the task processing model corresponding to the current reasoning request is installed, and the task processing model on the parallel computing unit is run to execute the current reasoning request.

[0120] The management module 410 is the core control unit of the distributed system 400, responsible for global resource scheduling, task distribution, node management and dynamic expansion and contraction strategy execution. Its core functions include: 1. Request perception and routing: receiving external reasoning requests, analyzing metadata such as request type, priority, resource requirements, and dynamically selecting reasonable nodes for distribution. 2. Node status monitoring: real-time collection of load indicators of each node (such as graphics processing unit utilization, memory usage, task queue length), and building a global resource view to support intelligent scheduling. 3. Elastic expansion and contraction decisions: trigger expansion or contraction operations according to preset strategies (such as queue overflow threshold, number of idle nodes) to maintain the resource efficiency and service stability of the distributed system. 4. Model lifecycle management: coordinate hot switching of models between nodes (such as unloading old models and loading new models) to ensure seamless connection of reasoning services.

[0121] The request distributor routes the inference request to the appropriate node based on a load balancing algorithm (such as weighted round-robin or minimum number of connections). For example, dynamic routing can be implemented using Nginx reverse proxy combined with a custom scripting language, or scheduling logic can be extended based on the Kubernetes Ingress Controller.

[0122] The resource monitoring engine collects node hardware indicators and container status through agents (such as Prometheus Exporter) to generate real-time monitoring data. For example, the Prometheus+Grafana monitoring stack is integrated and GPU MetricsExporter is used to collect indicators such as video memory utilization and computing core occupancy.

[0123] The policy executor analyzes the scaling rules (such as queue length thresholds, response time service level agreements), and calls the cloud platform application program interface or orchestration tool to perform operations. For example, automatic horizontal container scaling based on Kubernetes implements automatic scaling points.

[0124] The metadata repository stores metadata such as model version, node configuration, and scheduling strategy, and supports fast query and version rollback. For example, Redis is used as a distributed key-value storage to record key information such as model hash, node IP, and container image version.

[0125] In an optional embodiment of the present specification, a container corresponding to an inference service of at least one task processing model is running on the first node 420, and any container processes an inference request of a specific type of task processing model; The first node 420 is specifically used to identify whether the previous inference request of the first node 420 and the current inference request are from the same container on the first node 420 .

[0126] In an optional embodiment of the present specification, the first node 420 is further used to put the multiple inference requests into the node task queue of the first node 420, and send a capacity expansion strategy request to the management module 410 when the queue length of the node task queue of the first node 420 exceeds a preset threshold; The management module 410 is further configured to obtain, in response to a capacity expansion strategy request, candidate nodes that are idle and deployed with parallel computing units from the node pool, and add the candidate nodes to the distributed system.

[0127] In an optional embodiment of the present specification, the management module 410 is also used to release a second node from the distributed system when the number of nodes whose node task queue corresponding to at least one node is an empty queue exceeds a preset threshold, wherein the second node is a node whose node task queue is an empty queue among at least one node.

[0128] In the embodiments of this specification, in a distributed system, the management module distributes multiple reasoning requests for different types of task processing models to at least one node, thereby realizing large-scale concurrent processing, allowing different nodes to independently execute their respective reasoning tasks, reducing waiting time, and improving processing speed. On the first node, the task processing model type corresponding to the previous reasoning request of the task processing model type of the current reasoning request is dynamically identified, and when different types of task processing models are detected, the task reasoning model that is no longer needed is automatically uninstalled, and the task reasoning model applicable to the current reasoning request is installed at the same time, thereby ensuring efficient use of resources and rapid response, effectively solving the inefficiency problem caused by resource shortages and the use of mismatched task reasoning models, optimizing resource allocation, and improving the efficiency of reasoning task processing of the first node, thereby improving the overall performance and reliability of the distributed system.

[0129] The above is a schematic scheme of a distributed system of this embodiment. It should be noted that the technical scheme of the distributed system and the technical scheme of the above-mentioned reasoning task processing method belong to the same concept, and the details not described in detail in the technical scheme of the distributed system can be referred to the description of the technical scheme of the above-mentioned reasoning task processing method.

[0130] With the above Figure 4 The content corresponds to Figure 5 A schematic diagram of the architecture of a distributed system provided by an embodiment of this specification is shown. Figure 5 As shown: The management module distributes multiple inference requests to Node 1 and Node 2.

[0131] The node task queue of node 1 is used to store the inference requests assigned to node 1, and the node task queue of node 2 is used to store the inference requests assigned to node 2.

[0132] Multiple inference requests are executed in a first-in-first-out order. Any inference request is executed on the container of the corresponding inference service isolated on the node.

[0133] The node identifies whether the task processing model type of the current inference request is consistent with the previous inference request. If not, the device-to-host mode is adopted to unload the task processing model corresponding to the previous inference request from the graphics processing unit's video memory to the host system memory, and the host-to-device mode is adopted to install the task processing model corresponding to the current inference request from the host system memory to the graphics processing unit's video memory.

[0134] The above is a schematic scheme of a distributed system of this embodiment. It should be noted that the technical scheme of the distributed system and the technical scheme of the above-mentioned reasoning task processing method belong to the same concept, and the details not described in detail in the technical scheme of the distributed system can be referred to the description of the technical scheme of the above-mentioned reasoning task processing method.

[0135] Figure 6 The block diagram of a computing device provided by an embodiment of the present specification is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0136] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a Network Interface Controller (NIC)) of wired or wireless type, such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, and a Near Field Communication (NFC).

[0137] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0138] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 600 may also be a mobile or stationary server.

[0139] The processor 620 is used to execute the following computer program / instruction, which, when executed by the processor, implements the above Figure 1The steps of the method for processing an inference task shown in the embodiment.

[0140] The above is a schematic solution of a computing device of this embodiment. It should be noted that the technical solution of the computing device is similar to the above Figure 1 The technical solutions of the reasoning task processing methods shown in the embodiments belong to the same concept. For details not described in detail in the technical solutions of the computing device, please refer to the above Figure 1 Description of the technical solution of the reasoning task processing method shown in the embodiment.

[0141] Figure 7 The block diagram of a computing device provided by an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0142] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a Network Interface Controller (NIC)) of wired or wireless type, such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, and a Near Field Communication (NFC).

[0143] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0144] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0145] The processor 720 is used to execute the following computer program / instruction, which, when executed by the processor, implements the above Figure 2 The steps of the method for processing an inference task shown in the embodiment.

[0146] The above is a schematic solution of a computing device of this embodiment. It should be noted that the technical solution of the computing device is similar to the above Figure 2 The technical solutions of the reasoning task processing methods shown in the embodiments belong to the same concept. For details not described in detail in the technical solutions of the computing device, please refer to the above Figure 2 Description of the technical solution of the reasoning task processing method shown in the embodiment.

[0147] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned reasoning task processing method when executed by a processor.

[0148] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned reasoning task processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned reasoning task processing method.

[0149] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned reasoning task processing method when executed by a processor.

[0150] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned reasoning task processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned reasoning task processing method.

[0151] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0152] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the contents contained in the computer readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0153] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0154] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0155] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A method for processing an inference task, applied to a first node of a distributed system, wherein the first node is any one of at least one node of the distributed system, and a parallel computing unit is deployed on the first node, comprising: In response to an execution instruction of a current reasoning request on the first node, when the types of task processing models corresponding to a previous reasoning request and the current reasoning request are inconsistent, uninstall the task processing model corresponding to the previous reasoning request from the parallel computing unit, and install the task processing model corresponding to the current reasoning request; The task processing model is run on the parallel computing unit to execute the current reasoning request.

2. The method according to claim 1, further comprising: In response to the execution instruction of the current reasoning request on the first node, when the types of task processing models corresponding to the previous reasoning request and the current reasoning request are consistent, the task processing model on the parallel computing unit is run to execute the current reasoning request.

3. The method according to claim 1, wherein: A container corresponding to an inference service of at least one task processing model is running on the first node, and any container processes an inference request of a task processing model of a specific type; The method of identifying whether the types of task processing models corresponding to the previous reasoning request and the current reasoning request are consistent includes: Identify whether a previous inference request and the current inference request come from the same container on the first node.

4. According to the method of claim 1, the step of unloading the task processing model corresponding to the previous reasoning request from the parallel computing unit of the first node comprises: Migrating the parameters of the task processing model corresponding to the previous inference request from the unit storage medium of the parallel computing unit of the first node to the host system memory or persistent storage medium of the first node; The step of installing the task processing model corresponding to the current reasoning request includes: Migrate the parameters of the task processing model corresponding to the current reasoning request from the host system memory or persistent storage medium of the first node to the unit storage medium of the parallel computing unit of the first node.

5. The method according to any one of claims 1 to 4, wherein: The distributed system also includes a management module; In the response to the execution instruction of the current reasoning request on the first node, before uninstalling the task processing model corresponding to the previous reasoning request from the parallel computing unit and installing the task processing model corresponding to the current reasoning request when the types of the task processing models corresponding to the previous reasoning request and the current reasoning request are inconsistent, the method further includes: receiving at least one inference request sent by the management module; The at least one inference request is placed in a node task queue of the first node.

6. A method for processing an inference task, applied to a management module of a distributed system, wherein the distributed system further comprises at least one node, comprising: Acquire multiple inference requests for different types of task processing models, and distribute the multiple inference requests to the at least one node, wherein the first node is any one of the at least one node, and perform the following steps on the first node: In response to an execution instruction of a current reasoning request on the first node, when the types of task processing models corresponding to a previous reasoning request and the current reasoning request are inconsistent, uninstall the task processing model corresponding to the previous reasoning request from the parallel computing unit, and install the task processing model corresponding to the current reasoning request; The task processing model is run on the parallel computing unit to execute the current reasoning request.

7. The method according to claim 6, wherein distributing the plurality of inference requests to the at least one node comprises: Traversing the current node state of the at least one node recorded in the polling table, and assigning the first reasoning request to the next available node, wherein the polling table is a pre-built sequential circular list that records the node state of the at least one node in real time, and the first reasoning request is any one of the multiple reasoning requests; The first inference request is updated, and the step of traversing the current node state of the at least one node recorded in the polling table is returned to be executed until multiple inference requests are distributed to the at least one node.

8. The method according to claim 6 or 7, after distributing the plurality of inference requests to the at least one node, further comprising: When the queue length of the node task queue of the first node exceeds a preset threshold, a capacity expansion strategy for the first node is triggered to be executed.

9. The method according to claim 8, wherein when the queue length of the node task queue of the first node exceeds a preset threshold, triggering execution of a capacity expansion strategy for the first node comprises: When the queue length of the node task queue of the first node exceeds a preset threshold, a candidate node that is idle and has a parallel computing unit deployed thereon is obtained from a node pool, and the candidate node is added to the distributed system.

10. The method according to claim 6 or 7, after distributing the plurality of inference requests to the at least one node, further comprising: When the number of nodes whose node task queues corresponding to the at least one node are empty queues exceeds a preset threshold, a second node is released from the distributed system, wherein the second node is a node whose node task queues are empty queues among the at least one node.

11. A distributed system, comprising a management module and at least one node, any node being deployed with a parallel computing unit, and the first node being any one of the at least one node; The management module is used to obtain multiple reasoning requests for different types of task processing models and distribute the multiple reasoning requests to the at least one node; The first node is used to respond to the execution instruction of the current reasoning request on the first node. When the types of task processing models corresponding to the previous reasoning request and the current reasoning request are inconsistent, the task processing model corresponding to the previous reasoning request is uninstalled from the parallel computing unit, and the task processing model corresponding to the current reasoning request is installed, and the task processing model on the parallel computing unit is run to execute the current reasoning request.

12. The distributed system according to claim 11, wherein: A container corresponding to an inference service of at least one task processing model is running on the first node, and any container processes an inference request of a task processing model of a specific type; The first node is specifically used to identify whether the previous inference request and the current inference request come from the same container on the first node.

13. The distributed system according to claim 11, wherein the first node is further configured to place the multiple inference requests into a node task queue of the first node, and send a capacity expansion strategy request to the management module when the queue length of the node task queue of the first node exceeds a preset threshold; The management module is further used to obtain idle candidate nodes with parallel computing units deployed from the node pool in response to the expansion strategy request, and add the candidate nodes to the distributed system.

14. The distributed system according to claim 11, wherein the management module is further configured to release the second node from the distributed system when the number of nodes whose node task queues corresponding to the at least one node are empty queues exceeds a preset threshold, wherein: The second node is a node whose node task queue is an empty queue among the at least one node.

15. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 5 are implemented.

16. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 6 to 10 are implemented.

17. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

18. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images