Task processing method and device, computing system and computing equipment

Abstract

The invention provides a task processing method and device, a computing system and computing equipment, and relates to the technical field of data processing.The method comprises the steps that for a task executed based on multiple processes, when multiple model loading requests sent by the multiple processes are obtained to request to load the same AI model, the model loading requests are sent to the multiple processes; one model loading request is sent to the acceleration card related to the task to instruct the acceleration card to load the AI model, the method avoids the situation that the acceleration card repeatedly loads the same AI model, multiple processes share one AI model, resource waste is reduced, and calculation efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a task processing method, apparatus, computing system and computing device. Background Technology

[0002] In AI for science (AI4S) applications, high-performance computing (HPC) and artificial intelligence (AI) technologies are often combined to improve computational efficiency. For example, some computational processes within an HPC framework can be offloaded to AI models, thereby leveraging AI to enhance computational efficiency.

[0003] In related technologies, when running AI4S type applications, the computational task involves both HPC (High-Performance Computation) and AI computation in a single iteration step (e.g., in molecular dynamics simulation, an iteration step refers to simulating molecular motion over a period of time; multiple iteration steps can simulate molecular motion over a longer period). In HPC simulations, the data for a single iteration step is often divided into multiple grids or cells. The CPU side typically creates multiple processes for parallel computation based on this. To use the AI ​​model, a context is created for each process, allowing the accelerator card to load the AI ​​model based on the context.

[0004] However, in the above method, since there is a one-to-one correspondence between processes and contexts, the accelerator card needs to repeatedly load the same AI model based on multiple processes to achieve multiple AI inferences within one iteration step, resulting in resource waste and thus affecting the overall computing efficiency. Summary of the Invention

[0005] This application provides a task processing method, apparatus, computing system, and computing device that enables multiple processes to share the same AI model, reducing resource waste and improving computing efficiency.

[0006] In a first aspect, this application provides a task processing method applied to computational scenarios that combine HPC and AI technologies to solve complex inference computation problems. Schematic, the method includes:

[0007] Get the M first model loading requests sent by the M processes executing the task, where the M first model loading requests are for loading the same artificial intelligence AI model, and M is a positive integer;

[0008] Send a second model loading request to at least one accelerator card associated with the task to instruct at least one accelerator card to load the AI ​​model.

[0009] In this context, the task is initiated by a computing application, such as an AI4S-type application. The task instructs the invocation of an AI model to perform multiple AI inferences, that is, to use the AI ​​model to process different input data and obtain multiple output data, thereby achieving multiple AI inferences. In this embodiment, the execution process of the task is described using a single iteration step as an example. That is, the multiple processes executing the task refer to the multiple processes corresponding to a single iteration step of the task. By executing multiple iteration steps in the same way, the entire task can be completed. Through the above method, for tasks executed by multiple processes, when multiple model loading requests are received from multiple processes requesting the loading of the same AI model, a single model loading request is sent to the task-related accelerator card to instruct the accelerator card to load the AI ​​model. This method avoids the accelerator card repeatedly loading the same AI model, realizes multiple processes sharing a single AI model, reduces resource waste, and improves computing efficiency.

[0010] In some embodiments, the second model loading request is one of the M first model loading requests. In other words, a model loading request is selected from the M first model loading requests and sent to the accelerator card. For example, the second model loading request is the first first model loading request obtained among the M first model loading requests. In this way, the model loading request is sent to the accelerator card in a timely manner without waiting for subsequent model loading requests, thus improving computational efficiency.

[0011] In some embodiments, the method further includes:

[0012] Receive M data transmission requests from M processes. The M data transmission requests are used to request the transmission of M first inference data that the M processes need to use for inference through the AI ​​model.

[0013] Store the M first inference data corresponding to the M processes in shared memory.

[0014] By sharing memory, data exchange between M processes and the accelerator card was achieved, providing technical support for subsequent execution of inference requests.

[0015] In some embodiments, the method further includes:

[0016] Receive M first inference requests sent by M processes, merge the M first inference requests into N second inference requests, each second inference request corresponds to second inference data, and the second inference data includes at least one data corresponding to a first inference request, where N is a positive integer;

[0017] Send N second inference requests to at least one accelerator card to instruct the at least one accelerator card to execute the N second inference requests through the AI ​​model.

[0018] By using the above method, multiple inference requests for the execution task were merged, achieving the synergy between optimal multi-process data partitioning and optimal throughput batch processing of the accelerator card, thereby improving resource utilization.

[0019] In some embodiments, the method further includes:

[0020] Before receiving M data transmission requests, create N shared service processes and allocate shared memory space for the N shared service processes;

[0021] Store the M first inference data corresponding to the M processes in shared memory, including:

[0022] Store the X first inference data corresponding to X processes out of M processes into the shared memory space corresponding to the first shared service process out of N shared service processes. X processes can be any X processes out of M processes, and the first shared service process can be any one of the N shared service processes. X is a positive integer.

[0023] Merge M first inference requests into N second inference requests, including:

[0024] After receiving the first inference requests from X processes, the first accelerator card is determined through the first shared service process, and a context is created based on the first accelerator card to generate a third inference request from among N second inference requests. The third inference request is used to request the first accelerator card to infer the second inference data corresponding to the first shared service process through the AI ​​model.

[0025] In some embodiments, if the third inference request is executed successfully, the execution result of the third inference request is stored in the shared memory space corresponding to the first shared service process, the execution result is divided according to the first inference requests of X processes, and the execution result corresponding to each first inference request is returned.

[0026] By creating N shared service processes and allocating shared memory space to each shared service process, it is convenient for multiple processes to exchange data with the accelerator card, providing technical support for merging inference requests and returning execution results.

[0027] Secondly, embodiments of this application provide a task processing apparatus, which includes at least one functional module for executing the task processing method provided by the first aspect or any possible implementation thereof.

[0028] Thirdly, embodiments of this application provide a computing system including a host and at least one accelerator card, wherein the host is used to implement the task processing method provided by the first aspect or any possible implementation thereof.

[0029] Fourthly, this application provides a computing device including a processor and a memory, wherein the processor is configured to execute at least one piece of program code stored in the memory to enable the computing device to implement the task processing method provided by the first aspect or any possible implementation thereof.

[0030] Fifthly, this application provides a computer-readable storage medium for storing at least one piece of program code, which, when executed by a computing device, causes the computing device to implement the task processing method provided by the first aspect or any possible implementation thereof. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).

[0031] Sixthly, this application provides a computer program product that, when run on a computing device, causes the computing device to implement the task processing method provided by the first aspect or any possible implementation thereof. The computer program product can be a software installation package; when the aforementioned task processing method needs to be implemented, the computer program product can be downloaded and executed on the computing device. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0033] Figure 2 This is a schematic diagram of the architecture of a host provided in an embodiment of this application;

[0034] Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0035] Figure 4 This is a flowchart of a task processing method provided in an embodiment of this application;

[0036] Figure 5 This is a schematic diagram illustrating a process of performing data transmission and model inference provided in an embodiment of this application;

[0037] Figure 6 This is a schematic diagram of a task processing method provided in an embodiment of this application;

[0038] Figure 7 This is a schematic diagram of another task processing method provided in an embodiment of this application;

[0039] Figure 8 This is a schematic diagram of yet another task processing method provided in the embodiments of this application;

[0040] Figure 9 This is a schematic diagram of a task processing device provided in an embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, all kinds of requests, AI models, and input data of AI models involved in this application are obtained under fully authorized conditions.

[0042] To facilitate understanding, the key terms and concepts involved in this application will be explained below.

[0043] High-performance computing (HPC) is a high-performance computing architecture in which multiple nodes form at least one HPC cluster through a network. Nodes in the same HPC cluster work in parallel with each other, thereby improving data processing speed and achieving high-performance computing.

[0044] Artificial intelligence (AI) models are a class of mathematical algorithm models that use machine learning concepts to solve practical problems. Typically, AI models include a large number of parameters and calculation formulas (or calculation rules).

[0045] AI for Science (AI4S) refers to the use of artificial intelligence technologies such as machine learning and deep learning to analyze and process multi-dimensional, multi-modal, and multi-scenario simulated and real data, solve complex inference and computational problems, accelerate the discovery, verification, and application of basic and applied sciences, and create the next generation of scientific paradigms. AI4S applications include, but are not limited to, biopharmaceuticals such as protein structure analysis, new materials research and development, cutting-edge physics such as nuclear fusion, Earth simulation and astronomical exploration such as climate prediction, and industrial design such as aircraft engines and automotive powertrain structures. Currently, in AI4S applications, HPC and AI technologies are often combined to improve computational efficiency. For example, some computational processes in the HPC framework are delegated to AI models, thereby leveraging the computing power of the AI ​​models to improve computational efficiency. In some embodiments, applications combining HPC and AI technologies are called "HPC+AI coupled applications," where AI replaces several computationally intensive and time-consuming key steps in the original numerical computation, but the overall computational process still follows the traditional computational framework. Within such applications, there is both traditional computational framework and some CPU computation, as well as AI computation, which are interdependent and highly coupled throughout the solution cycle.

[0046] A process is a running instance of a program on a specific set of data in a computer. It is the basic unit for resource allocation in a system and forms the foundation of the operating system architecture. In some embodiments, processes created in an HPC environment are called HPC processes. Furthermore, each process has a unique identifier. In distributed computing, processes can run on different nodes. Each process has a rank as its identifier, typically an integer starting from 0 and incrementing. Different processes can identify and interact with each other using their ranks.

[0047] Interprocess communication (IPC) refers to the process of exchanging data and transferring information between different processes. In an operating system, each process has its own independent address space and resources, and they are isolated from each other. However, in some situations, processes need to cooperate and exchange information to accomplish more complex tasks. Common IPC methods include pipes, signals, message queues, shared memory, sockets, etc., which are not limited to this application.

[0048] Message Passing Interface (MPI) is a cross-language communication protocol used to write parallel computer programs. It supports point-to-point and broadcast communication. MPI is a message-passing application programming interface that includes protocol and semantic specifications that define how it performs its characteristics in various implementations. MPI aims for high performance, scalability, and portability. In other words, MPI is a message-passing-based parallel programming specification and environment that enables data exchange and synchronization between parallel processes or threads. Currently, MPI is commonly used in programming environments such as supercomputers, computer clusters, and HPC, serving as a standard for inter-process communication in parallel computing.

[0049] Remote Procedure Call (RPC) is a computer communication protocol. This protocol allows a program running on one computer to call a subroutine in another address space (usually a computer on an open network), as if calling a local program, without requiring additional programming for this interaction (i.e., without needing to concern themselves with the details). RPC is a client / server (C / S) model; a classic implementation is a system that exchanges information by sending requests and receiving responses. RPC is also a mode of inter-process communication, where programs are distributed across different address spaces. Specifically, if the programs are on the same host, RPC can communicate through different virtual address spaces (even if using the same physical address); if the programs are on different hosts, RPC can interact through different physical addresses.

[0050] Service sharing is a method in server / client architecture to improve resource utilization and enable rapid service replication. For example, service sharing can be divided into stateful and stateless modes. Typically, network services handle the internal distribution logic of multiple node services. Through service sharing, one or more sets of services can be implemented to meet the calling needs of a large number of clients.

[0051] Accelerator cards, also known as accelerators, acceleration devices, or acceleration chips, are a type of specialized hardware accelerator or computer system designed to accelerate computational processes in scenarios such as AI and HPC. Illustratively, accelerator cards may include, for example, graphics processing units (GPUs), neural network processing units (NPUs), intelligent processing units (IPUs), tensor processing units (TPUs), domain-specific architecture (DSA) chips, etc., and this application is not limited to these.

[0052] In computer science, a greedy strategy is a method used to solve problems where the algorithm chooses the best local solution at each step, without considering the overall optimal solution.

[0053] The application scenarios and implementation environment of this application are described below.

[0054] This application can be applied to computational scenarios that combine HPC and AI technologies to solve complex inference problems. For example, taking molecular dynamics simulation as an example, the goal of this task is to simulate molecular motion over a target time period. Based on this, the target time period is divided into multiple time periods, and the simulation process of molecular motion in one time period is called an iteration step. Multiple iteration steps can simulate the molecular motion over the target time period. With the support of HPC technology, multiple AI inferences within an iteration step can be implemented by multiple parallel HPC processes, thereby improving the processing efficiency of the computational task. It should be noted that this is only an example and does not constitute a limitation of this application. For example, this application can also be applied to computational scenarios such as climate simulation and fluid simulation. In the above computational scenarios, in HPC simulation, the data of a single iteration step is often divided into multiple grids (or cells), with one grid corresponding to one HPC process. That is, by utilizing the multi-process capability of HPC and adopting parallel computing, the data involved in the task is divided into multiple processes for parallel execution. Multiple processes call the AI ​​model in parallel to achieve multiple AI inferences. However, since each process corresponds one-to-one with its context, within a single iteration step, the accelerator card needs to repeatedly load the same AI model across multiple processes to perform multiple AI inferences. This means the accelerator card stores multiple copies of the same AI model in memory, resulting in significant waste of memory resources. Furthermore, in scenarios where multiple tasks share the accelerator card, the same AI model needs to be repeatedly loaded. For example, if tasks A and B share the accelerator card, and each task corresponds to a different AI model, by dividing the data for one iteration step in task A into grids corresponding to one process, the accelerator card can execute task A in time slice 1, task B in time slice 2, and task A in time slice 3. Specifically, executing task A in time slice 1 involves performing AI inference in grid 1, while executing task A in time slice 3 involves performing AI inference in grid 2. In other words, although the accelerator card is executing task A in both time slices 1 and 3, the processes are different. Therefore, the accelerator card needs to load the AI ​​model based on different processes, resulting in repeated model loading.

[0055] Based on this, this application provides a task processing method that creates a shared service process to provide shared services for multiple HPC processes, enabling multiple HPC processes to share the same AI model. This avoids the accelerator card repeatedly switching process contexts, thus avoiding the repeated loading of the same AI model, thereby saving resources and improving overall computing efficiency.

[0056] The following is for reference. Figure 1 The implementation environment of this application will be described.

[0057] Figure 1This is a schematic diagram of an implementation environment provided in an embodiment of this application. For example... Figure 1 As shown, the implementation environment includes a computing system 100, which includes a host 101 and at least one accelerator card 102 interconnected. In some embodiments, the accelerator card 102 is also referred to as an acceleration device, accelerator, etc., and this application is not limited thereto.

[0058] The computing system 100 is a distributed computing system capable of executing multiple tasks in parallel. This application does not limit the type of tasks. The host 101 refers to a device used to run computing applications, capable of accessing wired or wireless networks, providing users with HPC services and AI services, such as AI4S applications. In this embodiment, the host 101 can control at least one accelerator card 102 to execute various tasks of the computing system 100. For example, in response to a task execution request, the host 101 creates multiple processes for executing the task and assigns these processes to at least one accelerator card 102. Furthermore, the number of hosts 101 can be one or more, which is not limited in this application.

[0059] In this embodiment, host 101 runs multiple clients and servers, i.e., host 101 adopts a C / S many-to-one architecture, where multiple clients correspond to one server. Multiple clients act as service users, and the server acts as a service provider. Based on this, multiple clients can simultaneously send requests to the server. After processing the received requests, the server returns the processing results to the multiple clients. Illustratively, multiple clients run multiple processes to execute tasks, sending various requests to the server through these processes, such as model loading requests, data transfer requests, model inference requests, etc. In response to the requests sent by multiple clients, the server runs a shared service process to execute tasks. This shared service process merges the requests sent by multiple clients to generate instructions for controlling the accelerator card. That is, the server provides shared services to the multiple processes running by multiple clients by running the shared service process. Therefore, for the accelerator card, there is no need to repeatedly switch contexts to execute requests initiated by the processes running by each client; instead, it executes the corresponding requests based on the context of the shared service process, effectively reducing resource waste. For the specific architecture of host 101, please refer to the following... Figure 2 The content shown will not be repeated here.

[0060] For any given accelerator card 102, the accelerator card 102 possesses storage and computing capabilities, providing storage space and computing power for the tasks executed by the computing system 100. For example, the accelerator card 102 can be a GPU, NPU, IPU, TPU, DSA chip, etc., and this application is not limited to these. Furthermore, Figure 1 The number of accelerator cards 102 shown is for illustrative purposes only. The number of accelerator cards 102 may be more or less, and this application is not limited thereto.

[0061] The host 101 and accelerator card 102 described above can be integrated into one server or deployed in different servers, and this application does not limit this. Illustratively, taking the integration of host 101 and accelerator card 102 into one server as an example, host 101 and accelerator card 102 are connected via a peripheral component interconnect express (PCIe) link, and data exchange between host 101 and accelerator card 102 occurs through the PCIe link.

[0062] The aforementioned servers can be independent physical servers, server clusters composed of multiple physical servers, distributed file systems, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Taking a server as an example, a server can also be called a cloud platform (short for cloud computing platform), referring to services based on hardware and software resources that provide computing, networking, and storage capabilities. Through the network "cloud," massive amounts of data are processed and analyzed remotely before being returned to the user, featuring large-scale, distributed, virtualized, highly available, scalable, on-demand services, and security characteristics. Cloud platforms can achieve rapid deployment and release of configurable computing resources with relatively low management costs or low interaction complexity between users and service providers.

[0063] In some embodiments, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically a Transmission Control Protocol / Internet Protocol (TCP / IP) network or an RDMA network, such as an RDMA over Converged Ethernet (RoCE) network or an InfiniBand (IB) network, and is not limited thereto. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0064] The architecture of the host 101 in the aforementioned computing system 100 will be described below.

[0065] Figure 2 This is a schematic diagram of a host architecture provided in an embodiment of this application. For example... Figure 2 As shown, host 101 includes application module 1011, shared service module 1012 and execution module 1013.

[0066] Application module 1011 deploys computing applications, such as AI4S-like applications, to provide users with interfaces for intuitive and easy-to-use interaction with computing system 100, meeting various user business needs. Illustratively, application module 1011 supports M clients simultaneously running M processes (e.g., processes 0 to M, where M is a positive integer) to execute tasks, and sending requests to shared service module 1012 via RPC calls. The M processes running on the M clients can communicate with each other based on MPI.

[0067] The shared service module 1012 deploys a multi-process shared service (MPSS) engine to receive various requests sent by the application module 1011, process the received requests, generate instructions for controlling the accelerator card, and send the instructions to the execution module 1013. In this embodiment, the shared service module 1012 supports the server in processing received requests by running a shared service process and returning the processing results to the application module 1011 via RPC calls. Indicatively, the shared service module 1012 includes a shared service interface (MPSS API) and shared service components. The shared service interface can call various functions to provide corresponding functionalities. For example, it can call the `initial` function for initialization, the `malloc` function for dynamic memory allocation to help the program obtain the necessary memory resources at runtime, the `execute` function for executing specific operations or tasks, and the `finalize` function for cleanup or termination operations, such as releasing resources, cleaning up memory, and closing connections. It should be understood that this is merely an illustrative example, and the types of functions called by the shared service interface are not limited to this. Moreover, the specific implementation and usage of the aforementioned functions can be adjusted according to actual needs, and this application does not limit this. The shared service component is used to provide functions such as inter-process communication (e.g., implemented through Unix domain sockets), thread pool creation and management, fast serialization, shared memory, and separation of control data (i.e., control instructions and data are transmitted through different channels to ensure efficiency), and this application does not limit this.

[0068] The execution module 1013 is equipped with an execution interface to provide functions for calling computing resources such as accelerator cards. For example, the execution interface could be the Ascend computing language (AscendCL) interface, or ACLAPI for short, which provides API libraries for Device management, Context management, Stream management, memory management, model loading and execution, operator loading and execution, media data processing, and Graph management, for users to call when developing artificial intelligence applications. For example, the execution interface can call the initialization function, the malloc memory allocation function, the execute function, and the finalize function to provide corresponding functions; this application is not limited to these.

[0069] in addition, Figure 2 The architecture of host 101 shown is for illustrative purposes only. Host 101 may also include other functional modules to achieve more functions, and this application is not limited thereto.

[0070] The hardware structure of host 101 in the above computing system is described below.

[0071] This application provides a computing device that can be configured as the aforementioned host 101, see reference. Figure 3 , Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 3 As shown, the computing device 300 includes a memory 301, a processor 302, a communication interface 303, and a bus 304. The memory 301, processor 302, and communication interface 303 are interconnected via the bus 304.

[0072] Memory 301 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Illustratively, memory 301 is used to store at least one piece of program code. When the program code stored in memory 301 is executed by processor 302, processor 302 performs the steps performed by the host in the following method embodiments.

[0073] The processor 302 may be a network processor (NP), a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or an integrated circuit used to control the execution of the program of the present application. The processor 302 may be a single-core processor or a multi-core processor. The number of processors 302 may be one or more.

[0074] The communication interface 303 uses a transceiver module, such as a transceiver, to enable communication between the computing device 300 and other devices or communication networks. For example, data can be acquired through the communication interface 303.

[0075] The memory 301 and the processor 302 can be set separately or integrated together.

[0076] Bus 304 may include a pathway for transmitting information between various components of computing device 300 (e.g., memory 301, processor 302, communication interface 303).

[0077] Based on the above Figures 1 to 3 The content shown introduces the computing system provided in this application. The following describes the task processing method applied to this system through method embodiments.

[0078] Figure 4 This is a flowchart of a task processing method provided in an embodiment of this application. For example... Figure 4 As shown, the method is applied to a computing system, which includes a host and at least one accelerator card. M clients and servers run on the host. Schematic, the method includes the following steps 401 to 411.

[0079] 401. In response to the task execution request, the host creates M processes to be run by M clients. The M processes are used to process the data involved in the execution of the task in parallel, where M is a positive integer.

[0080] In this embodiment, a computing application runs on the host computer, capable of accessing wired or wireless networks to provide users with HPC and AI services. The computing application is, for example, an AI4S-type application; however, this application is not limited to this. Any application that combines HPC and AI technologies to solve computing problems is applicable to this application. It should be understood that the task is initiated by the computing application. Regarding the type of task, please refer to the foregoing description, which will not be repeated here. M processes are used to process the data involved in the task in parallel, meaning that the data involved in the task is divided and distributed to different processes for execution. Each process calls the same AI model to perform AI inference. In some embodiments, M processes are also referred to as HPC multi-process, MPI multi-process, etc. Furthermore, M clients running M processes means that one client runs one process, and one process corresponds to a portion of the data involved in the task. Alternatively, one client corresponds to a grid computation (or grid, cell, etc.) on the host CPU side. That is, each client is associated with a specific CPU process, clarifying the computing resources of each client, achieving clear division and isolation of tasks, thereby supporting HPC parallel computing and improving computing efficiency. For example, taking the task of simulating the motion of molecules within XX nanoseconds / picoseconds based on the initial state of the molecules as an example, the host responds to the execution request of the task by creating M processes run by M clients, and dividing the relevant data of molecular dynamics simulation into different processes to achieve parallel computing.

[0081] After step 401 above, the host creates M parallel running processes according to the task execution request to realize HPC parallel computing. The following describes the AI ​​inference process of the computing system executing the task using a single iteration step as an example. It should be understood that the entire task can be completed by executing multiple iteration steps in the same way. Of course, the same principle applies to tasks involving one iteration step, which will not be repeated here.

[0082] 402. M clients send M first model loading requests to the server through M processes executing tasks.

[0083] In this embodiment, M first model loading requests are used to request the loading of the same AI model. Illustratively, M clients, through M processes, generate M first model loading requests using RPC and send them to the server. Each of the M first model loading requests carries information obtained after parameter conversion of pointer variables in the M processes. Taking any one client as an example, to address the cross-process pointer passing problem, the client performs parameter conversion on the pointer variables in the process based on whether pointer variables exist in the input and output parameters. Based on the converted information, it generates a first model loading request and sends it to the server, ensuring that the server can identify the same pointer while distinguishing between different pointers. For example, pointer variables such as datatype, rank, and address are mapped to a specified index using a map, achieving automatic conversion of internal parameters, thereby ensuring the execution interface (or inference engine interface, see above) is maintained. Figure 2 The system is compatible with the host architecture shown, is non-intrusive to applications, and is more versatile. Furthermore, this application does not limit the RPC framework used in the computing system; options include Dubbo (an open-source, high-performance RPC framework), Grpc (a high-performance, cross-language RPC framework), Hessian (a lightweight RPC framework), etc., allowing for the selection of a suitable RPC framework based on the application scenario.

[0084] Furthermore, this application does not limit the order in which M clients send M first model loading requests to the server. For example, M clients can send M first model loading requests to the server simultaneously, or they can send M first model loading requests to the server sequentially according to a preset order. The mechanism for sending model loading requests can be configured according to actual business needs.

[0085] 403. The server obtains the M first model loading requests sent by the M processes executing the task.

[0086] 404. The server sends a second model loading request to at least one accelerator card related to the task, instructing at least one accelerator card to load the AI ​​model.

[0087] In this embodiment, the server creates a shared service process to execute tasks. Through this shared service process, a second model loading request is sent to at least one accelerator card related to the task. The shared service process provides shared services to M processes. This means that by running the shared service process, it enables the M processes to perform the functions required by each other. Therefore, for the accelerator cards, there is no need to repeatedly switch contexts across the M processes to load the same AI model for AI inference. Instead, the requests are executed based on the context of the shared service process. This eliminates performance loss caused by multiple context switches and avoids the drawback of linearly increasing accelerator card memory consumption as the number of client processes increases. After creating the shared service process, the server determines at least one accelerator card to execute AI inference and creates a context based on that accelerator card. The context is then sent to the accelerator card to bind the shared service process to the accelerator card, which allocates computing resources to the shared service process. The accelerator card then executes various requests sent by the server based on the context of the shared service process.

[0088] In some embodiments, the second model loading request is one of M first model loading requests. In other words, a model loading request is selected from the M first model loading requests and sent to at least one accelerator card associated with the task. For example, the second model loading request is the first first model loading request received among the M first model loading requests. In response to the first model loading request sent by the first process executing the task, the server creates a shared service process and context, and sends the first model loading request as the second model loading request to the accelerator card. That is, when the server receives the task-related model loading request for the first time, it promptly creates and runs the shared service process and sends the model loading request to the accelerator card without waiting for subsequent model loading requests, thus improving computational efficiency.

[0089] In some embodiments, the server determines a second model loading request based on the information indicated by M first model loading requests. For example, model loading request 1, model loading request 2, and model loading request 3 all indicate loading AI model-1. Based on the information indicated by each identified model loading request, the server determines that these three model loading requests indicate loading the same AI model (i.e., AI model-1), and then selects one request from model loading requests 1-3 and sends it as the second model loading request to the accelerator card, so that the accelerator card loads AI model-1 based on the shared service process. It should be understood that if the second model loading request is the first first model loading request received by the server, when the server receives other first model loading requests subsequently, it determines, based on the information carried in the request, that the AI ​​model requested by the second model loading request is the same, and therefore does not send other first model loading requests to the accelerator card. That is, for multiple first model loading requests that request to load the same AI model, the server sends one second model loading request to the accelerator card, so that the accelerator card loads the AI ​​model once, that is, one copy of the AI ​​model is stored on the accelerator card. In some embodiments, for any first model loading request, the first model loading request carries the storage address of the AI ​​model to be loaded. Based on the first model loading request, the server determines the storage address of the AI ​​model and writes the storage address of the AI ​​model into the context of the shared service process.

[0090] In some embodiments, the server calls the execution interface to send a second model loading request to the accelerator card. Upon receiving the second model loading request, the accelerator card calls the model loading function, determines the storage address of the AI ​​model based on the context of the shared service process, reads the model parameters of the AI ​​model based on the storage address of the AI ​​model, and loads the AI ​​model onto the accelerator card for subsequent AI inference execution.

[0091] After the aforementioned steps 401 to 404, during the execution of the task, the computing system provides shared services to the M processes running on the M clients through the shared service process running on the server. This eliminates the need for the accelerator card to repeatedly switch process contexts to reload the AI ​​model, meaning it does not need to store multiple copies of the AI ​​model. Instead, it executes the model loading request based on the context of the shared service process. In this way, while eliminating the performance loss caused by multiple context switching, it also avoids the drawback of the linear increase in memory consumption of the accelerator card as the number of client processes increases.

[0092] 405. M clients send M data transmission requests to the server through M processes. The M data transmission requests are used to request the transmission of M first inference data that the M processes need to infer through the AI ​​model.

[0093] In this embodiment, the M data transmission requests are used to request the transmission of M first inference data that M processes need to perform inference through the AI ​​model. This means transmitting the M first inference data to at least one accelerator card related to the task. One process corresponds to one data transmission request, and one data transmission request corresponds to one first inference data, which is also the input data of the AI ​​model.

[0094] 406. The server receives M data transmission requests from M processes.

[0095] 407. M clients store the M first inference data corresponding to the M processes in shared memory.

[0096] In this embodiment, the host and the accelerator card exchange data via shared memory. This application does not limit the timing of shared memory allocation; for example, the server may allocate shared memory upon receiving M first model loading requests, or upon creating a shared service process, and so on. In some embodiments, the shared memory includes a data queue, and M clients, through M processes, write M first inference data corresponding to the M processes into the data queue in the shared memory.

[0097] 408. The server will transfer M first inference data from the shared memory to at least one accelerator card related to the task.

[0098] In this embodiment, the server reads data from the shared memory based on M data transmission requests through a shared service process. If the data in the shared memory matches the M first inference data indicated by the M data transmission requests, the data in the shared memory is transferred to at least one accelerator card related to the task.

[0099] 409. M clients send M first inference requests to the server through M processes.

[0100] In this embodiment of the application, M first inference requests are used to request AI inference on M first inference data based on the AI ​​model.

[0101] It should be noted that this application does not limit the order in which M clients send M data transmission requests and M first inference requests. In some embodiments, M clients, through M processes, synchronously send M data transmission requests and M first inference requests to the server, and write the M first inference data indicated by the M data transmission requests into shared memory. This synchronous sending method saves communication resources. In other embodiments, M clients, through M processes, send M data transmission requests to the server, write the M first inference data indicated by the M data transmission requests into shared memory, and then send M first inference requests to the server. This application does not limit this method. For any given first inference request, the first inference request indicates that AI inference be performed based on the corresponding first inference data. It should be understood that each of the M clients sends a data transmission request and a first inference request to the server. For any given client, the server identifies the request sent by that client based on the rank of the process running that client. In addition, based on the same processing method as the aforementioned first model loading request, the data transmission request and the first inference request similarly carry information obtained after parameter conversion of the pointer variables in the process running on the client, which will not be elaborated here.

[0102] 410. The server sends M first inference requests to at least one accelerator card related to the task.

[0103] In some embodiments, the server processes the received data transmission requests and first inference requests sequentially according to a first-in, first-out (FIFO) principle. For example, if the server receives data transmission requests in the order of data transmission request 0 and data transmission request 1, the server will first transmit the inference data indicated by data transmission request 0 to the accelerator card, and then transmit the inference data indicated by data transmission request 1 to the accelerator card. Similarly, if the server receives the first inference requests in the order of inference request 0 and inference request 1, the server will first send inference request 0 to the accelerator card, and then send inference request 1 to the accelerator card. Of course, the server can also use other methods to process the received requests, and this application does not limit this.

[0104] After steps 405 to 410, the accelerator card receives M first inference data and M first inference requests, performs M AI inferences based on the loaded AI model, and returns the execution results obtained from the M AI inferences to the server.

[0105] In some embodiments, the server updates the execution status of each request in shared memory through a shared service process. Illustratively, updating the execution status of each request in shared memory through a shared service process includes at least one of the following: updating the execution status of each data transmission request in shared memory through a shared service process; and updating the execution status of each first inference request in shared memory through a shared service process. The execution status may be, for example, pending execution, executing, execution error, etc., and this application does not limit this. In this way, the server can promptly obtain the execution status of each request, and if the execution status indicates an execution error, it can promptly report the error or re-execute, ensuring that each request received by the server is executed successfully.

[0106] 411. The server writes the execution results of at least one accelerator card executing M first inference requests into shared memory, and notifies M processes to retrieve the execution results from shared memory.

[0107] In this embodiment, the server writes the execution results of the M first inference requests executed by the accelerator card into shared memory through a shared service process, and sends notification messages to the M processes so that the M processes can obtain the execution results from the shared memory based on the notification messages.

[0108] Furthermore, in steps 408 to 411 above, the example described is that the server sequentially processes the received data transmission requests and first inference requests. In some embodiments, the server can merge the received M data transmission requests and M first inference requests to improve the utilization of the accelerator card. Schematic, steps 408 to 411 above can be replaced by the following steps A01 to A04:

[0109] Step A01: M clients send M first inference requests to the server through M processes.

[0110] Step A02: The server merges the M first inference data corresponding to the M data transmission requests into N second inference data, and merges the M first inference requests into N second inference requests. Each second inference request corresponds to a second inference data.

[0111] The second inference data includes data corresponding to at least one first inference request, where N is a positive integer. Based on step 407, M clients, through M processes, store the M first inference data corresponding to the M processes in shared memory. In this step, the server, through a shared service process, detects the data already stored in the shared memory based on the M data transmission requests. Based on the batch processing data volume of the accelerator card, it merges the M first inference data indicated by the M data transmission requests in the shared memory to obtain N second inference data. Based on at least one first inference data included in each second inference data, N second inference requests are generated, that is, the M first inference requests are merged into N second inference requests.

[0112] Schematic illustration: The data size of each second inference data is less than or equal to the batch data size (also known as batchsize). The batch data size of the accelerator card can be adaptively set according to the operating performance of the accelerator card. This application does not limit the specific value of the batch data size. Furthermore, a second inference data can also be understood as a batch data packet, a batch data group, a data set, etc., and this application does not limit this. Schematic illustration: The server creates a first thread through a shared service process. Based on the received M data transmission requests, this first thread detects the data already written in the shared memory. Based on the size relationship between the M first inference data sizes indicated by the M data transmission requests and the batch data size, the M first inference data are merged into N second inference data. The first thread can also be understood as a thread used to implement heterogeneous data collaboration.

[0113] Taking X processes out of M processes as an example (X is a positive integer), this step includes: merging the X first inference data corresponding to the X processes out of M processes into a third inference data among N second inference data. X processes can be any X processes out of M processes, and the third inference data is any one of the N second inference data. After receiving the first inference requests from the X processes, based on the first inference requests from the X processes, a third inference request is generated among the N second inference requests. The third inference request is used to request the accelerator card to perform inference on the third inference data through the AI ​​model. For example, if M is 6, N is 2, and X is 3, the first inference data corresponding to the first 3 of the 6 first inference requests are merged into one second inference data, and a second inference request is generated based on these 3 first inference requests. The first inference data corresponding to the last 3 of the 6 first inference requests are merged into another second inference data, and another second inference request is generated based on these 3 first inference requests.

[0114] The following example illustrates the merging process using any two inference requests from M first inference requests (hereinafter referred to as the fourth and fifth inference requests) and the merged second inference data as a single data set. Illustratively, the server determines the data volume of the first data corresponding to the fourth inference request and the data volume of the second data corresponding to the fifth inference request. Based on the relationship between the data volumes of the first and second data and the batch data volume, the server merges the first and second data. Specifically, if the total data volume of the first and second data is less than or equal to the batch data volume, the first and second data are merged into one data set, resulting in one set of second inference data. For example, if the batch data volume is 10KB, the first data volume is 3KB, the second data volume is 5KB, and the total data volume of the first and second data is less than the batch data volume, then the first and second data are merged into the same data set. Conversely, if the first data volume is greater than the batch data volume, the first data is split into multiple data sets. For example, if the batch data processing size is 10KB and the first data size is 12KB, then the 10KB data in the first data is treated as one data set, and the remaining 2KB data is merged into other data sets. It should be understood that this explanation uses the first data as an example; the same applies to the second data. In other embodiments, if the first data size is greater than the remaining capacity of the target data set, based on the remaining capacity of the target data set, a portion of the first data is merged into the target data set, and the data in the first data that is not merged into the target data set is merged into other data sets besides the target data set. Here, the target data set refers to any data set with remaining capacity, meaning that the target data set already contains data corresponding to other inference requests. For example, if the first data size is 8KB and the remaining capacity of the target data set is 2KB, then the 2KB data in the first data is merged into the target data set, and the remaining 6KB data is merged into other data sets besides the target data set. It should be noted that the above is only an example. In some embodiments, the server can split or combine the M first inference data corresponding to the M first inference requests according to the FIFO principle based on a greedy strategy to form N data sets (i.e. N second inference data), so that the amount of data in each data set can adapt to the throughput of the accelerator card. This application does not limit the specific strategy for data merging.

[0115] In some embodiments, for any second inference data, the server stores the metadata of the second inference data in shared memory. The metadata includes a shared identifier, process number rank, data length, etc. This application does not limit this. By setting a shared identifier for each second inference data, it is possible to indicate which processes' data correspond to the same second inference data, providing technical support for subsequently reading the execution results corresponding to each process.

[0116] Step A03: The server sends N second inference requests to at least one accelerator card related to the task, instructing at least one accelerator card to execute the N second inference requests through the AI ​​model.

[0117] In this embodiment, the server transmits N second inference requests to the accelerator card via a shared service process, instructing the accelerator card to execute the N second inference requests through the AI ​​model. In some embodiments, the server uses a first thread to detect the data status of data already stored in shared memory. If it detects second inference data that meets certain conditions, it transmits the second inference data and the corresponding second inference request to the accelerator card. These conditions may include, for example, the amount of data in the second inference data reaching the batch processing capacity of the accelerator card, or the end of the time window for the second inference data, i.e., a relatively long waiting time for the second inference data. This application does not limit these conditions.

[0118] Schematic illustration: The server creates a second thread through a shared service process. If the first thread determines that the second inference data meets the conditions, it transmits the second inference data to the accelerator card. The first thread then notifies the second thread, which, based on the received notification message, sends the second inference request corresponding to the second inference data to the accelerator card. The second thread can also be understood as a thread used to execute the model.

[0119] Based on steps A01 and A03 above, the server adaptively processes the data required for AI inference based on the batch processing data volume of the accelerator card, thereby achieving the synergy between the optimal partitioning of multi-process data and the optimal throughput batch processing volume of the acceleration device, and thus improving the utilization rate of the host CPU and the accelerator card.

[0120] Step A04: The server writes the execution results of at least one accelerator card executing N second inference requests into shared memory, and notifies M processes to retrieve the execution results from shared memory.

[0121] In this process, after executing N second inference requests, the accelerator card returns the execution results to the server. The server writes the execution results to shared memory, divides the results according to the first inference requests corresponding to each second inference request, and returns the execution result corresponding to each first inference request. For example, taking the third inference request in step A02 above, if the third inference request executes successfully, the execution result of the third inference request is stored in shared memory, divided according to the first inference requests of X processes, and the execution result corresponding to each first inference request is returned.

[0122] It should be noted that the task processing methods shown in steps 401 to 411, steps 401 to 407, and steps A01 to A04 are described using the example of the server creating a shared service process to provide shared services to the M processes running on the M clients. That is, the M processes correspond to one shared service process and one context.

[0123] Schematic, in the task processing method shown in steps 401 to 411, the server receives M first model loading requests sent by M processes, creates a shared service process, allocates shared memory space for the shared service process, determines the accelerator card for performing AI inference, creates a context based on the accelerator card, sends the second model loading request from the M first model loading requests to the accelerator card, then receives M data transmission requests sent by M processes, processes the M data transmission requests sequentially according to the FIFO principle, and sequentially sends the M first inference data written by the M clients in the shared memory to the accelerator card, further receiving M first inference requests sent by M processes, and sequentially sending the M first inference requests to the accelerator card according to the FIFO principle, and after receiving the execution result returned by the accelerator card, writes the execution result to the shared memory, and sends a notification message to the M processes to return the execution result of each first inference request to the M processes.

[0124] Schematic, in the task processing method shown in steps 401 to 407 and steps A01 to A04, the server receives M first model loading requests sent by M processes, creates a shared service process, allocates shared memory space for the shared service process, determines the accelerator card for performing AI inference, creates a context based on the accelerator card, sends the second model loading request from the M first model loading requests to the accelerator card, then receives M data transmission requests and M first inference requests sent by M processes, detects the M first inference data written by M clients in the shared memory, merges the M first inference data into N second inference data based on the batch processing data volume of the accelerator card, generates a second inference request corresponding to any second inference data when any second inference data is ready, and sends the second inference data and the corresponding second inference request to the accelerator card. Further, after receiving the execution result of any second inference request returned by the accelerator card, the execution result is written to the shared memory, divided according to the first inference requests of X processes corresponding to the second inference request, and a notification message is sent to X processes to return the execution result of the X first inference requests to the X processes.

[0125] In other embodiments, the server can create N shared service processes to provide shared services for the M processes running by M clients. Illustratively, before receiving M data transmission requests, the server creates N shared service processes and allocates shared memory space for each of them; that is, one shared service process corresponds to one shared memory space. This application does not limit the timing of the creation of the N shared service processes. For example, the server creates N shared service processes and allocates shared memory space for each of them upon receiving M first model loading requests. By creating N shared service processes, the M first inference requests corresponding to the M processes are merged into N second inference requests, the M first inference data are merged into N second inference data, the N second inference data are stored in the shared memory spaces corresponding to the N shared service processes, and the N second inference requests are sent to the accelerator card through the N shared service processes, enabling the accelerator card to perform inference on the second inference data corresponding to the N shared service processes using the AI ​​model. Based on this, steps 408 to 411 are replaced by steps B01 to B04 as follows:

[0126] Step B01: M clients send M first inference requests to the server through M processes.

[0127] Step B02: The server stores the M first inference data corresponding to the M data transmission requests into the shared memory space corresponding to the N shared service processes, and merges the M first inference requests into N second inference requests, with each second inference request corresponding to second inference data.

[0128] The second inference data includes data corresponding to at least one first inference request, where N is a positive integer. Based on step 407, M clients, through M processes, write M sets of first inference data corresponding to the M processes into shared memory. In this step, the server stores the M sets of first inference data corresponding to the M processes into shared memory, including: storing X sets of first inference data corresponding to X processes out of the M processes into the shared memory space corresponding to the first shared service process out of the N shared service processes. X processes are any X processes out of the M processes, and the first shared service process is any one of the N shared service processes. Correspondingly, the M first inference requests are merged into N second inference requests, including: after receiving the first inference requests from X processes, determining the first accelerator card through the first shared service process, creating a context based on the first accelerator card, and generating a third inference request among the N second inference requests. The third inference request is used to request the first accelerator card to infer the second inference data corresponding to the first shared service process using an AI model. Here, the first accelerator card refers to any accelerator card.

[0129] It should be noted that the process of merging the M first inference data and the M first inference requests described above is the same as step A02 mentioned above, so it will not be repeated here.

[0130] Step B03: The server sends N second inference requests to at least one accelerator card related to the task, instructing at least one accelerator card to execute the N second inference requests through the AI ​​model.

[0131] Step B04: The server writes the execution results of at least one accelerator card executing N second inference requests into shared memory, and notifies M processes to retrieve the execution results from shared memory.

[0132] In this process, after executing N second inference requests, the accelerator card returns N execution results to the server. The server writes the N execution results into the shared memory space corresponding to N shared service processes. Then, it divides each execution result according to the first inference request corresponding to each second inference request and returns the execution result corresponding to each first inference request. For example, taking the third inference request in step B02 above, if the third inference request executes successfully, the execution result of the third inference request is stored in the shared memory space corresponding to the first shared service process. The execution result is then divided according to the first inference requests of X processes, and the execution result corresponding to each first inference request is returned.

[0133] Schematic, in the task processing method shown in steps 401 to 407 and steps B01 to B04, the server receives M first model loading requests sent by M processes, creates N shared service processes, allocates shared memory space for the N shared service processes, determines the accelerator card for performing AI inference, creates N contexts based on the accelerator card, sends the second model loading requests from the M first model loading requests to the accelerator card, then receives M data transmission requests and M first inference requests sent by M processes, merges the M first inference data into N second inference data, and then... The inference data is stored in the shared memory space corresponding to N shared service processes, and N second inference requests are generated and sent to the accelerator card. The accelerator card uses the AI ​​model to infer the second inference data corresponding to the N shared service processes. Furthermore, after receiving the execution results of the N second inference requests returned by the accelerator card, the N execution results are written into the shared memory space corresponding to the N shared service processes. The data is divided according to the first inference requests of the X processes corresponding to each second inference request, and notification messages are sent to the X processes to return the execution results of the X first inference requests to the X processes.

[0134] The following is for reference. Figure 5 Taking the example of the server creating a shared service process and merging M first inference requests into N second inference requests, this example illustrates the process of data transmission and model inference performed by the server. Figure 5 This is a schematic diagram of a process for performing data transmission and model inference provided in an embodiment of this application. Indicatively, this process includes the following steps 1 to 6.

[0135] Step 1: M clients, through their respective running processes (e.g., rank 0, rank 1... rank M, where M is a positive integer), write their first inference data into the data queue in shared memory, or in other words, each process pushes its first inference data to the batching queue in shared memory.

[0136] Step 2: The server runs the first thread (i.e., the heterogeneous data collaboration thread) created based on the shared service process to detect the arrival status of data in the data queue in real time.

[0137] Step 3: The server uses the first thread to combine or split the data in the data queue according to the FIFO principle and a greedy strategy to form at least one data set (or batching data packet). Each data set corresponds to a second inference data set, and the server stores the metadata of each second inference data set. The metadata includes the batch ID, process number rank, data length, etc.

[0138] Taking the three processes of rank 0, rank 1, and rank 2 as examples, if the total amount of data corresponding to rank 0, rank 1, and rank 2 is less than or equal to the batch size, the data corresponding to rank 0, rank 1, and rank 2 will be merged into one dataset, and this dataset will be used as the second inference data; if the amount of data corresponding to rank 0 is greater than the batch size or the remaining capacity of the target dataset, the data corresponding to rank 0 will be split into multiple datasets.

[0139] Step 4: For any second inference data, if the time window of the second inference data ends or the data volume reaches the batch processing data volume, the server sends a notification message to the second thread (i.e., the model execution thread) through the first thread. The notification message indicates that the second inference data can be used for AI inference.

[0140] Step 5: The server receives the notification message through a second thread. This second thread then generates a merged second inference request based on the notification message and the second inference data, and sends it to the accelerator card to instruct it to perform AI inference. It should be understood that since the accelerator card performs batch AI inference using the dataset as batch processing data, the inference results obtained by the accelerator card are also batched.

[0141] Step 6: The server writes the execution results obtained by the AI ​​inference performed by the accelerator card into the shared memory. Based on the metadata maintained in Step 3, the execution results corresponding to each process are restored from the batch processing results and returned to the corresponding process.

[0142] It should be understood that, taking computational tasks initiated by AI4S applications as an example, these tasks typically employ a grid partitioning approach. Different grids often involve largely independent computations. In related technologies, grid partitioning is often based on manual experience or CPU load analysis using MPI. However, the optimal partitioning of CPU multi-process data is often not the optimal batch processing data volume for the accelerator card. Using the method provided in this application, when the multi-process data is less than the batch processing data volume, the multi-process data can be merged into a single data set and pushed to the accelerator card for execution. When the multi-process data is greater than the batch processing data volume or the current remaining batch space, the multi-process data can be split into multiple data sets. This process can be adaptively handled, achieving synergy between optimal multi-process data partitioning and optimal accelerator card throughput batch processing volume, thereby improving the utilization of both the host CPU and the accelerator card.

[0143] The following is for reference. Figures 6 to 8 The task processing methods shown in steps 401 to 411 above are summarized. Figure 6 This is a schematic diagram of a task processing method provided in an embodiment of this application. For example... Figure 6 As shown, taking the creation of a shared service process on the server side as an example, this method includes the following stages.

[0144] Phase 1: The server responds to the first model loading request sent by the client, initializes the shared service process for the computing task, and creates the context of the shared service process.

[0145] Phase 2: The M processes running on the M clients send model loading requests to the shared service process. Taking processes rank 1, rank 2, and rank 3 as an example, the shared service process sends the model loading request corresponding to rank 1 from the model loading requests of rank 1, rank 2, and rank 3 to the accelerator card, so that the accelerator card loads a single-instance AI model, that is, loads the AI ​​model once. This process can also be understood as a merging process of the M model loading requests.

[0146] Phase 3: The shared service process allocates shared memory between the M processes running by the M clients and the shared service process. The shared memory is used for data exchange between the two parties.

[0147] Phase 4: M processes send data transfer requests and inference requests to the shared service process. The shared service process receives the data transfer requests and inference requests, completes the inference for the current iteration step according to the FIFO principle, and updates the execution status of each request in shared memory. Here, 4-1 represents M processes sending data transfer requests to the shared service process, 4-2 represents M processes sending inference requests to the shared service process, and 4-3 represents the shared service process controlling the accelerator card to execute AI inference.

[0148] Phase 5: The shared service process obtains the execution results and writes them into shared memory (based on the data address corresponding to each inference request) according to the FIFO principle, and updates the execution status of each request in shared memory. Here, 5-1 indicates that the shared service process sends the AI ​​execution results to M processes.

[0149] Next, refer to Figure 7 , Figure 7 This is a schematic diagram of another task processing method provided in an embodiment of this application. For example... Figure 7 As shown, taking a task involving L iterations (L is a positive integer) and M clients running on the host as an example, combined with the aforementioned... Figure 2As the host architecture described above shows, for shared service interfaces (such as the MPSS API), the interface can be divided into single-execution interfaces and batch processing interfaces. Mutexes and flags are used in conjunction to preempt and allocate resources, achieving thread synchronization and ensuring the effect of executing globally once or in batches. Illustratively, since the M processes running on M clients can share the same AI model through the shared service process, during the initialization phase of loading the AI ​​model, the shared service interface only needs to call the execution interface (such as the ACL API) once, i.e., through a single-execution interface. Similarly, during the final phase of destroying the AI ​​model, the shared service interface also only needs to call the execution interface once, i.e., through a single-execution interface. During AI inference, taking the example that the data from the M processes can be merged into a single data set, then within one iteration step, the shared service interface only needs to call the execution interface once. Correspondingly, within L iteration steps, the shared service interface needs to call the execution interface L times, i.e., through batch processing interfaces. Furthermore, in the diagram, host-to-device (H2D) memory copy refers to transferring (or moving) data from the host to the accelerator card device, while device-to-host (D2H) memory copy refers to transferring data from the accelerator card device to the host. It is evident that through the above call mapping, many-to-one sharing is achieved. The initialization and termination phases are executed globally only once, meaning the interface for reusing results throughout the inference lifecycle is reused. For the execution interface that actually performs the computation, the L iteration steps based on M clients can be mapped according to the system configuration to achieve multiple inference deployments. Moreover, the above shared service interface is compatible with the execution interface. In contrast, related technologies, during the initialization phase of loading the AI ​​model, require M calls to the execution interface due to the involvement of M processes. Similarly, the AI ​​model destruction phase also requires M calls to the execution interface. Additionally, during the AI ​​inference process, due to the involvement of L iteration steps and M processes, L×M calls to the execution interface are required, resulting in significant resource waste.

[0150] Further reference Figure 8 The above method will be illustrated with examples. Figure 8 This is a schematic diagram of another task processing method provided in the embodiments of this application, such as... Figure 8As shown, for the accelerator card, based on the MPSS provided in this application embodiment, the accelerator card executes the request of process A based on AI model-1 in time slice 1, and executes the merge request of process B and process C based on AI model-1 in time slice 2 (the merge request here is also any of the aforementioned second inference requests). It can be seen that the above method can effectively improve the utilization rate of the accelerator card. In addition, while MPSS realizes AI4S service sharing, it can also share the accelerator card with other tasks at the same time. That is, while encapsulating a set of efficient inference methods on the server side, it does not affect the upper-layer user calls, allowing users to maintain their original programming habits and logical expressions. The relevant configuration of MPSS can be adaptively optimized according to existing resources to maximize resource utilization and achieve optimal performance. Moreover, the MPSS API can be fully mapped to the execution interface and is a non-exclusive service, which can execute multiple tasks at the same time. The accelerator card resides in a single memory and maps multiple processes to the outside, realizing service sharing of compatible interfaces. In other words, during different iteration steps, the accelerator card can switch to the context of other tasks and share resources with other types of tasks. For example, task A includes iteration step 1 and iteration step 2, and task B includes iteration step 1 and iteration step 2. After the accelerator card executes iteration step 1 of task A, it can switch to execute iteration step 1 of task B, and then switch to execute iteration step 2 of task A. In this process, since the accelerator card executes task A based on the shared service process, it does not need to store multiple copies of the AI ​​model, thus avoiding the accelerator card repeatedly loading the same AI model and reducing resource waste.

[0151] In summary, this application provides a task processing method. For tasks executed by multiple processes, when multiple model loading requests are received from multiple processes requesting the loading of the same AI model, a model loading request is sent to the task-related accelerator card to instruct the accelerator card to load the AI ​​model. The above method avoids the accelerator card repeatedly loading the same AI model, realizes the sharing of one AI model by multiple processes, reduces resource waste, and improves computational efficiency.

[0152] This application also provides a task processing device that can be configured on the aforementioned host, such as... Figure 9 As shown, Figure 9 This is a schematic diagram of a task processing device provided in an embodiment of this application. The device includes an acquisition module 901 and a sending module 902.

[0153] The acquisition module 901 is used to acquire M first model loading requests sent by M processes executing the task, wherein the M first model loading requests are used to request the loading of the same artificial intelligence AI model, and M is a positive integer;

[0154] The sending module 902 is used to send a second model loading request to at least one accelerator card associated with the task, instructing at least one accelerator card to load the AI ​​model.

[0155] In some embodiments, the second model loading request is one of the M first model loading requests.

[0156] In some embodiments, the device further includes:

[0157] The receiving module is used to receive M data transmission requests sent by M processes. The M data transmission requests are used to request the transmission of M first inference data that the M processes need to use for inference through the AI ​​model.

[0158] The storage module is used to store the M first inference data corresponding to the M processes in shared memory.

[0159] In some embodiments, the device further includes:

[0160] The merging module is used to receive M first inference requests sent by M processes, merge the M first inference requests into N second inference requests, each second inference request corresponds to second inference data, and the second inference data includes at least one data corresponding to a first inference request, where N is a positive integer;

[0161] The sending module 902 is also configured to send N second inference requests to at least one accelerator card to instruct the at least one accelerator card to execute the N second inference requests through the AI ​​model.

[0162] In some embodiments, the device further includes:

[0163] Create a module to create N shared service processes and allocate shared memory space for the N shared service processes before receiving M data transmission requests;

[0164] The storage module is used to store X first inference data corresponding to X processes out of M processes into the shared memory space corresponding to the first shared service process out of N shared service processes, where X processes are any X processes out of M processes, and the first shared service process is any one of the N shared service processes, and X is a positive integer.

[0165] The merging module is used to: after receiving the first inference requests from X processes, determine the first accelerator card through the first shared service process, and create a context based on the first accelerator card to generate a third inference request from among N second inference requests. The third inference request is used to request the first accelerator card to infer the second inference data corresponding to the first shared service process through the AI ​​model.

[0166] In some embodiments, if the third inference request is executed successfully, the execution result of the third inference request is stored in the shared memory space corresponding to the first shared service process, the execution result is divided according to the first inference requests of X processes, and the execution result corresponding to each first inference request is returned.

[0167] In the aforementioned device, for tasks executed by multiple processes, when multiple model loading requests are received from multiple processes requesting the loading of the same AI model, a model loading request is sent to the task-related accelerator card to instruct the accelerator card to load the AI ​​model. This method avoids the accelerator card repeatedly loading the same AI model, realizes the sharing of one AI model by multiple processes, reduces resource waste, and improves computing efficiency.

[0168] It should be noted that the division of modules in the above-described task processing device is merely illustrative. In practical applications, the above functions can be assigned to different modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the task processing device and task processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0169] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the various examples described, a first client can be referred to as a second client, and similarly, a second client can be referred to as a first client. Both a first client and a second client can be clients, and in some cases, they can be separate and different clients.

[0170] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple clients means two or more clients.

[0171] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0172] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of program structure information. This program structure information includes one or more program instructions. When these program instructions are loaded and executed on a computing device, the processes or functions according to the embodiments of this application are generated, in whole or in part.

[0173] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0174] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A task processing method, characterized in that, The method includes: Obtain M first model loading requests sent by M processes executing the task, wherein the M first model loading requests are used to request the loading of the same artificial intelligence AI model, and M is a positive integer; A second model loading request is sent to at least one accelerator card associated with the task, instructing the at least one accelerator card to load the AI ​​model.

2. The method according to claim 1, characterized in that, The second model loading request is one of the M first model loading requests.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Receive M data transmission requests sent by the M processes, wherein the M data transmission requests are used to request the transmission of M first inference data that the M processes need to infer through the AI ​​model; The M first inference data corresponding to the M processes are stored in shared memory.

4. The method according to claim 3, characterized in that, The method further includes: Receive M first inference requests sent by the M processes, merge the M first inference requests into N second inference requests, each second inference request corresponds to second inference data, the second inference data includes at least one data corresponding to a first inference request, and N is a positive integer; The N second inference requests are sent to the at least one accelerator card to instruct the at least one accelerator card to execute the N second inference requests through the AI ​​model.

5. The method according to claim 4, characterized in that, The method further includes: Before receiving the M data transmission requests, N shared service processes are created, and shared memory space is allocated to the N shared service processes; The step of storing the M first inference data corresponding to the M processes into shared memory includes: X first inference data corresponding to X processes among the M processes are stored in the shared memory space corresponding to the first shared service process among the N shared service processes. The X processes are any X processes among the M processes, and the first shared service process is any one of the N shared service processes. X is a positive integer. The step of merging the M first inference requests into N second inference requests includes: After receiving the first inference request from the X processes, the first accelerator card is determined through the first shared service process, and a context is created based on the first accelerator card to generate a third inference request among the N second inference requests. The third inference request is used to request the first accelerator card to infer the second inference data corresponding to the first shared service process through the AI ​​model.

6. The method according to claim 5, characterized in that, If the third inference request is executed successfully, the execution result of the third inference request is stored in the shared memory space corresponding to the first shared service process. The execution result is divided according to the first inference requests corresponding to the X processes, and the execution result corresponding to each first inference request is returned.

7. A task processing device, characterized in that, The device includes: The acquisition module is used to acquire M first model loading requests sent by M processes executing the task, wherein the M first model loading requests are used to request the loading of the same artificial intelligence AI model, and M is a positive integer; A sending module is used to send a second model loading request to at least one accelerator card associated with the task, instructing the at least one accelerator card to load the AI ​​model.

8. The apparatus according to claim 7, characterized in that, The second model loading request is one of the M first model loading requests.

9. The apparatus according to claim 7 or 8, characterized in that, The device further includes: The receiving module is used to receive M data transmission requests sent by the M processes, wherein the M data transmission requests are used to request the transmission of M first inference data that the M processes need to infer through the AI ​​model; The storage module is used to store the M first inference data corresponding to the M processes into shared memory.

10. The apparatus according to claim 9, characterized in that, The device further includes: The merging module is used to receive M first inference requests sent by the M processes, merge the M first inference requests into N second inference requests, each second inference request corresponding to second inference data, the second inference data including at least one data corresponding to a first inference request, and N being a positive integer; The sending module is further configured to send the N second inference requests to the at least one accelerator card, so as to instruct the at least one accelerator card to execute the N second inference requests through the AI ​​model.

11. The apparatus according to claim 10, characterized in that, The device further includes: A creation module is used to create N shared service processes and allocate shared memory space for the N shared service processes before receiving the M data transmission requests; The storage module is used for: X first inference data corresponding to X processes among the M processes are stored in the shared memory space corresponding to the first shared service process among the N shared service processes. The X processes are any X processes among the M processes, and the first shared service process is any one of the N shared service processes. X is a positive integer. The merging module is used for: After receiving the first inference request from the X processes, the first accelerator card is determined through the first shared service process, and a context is created based on the first accelerator card to generate a third inference request among the N second inference requests. The third inference request is used to request the first accelerator card to infer the second inference data corresponding to the first shared service process through the AI ​​model.

12. The apparatus according to claim 11, characterized in that, If the third inference request is executed successfully, the execution result of the third inference request is stored in the shared memory space corresponding to the first shared service process. The execution result is divided according to the first inference requests of the X processes, and the execution result corresponding to each first inference request is returned.

13. A computing system, characterized in that, The computing system includes a host and at least one accelerator card, the host being used to implement the task processing method as described in any one of claims 1 to 6.

14. A computing device, characterized in that, The computing device includes a processor and a memory, the processor being configured to execute at least one piece of program code stored in the memory to enable the computing device to implement the task processing method as described in any one of claims 1 to 6.

15. A computer program product, characterized in that, When the computer program product is run on a computing device, the computing device performs the task processing method as described in any one of claims 1 to 6.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store at least one piece of program code, which is used to implement the task processing method as described in any one of claims 1 to 6.

Images