Model reasoning service instance starting method and device, equipment and medium

Abstract

The invention provides a model reasoning service instance starting method and device, equipment and a medium, and relates to the technical field of artificial intelligence and cloud computing, in particular to a reasoning service expansion technology. According to the implementation scheme, a first processing unit used for running a to-be-started first service instance is determined, and the first service instance is used for executing reasoning of a target model; a data transmission channel between the first processing unit and a second processing unit which already runs a second service instance used for reasoning the target model is established, and loading information corresponding to the target model is stored in a storage space corresponding to the second processing unit; the loading information comprises at least one of the weight of the target model, the model structure configuration information of the target model or the compiling cache information of the target model; reading loading information corresponding to the target model from a storage space corresponding to the second processing unit based on the data transmission channel; and performing initialization based on the loading information corresponding to the target model to start the first service instance.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and cloud computing technologies, and in particular to inference service expansion technology, specifically to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for launching a model inference service instance. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Cloud computing refers to a technology system that allows access to a flexible and scalable pool of shared physical or virtual resources via a network. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology provides efficient and powerful data processing capabilities for AI and other technological applications and model inference.

[0003] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have been widely used in natural language processing, content generation, and intelligent question answering. To support the operation of large-scale models, it is typically necessary to build large-scale model inference clusters, such as cloud computing clusters, utilizing containerization technology and heterogeneous computing resources to deploy and manage model inference services. Within these inference clusters, to cope with fluctuations in business traffic or to achieve fault tolerance, the system often needs to dynamically scale up or restart inference service instances.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for initiating a model inference service instance.

[0006] According to one aspect of this disclosure, a method for starting a model inference service instance is provided, comprising: determining a first processing unit for running a first service instance to be started, wherein the first service instance is used to perform inference of a target model; establishing a data transmission channel between the first processing unit and a second processing unit that has been running a second service instance for inference of the target model, wherein a corresponding storage space of the second processing unit stores loading information corresponding to the target model, the loading information including at least one of the target model's weights, the target model's model structure configuration information, or the target model's compilation cache information; reading the target model's loading information from the corresponding storage space of the second processing unit based on the data transmission channel; and initializing based on the target model's loading information to start the first service instance.

[0007] According to one aspect of this disclosure, a device for launching a model inference service instance is provided, comprising: a determining unit configured to determine a first processing unit for running a first service instance to be launched, wherein the first service instance is used to perform inference of a target model; an establishing unit configured to establish a data transmission channel between the first processing unit and a second processing unit that has already run a second service instance for inference of the target model, wherein a corresponding storage space of the second processing unit stores loading information corresponding to the target model, the loading information including at least one of the target model's weights, the target model's model structure configuration information, or the target model's compilation cache information; a reading unit configured to read the target model's loading information from the corresponding storage space of the second processing unit based on the data transmission channel; and a launching unit configured to initialize based on the target model's loading information to launch the first service instance.

[0008] According to one aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for launching a model inference service instance.

[0009] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to execute the above-described method for initiating a model inference service instance.

[0010] According to one aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program, when executed by a processor, is capable of implementing the above-described method for launching a model inference service instance.

[0011] According to one or more embodiments of this disclosure, the startup latency of model inference service instances can be reduced, hardware resources can be saved, and efficiency can be improved.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 A schematic diagram of an exemplary system in which various methods described herein may be implemented, according to exemplary embodiments of the present disclosure; Figure 2 A flowchart illustrating a method for launching a model inference service instance according to an exemplary embodiment of the present disclosure is shown; Figure 3 A schematic diagram illustrating the startup process of a model inference service instance according to an exemplary embodiment of the present disclosure is shown; Figure 4 A schematic diagram illustrating the startup process of a model inference service instance according to an exemplary embodiment of the present disclosure is shown; Figure 5 A schematic diagram illustrating the configuration process of a model inference service cluster according to an exemplary embodiment of the present disclosure is shown; Figure 6 A structural block diagram of a launch device for a model inference service instance according to an exemplary embodiment of the present disclosure is shown; Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0015] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0017] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0018] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0019] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0020] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of a startup method for a model inference service instance.

[0021] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0022] exist Figure 1In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0023] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to send model inference requests or scaling instructions for the model inference service. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to users through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0024] Client devices 101, 102, 103, 104, 105, and / or 106 may include various categories of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various categories and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0025] Network 110 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0026] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0027] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0028] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0029] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0030] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different categories. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0031] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be different categories of databases, such as key-value stores, object stores, or regular stores supported by a file system.

[0032] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0033] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have been widely used in natural language processing, content generation, and intelligent question answering. To support the operation of large-scale models, it is typically necessary to build large-scale model inference clusters, utilizing containerization technologies (such as Kubernetes) and heterogeneous computing resources (such as GPU clusters) to deploy and manage model inference services. Within these inference clusters, to cope with fluctuations in business traffic or to achieve fault tolerance, the system often needs to dynamically scale up or restart inference service instances.

[0034] Existing inference cluster technologies typically focus on optimizing model switching for the entire inference cluster, but lack optimization techniques for rapid scaling, restarting, and recovery of individual inference service instances. Therefore, current model inference service instances generally suffer from slow model loading speeds, especially in scenarios involving scaling up or disaster recovery of model inference services, where new service startup times are lengthy, thus impacting the availability of inference services.

[0035] Based on this, this disclosure provides a method for starting a model inference service instance. When it is necessary to start a new inference service instance corresponding to the target model, a data transmission channel is established between the first processing unit corresponding to the new inference service instance and the second processing unit corresponding to the existing inference service instance of the target model. The model loading information loaded by the second processing unit is reused to the first processing unit, so that the first processing unit can start the new inference service instance of the target model more efficiently, shorten the startup time of the model inference service instance, and improve the expansion efficiency of the model inference service.

[0036] Figure 2 A flowchart of a method 200 for launching a model inference service instance according to an exemplary embodiment of the present disclosure is shown. Figure 2 As shown, method 200 includes: Step S201: Determine a first processing unit for running the first service instance to be started, wherein the first service instance is used to perform inference of the target model; Step S202: Establish a data transmission channel between the first processing unit and the second processing unit that has been running the second service instance for inference target model. The storage space of the second processing unit stores the loading information of the target model. The loading information includes at least one of the following: the weight of the target model, the model structure configuration information of the target model, or the compilation cache information of the target model. Step S203: Read the corresponding loading information of the target model from the corresponding storage space of the second processing unit based on the data transmission channel; Step S204: Initialize based on the corresponding loading information of the target model to start the first service instance.

[0037] By applying the above method 200, a data transmission channel can be established between the first processing unit corresponding to the new inference service instance and the second processing unit corresponding to the existing target model inference service instance. The model loading information already loaded by the second processing unit can be reused in the first processing unit, which is equivalent to directly using the existing target model service instance to start the new service instance. Through this data transmission channel, the new service instance can directly read key loading information, including weights, configurations, and compilation caches, from the storage space (including but not limited to processor memory, video memory, or cache) of the existing service instance. By reusing the ready state of the existing service instance, the time-consuming hard disk information reading operation and repetitive compilation calculation process during the startup of the new service instance can be avoided, thereby shortening the startup time of the service instance, improving the scaling efficiency of the inference service, or quickly starting a new service instance when the inference service cluster fails, thus improving disaster recovery efficiency.

[0038] In some examples, the target model can be various types of neural network models built based on deep learning algorithms, such as large language models, multimodal large models, image processing models, video generation models, and so on. Model inference corresponds to the process of using the trained target model to compute input data to generate output results. For example, large language models can be used for intelligent question answering, content generation, and machine translation, or image processing models can be used to process input images. In practical applications, when the existing model inference service instance is overloaded, response latency increases, or a running instance is detected to have failed, the system needs to start a new first service instance for expansion or replacement. In this case, the host or scheduling process used to schedule the hardware resources of the inference service cluster can determine the first processing unit suitable for carrying the new instance based on the cluster's resource availability. In some examples, when a new processing unit is added to the inference service cluster, the new processing unit can be used to start a new model inference service instance to expand the scale of the inference service cluster and improve the data throughput of the inference service.

[0039] In some examples, the first processing unit and the second processing unit refer to the hardware devices or computing nodes used to perform model inference, such as hardware computing accelerators like CPUs (Central Processing Units) and GPUs (Graphics Processing Units). In one example, the above method 200 can be implemented by an inference service cluster. Specifically, an idle or reserved first processing unit is allocated within the inference service cluster to host an inference service instance of the new target model, and a second processing unit that has already successfully loaded and is running the target model is located as the source. Based on this, the operations of transmitting the target model's loading information and starting the new service instance are performed.

[0040] In some examples, the loading information for the target model refers to the data necessary to start the model inference service. The weights of the target model correspond to the parameters of each layer connection in the neural network (e.g., weight matrices and bias terms). Model structure configuration information includes architectural definition information such as the number of layers, hidden layer dimensions, and number of attention heads. Compilation and cache information corresponds to the optimization results of the computation graph generated when the model runs on hardware, operator compilation information, etc. In some examples, the loading information for the target model may also include intermediate computation results during the target model loading process, such as the number of cached blocks (e.g., KV-Cache) pre-computed based on the memory environment of the running hardware. By reusing model loading information based on existing model inference service instances, redundant computations when starting new instances can be avoided, data read latency can be reduced, and startup efficiency can be improved.

[0041] In some examples, the data transmission channel refers to the communication link established between the first processing unit and the second processing unit. When the first processing unit and the second processing unit are located in the same physical machine, they can establish a data transmission channel through the PCIe bus or a dedicated GPU interconnect bus (such as NVLink). When they belong to different physical machines, they can establish a network connection through high-speed Ethernet or other means to achieve efficient transmission of model weight information.

[0042] After obtaining the loading information of the target model, it can be loaded into the memory or video memory of the first processing unit, and the inference engine runtime environment can be built. This includes placing the weight tensor into the correct video memory address, parsing the model configuration to build the computation graph structure, loading the compiled operator kernel, and allocating the temporary buffer required for inference. After the initialization of the first service instance is completed, this instance can be used to receive and process user model inference requests.

[0043] According to some embodiments, the first processing unit includes multiple processors. Step S202, establishing a data transmission channel between the first processing unit and the second processing unit of the second inference service that has already run the target model, includes establishing a data transmission channel between each of the multiple processors and the second processing unit. Step S203, reading the corresponding loading information of the target model from the corresponding storage space of the second processing unit based on the data transmission channel, includes using multiple processors to read the corresponding loading information of the target model from the corresponding storage space of the second processing unit in parallel. By applying the method of multi-channel parallel data reading, data transmission efficiency can be effectively improved, the bandwidth bottleneck of single-point transmission can be avoided, thereby shortening the model loading time and improving the startup efficiency of new service instances.

[0044] In some examples, multiple processors can be multiple GPUs in a single-machine multi-GPU or multi-machine multi-GPU environment. The system establishes a one-to-one correspondence between each GPU in the source (second processing unit) and the corresponding GPU in the destination (first processing unit) based on a tensor parallel partitioning strategy (e.g., Rank 0 to Rank 0, Rank 1 to Rank 1). Each processor in the destination can independently pull its own partition's weight data from the source. All processors transmit simultaneously, improving data transmission efficiency, shortening model loading time, and increasing the startup efficiency of new service instances.

[0045] According to some embodiments, the data transmission channel includes a communication connection based on remote direct memory access technology or a communication connection based on inter-chip high-speed interconnect technology. Thus, data transmission at the memory or video memory level is achieved based on underlying communication technologies such as remote direct memory access or inter-chip high-speed interconnect, reducing data transmission latency and thereby improving model loading efficiency.

[0046] According to the solutions provided in this disclosure, communication connections based on Remote Direct Memory Access (RDMA) technology can be used for data transfer between different physical machines, allowing data to be transferred directly from the memory of one computer to another. Communication connections based on inter-chip high-speed interconnect technology can be used for direct access to the video memory between different GPUs within the same machine. In some examples, the data transfer channel can also be implemented in other ways, such as based on shared memory connections, Redis databases, or socket transmission methods. As long as the reuse of the ready state of existing inference service instances of the target model can be achieved, this disclosure does not limit the specific type or construction method of the data transfer channel.

[0047] In some examples, the model weight reading process can be implemented based on a separate background process, such as using a dedicated checkpoint-engine process to handle the weight information transmission. For instance, when the first and second processing units are on the same physical machine, the sending end (second processing unit) stores the partitioned weights in the GPU memory. When the receiving end (first processing unit) initiates a read request, the weight data can be synchronized from the sending end's GPU memory to the receiving end's GPU memory via the NVLink channel. As another example, when the first and second processing units are on different physical machines, the model weight information of the sending end (second processing unit) is stored in CPU memory and partitioned according to the Rank of each GPU. Each GPU in the receiving end (first processing unit) can be configured with an independent RDMA connection, for example, by establishing a connection with the sending end through an independent network card, thereby initiating read requests in parallel. Multiple GPUs can then pull model weight information from the second processing unit in parallel, reducing data transmission latency and improving model loading efficiency.

[0048] In some examples, the process of reusing the model loading information of the second processing unit described above can also be implemented by reusing the cache of an existing service instance running on the second processing unit. In practical application scenarios, the model startup process usually leaves behind cached content such as model structure configuration files, graph compilation information, and operator compilation information. In one example, the logical hash algorithm for cache hits can be optimized. For example, a hash retrieval key can be generated based on model information that remains unchanged before and after scaling, so that the first service instance to be started can generate a hash retrieval key consistent with the already running second service instance. This allows it to accurately identify and directly load the model structure configuration files, graph compilation information, or operator compilation information pre-generated by the source service, achieving cross-instance intermediate result reuse. The target service (first service instance) can accurately identify and utilize the cached content of the source service (second service instance), thereby saving compilation time and improving the startup efficiency of the service instance.

[0049] According to some embodiments, the initialization based on the loading information of the target model in step S204 to start the first service instance includes: after starting the main process of the first service instance, in response to determining that the main process has not created a hardware interaction context for the first processing unit, creating a subprocess of the first service instance for performing model inference by copying the context information of the main process. This allows the reuse of the context information already loaded in the main process during subprocess creation, reducing the initialization time of the subprocess. Simultaneously, by limiting the reuse of the main process's context information to when the main process has not created a hardware interaction context for the first processing unit, the risk of errors in the processing unit's hardware configuration information can be reduced, improving the stability and startup efficiency of the inference service.

[0050] In some examples, the main process of the first service instance is the service entry process (e.g., api_server) responsible for receiving model inference requests and scheduling model inference tasks, while the child processes used to perform model inference are the core processes (e.g., engine_core or GPU-worker) that actually load the model and call the processing unit for computation. Process context information includes loaded third-party libraries, global variables, file descriptors, etc. Hardware interaction context refers to the state information of communication with acceleration hardware such as processing units (e.g., GPUs), such as CUDA context information.

[0051] In related technologies, child processes are typically created using the `spawn` method. In this case, the child process needs to reload all dependent libraries, which is time-consuming. According to the exemplary embodiments provided in this disclosure, when the main process has not yet created a hardware interaction context (i.e., the hardware of processing units such as the GPU has not been initialized), a child process can be created using the `fork` method. This allows the child process to inherit the libraries and environment information already loaded in the main process without needing to reload them. Furthermore, when the main process has already created a hardware interaction context, to avoid conflicts caused by multiple processes operating on the same hardware interaction context, the `fork` method is no longer used to create child processes, thereby improving the stability and startup efficiency of the inference service.

[0052] According to some embodiments, method 200 further includes: in response to determining that the target model includes multiple computation graphs, determining at least one target computation graph from the multiple computation graphs based on the memory usage specifications of the multiple computation graphs; obtaining at least one target computation graph; and allocating memory to the first service instance based on the respective memory usage specifications of the at least one target computation graph. Thus, it is possible to determine the target computation graph and allocate memory accordingly by identifying memory usage specifications during the initialization phase, thereby reducing the amount of computation graphs obtained during the initialization phase, reducing initialization latency, and improving the startup efficiency of the service instance.

[0053] In some examples, one or more computation graphs with the largest memory usage are selected from multiple computation graphs as target computation graphs, and then memory is allocated based on the target computation graph. This forces the system to pre-allocate a sufficiently large contiguous pool of memory, reducing the probability of memory overflow when the service instance starts up and processes inference requests of various sizes, avoiding memory fragmentation caused by subsequent dynamic memory allocation, and improving the stability of the model inference service instance.

[0054] A computation graph (such as a CUDA graph) is a graph structure used to define the computational operations that processing units need to perform during model inference and their dependencies. The memory footprint of a computation graph is typically related to the maximum batch size and maximum sequence length supported by the graph. By capturing and acquiring only the target computation graph with the largest memory footprint during the initialization phase, a sufficiently large contiguous memory pool can be pre-allocated to meet the maximum inference requirements, while also being reusable by subsequent smaller inference tasks. This reduces the probability of memory overflow and memory fragmentation, improving the stability of the model inference service instance.

[0055] According to some embodiments, method 200 further includes: after starting the first service instance, acquiring other computation graphs among multiple computation graphs, excluding at least one target computation graph. This allows the acquisition process of some computation graphs corresponding to the target model to be postponed, saving startup latency caused by acquiring computation graphs and improving the startup efficiency of new service instances.

[0056] According to the above implementation, a new service instance only needs to complete basic memory pool initialization during the startup phase, without waiting for the computation graphs corresponding to all possible input specifications (e.g., different input length buckets) to be compiled. In some examples, the timing for triggering the retrieval of the remaining computation graph may be after the first service instance has finished starting (e.g., it has already sent its ready status to receive model inference requests), or during background asynchronous loading when the system is idle, to improve the startup efficiency of the new service instance.

[0057] According to some embodiments, after starting the first service instance, acquiring other computation graphs besides at least one target computation graph from multiple computation graphs includes: in response to receiving a request to perform inference of a target model using the started first service instance, acquiring other computation graphs besides at least one target computation graph from multiple computation graphs. By capturing all computation graphs corresponding to the target model only when the model inference request arrives, on-demand loading of computation graphs can be achieved, saving hardware resources for the model inference service.

[0058] In some examples, model inference requests are typically issued by the client, containing text or data to be processed. In one example, when the first actual inference request arrives, the specific computation graph required for the request, along with other computation graphs not yet retrieved, can be obtained based on the request's specific specifications (such as the current BatchSize or Sequence Length). In another example, timing information of the request can be incorporated (e.g., allowing the user to manually select whether there is a latency tolerance for inference requests) to determine whether to enable the optimization option of delayed computation graph retrieval. By applying these methods, the service instance startup time before the first request can be reduced without impacting the processing performance of subsequent requests.

[0059] According to some embodiments, determining the first processing unit for running the first service instance to be started in step S201 includes: determining the target processing unit corresponding to the running target model as the first processing unit, wherein the daemon instance is obtained by pre-starting the initial service instance for inference target model, and wherein initializing the first service instance based on the corresponding loading information of the target model in step S204 includes: filling the corresponding loading information of the target model back into the daemon instance to obtain the initialized first service instance. By introducing daemon instances, more efficient inference service scaling or disaster recovery can be achieved, and the startup efficiency of new service instances can be improved.

[0060] A daemon instance is a pre-started model inference service instance. It pre-occupies physical resources and completes partial initialization, and can be activated at any time to launch new model inference service instances. In some examples, when the inference service cluster has idle resources, the model inference service can be pre-started on an idle machine. After the service starts, it enters daemon mode. In daemon mode, the instance only retains a portion of the GPU memory (e.g., only the GPU memory for the computation graph portion) to maintain the GPU memory pool structure, while releasing all GPU memory for the larger data portions (e.g., model weight information). In this case, each pre-started daemon instance occupies less GPU memory, allowing multiple daemon instances of different models to be pre-embedded on a single physical machine, achieving multi-standby on a single machine. When scaling is required, the system only needs to backfill the target model's weight information to the corresponding daemon instance through the aforementioned data transmission channel (i.e., reusing the information of the already running target model's inference service instance), quickly transforming it into a complete new service instance, thus improving the scaling efficiency of the model inference service.

[0061] According to some embodiments, the daemon instance includes a main process for inferring the target model and a corresponding loading information management process for the target model. The loading information management process is used to perform operations to read the corresponding loading information of the target model. By introducing an independent loading information management process in the daemon instance, the loading information transmission task can be decoupled from the main inference process, avoiding the blocking of the main process during the data loading process, improving data transmission efficiency, and reducing the startup latency of the service instance.

[0062] In some examples, the loading information management process can be the checkpoint-engine process to achieve more efficient model weight transfer and improve model loading efficiency.

[0063] Figure 3 A schematic diagram illustrating the startup process of a model inference service instance according to an exemplary embodiment of this disclosure is shown. Figure 3 As shown, the startup process of the model inference service instance is designed to include a model inference process (the main process and child processes of the aforementioned first service instance) for executing model inference tasks and a weight transfer process (the aforementioned loading information management process, such as the checkpoint-engine process) for assisting data transfer.

[0064] In this example, the model inference process performs the following steps: Step S311: Child process initialization. In this step, the main process (such as api_server) uses the fork method to create a child process (such as engine_core or GPU-worker) to perform specific inference tasks. By reusing the context such as libraries or environment information already loaded in the parent process, the initialization time of the child process is reduced.

[0065] Step S312: Reuse the video memory calculation results. In this step, the model inference process directly reads and reuses the intermediate results already calculated by the running target model service instance (such as the calculation results of the number of KV-Cache blocks), avoiding repeated calculations at startup.

[0066] Step S313: Load weight information. In this step, the model inference process can be executed based on the dummy loading method, that is, only the skeleton structure of the model is initialized and the corresponding video memory space is allocated, without filling in the actual weight values, so as to save disk read overhead.

[0067] Step S314: Reuse compilation information. In this step, the model inference process can directly map and load the model structure configuration file, graph compilation information, and operator compilation information generated by the running target model service instance by matching hash keys, thereby skipping the time-consuming just-in-time compilation process.

[0068] After completing the above initialization steps, the model inference process starts the model inference service in step S315, for example, by setting the service status to the ready state. In this case, the weight transmission process on the right can complete step S321 (detecting successful service startup) after listening to the service ready state, and then trigger step S322 (transmitting weight information). In this step, the weight transmission process establishes a high-speed data transmission channel with the source service (i.e., the aforementioned second service instance and its corresponding second processing unit) to retrieve the real target model weight information. The real target model weight information is written into the video memory address pre-allocated by the model inference process, i.e., step S316 (model weight update) is executed to complete the replacement of the dummy data and realize the rapid startup of the inference service instance of the new target model.

[0069] Figure 4 A schematic diagram illustrating the startup process of a model inference service instance according to an exemplary embodiment of this disclosure is shown. Figure 4 As shown in this example, multiple service process groups (daemon instances) for different models (Model A, Model B, Model C, etc. as shown in the figure) are pre-started on the cold standby machine. Each group contains a model inference process and a corresponding weight transfer process. To achieve resource reuse and support disaster recovery for multiple models simultaneously on a single machine, these pre-started process groups are all in a daemon state. In this state, the model inference process retains only a small amount of resources, while actively releasing the memory for the larger data portions (such as model weight information).

[0070] When it is necessary to expand or perform disaster recovery on the target model (e.g., model B), it is necessary to start the model inference service, wake up the inference process of the target model from the sleep state, that is, put it into the start state. After the detection service is successfully started, the corresponding weight transmission process will perform the operation of transmitting weight information. The model inference process receives the model weight data and performs the model weight update operation to fill the weight back into the video memory, thereby completing the transition from the lightweight daemon state to the full-featured service state.

[0071] Figure 5 A schematic diagram illustrating the configuration process of a model inference service cluster according to an exemplary embodiment of this disclosure is shown. Figure 5 As shown, the configuration process for the inference service cluster includes the following steps: In step 501, it is determined whether a cold standby machine is configured (i.e., whether there are idle cluster resources).

[0072] If the determination is yes, proceed to step 502, pre-embed multiple guardian instances of different models, that is, pre-start multiple service instances of different models on the cold standby machine, and then release the video memory of the model weight part to enter the guardian state.

[0073] If the determination is negative, proceed to step 503 to start the initial service instance.

[0074] In step 504, the initial instance is used for inference. During this process, the load on the inference service cluster can be continuously monitored, or its failure can be detected, to determine whether a new service instance needs to be launched.

[0075] In step 505, it is determined whether capacity expansion is needed (e.g., due to excessive load or a failure). If capacity expansion is not needed, the reasoning continues.

[0076] If expansion is required, proceed to step 506 to determine if a guardian instance exists.

[0077] If no daemon instance exists, the process proceeds to step 507, reusing cache optimization startup, for example, by using the aforementioned methods of reusing the compilation configuration of the source service and the computation graph cache to initialize the new service instance.

[0078] In step 508, the weight reuse optimization is initiated, for example, by using the aforementioned method of quickly pulling model weight data from the source service instance to quickly load the target model.

[0079] In step 509, a new instance is launched to expand the inference service cluster.

[0080] If a guardian instance exists, the process proceeds to step 510. Since multiple guardian instances of a model may be pre-installed on a single machine, in order to ensure that the target model has exclusive access to computing resources, other guardian instances besides the target model are deleted in step 510.

[0081] In step 511, the guardian instance corresponding to the target model is woken up.

[0082] As mentioned earlier, the guardian instance has released the GPU memory for the model weight portion, so it is necessary to continue executing step 508 to pull the model weight data and quickly load the target model, and then launch a new instance to expand the inference service cluster.

[0083] By configuring the model inference service cluster using the above methods, the ready state of existing service instances can be reused when launching new service instances, avoiding time-consuming disk information reading operations and repetitive compilation calculations during the startup of new service instances. This shortens the startup time of service instances, improves the scaling efficiency of inference services, or allows for the rapid startup of new service instances when the inference service cluster fails, thereby improving disaster recovery efficiency.

[0084] According to one aspect of this disclosure, a device for launching a model reasoning service instance is also provided. Figure 6 A structural block diagram of a launch device 600 for a model inference service instance according to an exemplary embodiment of the present disclosure is shown. Figure 6 As shown, the device 600 includes: The determining unit 601 is configured to determine a first processing unit for running a first service instance to be started, wherein the first service instance is used to perform inference of the target model; Establishment unit 602 is configured to establish a data transmission channel between the first processing unit and the second processing unit that has been running a second service instance for inference target model. The storage space of the second processing unit stores the loading information of the target model. The loading information includes at least one of the following: the weight of the target model, the model structure configuration information of the target model, or the compilation cache information of the target model. The reading unit 603 is configured to read the corresponding loading information of the target model from the corresponding storage space of the second processing unit based on the data transmission channel; Startup unit 604 is configured to initialize based on the corresponding loading information of the target model in order to start the first service instance.

[0085] According to some embodiments, the first processing unit includes multiple processors, the establishment unit 602 is configured to establish a data transmission channel between each of the multiple processors and the second processing unit, and the reading unit 603 is configured to use the multiple processors to read the corresponding loading information of the target model from the corresponding storage space of the second processing unit in parallel.

[0086] According to some embodiments, the data transmission channel includes a communication connection based on remote memory direct access technology or a communication connection based on inter-chip high-speed interconnect technology.

[0087] According to some embodiments, the apparatus 600 further includes: a graph determination unit configured to determine at least one target computation graph from the plurality of computation graphs based on the memory usage specifications of the plurality of computation graphs in response to determining that the target model includes a plurality of computation graphs; a graph acquisition unit configured to acquire at least one target computation graph; and an allocation unit configured to allocate memory to a first service instance based on the respective memory usage specifications of the at least one target computation graph.

[0088] According to some embodiments, the graph acquisition unit is also configured to: after starting the first service instance, acquire other computation graphs among multiple computation graphs, excluding at least one target computation graph.

[0089] According to some embodiments, the graph acquisition unit is configured to: in response to receiving a request to perform inference of a target model using a first service instance after startup, acquire other computation graphs among a plurality of computation graphs, excluding at least one target computation graph.

[0090] According to some embodiments, the determining unit 601 is configured to: determine the target processing unit corresponding to the running target model as the first processing unit, wherein the daemon instance is obtained by pre-starting the initial service instance used for inference of the target model, and the starting unit 604 is configured to: obtain the initialized first service instance by backfilling the corresponding loading information of the target model to the daemon instance.

[0091] According to some embodiments, the guardian instance includes a main process for inference target model and a corresponding loading information management process for target model, the loading information management process being used to perform operations to read the corresponding loading information of target model.

[0092] According to some embodiments, the startup unit 604 is configured to: after starting the main process of the first service instance, in response to determining that the main process has not created a hardware interaction context for the first processing unit, create a subprocess of the first service instance for performing model inference by copying the context information of the main process.

[0093] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information and other types of information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0094] According to one aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for launching a model inference service instance.

[0095] According to one aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause the computer to execute the above-described method for initiating a model inference service instance.

[0096] According to one aspect of this disclosure, a computer program product is also provided, including a computer program, wherein the computer program, when executed by a processor, implements the above-described method for launching a model inference service instance.

[0097] refer to Figure 7The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0098] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0099] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to device 700. Input unit 706 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 707 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, a hard disk and an optical disk. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0100] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the method for launching a model inference service instance. For example, in some embodiments, the method for launching a model inference service instance may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the method for launching a model inference service instance described above may be performed. Alternatively, in other embodiments, computing unit 701 may be configured to execute the startup method of the model inference service instance by any other suitable means (e.g., by means of firmware).

[0101] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0103] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0105] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0106] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0107] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0108] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for starting a model inference service instance, comprising: A first processing unit is determined for running a first service instance to be started, wherein the first service instance is used to perform inference of the target model; A data transmission channel is established between the first processing unit and the second processing unit, which has been running a second service instance for inference of the target model. The storage space of the second processing unit stores the loading information of the target model. The loading information includes at least one of the following: the weights of the target model, the model structure configuration information of the target model, or the compilation cache information of the target model. Based on the data transmission channel, the corresponding loading information of the target model is read from the corresponding storage space of the second processing unit; and Initialization is performed based on the loading information corresponding to the target model to start the first service instance.

2. The method as described in claim 1, wherein, The first processing unit includes multiple processors, and the data transmission channel established between the first processing unit and the second processing unit that has already run the second inference service of the target model includes: Establish a data transmission channel between each of the plurality of processors and the second processing unit. Furthermore, the step of reading the corresponding loading information of the target model from the corresponding storage space of the second processing unit based on the data transmission channel includes: The multiple processors are used to read the corresponding loading information of the target model from the corresponding storage space of the second processing unit in parallel.

3. The method as described in claim 2, wherein, The data transmission channel includes a communication connection based on remote memory direct access technology or a communication connection based on inter-chip high-speed interconnect technology.

4. The method according to any one of claims 1-3, further comprising: In response to determining that the target model comprises multiple computation graphs, at least one target computation graph is determined from the multiple computation graphs based on the memory usage specifications of the multiple computation graphs; Obtain the at least one target computation graph; as well as Based on the respective memory usage specifications of the at least one target computation graph, memory is allocated to the first service instance.

5. The method of claim 4, further comprising: After starting the first service instance, obtain the other computation graphs in the plurality of computation graphs besides the at least one target computation graph.

6. The method of claim 5, wherein, After starting the first service instance, obtaining the other computation graphs in the plurality of computation graphs besides the at least one target computation graph includes: In response to receiving a request to perform inference of the target model using the first service instance after startup, obtain the other computation graphs in the plurality of computation graphs besides the at least one target computation graph.

7. The method according to any one of claims 1-6, wherein, The first processing unit for determining the first instance of the service to be started includes: The target processing unit corresponding to the running daemon instance of the target model is identified as the first processing unit, wherein the daemon instance is obtained by pre-starting the initial service instance used for inference of the target model. Furthermore, the initialization of the first service instance based on the loading information corresponding to the target model includes: By backfilling the corresponding loading information of the target model into the guardian instance, the first service instance after initialization is obtained.

8. The method of claim 7, wherein, The guardian instance includes a main process for inferring the target model and a corresponding loading information management process for the target model. The loading information management process is used to perform the operation of reading the corresponding loading information of the target model.

9. The method according to any one of claims 1-8, wherein, The initialization based on the loading information corresponding to the target model to start the first service instance includes: After starting the main process of the first service instance, in response to determining that the main process has not created a hardware interaction context for the first processing unit, a subprocess for performing model inference of the first service instance is created by copying the context information of the main process.

10. A device for launching a model inference service instance, comprising: The determining unit is configured to determine a first processing unit for running a first service instance to be started, wherein the first service instance is used to perform inference of the target model; The establishment unit is configured to establish a data transmission channel between the first processing unit and a second processing unit that has been running a second service instance for inference of the target model. The storage space of the second processing unit stores the loading information of the target model. The loading information includes at least one of the following: the weights of the target model, the model structure configuration information of the target model, or the compilation cache information of the target model. The reading unit is configured to read the corresponding loading information of the target model from the corresponding storage space of the second processing unit based on the data transmission channel; and The startup unit is configured to initialize based on the loading information corresponding to the target model in order to start the first service instance.

11. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

13. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-9.

Images