Service startup and control method, electronic device, storage medium, and program product

Abstract

The present application discloses a service startup and control method, electronic device, storage medium, and program product, and relates to the field of cloud service technology. The service startup method includes recording the processor process involved in deploying the current reasoning service in the computing cluster after the current reasoning service is running, forming a running process file, and using the current reasoning service as the target reasoning service that can improve startup efficiency. In this way, when obtaining a startup request for the first reasoning service belonging to the target reasoning service, a running process file can be written to omit steps such as reasoning the processor to be deployed. Therefore, the technical problem that the lengthy startup time of the reasoning service affects its performance is solved, and the inference service startup process is optimized to improve the efficiency of restarting the reasoning service, thereby improving the technical effect of improving the performance of the reasoning service.

Description

Technical Field

[0001] The present application relates to the field of cloud service technology, and in particular to a service startup method, a service control method, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Inference services are a key component of cloud services. Inference services deploy trained inference models online to perform inference tasks such as real-time prediction. However, due to the complexity of the inference model architecture and the large size of the model files, each startup or reload of the inference service takes a long time, which in turn affects its performance. Summary of the Invention

[0003] The present application provides a service startup method, a service control method, an electronic device, a computer-readable storage medium, and a computer program product to at least solve the problem in the related art that the lengthy startup time of an inference service affects its performance.

[0004] The present application provides a service startup method, which includes: in response to the current reasoning service running, obtaining the processor process when the current reasoning service runs on a computing cluster, recording the processor process to form a running process file; creating an executable file adapted to the running process file as a startup scheduling file; wherein the startup scheduling file is used to start the current reasoning service; using the current reasoning service for which the startup scheduling file is created as the target reasoning service; in response to obtaining a startup request for a first reasoning service, executing its startup scheduling file and writing the running process file to start the first reasoning service; wherein the first reasoning service belongs to the target reasoning service.

[0005] The present application also provides a service control method, which includes: in response to the initial operation of the current reasoning service, determining whether the current reasoning service is successfully executed; in response to the successful operation of the current reasoning service, executing the service startup method as described above; and scheduling the current reasoning service to execute the reasoning task.

[0006] The present application also provides an electronic device, which includes: a memory and a processor; the memory is used to store computer programs; the processor is used to implement the steps of the service startup method as described above when executing the computer program; or, implement the steps of the service control method as described above.

[0007] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the service startup method as described above are implemented; or, the steps of the service control method as described above are implemented.

[0008] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the service startup method as described above; or, implements the steps of the service control method as described above.

[0009] Through this application, the processor process identification of the target reasoning service is completed in advance and a running process file is formed. When obtaining the startup request of the first reasoning service belonging to the target reasoning service, the running process file can be written to omit the steps such as reasoning the processor to be deployed. Therefore, the technical problem that the long startup time of the reasoning service affects its performance can be solved, and the inference service startup process can be optimized to improve the efficiency of the reasoning service restart, thereby improving the technical effect of the inference service performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 This is a schematic diagram of an embodiment of an application scenario of the service startup method of this application;

[0012] Figure 2 This is a flowchart of an embodiment of the method for starting the application service;

[0013] Figure 3 A flowchart illustrating an embodiment of the current reasoning service for this application;

[0014] Figure 4 A flowchart of an embodiment of expanding the inference service capacity for this application;

[0015] Figure 5 This is a schematic diagram comparing the conventional reasoning service startup and the target reasoning service startup according to an embodiment of the present application;

[0016] Figure 6 This is a flow chart of an embodiment of the service control method of this application. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0019] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0020] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the service startup method depends, the specific application environment architecture or specific hardware architecture is described here.

[0021] See also Figure 1 , Figure 1 This is a scenario diagram of an embodiment of an application scenario of the service startup method of this application.

[0022] In this embodiment, the application scenario of the service startup method may include a computing cluster, a service management device, a first storage unit, and a second storage unit.

[0023] Among them, the computing cluster is capable of running inference services.

[0024] A computing cluster can be considered as a system consisting of multiple interconnected computers, where each computer can be considered as a computing node. For example, a computing node can be a server, etc. A computing node can include a processor that can support operations. For example, the processor type can include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an accelerator card (i.e., an accelerated processor). That is to say, when the processor type includes a CPU, a GPU, and an accelerated processor, the service startup method of the present application can be compatible with the processor deployment details of the inference service on the aforementioned processor types, thereby improving the functionality of service startup management and further improving the startup efficiency of the inference service.

[0025] An inference service can be understood as deploying a trained inference model to a computing cluster, allowing it to perform inference tasks such as real-time predictions. This allows machine learning or deep learning models to be applied to real-world problems. The inference model receives new input data, performs inference based on that data, and outputs predictions or decision recommendations. Inference services can be used for natural language processing, image recognition, recommendation systems, autonomous driving, and other applications, without specific limitations here.

[0026] The service management device can control the inference service's startup within the computing cluster and further control the inference service's execution of inference tasks. Specifically, the service management device can implement the steps of a service startup method or a service control method. The service startup and service control methods will be discussed in detail later and will not be discussed here. In other words, the service management device can have inference service lifecycle management capabilities, enabling the creation, expansion, and execution of inference file inference services.

[0027] The service management device can control the deployment of the current inference service on the computing cluster. The current inference service refers to the inference service currently being deployed or started. After the current inference service is deployed and reliably started, a running process file and a startup schedule file for the current inference service can be created. The running process file can be stored in the first storage unit, and the startup schedule file can be stored in the second storage unit.

[0028] For example, when the running process file is in the image file format, the first storage unit may be an image file library, etc., which is not limited here. The second storage unit may be a database, etc., which is not limited here.

[0029] It can be seen that the embodiment of the present application provides a service startup method. The following describes the working principle of the service startup method in detail in conjunction with the execution process of the service startup method.

[0030] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of the service startup method of this application.

[0031] S101: In response to the current reasoning service running, obtaining a processor process when the current reasoning service runs on a computing cluster, and recording the processor process to form a running process file.

[0032] In this embodiment, as described above, the current inference service refers to the inference service that is currently being started or being managed for startup.

[0033] You can monitor whether the current inference service is running. The fact that the current inference service is running indicates that the current inference service has been started and can be used to execute inference tasks. Optionally, this can be performed after the current inference service has been run for the first time in the computing cluster or after it has been started and run multiple times, without limitation here.

[0034] When monitoring the current inference service operation, the processor process of the current inference service when it is running in the computing cluster can be obtained, and the obtained processor process can be recorded to form the running process file of the current inference service, so as to record the processor process that the current inference service can run, so that when the current inference service is started again in the future, there is no need to immediately infer the processor that can be deployed, which is conducive to improving the startup efficiency of the inference service.

[0035] S102: creating an executable file adapted to the running process file as a startup scheduling file; wherein the startup scheduling file is used to start the current inference service.

[0036] In this embodiment, in response to the completion of the creation of the running process file, it can be considered that the original file for starting the current reasoning service, that is, the initial startup file, is less efficient in starting the reasoning service. Therefore, an executable file adapted to the running process file can be created as a startup scheduling file, so that when the current reasoning service is subsequently started, the startup scheduling file can be used to start the reasoning service, thereby improving its startup efficiency.

[0037] S103: The current inference service that has completed the creation of the startup scheduling file is used as the target inference service.

[0038] In this embodiment, in response to the completion of the creation of the startup scheduling file of the current reasoning service, it can be considered that the redeployment and startup efficiency of the current reasoning service can be improved, so the current reasoning service is used as the target reasoning service.

[0039] It should be noted that the term "target inference service" in this document refers to a specific type of inference service, and does not confer a specific meaning. The term "target inference service" is used in contrast to the term "regular inference service," which will be described in detail later. A target inference service is an inference service that utilizes a startup schedule file to improve its startup efficiency, while a regular inference service is an inference service that uses its initial startup file for startup.

[0040] S104: In response to obtaining a start request for the first reasoning service, executing its start scheduling file and writing a running process file to start the first reasoning service; wherein the first reasoning service is a target reasoning service.

[0041] In this embodiment, when obtaining a startup request for the first inference service belonging to the target inference service, it can be considered that the startup efficiency can be improved by utilizing the running process file and the startup scheduling file. Therefore, the scheduling file of the first inference service can be executed and the running process file can be written to realize the startup of the first inference service.

[0042] For example, the startup request may include one or more of a request to restart the execution of the first reasoning service, a request to expand the capacity of the first reasoning service (run the original first reasoning request and create a new first reasoning service), and a request to deploy another first reasoning service, without limitation here.

[0043] That is to say, in this embodiment, after the current reasoning service is running, the processor process involved in deploying the current reasoning service in the computing cluster is recorded to form a running process file, and the current reasoning service is used as the target reasoning service that can improve startup efficiency. In other words, in this embodiment, the processor process identification of the target reasoning service can be completed in advance and a running process file can be formed. In this way, when obtaining a startup request for the first reasoning service belonging to the target reasoning service, the running process file can be written to omit steps such as inferring the processor to be deployed. Therefore, the inference service startup process can be optimized to facilitate improving its startup efficiency when expanding the inference service, thereby improving the performance of the inference service.

[0044] See also Figure 3 , Figure 3 This is a flowchart of an embodiment of running the current reasoning service of this application.

[0045] S201: Obtain initial information of the current inference service.

[0046] In this embodiment, the model file of the current reasoning service and the reasoning framework image may be obtained as initial information.

[0047] In layman's terms, using executable files such as initial startup files and startup scheduling files as scripts as an example, when first creating an inference service, users can upload the model file and inference framework image of the current inference service to the computing cluster and service management device. In this case, the model file and inference framework image can be used as initial information.

[0048] S202: Construct an initial startup file based on the initial information and create a current inference service.

[0049] In this embodiment, the processor type and the number of processors of each type, scaling indicators, container images, and startup instructions indicated by the initial information can be parsed as resource requirement information for the current inference service; wherein the processor type includes at least one of a central processing unit, a graphics processing unit, and an accelerator processing unit.

[0050] The initial startup file can be generated by combining the initial information and the resource requirement information to obtain the initial startup file of the current inference service.

[0051] In simple terms, the resource requirements of the inference service are determined based on the initial information, and a deployment script for the current inference service, namely the initial startup file deployscripts_default, is adaptively generated. The initial startup file can include settings for the number of CPUs, the number of GPUs, scaling indicators, the container image inferenceImage_default, and the startup command startCommand_default.

[0052] Furthermore, the initial startup file can be run to load the inference model on which the current inference service relies, thereby creating the current inference service in the computing cluster.

[0053] Specifically, the processor type and number of processors of each type indicated in the initial startup file can be read as the target processor and its target number. The target number of target processors within the computing cluster is selected as the associated processors. The current inference service is deployed on the computing node with the associated processor. The processor process of the current inference service on the associated processor is obtained to form a running process file. In layman's terms, this allows the selection of computing nodes with the appropriate number of GPUs and CPUs for the current inference service to support the operation of the current inference service.

[0054] S203: Monitor the deployment status of the current inference service.

[0055] In this embodiment, during the process of creating the current reasoning service, the deployment status of the current reasoning service may be monitored to evaluate whether the current reasoning service is running.

[0056] Optionally, the loading status of the inference model can be monitored to represent the deployment status of the current inference service. Alternatively, deployment status monitoring can be achieved through feedback from deployment completion, configuration identification based on configuration files, and other methods. These will not be discussed further here.

[0057] S204: Determine whether the current inference service is running.

[0058] In this embodiment, when it is determined that the current reasoning service is running, step S205 may be executed. When it is determined that the current reasoning service is running, step S204 may be continued to be executed.

[0059] Optionally, whether the current inference service is in a ready state may be determined based on the loading state of the inference model.

[0060] S205: Obtain the processor process of the current inference service running on the computing cluster.

[0061] In this embodiment, the computing cluster may include one or more computing nodes. In this embodiment, the number of computing nodes in the computing cluster is multiple as an example.

[0062] At the same time, in this embodiment, in response to the current reasoning service being in a ready state, it can be determined that the current reasoning service is running, and the current reasoning service is not loaded to the load balancer temporarily.

[0063] That is, in this embodiment, when the current inference service is deemed to be in a ready state, the current inference service is controlled to temporarily not provide inference services externally, and a running process file for the processor process is created for the current inference service. This reduces the acquisition of additional processor processes that may exist when the current inference service executes inference tasks, thereby ensuring the simplicity of the running process file. This reduces the redundant processor process generation when the current inference service is started using the running process file and the startup scheduling file, further improving the startup efficiency of the current inference service. At the same time, the impact of the running process file generation process on the inference performance of the current inference service can be reduced, thereby improving the startup efficiency of the current inference service while also ensuring its inference performance when executing inference tasks, thereby reliably improving service inference and computing cluster performance. For example, by calling the computing cluster API (Application Programming Interface), a processor process snapshot of the container (i.e., the current inference service) can be obtained as a running process file. Once the running process file is created, the current inference service is added to the load balancer, where it can be scheduled by the load balancer to provide inference services externally, thus enabling the execution of inference tasks.

[0064] S206: Determine whether the processor process is successfully acquired.

[0065] In this embodiment, when the processor process is successfully acquired, step S207 may be executed. When the processor process is unsuccessful, step S210 may be executed.

[0066] S207: Generate an operating process file and store it in the first storage unit.

[0067] In this embodiment, the processor process may be recorded to form a running process file.

[0068] Specifically, the compute nodes included in the computing cluster can be pre-acquired, and a process inspection component can be deployed to the compute nodes. This component can control the container management component and node communication component of the compute nodes to enable the container snapshot function. For example, the process inspection component can be designed independently or it can leverage an existing process inspection component. For example, the process inspection component can be CRIU (Checkpoint and Restore in Userspace), an open source tool that enables checkpoint and restore functions for userspace processes. In this embodiment, it can be used to identify and obtain the current inference service processor process.

[0069] The scheduling process check component performs process checks on the compute nodes to obtain the processor process of the current inference service. The container snapshot function is used to create a process snapshot of the processor process.

[0070] Convert the process snapshot into an image format and use it as the running process file.

[0071] Furthermore, the first storage unit and the second storage unit may be connected.

[0072] The running process file can be stored in the first storage unit. The startup schedule file can be stored in the second storage unit. Both the running process file and the startup schedule file can carry the service tag of the current inference service. When the current inference service is created, the running process file and the startup schedule file can be obtained from the first storage unit and the second storage unit.

[0073] With such a design, the first storage unit and the second storage unit can be used to store the running process files and startup scheduling files of multiple reasoning services, that is, the target reasoning service includes multiple reasoning services, so that when multiple reasoning services belonging to the target reasoning service are started, the startup efficiency can be improved, further improving the service startup and the performance of the target reasoning service.

[0074] S208: Create an executable file adapted to the running process file as a startup scheduling file.

[0075] In this embodiment, an initial startup file for deploying and running the current inference service to the computing cluster may be obtained.

[0076] Controlling the hardware information and computing resources constrained within the startup schedule file to match those in the initial startup file; wherein the computing resources and hardware information are associated with an associated processor, which is a processor in which a processor process exists. Thus, by extracting the hardware and computing resource information involved in the initial startup file and controlling the startup schedule file to contain content that matches the extracted information, it is possible to ensure that the startup schedule file supports the reliable startup of the inference service to which it belongs, and to ensure that the inference service launched using the startup schedule file has the corresponding inference capabilities. This improves the startup efficiency of the inference service while also taking into account the reliability of the inference service, thereby optimizing the performance of the inference service and its computing cluster.

[0077] Optionally, the inference image of the current inference service deployed in the initial startup file may be located; wherein the inference image is used to identify the inference image to select an associated processor and form a processor process on the associated processor when deploying and starting the current inference service.

[0078] In this way, the inference image can be replaced with a running process file, and the startup instructions of the inference image can be located and removed to form a startup scheduling file. In this way, the code related to the inference image can be deleted and adjusted based on the initial startup file to form a startup scheduling file, thereby reducing the difficulty of generating the startup scheduling file and reducing the risk of unreliable startup scheduling files. In other words, the efficiency and reliability of startup scheduling file generation can be improved, so that the target inference service can be started efficiently and reliably based on the startup scheduling file, further improving the reliability of service startup, and improving the reliability of the inference service and computing cluster.

[0079] The specific initial startup file and startup scheduling file will be given as examples in the following text in combination with the file code, so they will not be described in detail here.

[0080] In this embodiment, it can be determined whether the startup scheduling file of the current reasoning service has been created. If it is determined that the startup scheduling file of the current reasoning service has not been created, it can wait for the startup scheduling file to be created. If it is determined that the startup scheduling file of the current reasoning service has been created, step S209 can be executed.

[0081] Furthermore, when the startup schedule file is detected to have been created, it can be determined whether the creation of the startup schedule file has failed. That is, completion of creation is not equivalent to success, thereby further enhancing the robustness of service startup. If it is determined that the startup schedule file has failed to be created, step S210 can be executed. If it is determined that the startup schedule file has been created successfully, step S209 can be executed.

[0082] S209: Load the current inference service as the target inference service to the load balancer.

[0083] In this embodiment, in response to the completion of the scheduling file creation, the current inference service is used as the target inference service. The target inference service can be loaded into the load balancer of the computing cluster so that the target inference service can be scheduled by the load balancer to execute the inference task.

[0084] S210: Load the current inference service as a regular inference service to the load balancer.

[0085] In this embodiment, in response to the failure in creating the startup schedule file of the current inference service, it is treated as a regular inference service.

[0086] In this way, this embodiment can also take into account the inference services that are difficult to obtain processor processes and start scheduling file creation, and treat this type of inference service as a regular inference service. When there is a need to start it, it can still be started using the initial startup file to ensure the reliable startup of the regular inference service, thereby improving the compatibility of service startup.

[0087] S211: Determine whether the current inference service startup is complete.

[0088] Taking the aforementioned running process file as an example of a mirror format file, in layman's terms, after the processor process snapshot is successfully created in this embodiment, the snapshot file can be converted into a snapshot image inferenceImage_snash, and the snapshot image can be uploaded to the image warehouse, i.e., the first storage unit. Based on the snapshot image of the current inference service, a startup scheduling file for the current inference service, i.e., a startup scheduling script file deployscripts_snash, can be created. Deployscripts_snash can include the number of CPUs, number of GPUs, scaling indicators, container images, GPU models, CPU models, inferenceImage_snash, startup commands startCommand_snash, etc. required by the current inference service.

[0089] At the same time, the GPU process's hardware requirements for operation can be met, and the newly launched first inference service can be controlled to use the same hardware as that used when its startup schedule file was generated. For example, resource information about the compute node running the first inference service can be queried, such as the GPU card model and CPU model. Details related to controlling the newly launched first inference service to use the same hardware as that used when its startup schedule file was generated will be explained in detail later.

[0090] The following is an example of a specific initial startup file and a startup scheduling file combined with file codes.

[0091] The initial startup file can contain the following code:

[0092] apiVersion: inference.inais / v1alpha1

[0093] kind: DistributedInference

[0094] metadata:

[0095] name: vllm

[0096] namespace: ds-dist

[0097] spec:

[0098] hpa:

[0099] maxReplicas: 1

[0100] metrics:

[0101] - resource:

[0102] name: cpu

[0103] target:

[0104] averageUtilization: 2

[0105] type: Utilization

[0106] type: Resource

[0107] minReplicas: 1

[0108] scaleTargetRef:

[0109] apiVersion: inference.inais / v1alpha1

[0110] kind: DistributedInference

[0111] name: vllm

[0112] mpiReplicaSpecs:

[0113] Launcher:

[0114] replicas: 1

[0115] template:

[0116] spec:

[0117] nodeSelector:

[0118] accelerator: "NVIDIA-H20-3e"

[0119] containers:

[0120] - command:

[0121] - / bin / sh

[0122] - -c

[0123] - / vllm-workspace / start.sh --tensor-parallel-size 4 --pipeline-parallel-size 4 --model-path / huggingface / --model-id model

[0124] image: 192.168.16.0.151:5000 / mine / vllm / vllm:v0.7.2

[0125] imagePullPolicy: IfNotPresent

[0126] env:

[0127] - name: "h_parameter"

[0128] value: "--max-model-len 65536 --gpu-memory-utilization 0.98 --max-num-seqs 256"

[0129] name: launcher

[0130] ports:

[0131] - containerPort: 8080

[0132] protocol: TCP

[0133] resources:

[0134] limits:

[0135] cpu: 10

[0136] memory: 100G

[0137] nvidia.com / gpu: 4

[0138] requests:

[0139] cpu: 10

[0140] memory: 100G

[0141] nvidia.com / gpu: 4

[0142] volumeMounts:

[0143] - mountPath: / huggingface

[0144] name: ww

[0145] - mountPath: / dev / shm

[0146] name: shared-mem

[0147] volumes:

[0148] - hostPath:

[0149] path: / data0 / Models / DeepSeek-R1-671B

[0150] type: Directory

[0151] name: ww

[0152] - emptyDir:

[0153] medium: Memory

[0154] sizeLimit: 10.24G

[0155] name: shared-mem

[0156] terminationGracePeriodSeconds: 0

[0157] Worker:

[0158] replicas: 3

[0159] template:

[0160] spec:

[0161] nodeSelector:

[0162] accelerator: "NVIDIA-H20-3e"

[0163] containers:

[0164] - image: 172.16.0.151:5000 / mine / vllm / vllm:v0.7.2-dis

[0165] imagePullPolicy: IfNotPresent

[0166] name: worker

[0167] resources:

[0168] limits:

[0169] cpu: 10

[0170] memory: 100G

[0171] nvidia.com / gpu: 4

[0172] requests:

[0173] cpu: 10

[0174] memory: 100G

[0175] nvidia.com / gpu: 4

[0176] volumeMounts:

[0177] - mountPath: / huggingface

[0178] name: ww

[0179] - mountPath: / dev / shm

[0180] name: shared-mem

[0181] command:

[0182] - / bin / sh

[0183] - -c

[0184] - / vllm-workspace / start_worker.sh

[0185] volumes:

[0186] - hostPath:

[0187] path: / data0 / Models / DeepSeek-R1-671B

[0188] type: Directory

[0189] name: ww

[0190] - emptyDir:

[0191] medium: Memory

[0192] sizeLimit: 10.24G

[0193] name: shared-mem

[0194] terminationGracePeriodSeconds: 0

[0195] replicas: 1

[0196] slots: 1

[0197] sshAuthMountPath: / root / .ssh

[0198] The startup schedule file can contain the following code:

[0199] apiVersion: inference.inais / v1alpha1

[0200] kind: DistributedInference

[0201] metadata:

[0202] name: vllm

[0203] namespace: ds-dist

[0204] spec:

[0205] hpa:

[0206] maxReplicas: 1

[0207] metrics:

[0208] - resource:

[0209] name: cpu

[0210] target:

[0211] averageUtilization: 2

[0212] type: Utilization

[0213] type: Resource

[0214] minReplicas: 1

[0215] scaleTargetRef:

[0216] apiVersion: inference.inais / v1alpha1

[0217] kind: DistributedInference

[0218] name: vllm

[0219] mpiReplicaSpecs:

[0220] Launcher:

[0221] replicas: 1

[0222] template:

[0223] spec:

[0224] nodeSelector:

[0225] accelerator: "NVIDIA-H20-3e"

[0226] containers:

[0227] image: 192.168.16.0.151:5000 / mine / vllm / vllm:v0.7.2-snashpot

[0228] imagePullPolicy: IfNotPresent

[0229] env:

[0230] - name: "h_parameter"

[0231] value: "--max-model-len 65536 --gpu-memory-utilization 0.98 --max-num-seqs 256"

[0232] name: launcher

[0233] ports:

[0234] - containerPort: 8080

[0235] protocol: TCP

[0236] resources:

[0237] limits:

[0238] cpu: 10

[0239] memory: 100G

[0240] nvidia.com / gpu: 4

[0241] requests:

[0242] cpu: 10

[0243] memory: 100G

[0244] nvidia.com / gpu: 4

[0245] volumeMounts:

[0246] - mountPath: / huggingface

[0247] name: ww

[0248] - mountPath: / dev / shm

[0249] name: shared-mem

[0250] volumes:

[0251] - hostPath:

[0252] path: / data0 / Models / DeepSeek-R1-671B

[0253] type: Directory

[0254] name: ww

[0255] - emptyDir:

[0256] medium: Memory

[0257] sizeLimit: 10.24G

[0258] name: shared-mem

[0259] terminationGracePeriodSeconds: 0

[0260] Worker:

[0261] replicas: 3

[0262] template:

[0263] spec:

[0264] nodeSelector:

[0265] accelerator: "NVIDIA-H20-3e"

[0266] containers:

[0267] - image: 172.16.0.151:5000 / mine / vllm / vllm:v0.7.2-dis

[0268] imagePullPolicy: IfNotPresent

[0269] name: worker

[0270] resources:

[0271] limits:

[0272] cpu: 10

[0273] memory: 100G

[0274] nvidia.com / gpu: 4

[0275] requests:

[0276] cpu: 10

[0277] memory: 100G

[0278] nvidia.com / gpu: 4

[0279] volumeMounts:

[0280] - mountPath: / huggingface

[0281] name: ww

[0282] - mountPath: / dev / shm

[0283] name: shared-mem

[0284] command:

[0285] - / bin / sh

[0286] - -c

[0287] - / vllm-workspace / start_worker.sh

[0288] volumes:

[0289] - hostPath:

[0290] path: / data0 / Models / DeepSeek-R1-671B

[0291] type: Directory

[0292] name: ww

[0293] - emptyDir:

[0294] medium: Memory

[0295] sizeLimit: 10.24G

[0296] name: shared-mem

[0297] terminationGracePeriodSeconds: 0

[0298] replicas: 1

[0299] slots: 1

[0300] sshAuthMountPath: / root / .ssh

[0301] It can be seen that the difference between the startup scheduling file and the initial startup file is that the code part of the inference mirror part in the initial startup file code is deleted. Specifically, the code content is as follows:

[0302] " - command:

[0303] - / bin / sh

[0304] - -c

[0305] - / vllm-workspace / start.sh --tensor-parallel-size 4 --pipeline-parallel-size 4 --model-path / huggingface / --model-id model”

[0306] At the same time, the startup scheduling file also adds relevant information about the running process file, and the specific code is as follows:

[0307] The "-snashpot" code field has been added to "image: 192.168.16.0.151:5000 / mine / vllm / vllm:v0.7.2-snashpot." "snashpot" is an example of a running process file in image format. This means that the inference image and its startup instructions no longer need to be configured in the startup schedule file.

[0308] See also Figure 4 , Figure 4 A flowchart illustrating an embodiment of expanding the inference service capacity for this application.

[0309] S301: Obtain the current expansion instruction, and parse the inference service indicated by it as the service to be verified.

[0310] In this embodiment, the current expansion instruction refers to the instruction currently obtained and based on which the inference service expansion needs to be performed.

[0311] In this embodiment, the current expansion instruction can be parsed to obtain the reasoning service executed for expansion, and the reasoning service obtained by parsing is used as the service to be verified. The content to be verified is to evaluate the startup files (initial startup files and startup scheduling files) on which the expansion relies.

[0312] S302: Identify whether the service to be verified belongs to the target reasoning service.

[0313] In this embodiment, when it is determined that the service to be verified belongs to the target reasoning service, step S303 is executed. When it is determined that the service to be verified does not belong to the target reasoning service, step S313 is executed.

[0314] Optionally, the existence of a running process file and / or a startup schedule file for the service to be verified can be identified to determine whether it belongs to the target reasoning service. That is, if a running process file and / or a startup schedule file exist for the service to be verified, the service to be verified is considered to belong to the target reasoning service; if a running process file and / or a startup schedule file do not exist for the service to be verified, the service to be verified is considered not to belong to the target reasoning service, that is, the service to be verified belongs to a regular reasoning service.

[0315] S303: The service to be verified is used as the first inference service, and its startup scheduling file is retrieved.

[0316] In this embodiment, in response to the fact that the service to be verified belongs to the target inference service, it can be considered that it can be expanded using the startup scheduling file to improve its creation and startup efficiency during expansion. Therefore, the service to be verified can be used as the first inference service, and the startup scheduling file of the service to be verified can be obtained.

[0317] S304: Evaluate a scheduling factor of the processor specification associated with the first inference service.

[0318] In this embodiment, considering that there may be insufficient remaining resources of computing nodes and / or associated processors when the first inference service is expanded, a scheduling factor representing whether the remaining resources are sufficient may be evaluated.

[0319] S305: Check whether the associated processor has sufficient resources to expand the first inference service.

[0320] In this embodiment, when the associated processor has sufficient resources to expand the first reasoning service, step S306 is executed. When the associated processor does not have sufficient resources to expand the first reasoning service, step S308 is executed.

[0321] The scheduling factor can be used to verify whether the associated processor has sufficient resources to expand the first inference service. Optionally, in this embodiment, whether the associated processor of the first inference service has sufficient resources can be determined; or, whether the associated processor of the first inference service has sufficient resources can be determined to indicate whether the associated processor has sufficient resources; or, whether the associated processor of the first inference service and its deployed computing nodes have sufficient resources can be determined. Only when both have sufficient resources is the associated processor considered to have sufficient resources; otherwise, the associated processor is considered to have insufficient resources. This takes into account both the associated processor and its computing nodes, and can improve the reliability of the expansion of the first inference service, thereby further improving the performance of the inference service and the computing cluster.

[0322] S306: The resource scheduling associated processor executes its startup scheduling file and writes the running process file to create and run the new first reasoning service as the second reasoning service.

[0323] In this embodiment, in response to the associated processor having sufficient resources to expand the first inference service, resource scheduling can be performed on the associated processor. The startup scheduling file for the first inference service is executed and the running process file is written to create and run a new first inference service as a second inference service. In other words, the second inference service is a new first inference service formed by the expansion.

[0324] S307: Load the second inference service to the load balancer.

[0325] In this embodiment, in response to the second reasoning service being created, the second reasoning service can be loaded into the load balancer, so that the load balancer knows that the second reasoning service is in a ready state and can schedule reasoning tasks to it.

[0326] S308: Delete the first inference service from the associated processor.

[0327] In this embodiment, in response to the associated processor not having sufficient resources to expand the first reasoning service, that is, the associated processor may be deemed to have insufficient resources, the first reasoning service may be deleted from the associated processor so that the first reasoning service and its second reasoning service can be reliably deployed.

[0328] S309: Inferring the inference image in the initial startup file to select another processor with sufficient resources as the update processor.

[0329] In this embodiment, in response to the associated processor not having sufficient resources to expand the first inference service, another processor for deploying the first inference service and its expanded second inference service can be selected. Therefore, the initial startup file of the first inference service can be obtained to extract the inference image in the initial startup file and perform re-inference at the current moment, so as to select another processor with sufficient resources as the update processor.

[0330] S310: Deploy the first inference service on the update processor using the initial startup file, and use the update processor as a new associated processor.

[0331] In this embodiment, in response to completing the inference and selection of the update processor, the first inference service may be deployed on the update processor using the initial startup file, and the update processor may be used as a new associated processor.

[0332] S311: Deploy the second inference service on the new associated processor.

[0333] In this embodiment, the second inference service can also be deployed with the newly inferred associated processor. Optionally, the second inference service can be deployed synchronously or asynchronously when the first inference service is deployed. Alternatively, when the first inference service is deployed and running, a new running process file and startup scheduling file are created for it, and the second inference service is deployed based on the newly created running process file and startup scheduling file. This is not strictly limited here.

[0334] S312: Load the first inference service and the second inference service to the load balancer.

[0335] In this embodiment, in response to the first and second inference services completing deployment on the new associated processor, the first and second inference services can be loaded onto the load balancer. This allows the load balancer to be aware that both the first and second inference services are in a ready state and to schedule inference tasks to them.

[0336] S313: The service to be verified is used as the third reasoning service, and its initial startup file is used to create a new third reasoning service as the fourth reasoning service.

[0337] In this embodiment, in response to the service to be verified not belonging to the target reasoning service, it can be considered that the service to be verified belongs to the regular reasoning service, and the service to be verified can be used as the third reasoning service, and its initial startup file can be used to create a new third reasoning service as the fourth reasoning service.

[0338] S314: Load the fourth inference service to the load balancer.

[0339] In this embodiment, in response to the completion of the creation of the fourth inference service, the fourth inference service may be loaded into the load balancer, so that the load balancer knows that the fourth inference service is in a ready state and can schedule inference tasks to the fourth inference service.

[0340] S315: Determine whether the expansion and startup of the inference service indicated by the current expansion instruction are completed.

[0341] That is to say, in this embodiment, in response to the startup request being an expansion request, the startup scheduling file of the first reasoning service can be obtained as the target startup file, and the running process file of the first reasoning service can be obtained as the target process file. The target startup file is executed and the target process file is written to create a new first reasoning service as the second reasoning service. Resource usage information is obtained when the computing cluster schedules the first reasoning service. The computing node corresponding to the physical resource in the resource usage information is identified as the associated node. When scheduling the second reasoning service, the physical resources of the associated node are used to perform the reasoning task. With such a design, when creating a new first reasoning service, that is, its second reasoning service, the node resources, processor resources and other resource usage involved in the second reasoning service can be strongly matched with the resource usage of the first reasoning service, thereby facilitating the reliable operation of the second reasoning service and the reliable execution of the reasoning task, so as to ensure the performance of the first reasoning service and the second reasoning service.

[0342] In response to obtaining a startup request for the third reasoning service, the third reasoning service is restarted using the initial startup file of the third reasoning service. The third reasoning service is a regular reasoning service. As previously described, the regular reasoning service is formed by the current reasoning service that failed to create the startup schedule file.

[0343] like Figure 5 As shown in the example, Figure 5 This is a comparative diagram of an embodiment of the conventional reasoning service startup and the target reasoning service startup of this application. Combined with the implementation methods described above, it can be seen that when starting the target reasoning service, the process of immediately reasoning on the deployed processors when starting the conventional reasoning service can be omitted, thereby improving the efficiency of the reasoning service startup.

[0344] In layman's terms, taking the running process file as a snapshot image as an example, when the service management device receives a request to expand the inference service or an administrator instructs to create an inference service with the same configuration, it can be considered as obtaining a request to start the inference service. The service management device can determine whether a snapshot image exists for the inference service.

[0345] When a snapshot image of an inference service exists, the inference service can be used as the first inference service. The first inference service can be started based on the startup request based on the new inference service deployment script deployscripts_snash (i.e., the startup scheduling file) and the snapshot image inferenceImage_snash (i.e., the running process file).

[0346] For example, if the startup process is an expansion process, the GPU model and CPU model required by the first inference service can be used as scheduling factors for the expanded inference service. The scheduler then selects a compute node with GPU and / or CPU specifications that strictly match the first inference service to start the first inference service. If the compute cluster does not have the required number of GPU cards, GPU card models, and CPU models, the inference service rollback is triggered, deleting the first inference service and reusing the default inference service deployment script deployscripts_defalut (i.e., the initial startup file) to create the first inference service.

[0347] If the inference service snapshot image inferenceImage_snash does not exist, the default inference service deployment script deployscripts_default is used to create the inference service. The scheduler selects a node that meets the required number of GPU cards and CPUs based on the cluster's GPU resource usage and starts the inference service. Furthermore, you can create a snapshot image for the inference service, using it as the current inference service.

[0348] When the inference service requested by the start request runs successfully, access information of the inference service replica can be loaded into the complex balancer to be scheduled by the load balancer.

[0349] The embodiments of the present application provide a service control method, and the working principle of the service control method is described in detail in combination with the execution process of the service control method.

[0350] See also Figure 6 , Figure 6 This is a flow chart of an embodiment of the service control method of this application:

[0351] S401: In response to the initial execution of the current reasoning service, determining whether the current reasoning service is successfully executed.

[0352] S402: In response to the current inference service running successfully, executing the service startup method.

[0353] S403: Schedule the current inference service to execute the inference task.

[0354] In simple terms, large-scale AI (Artificial Intelligence) inference services typically require the training framework to load model files from shared storage and read them into GPU memory. Large model files can take a long time to load, potentially taking 30 minutes or even longer. This can slow down the inference service startup and waste GPU resources.

[0355] Especially in the online environment of inference services, when the inference service has a large access pressure, it can usually trigger automatic expansion of the inference service. The newly added inference service formed by the expansion also takes a long time to operate normally, and during this period, the service access pressure cannot be quickly relieved.

[0356] To this end, the service startup method provided in this application proposes a unified processor process snapshot generation and recovery method based on running process files such as process images, which can be used in processing such as CPU, GPU, and acceleration cards.

[0357] Taking the CPU and GPU as an example, CPU and GPU process snapshots can be encapsulated as container images, which supports node migration and fast startup across compute nodes. Specifically, after the inference service successfully runs for the first time, you can add a CPU and GPU process snapshot operation to save the initial running GPU and CPU process states as container images.

[0358] If so, when the inference service needs to be replicated or the platform creates an inference service with the same specifications, the resource scheduling mechanism that meets the GPU and CPU specifications can be used to select a compute node that matches the snapshot image and quickly start the inference service using the snapshot container image. This reduces service startup time and ensures that the inference service can run when sufficient GPU resources are available. Furthermore, if cluster resources do not meet the GPU and CPU specifications, the default inference service deployment script can be used to deploy the inference service, ensuring reliable deployment and startup of the inference service and optimizing the performance of the inference service and the computing cluster.

[0359] In other words, this application is an optimization solution for the startup mechanism of large-model inference services. When the AI ​​platform manages inference services, it can add a GPU process snapshot generation and use mechanism, optimize and remove the process of loading model files into GPU memory, and thus reduce the startup time of the inference service. When the inference service is deployed for the first time, when the inference service runs successfully, the platform first takes a snapshot of the CPU process and GPU process of the inference service. After the snapshot is completed, the inference service is exposed to participate in the execution of the inference task. At the same time, an asynchronous process is started to convert the snapshot file into a container image and upload it to the image warehouse. It can define an inference service deployment script based on the snapshot image. When the administrator creates an inference service of the same specification, or expands the inference service, it first determines whether there is a snapshot image, and selects a computing node with the same GPU specification and the same CPU specification to run the inference service based on the snapshot image during resource scheduling. When the inference service starts, there is no need to read the model file, thereby speeding up the startup of the inference service.

[0360] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, which is not strictly limited here.

[0361] An embodiment of the present application also provides a service management device.

[0362] In this embodiment, the service management device may include a connection module and a control module.

[0363] The connection module can be connected to the computing cluster.

[0364] The control module can be connected to the connection module to implement the service startup method or service control method as described in any of the above embodiments.

[0365] Specifically, the service startup method may at least include: in response to the current reasoning service running, obtaining the processor process when the current reasoning service runs in the computing cluster, recording the processor process to form a running process file; creating an executable file adapted to the running process file as a startup scheduling file; wherein, the startup scheduling file is used to start the current reasoning service; using the current reasoning service that has completed the creation of the startup scheduling file as the target reasoning service; in response to obtaining a startup request for the first reasoning service, executing its startup scheduling file and writing the running process file to start the first reasoning service; wherein, the first reasoning service belongs to the target reasoning service.

[0366] The service control method may at least include: in response to the initial operation of the current reasoning service, determining whether the current reasoning service is successfully executed; in response to the successful operation of the current reasoning service, executing the service startup method as described in any of the above embodiments; and scheduling the current reasoning service to execute the reasoning task.

[0367] Furthermore, for descriptions of features in the embodiments corresponding to the service management device, reference can be made to the relevant descriptions of the embodiments corresponding to the service management device method and the service control method, which will not be repeated here.

[0368] An embodiment of the present application also provides an electronic device.

[0369] An electronic device may include a memory and a processor. The memory stores a computer program, and the processor is configured to execute the computer program to perform the steps of any of the aforementioned service activation methods or service control method embodiments. That is, the memory is configured to store the computer program. When the processor executes the computer program, the steps of the aforementioned service activation method are implemented; or, the steps of the aforementioned service control method are implemented.

[0370] Embodiments of the present application further provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute, when executed, the steps of any of the aforementioned service startup method or service control method embodiments. Specifically, when the computer program is executed by a processor, the steps of the aforementioned service startup method or the aforementioned service control method are implemented.

[0371] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0372] Embodiments of the present application also provide a computer program product. The computer program product may include a computer program that, when executed by a processor, implements the steps of any of the aforementioned service startup method or service control method embodiments. Specifically, when executed by a processor, the computer program implements the steps of the aforementioned service startup method; or, alternatively, the steps of the aforementioned service control method.

[0373] Embodiments of the present application also provide another computer program product. The computer program product may include a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-mentioned service startup method or service control method embodiments.

[0374] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0375] The above describes in detail a service startup method, a service control method, an electronic device, a computer-readable storage medium, and a computer program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A service startup method, characterized in that: The service startup method includes: In response to the current reasoning service running, obtaining a processor process when the current reasoning service runs on the computing cluster, and recording the processor process to form a running process file; Creating an executable file adapted to the running process file as a startup scheduling file; wherein the startup scheduling file is used to start the current reasoning service; The current inference service that has completed the creation of the startup scheduling file is used as the target inference service; In response to obtaining a startup request for a first reasoning service, executing its startup scheduling file and writing the running process file to start the first reasoning service; wherein the first reasoning service belongs to the target reasoning service; The response before the current reasoning service is executed includes: Obtaining the initial startup file of the current inference service; Reading the processor type and the number of processors of each type indicated by the initial startup file as the target processor and its target number; Selecting the target number of target processors in the computing cluster as associated processors; The current reasoning service is deployed on a computing node carrying the associated processor; and a processor process of the current reasoning service on the associated processor is obtained to form the running process file.

2. The service startup method according to claim 1, characterized in that: The step of creating an executable file adapted to the running process file as a startup scheduling file includes: Obtaining an initial startup file for deploying and running the current inference service to the computing cluster; Control the hardware information and computing resources constrained in the startup scheduling file to match the initial startup file; wherein the computing resources and the hardware information are associated with an associated processor, and the associated processor is a processor where the processor process exists.

3. The service startup method according to claim 1, characterized in that: The current inference service that has completed the creation of the startup scheduling file as the target inference service includes: Determine whether the startup scheduling file of the current inference service is created; In response to completion of creation of the scheduling file, using the current inference service as the target inference service; The target inference service is loaded to the load balancer of the computing cluster, so that the target inference service is scheduled by the load balancer to execute the inference task.

4. The service startup method according to claim 3, characterized in that: The response to the current reasoning service operation includes: Obtaining the initial startup file of the current inference service; Running the initial startup file to load the inference model on which the current inference service relies; Determining whether the current reasoning service is in a ready state based on the loading state of the reasoning model; In response to the current reasoning service being in the ready state, it is determined that the current reasoning service is running, and the current reasoning service is not temporarily loaded to the load balancer.

5. The service startup method according to claim 4, characterized in that: The obtaining of the initial startup file of the current reasoning service includes: Obtain the model file and the inference framework image of the current inference service as initial information; Parsing the processor type and the number of processors of each type, scaling indicators, container images, and startup instructions indicated by the initial information as resource requirement information for the current inference service; wherein the processor type includes at least one of a central processing unit, a graphics processing unit, and an accelerator processing unit; The initial startup file is generated by combining the initial information and the resource requirement information.

6. The service startup method according to claim 1, characterized in that: After creating an executable file adapted to the running process file as a startup scheduling file, the method further includes: connecting the first storage unit and the second storage unit; The running process file is stored in the first storage unit, and the startup scheduling file is stored in the second storage unit, and both the running process file and the startup scheduling file carry the service tag of the current inference service; when the current inference service is created, its running process file and startup scheduling file are obtained from the first storage unit and the second storage unit.

7. The service startup method according to claim 1, characterized in that: The executing the startup scheduling file and writing the running process file to start the first reasoning service also includes: In response to the startup request being a capacity expansion request, obtaining a startup scheduling file of the first reasoning service as a target startup file, and obtaining a running process file of the first reasoning service as a target process file; executing the target startup file and writing the target process file to create a new first inference service as a second inference service; Obtaining resource usage information when the computing cluster schedules the first inference service; wherein the computing cluster includes multiple computing nodes; Identifying a computing node corresponding to a physical resource in the resource usage information as an associated node; When scheduling the second inference service, the physical resources of the associated node are used to perform the inference task.

8. The service startup method according to claim 1 or 7, characterized in that: The obtaining of the processor process when the current inference service is running on the computing cluster and recording the processor process to form a running process file includes: Pre-acquire computing nodes included in the computing cluster, deploy a process inspection component to the computing nodes, and control a container management component and a node communication component of the computing nodes to enable a container snapshot function; Scheduling the process check component to perform process check processing on the computing node to obtain the processor process of the current reasoning service; Creating a process snapshot of the processor process by using a container snapshot function; The process snapshot is formatted into a mirror image format as the running process file.

9. The service startup method according to claim 1, characterized in that: After creating an executable file adapted to the running process file as a startup scheduling file, the method further includes: In response to a failure in creating a startup scheduling file for the current reasoning service, treating it as a regular reasoning service; In response to obtaining a startup request for a third reasoning service, the third reasoning service is restarted using an initial startup file of the third reasoning service; wherein the third reasoning service belongs to the conventional reasoning service.

10. The service startup method according to claim 1 or 9, characterized in that: The service startup method further includes: Get the current expansion instruction; Parsing the reasoning service indicated by the current expansion instruction as the service to be verified; Identifying whether the service to be verified belongs to the target reasoning service; In response to the service to be verified belonging to the target reasoning service, it is determined to be the first reasoning service; otherwise, it is determined to be a third reasoning service belonging to a regular reasoning service.

11. A service control method, characterized in that: The service control method includes: In response to the initial execution of the current reasoning service, determining whether the current reasoning service is successfully executed; In response to the current reasoning service running successfully, executing the service startup method according to any one of claims 1 to 10; The current reasoning service is scheduled to execute the reasoning task.

12. An electronic device, characterized in that: The electronic device comprises: memory for storing computer programs; A processor, configured to implement the steps of the service startup method according to any one of claims 1 to 10 when executing the computer program; or implement the steps of the service control method according to claim 11.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the service startup method according to any one of claims 1 to 10; or implements the steps of the service control method according to claim 11.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the service startup method according to any one of claims 1 to 10; or implements the steps of the service control method according to claim 11.

Images