General distributed execution method, device and system based on Ray framework and medium

Abstract

The invention discloses a general distributed execution method, device and system based on a Ray framework and a medium. The method comprises the steps that a Ray main node and a working node are started through a task runner of a cloud platform, and environment variables are injected for task configuration; environment variables injected by each working node are obtained and analyzed, the number of inference instances is automatically calculated according to resources needed by tasks, and an inference engine is started; monitoring a health check port of a local reasoning instance of each working node, and after the local reasoning instance is successfully started, reporting a URL of the successfully started local reasoning instance to the Ray host process; the Ray host process collects all reasoning instance URLs, and a sglang router is started and configured; and monitoring the state of the sglang router, and running the task script after the state of the router is confirmed to be ready. According to the method, coordination and execution of the multi-node reasoning task can be completed on a mainstream cloud platform with minimum configuration, so that the development cost is remarkably reduced, and the experiment efficiency of a large-scale model in evaluation and sampling tasks can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed computing and artificial intelligence reasoning, and particularly relates to a general distributed execution method, device, system and medium based on a Ray framework. BACKGROUND

[0002] The distributed reasoning method is a systematic solution proposed to solve the bottleneck of modern large-scale artificial intelligence models in computing resources, efficiency and scalability. At present, the existing distributed solutions such as Swarm (a lightweight multi-agent orchestration framework) of HuggingFace (an open-source machine learning platform) need to strongly depend on specific cluster environments such as Docker (an open-source application container engine) and Slurm (an open-source cluster management and job scheduling system), resulting in the need to rewrite the underlying communication code for cross-platform deployment; in addition, in order to test the model performance on multiple evaluation tasks, the existing distributed solutions often need to integrate the evaluation data with the internal framework, or rewrite a large amount of open-source evaluation code to realize distributed reasoning, thereby greatly reducing the task adaptation efficiency and easily increasing the potential errors introduced by human modification of the code; and in the sampling task aspect, the existing distributed solutions need to manually schedule multiple tasks for each data partition to sample in parallel, thereby greatly increasing the development and operation complexity. SUMMARY

[0003] The present application aims to at least partially solve the technical problems in the related art. To this end, the first object of the present application is to provide a general distributed execution method based on a Ray framework, which does not need to integrate specific cluster environments and can coordinate and execute multi-node reasoning tasks on mainstream cloud platforms with minimal configuration, thereby significantly reducing development costs and improving the experimental efficiency of large-scale models in evaluation and sampling tasks.

[0004] The second object of the present application is to provide a general distributed execution device based on a Ray framework.

[0005] The third object of the present application is to provide a computer-readable storage medium.

[0006] The fourth object of the present application is to provide a distributed reasoning system.

[0007] To achieve the above objects, the present application realizes the following technical solutions:

[0008] A general distributed execution method based on a Ray framework, comprising:

[0009] starting a Ray master node and a worker node through a task runner of a cloud platform, and injecting environment variables to each worker node for task configuration;

[0010] Acquire and parse the environment variables injected by each worker node to obtain the required resources of the task, automatically calculate the number of inference instances according to the required resources of the task, and start the inference engine;

[0011] Monitor the health check port of each worker node local inference instance, and after confirming that the local inference instance is healthy and successfully started, report the URL of the successfully started local inference instance to the Ray master process;

[0012] After the Ray master process collects all the inference instance URLs, start and configure the sglang router;

[0013] Monitor the state of the sglang router through the Ray master node, and after confirming that the router state is ready, run the task script.

[0014] Preferably, the method further comprises: constructing an isolated virtual environment to enable the task script to run in the isolated virtual environment, and using environment variables to achieve connection configuration with the sglang router to realize network connectivity.

[0015] Preferably, the environment variables at least include tensor parallel size, single card GPU memory utilization upper limit and inference engine type.

[0016] Preferably, when configuring the sglang router, the configuration is performed according to the strategy specified by the environment variables.

[0017] Preferably, the strategy specified by the environment variables includes cache-aware routing, load balancing routing and hybrid routing strategy, wherein the hybrid routing strategy is a mixed strategy including cache-aware routing and load balancing routing.

[0018] Preferably, the cache-aware routing strategy detects the prompt prefix of a new request and matches the existing cache of the worker node, wherein the request type at least includes long text generation and short interactive question and answer;

[0019] The load balancing routing strategy is to monitor the queue length and GPU memory utilization of each worker node in real time, so as to preferentially allocate new requests to the worker node with the shortest queue length and sufficient GPU memory.

[0020] Preferably, the router state readiness determination conditions include:

[0021] Detecting that the router service has been continuously active for more than a preset value;

[0022] Detecting that the resource load state includes CPU or memory usage, and request queue length, respectively less than the corresponding set value;

[0023] Detecting that all worker node URLs are consistent with the Ray global control service registration list;

[0024] When the determination conditions are simultaneously satisfied, the router state is determined to be ready.

[0025] To achieve the above object, the second aspect of the present application provides a general distributed execution device based on Ray framework, comprising:

[0026] A node management module is configured to start Ray master nodes and worker nodes through a task runner of a cloud platform, and inject environment variables into each worker node for task configuration.

[0027] An inference instance management module is configured to obtain and parse the environment variables injected by each worker node to obtain the required resources of the task, automatically calculate the number of inference instances according to the required resources of the task, and start an inference engine.

[0028] A health monitoring module is configured to monitor the health check port of the local inference instance of each worker node, and report the URL of the successfully started local inference instance to the Ray master process after confirming that the local inference instance is healthy and successfully started.

[0029] A routing configuration module is configured to start and configure the sglang router after the Ray master process collects all inference instance URLs.

[0030] A state coordination module is configured to monitor the state of the sglang router through the Ray master node, and generate a ready signal after confirming that the router state is ready.

[0031] A virtualization execution module is configured to build an isolated virtual environment, implement connection configuration with the sglang router by using environment variables, and run a task script in the isolated virtual environment after the ready signal is generated.

[0032] To achieve the above object, the third aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the general distributed execution method based on the Ray framework.

[0033] To achieve the above object, the fourth aspect of the present application provides a distributed inference system, comprising the general distributed execution device based on the Ray framework, the computer readable storage medium, or a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the general distributed execution method based on the Ray framework.

[0034] The present application has at least the following technical effects:

[0035] (1) The distributed scheme provided by the present application has low dependence on the underlying scheduling system, and the entire scheme does not require cluster support for open source application container engines and open source cluster management and job scheduling systems, so that distributed execution of tasks can be realized, and the deployment of the distributed scheme has high flexibility.

[0036] (2) In order to test the model performance on multiple evaluation tasks, the existing distributed scheme often needs to integrate the evaluation data with the internal framework, or to rewrite a large amount of open source evaluation code to realize distributed reasoning. However, the present application only needs to configure the task-related logic through environment variables, so that the task can be quickly adapted. This change greatly improves the experimental efficiency and significantly reduces the potential errors introduced by manual code modification.

[0037] (3) In terms of sampling tasks, the method also significantly simplifies the process. The existing distributed scheme needs to manually schedule multiple tasks for each data partition to sample in parallel, while the present application only needs to adjust the environment variables to complete the configuration, greatly reducing the complexity of development and operation.

[0038] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A flowchart of a general distributed execution method based on a Ray framework for an embodiment of the present application.

[0040] Figure 2 A structure block diagram of a general distributed execution device based on a Ray framework for an embodiment of the present application.

[0041] Figure 3 An entity structure diagram of a distributed reasoning system is exemplified. DETAILED DESCRIPTION

[0042] The embodiments are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0043] The Ray framework-based general distributed execution method, device, system and medium of the embodiments are described below with reference to the accompanying drawings.

[0044] It should be noted that the underlying of the general distributed execution method based on the Ray framework in the embodiment is built based on the following components, that is, Ray (an open source distributed computing framework), sglang-router (an intelligent router specially designed for distributed deployment), and inference engine (such as vLLM or sglang, which supports extension to other engines such as TensorRT). Ray is used for multi-node resource coordination and job scheduling, and sglang-router is used for distributed routing of inference requests, that is, subsequent sglang router.

[0045] Figure 1 The flowchart of the general distributed execution method based on the Ray framework in the embodiment is shown in FIG. 1. Figure 1 As shown in the figure, the method comprises the following steps.

[0046] Step S100: Start the Ray master node and worker nodes through the task runner of the cloud platform, and inject environment variables into each worker node for task configuration.

[0047] On a common cloud platform, the Ray master node and worker nodes can be started through the task runner system, and then environment variables are injected into the worker nodes for task configuration. The node is an independent computing entity in a distributed system, which can be a physical server, a virtual machine, a container or a cloud instance, has independent computing, storage and network capabilities, and cooperates with other nodes through the network to complete tasks. In the embodiment, the task runner can be used to manage the node life cycle and perform node initialization. The master node acts as the brain of the cluster, and the worker nodes dynamically join the cluster to realize cluster management of the nodes. In the embodiment, the injected environment variables are used to control resource allocation, scheduling strategy and runtime behavior.

[0048] Step S200: Obtain and parse the environment variables injected by each worker node to obtain the required resources of the task, automatically calculate the number of inference instances according to the required resources of the task, and start the inference engine.

[0049] In the embodiment, each worker node automatically calculates the number of inference instances according to the required resources of the task such as tensor parallel size and GPU (Graphics Processing Unit, graphics processor) quantity, and starts the selected inference engine.

[0050] Specifically, the environment variables injected to each worker node, such as core parameters including tensor parallel size, upper limit of single-card memory utilization, inference engine type, etc., are parsed to obtain the required resources of the task, and then the number of inference instances is automatically calculated according to the required resources of the task. In this embodiment, resource declaration is completed through environment variables such as tensor parallel size, GPU quantity, and inference engine type, without the need to modify the underlying code. The developer only needs to adjust the environment variable values, and the system automatically calculates the instance number and resource allocation, which can avoid manually writing distributed scheduling logic. Thus, this embodiment only needs to be configured through environment variables, the resource requirements and task types are declared through environment variables, the hardware resource allocation such as GPU quantity and data sharding strategy is abstracted as environment variables, the framework automatically parses the environment variables, and a distributed task flow is generated as needed to configure the task-related logic, so that the task is quickly adapted, which greatly improves the experimental efficiency and significantly reduces the potential errors introduced by manual code modification.

[0051] In addition, in terms of sampling tasks, this embodiment only needs to adjust environment variables, such as declaring corresponding environment variables for parameters such as data sharding quantity, sampling quantity of each partition, and single-node GPU quantity, and then the underlying framework automatically generates a parallel task topology according to the environment variables, dynamically allocates resources according to the parallel task topology, and processes inter-node communication to complete the configuration, without the need to manually schedule multiple tasks for each data partition to sample in parallel, thereby greatly reducing the complexity of development and operation.

[0052] It should be noted that, unlike traditional inference services, this embodiment organizes the inference process in a task running mode, and the inference engine will automatically shut down after the task is completed, saving the step of manually cleaning up resources, which can effectively release the GPU and improve resource utilization.

[0053] Step S300: Monitor the health check port of each worker node local inference instance, and after confirming that the local inference instance is healthy and starts successfully, the URL of the successfully started local inference instance is reported to the Ray master process.

[0054] Specifically, each worker node continuously checks the health status of the inference instance running locally. For example, the health check port exposed by the service is accessed, and if a predetermined successful response is returned, it is considered that the corresponding inference instance starts successfully. After confirming that the local inference instance is healthy, the worker node actively reports the access URL (Uniform Resource Locator, Uniform Resource Locator) of the inference instance to the Ray master process. The reporting content specifically includes the protocol, node IP (Internet Protocol), port, and service path of the inference instance, wherein the reporting is performed only when the health check is passed for the first time to avoid repeated registration.

[0055] Step S400: After Ray master process collects all inference instance URLs, start and configure the sglang router.

[0056] Specifically, the master node of Ray dynamically collects the URL addresses of all inference instances on the work nodes through the global control service (GCS). Ray, as a distributed framework, uniformly manages the cluster state, automatically senses node changes, and does not need manual maintenance of the node list. The sglang router is started as an independent component and dynamically connects all inference instances through the URL list provided by Ray. The sglang router does not need to be manually configured with backend node addresses, but automatically obtains the latest node information through the real-time service discovery mechanism of Ray.

[0057] Among them, when configuring the sglang router, the strategy specified by the environment variable is configured. The strategy specified by the environment variable includes cache-aware routing, load balancing routing, and hybrid routing strategy. Among them, the hybrid routing strategy simultaneously enables the above two strategies.

[0058] The cache-aware routing strategy is suitable for scenarios with high similarity of prompt words, and the load balancing routing strategy is suitable for scenarios with uneven distribution of requests. Among them, the essence of the prompt word is to control the underlying scheduling logic of the router through simple instructions, without the need for users to write complex code. For example, the environment variable is also a prompt word, which is used to specify the health check rules and resource allocation strategy in the subsequent steps. The request types in this embodiment include long text generation, short interactive question and answer, etc. The cache-aware routing strategy specifically detects the prefix of the prompt word of the new request, such as please write a story about spring, and matches the existing cached nodes, such as node A has already processed similar prefixes.

[0059] The load balancing routing strategy in this embodiment is a dynamic strategy, which specifically monitors the queue length and GPU memory utilization of each work node in real time to preferentially allocate new requests to the work node with the shortest queue length and sufficient GPU memory.

[0060] In this embodiment, the routing strategy of the sglang router can be specified through the environment variable, and the user can switch the strategy through the environment variable without the need to restructure the routing code.

[0061] Step S500: Monitor the state of the sglang router through the Ray master node, and run the task script after the router state is confirmed ready.

[0062] In this embodiment, the Ray master node can track the following states of the sglang router in real time through the global control service heartbeat detection mechanism: service activity, resource load state, and routing table consistency. Among them, the service activity is, for example, whether the health check port response status code is 200 (200 is normal), and whether the routing service continuous active time is greater than a preset value such as 30s; the resource load state detection includes CPU or memory usage, request queue length detection, such as whether the request queue backlog is less than a set value such as 100; the routing table consistency is to check whether all worker node URLs are consistent with the Ray global control service registration list.

[0063] When the health check port response status code is the set value, the routing service continuous active time is greater than the preset value, the CPU or memory usage, and the request queue length are all less than the corresponding set value, and all worker node URLs are consistent with the Ray global control service registration list, that is, all conditions are met at the same time, then it is determined that the router state is ready.

[0064] Further, the method further includes constructing an isolated virtual environment to enable the task script to run in the isolated virtual environment, and using an environment variable to achieve connection configuration with the sglang router to realize network connectivity.

[0065] Specifically, the constructed virtual environment is only used to solve the isolation problem of the task script running environment, but does not automatically provide network connection capability. When the task script starts in the isolated virtual environment, there is no network configuration information by default, so it is necessary to externally inject connection parameters.

[0066] In a distributed system, the router address may change dynamically, for example, node expansion or failover. In this embodiment, the environment variable is used to achieve connection configuration with the sglang router, which can realize network connectivity, and the environment variable can allow the task script to obtain the latest routing address in real time, avoiding the connection failure or service interruption caused by manually modifying the code and redeploying when the routing address is directly written into the code (hard coding) but the routing address changes dynamically.

[0067] The general distributed execution method based on the Ray framework does not need to integrate a specific cluster environment, and can complete the coordination and execution of multi-node inference tasks with minimal configuration, that is, environment variable configuration, on mainstream cloud platforms, thereby significantly reducing development costs and improving the experimental efficiency of large-scale models in evaluation and sampling tasks. Two typical application scenarios of this method to reduce costs include quickly running open-source evaluation benchmarks that do not originally support multi-node inference by using a small amount of configuration changes, and performing large-scale synthetic data generation in parallel on multiple nodes without the need to specially write code to handle distributed logic.

[0068] In addition, the method realizes automatic arrangement of an inference process and dynamic scheduling of resources through a modular and configurable architecture, and is especially suitable for scenarios that need to frequently change models, tasks or data sets, and effectively improves engineering efficiency and system flexibility.

[0069] Figure 2 A structural block diagram of a general distributed execution device based on a Ray framework in an embodiment of the application is shown in FIG. 1. Figure 2 As shown in the figure, the general distributed execution device 100 based on the Ray framework includes a node management module 10, an inference instance management module 20, a health monitoring module 30, a routing configuration module 40, a state coordination module 50 and a virtualization execution module 60 connected in sequence.

[0070] The node management module 10 is configured to start a Ray master node and worker nodes through a task runner of a cloud platform, and inject environment variables into each worker node for task configuration. The inference instance management module 20 is configured to obtain and parse the environment variables injected by each worker node, obtain the required resources of the task, automatically calculate the number of inference instances according to the required resources of the task, and start an inference engine. The health monitoring module 30 is configured to monitor a health check port of a local inference instance of each worker node, and after confirming that the local inference instance is healthy and successfully started, report the URL of the successfully started local inference instance to a Ray master process. The routing configuration module 40 is configured to start and configure an sglang router after the Ray master process collects all inference instance URLs. The state coordination module 50 is configured to monitor the state of the sglang router through the Ray master node, and generate a ready signal after confirming that the state of the router is ready. The virtualization execution module 60 is configured to build an isolated virtual environment, implement connection configuration with the sglang router by using the environment variables, and run a task script in the isolated virtual environment after the ready signal is generated.

[0071] In an embodiment of the application, the environment variables at least include a tensor parallel size, an upper limit of single-card GPU memory utilization and an inference engine type.

[0072] In an embodiment of the application, when configuring the sglang router, the routing configuration module 40 configures according to a strategy specified by the environment variables. The strategy specified by the environment variables includes a cache-aware routing, a load balancing routing and a hybrid routing strategy. The hybrid routing strategy is a mixed strategy including the cache-aware routing and the load balancing routing.

[0073] The cache-aware routing strategy is to detect the prompt word prefix of a new request and match the existing cache of a worker node, wherein the request type at least includes long text generation and short interactive question and answer; and the load balancing routing strategy is to monitor the queue length and GPU memory utilization of each worker node in real time, so as to preferentially assign the new request to the worker node with the shortest queue length and sufficient GPU memory.

[0074] In an embodiment of the present application, the state coordination module 50 is specifically configured to determine that the router state is ready when it is detected that the router service continuous active time is greater than a preset value; the resource load state includes that the CPU or memory usage and the request queue length are less than the corresponding set values respectively; the URL of all worker nodes is consistent with the Ray global control service registration list; and all the determination conditions are met at the same time.

[0075] It should be noted that the specific implementation of the Ray framework-based general distributed execution device in the embodiment can refer to the specific implementation of the Ray framework-based general distributed execution method described above. To avoid redundancy, details are not repeated here.

[0076] Further, the present application also provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the Ray framework-based general distributed execution method described above.

[0077] Further, the present application also provides a distributed inference system, which includes the Ray framework-based general distributed execution device described above or the computer readable storage medium described above, or includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the Ray framework-based general distributed execution method described above.

[0078] Figure 3 An entity structure diagram of a distributed inference system is shown. Figure 3 As shown, the distributed inference system can include a processor 210, a communications interface 220, a memory 230 and a communications bus 240, wherein the processor 210, the communications interface 220 and the memory 230 complete communication with each other through the communications bus 240. The processor 210 can invoke the logical instructions in the memory 230 to execute the Ray framework-based general distributed execution method described above.

[0079] In addition, the logic instructions in the memory 230 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0080] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0081] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software and the necessary general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions essentially or the parts that contribute to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0082] Finally, it should be pointed out that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A general distributed execution method based on the Ray framework, characterized in that, include: The Ray master node and worker nodes are started through the task runner of the cloud platform, and environment variables are injected into each worker node to configure the task. Obtain and parse the environment variables injected by each worker node to get the resources required for the task. Calculate the number of inference instances based on the resources required for the task and start the inference engine. Monitor the health check port of the local inference instance on each worker node. After confirming that the local inference instance is healthy and has started successfully, report the URL of the successfully started local inference instance to the Ray main process. After the Ray main process collects all inference instance URLs, the sglang router is started and configured. Monitor the status of the sglang router using the Ray master node, and run the task script after confirming that the router status is ready.

2. The method as described in claim 1, characterized in that, Also includes: An isolated virtual environment is constructed so that the task scripts can run in the isolated virtual environment. At the same time, environment variables are used to configure the connection with the sglang router to achieve network connectivity.

3. The method as described in claim 1, characterized in that, The environment variables include at least the tensor parallel size, the upper limit of single-card memory utilization, and the inference engine type.

4. The method as described in claim 1, characterized in that, When configuring a sglang router, configure it according to the policy specified in the environment variables.

5. The method as described in claim 4, characterized in that, The policies specified by the environment variables include cache-aware routing, load-balanced routing, and hybrid routing policies. The hybrid routing policy is a combination of cache-aware routing and load-balanced routing.

6. The method as described in claim 5, characterized in that, The cache-aware routing strategy detects the prompt word prefix of new requests and matches them with existing cached worker nodes. The request types include at least long text generation and short interactive Q&A. The load balancing routing strategy monitors the queue length and GPU memory utilization of each worker node in real time to prioritize the allocation of new requests to the worker node with the shortest queue length and sufficient GPU memory.

7. The method as described in claim 1, characterized in that, The conditions for determining whether a router is ready include: The router service has been continuously active for a longer period than a preset value. The resource load status is detected, including CPU or memory utilization and request queue length, both of which are less than the corresponding set values. Check that all worker node URLs match the Ray global control service registration list; When all the detection conditions are met simultaneously, the router is considered ready.

8. A general-purpose distributed execution device based on the Ray framework, characterized in that, include: The node management module is used to start Ray master nodes and worker nodes through the task runner of the cloud platform, and to inject environment variables into each worker node for task configuration. The inference instance management module is used to obtain and parse the environment variables injected by each working node, get the resources required for the task, automatically calculate the number of inference instances based on the resources required for the task, and start the inference engine. The health monitoring module is used to monitor the health check port of the local inference instance of each working node. After confirming that the local inference instance is healthy and has started successfully, the URL of the successfully started local inference instance is reported to the Ray main process. The routing configuration module is used to start and configure the sglang router after the Ray main process collects all inference instance URLs; The state coordination module is used to monitor the state of the sglang router through the Ray master node and generate a ready signal after the router state is confirmed to be ready. The virtualization execution module is used to build an isolated virtual environment, use environment variables to configure the connection with the sglang router, and run task scripts in the isolated virtual environment after a ready signal is generated.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

10. A distributed reasoning system, characterized in that, It includes the apparatus of claim 8 or the computer-readable storage medium of claim 9, or includes a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the method of any one of claims 1-7.

Images