A model service distributed inference method and device, equipment and medium

Abstract

This application provides a method, apparatus, device, and medium for distributed inference of model services. The method includes: preparing a base image containing dependency libraries; configuring inference service parameters according to business needs; integrating image information and parameters through a Java client to construct a YAML configuration file defining Ray service application configuration and Ray cluster configuration; and distributing the configuration file to a Kubernetes cluster to start the Ray cluster and distributed inference service. This application can automatically generate configurations through a Java client, combining Ray distributed computing and Kubernetes container orchestration to achieve standardized deployment and elastic scaling of model services, improve resource utilization, reduce deployment and maintenance costs, and adapt to the high-concurrency inference needs of large neural network models.

Description

Technical Field

[0001] This application relates to the field of distributed computing technology, and in particular to a distributed reasoning method, apparatus, device and medium for model services. Background Technology

[0002] With the rapid development of artificial intelligence technology, large neural network models (LNNMs) have demonstrated powerful capabilities in numerous fields. Model inference, as a crucial step in applying trained models to actual prediction tasks, directly impacts the availability of model services and user experience in terms of efficiency, latency, and throughput. To meet the growing demand for online inference and high-concurrency requests, model service technology has emerged to efficiently and stably host and run models and handle client requests.

[0003] The current mainstream model service deployment methods have many drawbacks: single-node deployment is simple to implement, but when faced with large-scale models or high-concurrency requests, computing resources are easily saturated, resulting in obvious performance bottlenecks and extremely poor scalability; although load balancing request distribution improves throughput and fault tolerance, each request still requires a complete model instance on a single backend node to process independently, resulting in high latency for single request processing, low hardware resource utilization, high costs, and linearly increasing costs for horizontal scaling; although model parallelism solves the problem of insufficient memory on a single device, it is complex to implement, highly coupled, has poor versatility, uneven resource utilization, communication overhead may become a bottleneck, and management and maintenance are difficult.

[0004] In summary, existing technologies suffer from low resource utilization and high cost, limited scalability, high latency jitter, weak ability to handle sudden traffic surges, and high deployment and maintenance complexity. There is an urgent need for a more efficient, flexible, scalable, and resource-efficient distributed model inference method and system to overcome these shortcomings and meet the needs of large-scale, high-performance model services. Summary of the Invention

[0005] In view of this, embodiments of this application provide a model service distributed inference method, apparatus, device and medium, which can realize the standardization and configurable deployment of model service distributed inference, improve resource utilization and service scalability, reduce deployment and maintenance costs, and adapt to the high-concurrency inference requirements of large neural network models.

[0006] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a distributed inference method for model services, comprising the following steps: Prepare a base image; wherein the base image includes the dependency libraries required for the model service to run and meets the runtime environment requirements of the distributed inference service; Configure inference service parameters according to business needs; wherein, the inference service parameters include at least the inference framework, service uniform resource locator, model file address, environment variables, and number of instances; A configuration file is built using a Java client; the configuration file integrates the base image information and the inference service parameters, and defines the Ray service application configuration and Ray cluster configuration. The configuration file is distributed to the Kubernetes cluster, the Ray cluster and the corresponding distributed inference service are started, and the distributed deployment and inference operation of the model service are completed.

[0007] Secondly, embodiments of this application also provide a distributed inference apparatus for model services, the apparatus comprising: The base image module is used to prepare the base image; wherein, the base image includes the dependency libraries required for the model service to run and meets the runtime environment requirements of the distributed inference service; The parameter configuration module is used to configure inference service parameters according to business requirements; wherein, the inference service parameters include at least the inference framework, service uniform resource locator, model file address, environment variables, and number of instances; The Java client module is used to build a configuration file through the Java client; wherein, the configuration file integrates the base image information and the inference service parameters, and the configuration file defines the Ray service application configuration and Ray cluster configuration; The deployment and operation module is used to distribute the configuration file to the Kubernetes cluster, start the Ray cluster and the corresponding distributed inference service, and complete the distributed deployment and inference operation of the model service.

[0008] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the model service distributed inference method described in any of the first aspects.

[0009] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the distributed inference method for model services as described in any of the first aspects.

[0010] The embodiments of this application have the following beneficial effects: By constructing a standardized and configurable distributed inference solution for model services, this solution effectively addresses the core pain points of existing technologies, such as low utilization of model inference resources, poor scalability, and complex deployment and maintenance. Specifically, the preparation of the base image enables the establishment of a unified and compatible runtime environment, avoiding service startup failures or inference anomalies caused by missing dependencies or version incompatibility between different deployment nodes, providing stable environmental support for distributed inference. Personalized configuration of inference service parameters can accurately meet the needs of different business scenarios, such as model types and inference throughput, balancing the solution's versatility and flexibility. Automated configuration file construction via Java clients breaks away from the tedious traditional manual configuration process, significantly lowering the deployment threshold, reducing human configuration errors, and improving deployment efficiency. Leveraging Ray distributed computing and Kubernetes container orchestration capabilities, it achieves distributed allocation and cluster collaborative operation of inference tasks. Compared to single-node deployment or simple replica deployment, this significantly improves hardware resource utilization, adapts to the high-concurrency inference requirements of large neural network models, and allows for flexible cluster expansion through configuration, solving the problems of limited scalability and insufficient high-concurrency capabilities in existing technologies, comprehensively improving the stability and efficiency of model inference. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating steps S101-S104 provided in the embodiments of this application; Figure 2 This is a flowchart illustrating steps S201-S203 provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the distributed inference device for model services provided in the embodiments of this application; Figure 4 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0014] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0015] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0016] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0017] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application and is not intended to limit this application.

[0019] See Figure 1 , Figure 1This is a flowchart illustrating steps S101-S104 of the distributed inference method for model services provided in this application embodiment, which will be combined with... Figure 1 Steps S101-S104 are explained below.

[0020] In step S101, a base image is prepared; wherein the base image includes the dependency libraries required for the model service to run and meets the runtime environment requirements of the distributed inference service. In step S102, inference service parameters are configured according to business requirements; wherein, the inference service parameters include at least the inference framework, service uniform resource locator, model file address, environment variables, and number of instances; In step S103, a configuration file is built using a Java client; wherein the configuration file integrates the base image information and the inference service parameters, and defines the Ray service application configuration and Ray cluster configuration; In step S104, the configuration file is distributed to the Kubernetes cluster, the Ray cluster and the corresponding distributed inference service are started, and the distributed deployment and inference operation of the model service are completed.

[0021] First, prepare a base image to build a unified and compatible runtime environment. This avoids service startup failures or inference anomalies caused by missing dependencies or version incompatibility on different deployment nodes. It includes all the necessary dependency components for the model service to run, providing stable environmental support for the subsequent startup of the Ray cluster, node communication, and execution of inference tasks, and avoiding deployment risks caused by environmental differences.

[0022] Next, configure the inference service parameters. Users can flexibly configure the inference framework (inference engine adapted to different models), service unified resource locator (used for service access and identification), model file address (specifying the storage path of the model to be deployed to ensure that the cluster can read the model normally), environment variables (adapting to customized business needs), and number of instances (controlling concurrent processing capabilities) according to actual business scenarios, such as model type (deep learning model, machine learning model, etc.), inference throughput requirements, response latency standards, etc., to take into account the inference service needs in different scenarios.

[0023] Then, a configuration file is built using a Java client to integrate the base image information with the inference service parameters. The configuration file must define both the Ray Service application configuration and the Ray cluster configuration, providing a standardized configuration basis for subsequent cluster startup and service deployment, significantly reducing the deployment threshold and the probability of manual configuration errors.

[0024] Finally, the configuration file is deployed to the Kubernetes cluster and the service is started. Relying on Kubernetes' powerful container orchestration capabilities, the Ray cluster can be quickly deployed, node managed and elastically scheduled. At the same time, combined with Ray's distributed computing capabilities, inference tasks are distributed to cluster nodes to improve inference efficiency and resource utilization, and finally the distributed deployment and inference operation of the model service are completed.

[0025] In some embodiments, the dependency library includes at least the Ray dependency library, the programming language dependency library, and the runtime environment library, and the base image is built based on a preset operating system.

[0026] Ray dependency libraries are the core components supporting distributed inference, directly responsible for node communication, task scheduling, distributed computing, and resource management within the Ray cluster, forming the foundation for collaborative inference across the cluster. Programming language dependency libraries are used to adapt to the programming language environments (such as Python and Java) required for model development and inference, ensuring that model code can be compiled and executed correctly, covering the inference needs of models developed in different languages. Runtime environment libraries are used to supplement basic dependencies at the operating system level, such as system tools and library files, resolving compatibility issues at the operating system level and ensuring stable service operation.

[0027] The default operating system can be flexibly selected according to the actual deployment scenario, such as Linux series operating systems (adapting to the mainstream scenario of containerized deployment), ensuring that the base image can adapt to different deployment environments, while providing stable underlying support for the installation and operation of dependency libraries, avoiding dependency installation failures or service operation abnormalities caused by operating system incompatibility, and improving the building specifications of the base image.

[0028] In some embodiments, the configuration file is in YAML format, and the Ray service application configuration includes at least the application name, import path, runtime environment, and deployment configuration. The deployment configuration includes the number of replicas and Ray role options. The Ray cluster configuration includes at least the Ray version, head node group configuration, and worker node group configuration.

[0029] The configuration file is in YAML format. YAML format is characterized by its simplicity, clarity, and distinct hierarchy. It can accurately describe the various configuration parameters required for containerized deployment, making it easy for Kubernetes clusters to parse quickly. It is also easy for developers to view and modify the configuration, balancing execution efficiency and maintainability. Unlike other configuration formats (such as JSON), it is more suitable for distributed deployment configuration scenarios.

[0030] Ray service application configuration is the deployment configuration for the distributed inference application itself. The application name is used to uniquely identify the inference service instance and avoid conflicts between multiple services in the cluster. The import path specifies the execution code entry point of the inference task to ensure that the cluster can correctly load and execute the inference task. The runtime environment configuration is used to supplement the special dependencies and paths required for the application to run, ensuring that the application is compatible with the cluster environment. The number of replicas in the deployment configuration is used to control the application's concurrent processing capabilities. The Ray role option is used to assign the application's role in the cluster (such as execution node, management node) to adapt to the needs of distributed collaborative inference.

[0031] Ray cluster configuration is for setting up and configuring the Ray cluster architecture. Ray version is used to ensure version compatibility of all components in the cluster (head node, worker nodes) to avoid cluster startup failure or communication anomalies caused by version conflicts. Head node group configuration is used to define the running parameters of the cluster management node, which is the core management core of the cluster. Worker node group configuration is used to define the execution node parameters of inference tasks. The two work together to build a complete Ray distributed cluster architecture, providing hardware and software support for the distributed allocation and execution of inference tasks.

[0032] In some embodiments, the head node group configuration includes a head node template, which defines the container name, image address, environment variables, resource limits, resource requests, port configuration, and volume mount configuration; the worker node group configuration includes a worker node template and elastic scaling parameters, which include the number of replicas, the minimum number of replicas, and the maximum number of replicas.

[0033] As the core management node of the Ray cluster, the head node is responsible for core functions such as cluster node management, task scheduling, and metadata synchronization. For example, the container name is used to identify the head node container and avoid conflicts with other containers; the image address specifies the image required for the head node to run, ensuring that the head node environment is consistent with the cluster; environment variable configuration adapts to customized business needs and cluster communication needs, such as configuring node communication addresses and authentication information; resource limits and resource requests are used to reasonably allocate hardware resources such as CPU and GPU to prevent the head node from experiencing management function lag or resource waste due to insufficient resources.

[0034] The port configuration and volume mount configuration in the head node template ensure the normal operation of the head node. The port configuration is used to open the ports required for communication between the head node and the client and worker nodes, ensuring that nodes within the cluster can communicate with each other and that the client can manage the cluster normally. The volume mount configuration is used to realize the sharing and mounting of model files, inference code, and log files, ensuring that the head node can access the various types of files required, and facilitating unified file management.

[0035] As the core execution nodes for inference tasks in a Ray cluster, worker nodes maintain environmental compatibility with the head node's template configuration, ensuring normal communication and collaborative task execution between nodes. Among the elastic scaling parameters, the replica count sets the initial number of worker nodes, the minimum replica count ensures basic inference capabilities, and the maximum replica count limits maximum resource consumption. These three parameters work together to achieve dynamic elastic scheduling of worker nodes, automatically increasing or decreasing the number of worker nodes based on changes in inference request volume. This avoids excessive inference latency caused by sudden traffic spikes and prevents resource waste during idle periods, addressing the weakness of existing technologies in handling sudden traffic surges.

[0036] In some embodiments, the resource limits and resource requests include at least configuration parameters for the central processing unit (CPU), memory, graphics processing unit (GPU), GPU memory, and the number of GPU cores; the port configuration includes at least a GCS server port, a console port, a client port, and a service port.

[0037] CPU and memory configurations are used to support cluster management, node communication, and auxiliary computing for inference tasks, ensuring stable service operation and avoiding task lag and service crashes due to insufficient CPU or memory. The configuration of GPU, GPU memory, and GPU cores is specifically designed for computationally intensive large model inference scenarios (such as deep learning models and large neural network models). These models require a large amount of parallel computing power, and sufficient GPU resources can significantly improve inference efficiency, reduce inference latency, and meet the needs of high-concurrency inference.

[0038] The GCS server port is the core communication port of the Ray cluster, used for metadata synchronization and task scheduling instruction transmission among all nodes in the cluster, and is the foundation for cluster collaboration. The console port provides the monitoring and management entry point for the cluster, allowing developers to view cluster node status, task execution progress, resource usage, etc., facilitating operation and maintenance management. The client port is used for interaction between the Java client and the Ray cluster, enabling functions such as configuration distribution, task submission, and status query, and is crucial for client-side cluster management. The service port receives external inference requests, enabling the provision of inference services to the outside world and ensuring that external systems can normally call the distributed inference service.

[0039] In some embodiments, see Figure 2 , Figure 2 This is a flowchart illustrating steps S201-S203 provided in an embodiment of this application. The process of building the configuration file by the Java client includes steps S201-S203, which will be explained in conjunction with each step.

[0040] In step S201, a Ray cluster configuration object is constructed, which integrates the head node group configuration and the worker node group configuration. In step S202, a service configuration string is constructed, which defines the deployment parameters of the Ray service application; In step S203, the Ray cluster configuration object and service configuration string are integrated into a YAML format configuration file.

[0041] Building a Ray cluster configuration object is the first step in generating configuration files. The core of this process is to standardize and structurally integrate the configuration information of the head node group and worker node group, forming a unified cluster configuration object. This process organizes scattered node configuration parameters (such as resource configuration, port configuration, and elastic scaling parameters), avoiding omissions or conflicts, laying the foundation for subsequent integration with service configurations, and facilitating configuration modification and maintenance.

[0042] Building the service configuration string is the second step in generating the configuration file. The core is to define the deployment parameters of the Ray service application, including the application name, import path, runtime environment, deployment configuration, etc. These parameters are converted into a standardized string format to ensure compatibility with the cluster configuration object, while also adapting to the parsing requirements of the YAML format, and clearly defining the running rules and concurrent processing capabilities of the inference application.

[0043] Finally, the two types of configuration information are integrated into a YAML format configuration file. The Java client uses preset integration logic to merge the structured Ray cluster configuration object with the standardized service configuration string, automatically generating a complete YAML configuration file. This replaces the traditional method of manually writing YAML files, which not only reduces deployment difficulty and the probability of manual configuration errors, but also improves the maintainability and flexibility of the configuration. It can quickly adapt to the configuration adjustment needs under different business scenarios, realizing the automation and standardization of configuration generation.

[0044] In some embodiments, a log collection step is also included: adding a log collection configuration to the configuration file, the log collection configuration being built based on the Fluent Bit tool, for collecting and outputting the runtime logs of the Ray service and the distributed inference service.

[0045] The purpose of this application embodiment is to integrate the log collection configuration into the constructed configuration file, so as to realize the synchronous deployment and management of the log collection function and the distributed inference service. There is no need to deploy the log collection component separately, which greatly simplifies the deployment process, reduces the operation and maintenance cost, and ensures that the log collection function is compatible with the cluster and service operation environment, avoiding configuration conflicts or log collection failures caused by separate deployment.

[0046] The log collection configuration is built on the Fluent Bit tool, which is lightweight, efficient, and has low resource consumption. It can adapt to the log collection needs in containerized environments. Compared with other log collection tools (such as Fluentd), it is more suitable for deployment in resource-constrained or high-concurrency inference scenarios, and will not consume too much CPU and memory resources, thus avoiding affecting the running performance of the inference service.

[0047] The core function of the log collection configuration is to collect all the runtime logs of the Ray service and the distributed inference service, including the runtime status logs of the Ray cluster head node and worker nodes, the execution logs of inference tasks (such as task submission, execution progress, and execution results), and error logs (such as service crashes, task failures, and node communication anomalies). This provides accurate data support for service operation and maintenance and problem troubleshooting, enabling rapid location of fault causes and improving the maintainability and stability of the service.

[0048] In some embodiments, the log collection configuration includes a log input path, log tags, refresh interval, and log output method, wherein the log input path points to the log storage directory of the Ray service.

[0049] The log input path points to the log storage directory of the Ray service, which can accurately locate the log files of all nodes (head node, worker node) and inference service in the Ray cluster. This avoids collecting irrelevant logs or missing key logs, ensuring that all collected logs are valid logs related to the distributed inference service, and providing reliable data for subsequent log analysis and troubleshooting.

[0050] Log tags are used to classify and identify the collected logs. Different tags can be set according to log type (such as node operation log, inference task log, error log) and node role (head node log, worker node log), which facilitates subsequent log retrieval, filtering and classification analysis, improves log analysis efficiency and avoids difficulties in fault location caused by log chaos.

[0051] The refresh interval is used to control the frequency of log collection. Setting a reasonable refresh interval can ensure that logs can be collected in real time to capture service operation status and fault information in a timely manner, while avoiding excessive system resources (CPU, memory) from frequent collection, which may affect the normal operation of the inference service. It can be flexibly adjusted according to the log generation frequency of the actual inference scenario.

[0052] The flexible configuration of log output methods adapts to different operation and maintenance needs. Depending on the actual scenario, you can choose to output to standard output (for easy real-time viewing), log server (for easy long-term log storage and batch analysis), local file (for easy temporary troubleshooting), etc., to take into account the needs of real-time operation and maintenance and long-term log management, and improve the whole process management of log collection.

[0053] In summary, the embodiments of this application have the following beneficial effects: By combining Ray distributed computing with Kubernetes container orchestration, distributed allocation of inference tasks is achieved. Coupled with a worker node elastic scaling mechanism, resource allocation can be dynamically adjusted based on request volume, effectively avoiding resource idleness or insufficiency, significantly improving hardware resource utilization, and reducing hardware costs for large model inference. It also supports flexible cluster scaling by configuring the number of instances and elastic scaling parameters, eliminating the need to copy complete model instances. This results in low scaling costs and fine-grained scaling, efficiently adapting to high-concurrency inference needs of large models and sudden traffic fluctuations, solving the scalability limitations of existing technologies. The automated construction of YAML configuration files via a Java client replaces the traditional manual writing mode, reducing configuration errors and lowering deployment difficulty. Log collection is deployed synchronously with the service, relying on Fluent Bit tools for accurate log collection and management, facilitating operation and maintenance troubleshooting, effectively simplifying deployment and maintenance processes, and lowering the operational threshold. Furthermore, a unified base image environment avoids dependency compatibility issues, and the standardized configuration process and modular system architecture adapt to different types of model inference scenarios, balancing versatility and stability. This reduces latency jitter, meets stringent SLA requirements, and comprehensively improves the shortcomings of existing distributed inference services for model services.

[0054] Based on the same inventive concept, this application also provides a model service distributed reasoning device corresponding to the model service distributed reasoning method in the first embodiment. Since the principle of the device in this application is similar to the above-mentioned model service distributed reasoning method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0055] like Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of the model service distributed inference device 300 provided in an embodiment of this application. The model service distributed inference device 300 includes: The base image module 301 is used to prepare a base image; wherein, the base image includes the dependency libraries required for the model service to run and meets the runtime environment requirements of the distributed inference service. The parameter configuration module 302 is used to configure inference service parameters according to business requirements; wherein, the inference service parameters include at least the inference framework, service uniform resource locator, model file address, environment variables and number of instances; Java client module 303 is used to build a configuration file through a Java client; wherein, the configuration file integrates the base image information and the inference service parameters, and the configuration file defines the Ray service application configuration and Ray cluster configuration; The deployment and operation module 304 is used to distribute the configuration file to the Kubernetes cluster, start the Ray cluster and the corresponding distributed inference service, and complete the distributed deployment and inference operation of the model service.

[0056] Those skilled in the art should understand that Figure 3 The functions of each unit in the model service distributed inference device 300 shown can be understood by referring to the relevant description of the aforementioned model service distributed inference method. Figure 3 The functions of each unit in the model service distributed inference device 300 shown can be implemented by a program running on a processor or by specific logic circuits.

[0057] In one possible implementation, the dependency library includes at least the Ray dependency library, the programming language dependency library, and the runtime environment library, and the base image is built based on a preset operating system.

[0058] In one possible implementation, the configuration file is in YAML format, and the Ray service application configuration includes at least the application name, import path, runtime environment, and deployment configuration. The deployment configuration includes the number of replicas and Ray role options. The Ray cluster configuration includes at least the Ray version, head node group configuration, and worker node group configuration.

[0059] In one possible implementation, the head node group configuration includes a head node template that defines the container name, image address, environment variables, resource limits, resource requests, port configuration, and volume mount configuration; the worker node group configuration includes a worker node template and elastic scaling parameters, including the number of replicas, the minimum number of replicas, and the maximum number of replicas.

[0060] In one possible implementation, the resource restrictions and resource requests include at least configuration parameters for the central processing unit (CPU), memory, graphics processing unit (GPU), GPU memory, and the number of GPU cores; the port configuration includes at least a GCS server port, a console port, a client port, and a service port.

[0061] In one possible implementation, the Java client module is also used for: Construct a Ray cluster configuration object, which integrates the head node group configuration and the worker node group configuration; Construct a service configuration string, which defines the deployment parameters of the Ray service application; The Ray cluster configuration object and service configuration strings are integrated into a YAML format configuration file.

[0062] In one possible implementation, a log collection step is also included: adding a log collection configuration to a configuration file, the log collection configuration being built based on the Fluent Bit tool, for collecting and outputting the runtime logs of the Ray service and the distributed inference service.

[0063] In one possible implementation, the log collection configuration includes a log input path, log tags, refresh interval, and log output method, wherein the log input path points to the log storage directory of the Ray service.

[0064] The aforementioned distributed inference device for model services combines Ray distributed computing with Kubernetes container orchestration technology to achieve distributed allocation of inference tasks. Coupled with a worker node elastic scaling mechanism, resource allocation can be dynamically adjusted based on request volume, effectively avoiding resource idleness or insufficiency, significantly improving hardware resource utilization, and reducing hardware costs for large model inference. It also supports flexible expansion of the cluster size by configuring the number of instances and elastic scaling parameters, without needing to copy complete model instances. This results in low expansion costs and fine-grained scaling, efficiently adapting to the high-concurrency inference needs of large models and sudden traffic fluctuations, solving the problem of limited scalability in existing technologies. The automated construction of YAML configuration files via Java clients replaces the traditional manual writing mode, reducing configuration errors and lowering deployment difficulty. Log collection is deployed synchronously with the service, relying on Fluent Bit tools for accurate log collection and management, facilitating operation and maintenance troubleshooting, effectively simplifying deployment and maintenance processes, and lowering the operational threshold. Furthermore, a unified base image environment avoids dependency compatibility issues, and the standardized configuration process and modular system architecture adapt to different types of model inference scenarios, balancing versatility and stability. This reduces latency jitter, meets stringent SLA requirements, and comprehensively improves the shortcomings of existing distributed inference devices for model services.

[0065] like Figure 4 As shown, Figure 4 This is a schematic diagram of the composition structure of the electronic device 400 provided in the embodiments of this application. The electronic device 400 includes: The device 400 includes a processor 401, a storage medium 402, and a bus 403. The storage medium 402 stores machine-readable instructions that can be executed by the processor 401. When the electronic device 400 is running, the processor 401 communicates with the storage medium 402 via the bus 403. The processor 401 executes the machine-readable instructions to perform the steps of the distributed inference method for model services described in the embodiments of this application.

[0066] In practical applications, the various components in the electronic device 400 are coupled together via a bus 403. It is understood that the bus 403 is used to achieve communication between these components. In addition to a data bus, the bus 403 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 4 The general designated all buses as Bus 403.

[0067] The aforementioned electronic devices combine Ray distributed computing with Kubernetes container orchestration technology to achieve distributed allocation of inference tasks. Coupled with a worker node elastic scaling mechanism, resource allocation can be dynamically adjusted based on request volume, effectively avoiding resource idleness or insufficiency, significantly improving hardware resource utilization, and reducing hardware costs for large model inference. Simultaneously, it supports flexible expansion of the cluster size by configuring the number of instances and elastic scaling parameters, without needing to copy complete model instances. This results in low expansion costs and fine-grained scaling, efficiently adapting to the high-concurrency inference needs of large models and sudden traffic fluctuations, solving the problem of limited scalability in existing technologies. The automated construction of YAML configuration files via Java clients replaces the traditional manual writing mode, reducing configuration errors and lowering deployment difficulty. Log collection functionality is deployed synchronously with the service, relying on Fluent Bit tools for accurate log collection and management, facilitating operation and maintenance troubleshooting, effectively simplifying deployment and maintenance processes, and lowering the operational threshold. Furthermore, a unified base image environment avoids dependency compatibility issues, and the standardized configuration process and modular system architecture adapt to different types of model inference scenarios, balancing versatility and stability. This reduces latency jitter, meets stringent SLA requirements, and comprehensively improves the shortcomings of existing distributed inference services for model services.

[0068] This application also provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by at least one processor 401, the distributed inference method for model services described in this application is implemented.

[0069] In some embodiments, the storage medium may be a magnetic random access memory (FRAM), a read-only memory (ROM), or a programmable read-only memory (PROM). Erasable Programmable Read-Only Memory (EPROM) Electrically Erasable Programmable Read-Only Memory (EEPROM) Read-only memory, flash memory, magnetic surface storage, optical disc, or CD-ROM ROM, Compact Disc Read It can be a memory such as a memory only; or it can be a device that includes one or any combination of the above-mentioned memories.

[0070] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0071] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0072] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0073] The aforementioned computer-readable storage media, combined with Ray distributed computing and Kubernetes container orchestration technology, enables distributed allocation of inference tasks. Coupled with a worker node elastic scaling mechanism, resource allocation can be dynamically adjusted based on request volume, effectively avoiding resource idleness or insufficiency, significantly improving hardware resource utilization, and reducing hardware costs for large model inference. It also supports flexible cluster expansion by configuring the number of instances and elastic scaling parameters, eliminating the need to copy complete model instances. This results in low expansion costs and fine-grained scaling, efficiently adapting to high-concurrency inference needs of large models and sudden traffic fluctuations, solving the scalability limitations of existing technologies. The automated construction of YAML configuration files via Java clients replaces the traditional manual writing mode, reducing configuration errors and deployment difficulty. Log collection is deployed synchronously with the service, relying on Fluent Bit tools for accurate log collection and management, facilitating operation and maintenance troubleshooting, effectively simplifying deployment and maintenance processes, and lowering the operational threshold. Furthermore, a unified base image environment avoids dependency compatibility issues, and the standardized configuration process and modular system architecture adapt to different types of model inference scenarios, balancing versatility and stability. This reduces latency jitter, meets stringent SLA requirements, and comprehensively improves the shortcomings of existing distributed inference services for model services.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed methods and electronic devices can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0075] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0077] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0078] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A distributed reasoning method for model services, characterized in that, Includes the following steps: Prepare a base image; wherein the base image includes the dependency libraries required for the model service to run and meets the runtime environment requirements of the distributed inference service; Configure inference service parameters according to business needs; wherein, the inference service parameters include at least the inference framework, service uniform resource locator, model file address, environment variables, and number of instances; A configuration file is built using a Java client; the configuration file integrates the base image information and the inference service parameters, and defines the Ray service application configuration and Ray cluster configuration. The configuration file is distributed to the Kubernetes cluster, the Ray cluster and the corresponding distributed inference service are started, and the distributed deployment and inference operation of the model service are completed.

2. The distributed inference method for model services according to claim 1, characterized in that, The dependency libraries include at least the Ray dependency library, programming language dependency libraries, and runtime environment libraries, and the base image is built based on a preset operating system.

3. The distributed inference method for model services according to claim 1, characterized in that, The configuration file is in YAML format. The Ray service application configuration includes at least the application name, import path, runtime environment, and deployment configuration. The deployment configuration includes the number of replicas and Ray role options. The Ray cluster configuration includes at least the Ray version, head node group configuration, and worker node group configuration.

4. The distributed inference method for model services according to claim 3, characterized in that, The head node group configuration includes a head node template, which defines the container name, image address, environment variables, resource limits, resource requests, port configuration, and volume mount configuration; the worker node group configuration includes a worker node template and elastic scaling parameters, which include the number of replicas, minimum number of replicas, and maximum number of replicas.

5. The distributed inference method for model services according to claim 4, characterized in that, The resource restrictions and resource requests include at least the configuration parameters for the central processing unit (CPU), memory, graphics processing unit (GPU), GPU memory, and the number of GPU cores; the port configuration includes at least the GCS server port, console port, client port, and service port.

6. The distributed inference method for model services according to claim 3, characterized in that, The process of building the configuration file for the Java client includes: Construct a Ray cluster configuration object, which integrates the head node group configuration and the worker node group configuration; Construct a service configuration string, which defines the deployment parameters of the Ray service application; The Ray cluster configuration object and service configuration strings are integrated into a YAML format configuration file.

7. The distributed reasoning method for model services according to claim 1, characterized in that, It also includes a log collection step: adding a log collection configuration to the configuration file. The log collection configuration is built based on the Fluent Bit tool and is used to collect and output the running logs of the Ray service and the distributed inference service.

8. The distributed inference method for model services according to claim 7, characterized in that, The log collection configuration includes the log input path, log tags, refresh interval, and log output method. The log input path points to the log storage directory of the Ray service.

9. A model-service distributed inference device, characterized in that, The device includes: The base image module is used to prepare the base image; wherein, the base image includes the dependency libraries required for the model service to run and meets the runtime environment requirements of the distributed inference service; The parameter configuration module is used to configure inference service parameters according to business requirements; wherein, the inference service parameters include at least the inference framework, service uniform resource locator, model file address, environment variables, and number of instances; The Java client module is used to build a configuration file through the Java client; wherein, the configuration file integrates the base image information and the inference service parameters, and the configuration file defines the Ray service application configuration and Ray cluster configuration; The deployment and operation module is used to distribute the configuration file to the Kubernetes cluster, start the Ray cluster and the corresponding distributed inference service, and complete the distributed deployment and inference operation of the model service.

10. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the model-service distributed inference method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the model-service distributed inference method as described in any one of claims 1 to 8.

Images