Method for dynamically calling multiple models for deep learning algorithm and service architecture

Abstract

The invention discloses a method for dynamically calling multiple models for a deep learning algorithm and a service architecture. The method comprises the steps that a client sends a request to a scheduling service through a standard AP I interface; after the scheduling service receives the request, key information of the request is analyzed and extracted to generate a task queue, and a Server server is selected; the scheduling service acquires state data provided by the Agent service in real time, and performs task allocation according to the load condition of the Server; the scheduling service stores the selected task information into a database, and marks a'to-be-processed 'state in a task state table; after receiving the scheduling task, the Server calls a model management service to load a specified reasoning model; and S6, after the reasoning task is executed, the reasoning service pool informs a scheduling service of a result through a callback mechanism. The method has the advantages that the processing capacity and efficiency of the deep learning task are improved, the requirements of different tasks are met by dynamically calling multiple models, meanwhile, resource use is optimized, and the expandability of the system is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and service architecture for dynamically calling multiple models for deep learning algorithms. Background Art

[0002] The traditional deep learning inference service architecture has the following problems:

[0003] 1. Waste of resources: In traditional deep learning inference services, especially the static model loading method, the system often loads a large number of unnecessary models, resulting in a large waste of memory and computing resources. Especially when different tasks require different models, traditional methods cannot be flexibly adjusted according to real-time task requirements, resulting in inefficient use of resources.

[0004] 2. Insufficient ability to handle diverse tasks: Traditional reasoning architectures usually only support a fixed model, but in actual applications, the types of tasks are very diverse and require different models to handle. These architectures cannot dynamically select and load models suitable for the current task, resulting in performance degradation when processing multiple tasks.

[0005] 3. Load imbalance: Traditional load balancing algorithms cannot effectively handle the complexity of deep learning tasks. Different deep learning models and tasks may have very different requirements for computing resources. Traditional load balancing algorithms may not be able to reasonably allocate resources according to the complexity of the task, resulting in some servers being overloaded while other servers are idle.

[0006] 4. Long response delay: When a task requires loading and unloading multiple models, the lack of dynamic management mechanisms in traditional architectures may cause model loading delays, thereby increasing the overall task response time. This may significantly affect system performance in application scenarios with high real-time requirements (such as intelligent monitoring, real-time speech recognition, etc.).

[0007] 5. Poor scalability: Traditional system architecture is usually statically configured. Once the design is completed, the system has poor scalability. As the amount of tasks increases or new model requirements emerge, expanding the system requires complex configuration and adjustment, resulting in increased system maintenance costs and difficulty in flexibly responding to business changes.

[0008] 6. High system complexity: Some current complex deep learning service architectures often separate model reasoning from other responsibilities such as task scheduling and resource management, resulting in excessive modularization of the system, increasing the difficulty of system maintenance and the complexity of troubleshooting. The high degree of coupling between different components may affect the stability of the system. Summary of the invention

[0009] The purpose of the present invention is to provide a method and service architecture for dynamically calling multiple models for deep learning algorithms, thereby solving the aforementioned problems existing in the prior art.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] A method for dynamically calling multiple models for a deep learning algorithm comprises the following steps:

[0012] S1. The client sends a request to the scheduling service through the standard API interface;

[0013] S2. After receiving the request, the scheduling service parses and extracts the key information contained in the request, generates a task queue based on the key information, and selects the Server server;

[0014] S3. The scheduling service and the Server server communicate regularly through the Agent service. The scheduling service obtains the status data provided by the Agent service in real time and distributes tasks according to the load of the Server server.

[0015] S4, the scheduling service stores the selected task information in the database and marks the "pending" status in the task status table;

[0016] S5. After receiving the scheduling task, the Server calls the model management service to load the specified inference model. The model management service loads the appropriate version of the inference model from the local storage or remote warehouse according to the task type, and ensures that the inference model has been initialized and is ready to perform inference.

[0017] S6. When the reasoning task is completed, the reasoning service pool notifies the scheduling service of the result through the callback mechanism.

[0018] Preferably, the key information includes: task type: specifies the category of reasoning task to be performed; input data: input data used for model reasoning; computing resource requirements: the client specifies the required computing resources.

[0019] Preferably, generating a task queue specifically comprises:

[0020] The scheduling service constructs task parameters based on task type and data and adds tasks to the global task queue; the task queue supports a priority-based scheduling mechanism to prioritize high-priority or urgent tasks;

[0021] Select the Server server as follows:

[0022] The scheduling service obtains the load status of each server in real time and selects a server with low current load and available resources to execute tasks. The load status is evaluated based on the usage of the server's CPU, GPU, memory, and network bandwidth.

[0023] Preferably, in step S3, if there are multiple Server servers in the system, the scheduling service will select an idle Server server to process the task based on a load balancing algorithm.

[0024] Preferably, in step S4,

[0025] The database stores all metadata related to the task, including the task submission time, task type, and basic information of input data;

[0026] The scheduling service records the current server load status, task allocation, and expected task completion time to ensure high availability and efficiency during task execution.

[0027] Preferably, in step S5,

[0028] During the model loading process, the model management service automatically selects the most suitable resources for loading based on the current system resources to reduce loading time and improve resource utilization efficiency;

[0029] After loading is completed, the Server passes the input data to the inference service pool, which performs inference operations in multiple inference threads or processes; the inference service pool dynamically adjusts the thread pool size and computing resource allocation according to the computing requirements of the current task to maximize parallelism and reduce latency.

[0030] Preferably, in step S6,

[0031] When the inference task is completed, the inference service pool notifies the scheduling service of the result through the callback mechanism; the inference result includes the inference output of the task and the performance data during the execution process;

[0032] After receiving the inference result, the scheduling service updates the task status in the database and stores the task result in the result table; the task status field in the database will be marked as "completed", and the inference result will be saved as a queryable record for use by the client or other systems.

[0033] Preferably, the method further comprises,

[0034] S7. The scheduling service continuously optimizes the task scheduling strategy based on the historical task data in the database and the current server load. The scheduling service can dynamically adjust the priority of the task queue and the server load distribution strategy to achieve high efficiency and low latency in the task processing process.

[0035] The Agent service periodically sends the Server load information to the Scheduling Service so that the Scheduling Service can understand the real-time status of the Server and avoid assigning tasks to an overloaded Server.

[0036] Preferably, the method further comprises,

[0037] S8. If a server fails or fails to respond in time, the scheduling service will automatically transfer the task to other available servers through the fault detection mechanism; based on the fault-tolerant recovery mechanism, it ensures that the task will not be lost or interrupted due to server failure;

[0038] The task queue and task results will be backed up in multiple database nodes to avoid the impact of single point failure on the task processing flow.

[0039] The present invention also aims to provide a service architecture for dynamically calling multiple models for deep learning algorithms, the service architecture can implement the above-mentioned method, and the service architecture includes:

[0040] Client: Sends a task request to the scheduling service, specifies the task type and related data, and waits for the task execution result;

[0041] Scheduling service: responsible for receiving client requests, parsing task types and data, selecting appropriate servers to execute tasks, and performing task scheduling and load balancing;

[0042] Server: responsible for loading the specified machine learning model, executing the inference task and returning the inference result; Server reports the load status in real time through the instant communication mechanism with the scheduling service, and collaborates in scheduling during task execution; the Server includes Agent service, model management service and inference service pool;

[0043] Agent service: provides a real-time heartbeat mechanism and regularly sends the current load status of the Server to the scheduling service;

[0044] Model management service: responsible for model loading, unloading and version management, supporting dynamic updates and resource optimization;

[0045] Inference service pool: provides an efficient inference execution environment and is responsible for executing model inference tasks in multiple inference threads or processes according to scheduling requests;

[0046] Database: used to store key metadata, including current server status, task execution records, and model metadata; the database is also used to manage task scheduling queues and access to inference results.

[0047] The beneficial effects of the present invention are as follows: 1. The present invention provides a more efficient, flexible and scalable scheduling method and service architecture, which can significantly improve the resource utilization, task processing speed and system scalability of the system, and optimize energy efficiency. Compared with the existing static scheduling scheme, the present invention can provide more stable and efficient scheduling services under multi-task and high load conditions, adapt to the ever-changing computing needs, and improve the overall system performance. 2. Improve resource utilization: Improve resource utilization: Traditional scheduling systems usually fail to fully consider the changes in server load, resulting in unbalanced use of computing resources, some servers are overloaded and other servers are idle. In contrast, the present invention can monitor the load information of the server in real time, and intelligently allocate tasks according to the current state of the server, thereby avoiding resource waste and improving the overall resource utilization of the system. 3. Optimize task scheduling efficiency: The static task allocation method in the prior art cannot dynamically adjust tasks to suitable servers, which easily leads to task backlog or execution delay. The present invention uses a flexible load balancing mechanism to ensure that tasks can be promptly allocated to servers with lighter loads, thereby greatly reducing the delay in task processing and improving the response speed and processing efficiency of tasks. 4. Enhance the scalability of the system: The task scheduling system in the prior art may encounter bottlenecks when the system is expanded and cannot efficiently handle more task loads. The present invention adopts dynamic load monitoring and intelligent scheduling strategies, so that the system can automatically expand according to demand and maintain efficient task processing capabilities, ensuring that even in multi-node and high-concurrency scenarios, the system can still maintain good performance and stability. 5. Save computing resources and reduce energy consumption: By dynamically adjusting task allocation, unnecessary overload and waste of resources are avoided. Especially in high-load conditions, the present invention can automatically unload unnecessary tasks to servers with lower loads through load balancing. This mechanism improves the overall energy efficiency of the system, saves computing resources and reduces energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the service architecture in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] like Figure 1 As shown, in this embodiment, a service architecture for dynamically calling multiple models for deep learning algorithms is provided to solve the problems of unbalanced resource management, low task scheduling efficiency, and time-consuming model loading in traditional model reasoning systems. In the present invention, the client's reasoning request can be efficiently distributed to the server with a suitable load state, and the loading and reasoning process of the model can be dynamically scheduled, thereby achieving more efficient computing resource utilization and better response time. Improve the execution efficiency of reasoning tasks, enhance the concurrent processing capability of the system, and ensure the stability of the entire system under high load. The service architecture mainly includes the following components:

[0051] 1. Client: used to send task requests to the scheduling service, specify the task type and related data, and wait for the task execution results.

[0052] 2. Scheduling service: responsible for receiving client requests, parsing task types and data, selecting appropriate servers to execute tasks, and performing task scheduling and load balancing.

[0053] 3. Server: Responsible for loading the specified machine learning model, executing the inference task and returning the inference result. The server reports the load status in real time through the instant communication mechanism with the scheduling service and collaborates in scheduling during task execution.

[0054] (1) Agent Service: Provides a real-time heartbeat mechanism and regularly sends the current load status of the server to the scheduling service, including hardware resource information such as CPU, GPU usage, memory usage, and network bandwidth.

[0055] (2) Model Management Service: Responsible for loading, unloading and version management of models, supporting dynamic updates and resource optimization.

[0056] (3) Inference Pool Service: Provides an efficient inference execution environment and is responsible for executing model inference tasks in multiple inference threads or processes based on scheduling requests.

[0057] 4. Database: used to store key metadata of the system, including the current server status, task execution records, model metadata, etc. The database is also used to manage the task scheduling queue and the access of inference results.

[0058] In this embodiment, a method for dynamically calling multiple models for a deep learning algorithm is provided. The method relies on the above-mentioned service architecture implementation and specifically includes the following steps:

[0059] 1. Client request sending: The client sends a request containing task type, data and computing requirements to the scheduling service through the standard API interface. The request contains the following key information:

[0060] Task Type: Specifies the type of reasoning task to be performed, such as image classification, text analysis, etc.

[0061] Input Data: Input data used for model reasoning, which can be images, text, or data in other formats.

[0062] Resource Requirements: The client can specify the required computing resources (such as GPU, memory, computing power, etc.) to ensure the correct execution of the task.

[0063] 2. The scheduling service receives the request: After receiving the client request, the scheduling service first parses the request content to extract the task type, input data, and resource requirements. Then, the scheduling service performs the following operations:

[0064] Task Queue Generation: According to the task type and data, build task parameters and add the task to the global task queue. The task queue supports a priority-based scheduling mechanism, giving priority to high-priority or urgent tasks.

[0065] Server Selection: The scheduling service obtains the load status of each server in real time and selects a server with low current load and available resources to execute the task. The load evaluation is based on the usage of the server's hardware resources such as CPU, GPU, memory, and network bandwidth.

[0066] 3. Server load monitoring and task allocation: The scheduling service and the server communicate regularly through the Agent service. The Agent service provides information about the hardware status, including CPU, memory, GPU usage, etc. The scheduling service obtains this status data in real time and allocates tasks according to the load situation.

[0067] If there are multiple servers in the system, the scheduling service will select an idle server to process tasks based on a load balancing algorithm (such as minimum load priority, round-robin scheduling, etc.).

[0068] 4. Task queuing and database recording: The scheduling service stores the selected task information in the database and marks it as "pending" in the task status table. The database also stores all metadata related to the task, including task submission time, task type, basic information of input data, etc.

[0069] The scheduling service also records the current server load status, task allocation, and expected task completion time to ensure high availability and efficiency during task execution.

[0070] 5. Task execution: After receiving the scheduled task, the server calls the model management service to load the specified inference model. The model management service loads the appropriate version of the model from the local storage or remote warehouse according to the task type, and ensures that the model has been initialized and is ready to perform inference.

[0071] During the model loading process, the model management service automatically selects the most suitable resources for loading based on the current system resources (such as memory, GPU availability, etc.) to reduce loading time and improve resource utilization efficiency.

[0072] After the model is loaded, the Server passes the input data to the inference service pool, which performs inference operations in multiple inference threads or processes. The inference service pool dynamically adjusts the thread pool size and computing resource allocation according to the computing requirements of the current task to maximize parallelism and reduce latency.

[0073] 6. Inference result transmission and database update: When the inference task is completed, the inference service pool notifies the scheduling service of the result through the callback mechanism. The inference result includes the inference output of the task and the performance data during the execution process (such as latency, inference time, etc.).

[0074] After receiving the inference result, the scheduling service updates the task status in the database and stores the task result in the result table. The task status field in the database will be marked as "completed" and the inference result will be saved as a queryable record for use by clients or other systems.

[0075] 7. System monitoring and optimization: The scheduling service continuously optimizes the task scheduling strategy based on the historical task data in the database and the current server load. The scheduling service can dynamically adjust the priority of the task queue and adjust the server load distribution strategy to achieve high efficiency and low latency in the task processing process.

[0076] The Agent service periodically sends the Server load information to the Scheduling Service so that the Scheduling Service can understand the real-time status of the Server and avoid assigning tasks to an overloaded Server.

[0077] 8. System fault tolerance and high availability: If a server fails or fails to respond in time, the scheduling service will automatically transfer the task to other available servers through the fault detection mechanism. The system supports a fault-tolerant recovery mechanism to ensure that tasks will not be lost or interrupted due to server failures.

[0078] To ensure high availability of tasks, the system supports multiple backup nodes. Task queues and task results will be backed up in multiple database nodes to avoid the impact of single point failures on the task processing flow.

[0079] In this embodiment, the method and service architecture of the present invention include the following application scenarios:

[0080] 1. Intelligent monitoring: According to different monitoring tasks (such as face recognition, object detection, etc.), the required models are dynamically loaded, and the appropriate server is allocated according to the computing needs of the monitoring area to improve real-time monitoring capabilities.

[0081] 2. Autonomous driving: In the autonomous driving system, the appropriate deep learning model is selected for dynamic calling according to the complexity of different road conditions and environments to ensure efficient reasoning and response.

[0082] 3. Personalized recommendation system: Dynamically load different recommendation models based on user behavior data to ensure the real-time and accuracy of the recommendation algorithm.

[0083] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0084] The present invention provides a method and service architecture for dynamically calling multiple models for deep learning algorithms, which can significantly improve the resource utilization, task processing speed and system scalability of the system, while optimizing energy efficiency. Compared with the existing static scheduling scheme, the present invention can provide more stable and efficient scheduling services under multi-task and high load conditions, adapt to changing computing needs, and improve overall system performance. Improve resource utilization: Improve resource utilization: Traditional scheduling systems usually fail to fully consider changes in server loads, resulting in unbalanced use of computing resources, with some servers overloaded and other servers idle. In contrast, the present invention can monitor the load information of the server in real time and intelligently allocate tasks according to the current state of the server, thereby avoiding resource waste and improving the overall resource utilization of the system. Optimize task scheduling efficiency: The static task allocation method in the prior art cannot dynamically adjust tasks to appropriate servers, which easily leads to task backlogs or execution delays. The present invention uses a flexible load balancing mechanism to ensure that tasks can be promptly allocated to servers with lighter loads, thereby greatly reducing the delay in task processing and improving the response speed and processing efficiency of tasks. Enhance the scalability of the system: The task scheduling system in the prior art may encounter bottlenecks when the system is expanded and cannot efficiently handle more task loads. The present invention adopts dynamic load monitoring and intelligent scheduling strategies, so that the system can automatically expand according to demand and maintain efficient task processing capabilities, ensuring that even in multi-node and high-concurrency scenarios, the system can still maintain good performance and stability. Save computing resources and reduce energy consumption: By dynamically adjusting task allocation, unnecessary overload and waste of resources are avoided. Especially in high-load conditions, the present invention can automatically unload unnecessary tasks to servers with lower loads through load balancing. This mechanism improves the overall energy efficiency of the system, saves computing resources and reduces energy consumption.

[0085] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A method for dynamically calling multiple models for a deep learning algorithm, characterized in that: The following steps are included: S1. The client sends a request to the scheduling service through the standard API interface; S2. After receiving the request, the scheduling service parses and extracts the key information contained in the request, generates a task queue based on the key information, and selects the Server server; S3. The scheduling service and the Server server communicate regularly through the Agent service. The scheduling service obtains the status data provided by the Agent service in real time and distributes tasks according to the load of the Server server. S4. The scheduling service stores the selected task information in the database and marks the "pending" status in the task status table; S5. After receiving the scheduling task, the Server calls the model management service to load the specified inference model. The model management service loads the appropriate version of the inference model from the local storage or remote warehouse according to the task type, and ensures that the inference model has been initialized and is ready to perform inference. S6. When the reasoning task is completed, the reasoning service pool notifies the scheduling service of the result through the callback mechanism.

2. The method for dynamically calling multiple models for a deep learning algorithm according to claim 1, characterized in that: The key information includes: Task type: specifies the category of reasoning task to be performed; Input data: input data used for model inference; computing resource requirements: the client specifies the required computing resources.

3. The method for dynamically calling multiple models for a deep learning algorithm according to claim 1, characterized in that: Generate task queue specifically as follows: The scheduling service constructs task parameters based on task type and data and adds tasks to the global task queue; the task queue supports a priority-based scheduling mechanism to prioritize high-priority or urgent tasks; Select the Server server as follows: The scheduling service obtains the load status of each server in real time and selects a server with low current load and available resources to execute tasks. The load status is evaluated based on the usage of the server's CPU, GPU, memory, and network bandwidth.

4. The method for dynamically calling multiple models for a deep learning algorithm according to claim 1, characterized in that: In step S3, if there are multiple Server servers in the system, the scheduling service will select an idle Server server to process the task based on the load balancing algorithm.

5. The method for dynamically calling multiple models for a deep learning algorithm according to claim 1, characterized in that: In step S4, The database stores all metadata related to the task, including the task submission time, task type, and basic information of input data; The scheduling service records the current server load status, task allocation, and expected task completion time to ensure high availability and efficiency during task execution.

6. The method for dynamically calling multiple models for a deep learning algorithm according to claim 1, characterized in that: In step S5, During the model loading process, the model management service automatically selects the most suitable resources for loading based on the current system resources to reduce loading time and improve resource utilization efficiency; After loading is completed, the Server passes the input data to the inference service pool, which performs inference operations in multiple inference threads or processes; the inference service pool dynamically adjusts the thread pool size and computing resource allocation according to the computing requirements of the current task to maximize parallelism and reduce latency.

7. The method for dynamically calling multiple models for a deep learning algorithm according to claim 1, characterized in that: In step S6, When the inference task is completed, the inference service pool notifies the scheduling service of the result through the callback mechanism; the inference result includes the inference output of the task and the performance data during the execution process; After receiving the inference result, the scheduling service updates the task status in the database and stores the task result in the result table; The task status field in the database will be marked as "completed", and the inference result will be saved as a queryable record for use by clients or other systems.

8. The method for dynamically calling multiple models for a deep learning algorithm according to claim 1, characterized in that: The method also includes, S7, the scheduling service continuously optimizes the task scheduling strategy based on the historical task data in the database and the current server load; The scheduling service can dynamically adjust the priority of the task queue and the load distribution strategy of the server to achieve high efficiency and low latency in the task processing process; The Agent service periodically sends the Server load information to the Scheduling Service so that the Scheduling Service can understand the real-time status of the Server and avoid assigning tasks to an overloaded Server.

9. The method for dynamically calling multiple models for a deep learning algorithm according to claim 1, characterized in that: The method also includes, S8. If a server fails or fails to respond in time, the scheduling service will automatically transfer the task to other available servers through the fault detection mechanism; based on the fault-tolerant recovery mechanism, it ensures that the task will not be lost or interrupted due to server failure; The task queue and task results will be backed up in multiple database nodes to avoid the impact of single point failure on the task processing flow.

10. A service architecture for dynamically calling multiple models for deep learning algorithms, characterized by: The service architecture can implement the method described in any one of claims 1 to 9 above, and the service architecture includes: Client: Sends a task request to the scheduling service, specifies the task type and related data, and waits for the task execution result; Scheduling service: responsible for receiving client requests, parsing task types and data, selecting appropriate servers to execute tasks, and performing task scheduling and load balancing; Server: responsible for loading the specified machine learning model, executing the inference task and returning the inference result; Server reports the load status in real time through the instant communication mechanism with the scheduling service, and collaborates in scheduling during task execution; the Server includes Agent service, model management service and inference service pool; Agent service: provides a real-time heartbeat mechanism and regularly sends the current load status of the Server to the scheduling service; Model management service: responsible for model loading, unloading and version management, supporting dynamic updates and resource optimization; Inference service pool: provides an efficient inference execution environment and is responsible for executing model inference tasks in multiple inference threads or processes according to scheduling requests; Database: used to store key metadata, including current server status, task execution records, and model metadata; the database is also used to manage task scheduling queues and access to inference results.

Images