Model task scheduling method and apparatus, device, medium, and program product

Abstract

The present application relates to a kind of model task scheduling method, device, equipment, medium and program product, it relates to model service technical field.The method includes: by obtaining the configuration information of task, multiple tasks are distributed to corresponding scheduling group, can generate flow line based on the scheduling strategy of scheduling group and the data flow between different tasks, then according to the model identifier of task, the corresponding model instance can be loaded, so that the task in flow line can be sequentially model inference.By this way, suitable scheduling strategy can be selected according to task type, and grouping generation flow line management can optimize scheduling and improve execution efficiency, solve the contradiction between resource limitation and complex task scheduling in end-side scene, realize the efficient inference service of end-side model under limited resource.

Description

Technical Field

[0001] This invention relates to the field of model services, specifically to a model task scheduling method, apparatus, equipment, medium, and program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, the local deployment of deep learning models is becoming increasingly common. These locally deployed models have advantages such as low latency, data privacy protection, and no need for network connection, and have been widely used in various terminal devices (such as smart cockpits, IoT devices, embedded systems, etc.).

[0003] Currently, the model needs to handle a large number of inference tasks, but the terminals where the model is deployed are limited by insufficient computing resources and memory capacity. In addition, the task scheduling mechanism is simple, which often leads to problems such as task backlog and high-priority tasks being blocked, which seriously affects the efficiency of task processing. Summary of the Invention

[0004] One of the objectives of this invention is to provide a model task scheduling method, apparatus, device, medium, and program product to improve the efficiency of model task scheduling.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A model task scheduling method, comprising:

[0007] Obtain the task's configuration information, which includes the task type and model identifier;

[0008] Multiple tasks are assigned to at least one scheduling group based on task type, and the tasks in each scheduling group use the same scheduling strategy.

[0009] Based on the scheduling strategy of the scheduling group and the data flow between different tasks, a pipeline containing multiple tasks is generated.

[0010] Load the model instance corresponding to the task according to the model identifier, and execute the pipeline through the model instance.

[0011] In some possible implementations, a pipeline containing multiple tasks is generated based on the scheduling policy of the scheduling group and the data flow between different tasks, including:

[0012] For each task in a scheduling group, determine the scheduling priority of the task as specified by the scheduling policy of the scheduling group;

[0013] Determine the dependencies between tasks based on the data flow between different tasks;

[0014] Based on scheduling priority and dependencies, the execution order of multiple tasks is determined, and a pipeline is generated according to the execution order.

[0015] In some possible implementations, the pipeline is generated according to the execution order, including:

[0016] Obtain the parallelism parameters and resource allocation method pre-set for the scheduling group. The parallelism parameters include the number of tasks that are allowed to be executed concurrently within the scheduling group, and the resource allocation method includes the computing cores bound to each scheduling group.

[0017] A pipeline is generated by combining the execution order, parallelism parameters, and resource allocation methods.

[0018] In some possible implementations, dependencies include downstream tasks in the data flow depending on upstream tasks, and also include the following before the pipeline is executed:

[0019] Set the communication topic for the task so that the downstream task can subscribe to the data published by the upstream task in the preset message middleware;

[0020] Based on the communication topic of the task, a corresponding callback function is registered with the message middleware. The callback function is used to forward the data received by the message middleware to the corresponding task.

[0021] In some possible implementations, the model instance corresponding to the task is loaded according to the model identifier, including:

[0022] Upon receiving a request to use the model, the system queries the pre-defined model service pool to see if a corresponding model instance exists, based on the model identifier carried in the request.

[0023] If the model does not exist, the corresponding model file is loaded from the preset model library into the independent runtime environment, and the loaded model instance is registered to the model service pool.

[0024] In some possible implementations, when executing a pipeline through a model instance, if multiple tasks with the same model identifier are executed simultaneously, then the multiple tasks with the same model identifier share the same model instance.

[0025] A model task scheduling device, comprising:

[0026] The acquisition module is used to acquire the task's configuration information, which includes the task type and model identifier.

[0027] The allocation module is used to assign multiple tasks to at least one scheduling group based on task type, with tasks in each scheduling group using the same scheduling strategy.

[0028] The generation module is used to generate a pipeline containing multiple tasks based on the scheduling strategy of the scheduling group and the data flow between different tasks.

[0029] The loading module is used to load the model instance corresponding to the task according to the model identifier, and execute the pipeline through the model instance.

[0030] An electronic device includes: a processor, and a memory communicatively connected to the processor;

[0031] The memory stores the instructions that the computer executes;

[0032] The processor executes computer execution instructions stored in memory to implement any of the above-mentioned model task scheduling methods.

[0033] A computer-readable storage medium includes: computer-executable instructions stored in the computer-readable storage medium, which, when executed by a processor, are used to implement a model task scheduling method as described above.

[0034] A computer program product includes a computer program that, when executed by a processor, implements a model task scheduling method as described above.

[0035] The beneficial effects of this invention are as follows: By acquiring the configuration information of tasks, multiple tasks can be assigned to corresponding scheduling groups. Based on the scheduling strategy of the scheduling groups and the data flow between different tasks, a pipeline can be generated. Then, according to the model identifier of the task, the corresponding model instance can be loaded, thereby enabling model inference for the tasks in the pipeline in sequence. In this way, appropriate scheduling strategies can be selected according to task type, and grouping to generate pipeline management can optimize scheduling and improve execution efficiency, resolving the contradiction between resource constraints and complex task scheduling in edge scenarios. Attached Figure Description

[0036] Figure 1 A flowchart illustrating a model task scheduling method provided for an exemplary embodiment of the present invention;

[0037] Figure 2 A schematic diagram of task data flow in an intelligent cockpit scenario provided as an exemplary embodiment of the present invention;

[0038] Figure 3 A schematic diagram of the architecture of an edge-side model inference service system provided for an exemplary embodiment of the present invention;

[0039] Figure 4 A flowchart illustrating a modular task management process provided for an exemplary embodiment of the present invention;

[0040] Figure 5 A schematic diagram of a model lifecycle management process provided for an exemplary embodiment of the present invention;

[0041] Figure 6A schematic diagram of the structure of a model task scheduling device provided for an exemplary embodiment of the present invention;

[0042] Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present invention.

[0043] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0044] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0045] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0047] The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, product, or apparatus. Without further limitation, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes elements is not excluded. For example, the use of terms such as "first," "second," etc., to indicate names does not imply any particular order.

[0048] With the rapid development of artificial intelligence technology, deep learning models have been widely used in various terminal devices (such as smart cockpits, IoT devices, embedded systems, etc.). By deploying models locally, models on the terminal can operate without an internet connection and enjoy lower latency. These locally deployed models on the terminal can also be called "edge-side models." However, edge-side environments are limited by computing resources, memory capacity, and power consumption budgets, posing a severe challenge to the stable and efficient operation of model inference services.

[0049] Taking models deployed in smart cockpits as an example, as deep learning models become increasingly powerful, tasks such as object detection, driver posture recognition, behavior analysis, and intelligent driving decision-making all rely on model inference. Currently, edge-side model inference tasks are still generally scheduled using traditional queue methods (such as FIFO, the first-in, first-out principle). This approach lacks effective management of non-reentrant tasks, which often mistakenly enter threads, especially when facing high-concurrency, multi-priority inference requests. This can easily cause high-priority tasks to be blocked, leading to low task processing efficiency and insufficient model utilization. Therefore, there is an urgent need for an edge-side model inference framework that can achieve flexible and efficient task scheduling.

[0050] To address the aforementioned issues, a technical concept is proposed. This involves modular task configuration and scheduling processes, grouping and scheduling tasks based on their configured task types. Multiple tasks are grouped and implemented with different scheduling strategies. By combining data flow constraints between tasks to prevent reentrancy, a multi-task pipeline that can be executed sequentially is generated. By constructing a unified framework for an edge-side model inference service system, different types of inference tasks can be flexibly scheduled, enabling efficient inference services for edge devices with limited resources.

[0051] The application scenarios described above are only partial examples. Those skilled in the art can expand the applications according to specific needs and scenarios, and the embodiments of the present invention do not impose specific limitations in this regard. The method according to an exemplary embodiment of the present invention will now be described with reference to the accompanying drawings.

[0052] Figure 1 This is a flowchart illustrating a model task scheduling method provided as an exemplary embodiment of the present invention. Figure 1 As shown, the method may include:

[0053] Step S101: Obtain the task configuration information.

[0054] The configuration information includes the task type and model identifier.

[0055] The model task scheduling method of this invention can be applied to an edge-side model inference service system. This system can interact with the hardware and software related to model inference tasks to achieve effective management of multiple model inference tasks. Taking a smart cockpit application scenario as an example, the edge-side model inference service system can communicate with sensors such as vehicle cameras and radar, receive data collected by the sensors for model inference, or transmit the inference results to the vehicle ECU (electronic control unit) for execution. The edge-side model inference service system can also read model files from the smart cockpit's storage unit to load at least one model instance and manage the model instance.

[0056] In this embodiment of the invention, the system can obtain information such as task identifier, task type, and model identifier predefined for a task by reading configuration files. For example, in a smart cockpit, some services rely on models to identify road images captured by cameras and detect targets in the images. The smart cockpit can generate a configuration file for this task, writing the task type and the identifier of the model to be used into the configuration file.

[0057] It should be noted that, unless otherwise specified, the "system" mentioned in the embodiments of the present invention refers to the terminal-side model inference service system.

[0058] Step S102: Assign multiple tasks to at least one scheduling group according to task type.

[0059] In this process, the tasks in each scheduling group adopt the same scheduling strategy.

[0060] In this embodiment of the invention, task types may include recurring tasks, event tasks, and short tasks. Recurring tasks are tasks that run continuously or periodically, and these tasks can reside in the execution thread pool for an extended period. Event tasks are tasks triggered by receiving sensor data or data from other services. Short tasks are tasks with a lifecycle, and these tasks automatically exit the thread pool after their lifecycle ends.

[0061] This step involves assigning a corresponding scheduling policy to each task, and then grouping some or all tasks using the same scheduling policy into the same scheduling group. All tasks within each scheduling group execute according to the same scheduling policy, thereby achieving differentiated control over different types of tasks. In some possible implementations, when dividing scheduling groups, the resource allocation method for the scheduling groups can be set according to hardware resources such as CPU, for example, each scheduling group can be bound to a fixed corresponding computing core.

[0062] For example, recurring tasks (such as continuously running camera acquisition tasks or model-dependent inference tasks) can employ scheduling strategies such as time-slice round-robin or dynamic priority to ensure their long-term stable occupation of computing resources and periodic execution. Event-driven tasks (such as target detection tasks triggered by sensor data) can employ event-triggered or real-time response strategies, allowing them to be scheduled for execution only when data arrives. Short tasks (such as one-off data preprocessing tasks) can employ a first-in-first-out (FIFO) strategy or a short-task-first job strategy, with lifecycle timers set to automatically release resources upon task completion.

[0063] Step S103: Based on the scheduling strategy of the scheduling group and the data flow between different tasks, generate a pipeline containing multiple tasks.

[0064] In this embodiment of the invention, an efficient and non-blocking task pipeline can be generated by integrating two key dimensions: the optimization orientation of the scheduling strategy and the structured constraints of the data flow.

[0065] For example, on the one hand, local optimization conditions for the execution order and resource allocation of tasks within each scheduling group can be formed based on the different scheduling strategies bound to each scheduling group; on the other hand, the temporal dependencies and execution conditions that tasks must follow can be determined based on the data flow established by the task input-output relationship. Combining these two aspects, a global execution sequence that can simultaneously satisfy both conditions is constructed, enabling flexible application of multi-task scheduling strategies within each scheduling group while ensuring that all data flows are strictly followed.

[0066] Step S104: Load the model instance corresponding to the task according to the model identifier, and execute the pipeline through the model instance.

[0067] In this embodiment of the invention, the system can retrieve the model file corresponding to the model identifier configured for each inference task in the pipeline from the local storage of the terminal, and create an independent model runtime environment in memory or the computing core bound to the scheduling group to load the model instance. Then, the system can take the data produced by the upstream task (such as image frames and sensor signals) as input and pass it to the loaded model instance of the downstream task for inference calculation, according to the execution order of the pipeline and the task data flow, and continue to pass or output the inference results (such as detection boxes and classification labels).

[0068] In the above embodiments, by obtaining the task configuration information, multiple tasks can be assigned to corresponding scheduling groups. Based on the scheduling strategy of the scheduling groups and the data flow between different tasks, a pipeline can be generated. Then, according to the model identifier of the task, the corresponding model instance can be loaded, thereby enabling model inference to be performed on the tasks in the pipeline sequentially. In this way, appropriate scheduling strategies can be selected according to task type, and grouping to generate pipeline management can optimize scheduling and improve execution efficiency, resolving the contradiction between resource constraints and complex task scheduling in edge scenarios, and realizing efficient inference services for edge models under limited resources.

[0069] For multiple tasks, the data flow indicates the upstream and downstream relationships between them. The execution of some downstream tasks (such as event-triggered inference tasks) must strictly depend on the real-time output of upstream tasks (such as data acquisition tasks) as a prerequisite. If this condition is not met, the downstream task is actually in a non-reentrant state. If the non-reentrancy issue of tasks is not considered and non-reentrant tasks are allowed to enter the execution queue or thread too early, it can easily lead to task backlog and a decrease in overall throughput.

[0070] Figure 2 This is a schematic diagram illustrating the task data flow in an intelligent cockpit scenario, provided as an exemplary embodiment of the present invention. (See diagram below.) Figure 2 As shown, the arrows indicate that data flows from upstream tasks to downstream tasks. Some upstream tasks are prerequisites for the execution of downstream tasks. For example, the two third-level models, the integrated decision-making model and the risk assessment model, both depend on the output of the scenario understanding model in the second-level model. This means that the two downstream reasoning tasks, integrated decision-making and risk assessment, depend on the upstream reasoning task, scenario understanding.

[0071] In one embodiment, generating a pipeline containing multiple tasks based on the scheduling policy of the scheduling group and the data flow between different tasks may include:

[0072] For each task in a scheduling group, determine the scheduling priority of the task as specified by the scheduling policy of the scheduling group; determine the dependencies between tasks based on the data flow between different tasks; determine the execution order of multiple tasks based on scheduling priority and dependencies, and generate a pipeline according to the execution order.

[0073] In this embodiment of the invention, the system can first determine the initial scheduling order of tasks within a scheduling group based on the scheduling policy of that group. Simultaneously, the system analyzes the data flow between tasks to identify non-reentrant tasks with upstream and downstream constraints. When generating the final pipeline, the system can impose constraints based on these upstream and downstream relationships, ensuring that non-reentrant tasks are only scheduled and executed after the preconditions are met, thus resolving the task backlog problem caused by non-reentrant tasks.

[0074] In the above embodiments, by analyzing scheduling priority and data dependencies in a coordinated manner, the task backlog problem easily caused by non-reentrant tasks can be effectively solved. In traditional scheduling schemes, non-reentrant downstream tasks are often forced to occupy execution resources due to incomplete preceding data, leading to the blocking of subsequent tasks. In the pipeline generation embodiment of this invention, all non-reentrant tasks and their dependent upstream tasks are first determined based on the data flow. Then, when arranging the execution order, the dependency relationship is used as a hard constraint to ensure that downstream tasks are only included in the executable queue after their required input data has been generated (i.e., the upstream task has been executed). This can avoid the problem of non-reentrant tasks occupying threads prematurely due to unmet conditions, causing task backlog, and improve the effective utilization of model resources.

[0075] In one embodiment, generating a pipeline in execution order includes:

[0076] Obtain the parallelism parameters and resource allocation methods pre-set for the scheduling group. Combine the execution order, parallelism parameters, and resource allocation methods to generate the pipeline.

[0077] The parallelism parameter includes the number of tasks that can be executed concurrently within a scheduling group, and the resource allocation method includes the computing cores bound to each scheduling group.

[0078] In this embodiment of the invention, in addition to determining the task execution order based on the scheduling strategy of the scheduling group and the data flow between tasks, two resource control dimensions can also be introduced: the first is the parallelism parameter of the scheduling group, which can be used to limit the upper limit of the number of tasks that are allowed to be in the execution state at the same time within the group; the second is the resource allocation method of the scheduling group, which can specifically be to statically or dynamically bind one or more dedicated computing cores (such as CPU cores) to a specific scheduling group.

[0079] In the above embodiments, when generating the final pipeline, the system can integrate the determined task execution order, the parallelism constraints of each scheduling group, and the allocation relationship of computing cores to perform resource mapping and timing fine-tuning. This can reduce the additional context switching overhead caused by the migration of tasks between different cores, thereby forming a deployable pipeline that satisfies both the task execution logic and the physical resource constraints of the edge.

[0080] The inventors discovered that when deploying models to handle service requests on the edge, in addition to the scheduling mechanism affecting task processing efficiency, there is also the problem of high latency in event task response. The core issue lies in the overly tight coupling between task execution and event processing. This tightly coupled architecture makes it difficult for the entire system to handle scattered inference needs triggered by user interactions, sensor data streams, or other asynchronous events. It also makes it difficult to accurately match events to generate inference tasks for the model, which also affects the overall scheduling efficiency of the system. Furthermore, in practical applications, some upstream tasks may need data to flow to multiple downstream tasks simultaneously, and some downstream tasks may need the output results of multiple upstream tasks simultaneously. Traditional event-driven task mechanisms only support single-threaded operation and cannot handle one-to-many or many-to-one workflow requirements between tasks.

[0081] To address the issue of overly tight coupling between tasks and events, this invention proposes a task communication mechanism based on subscription and publication. This mechanism decouples event triggering from the generation of inference tasks. By configuring the communication topics of tasks, data forwarding can be achieved with the help of message middleware, providing flexible support for workflows with multiple data sources, multiple models, and multiple levels of interconnection in complex scenarios.

[0082] In one embodiment, the dependency includes downstream tasks in the data flow depending on upstream tasks, and before executing the pipeline, it also includes:

[0083] Set the communication topic for the task so that downstream tasks can subscribe to the data published by the upstream task in the preset message middleware; based on the communication topic of the task, register the corresponding callback function with the message middleware, and the callback function is used to forward the data received by the message middleware to the corresponding task.

[0084] The communication topics can include subscription topics (SubTopic) and publication topics (PubTopic).

[0085] In this embodiment of the invention, the data flow can be explicitly defined by configuring a communication topic for each task. Relying on a pre-defined message middleware, upstream tasks publish their output data to a specified topic, while downstream tasks that depend on this data subscribe to that topic. During system initialization, based on the subscription relationship, a dedicated callback function can be registered in the message middleware for each downstream task. When an upstream task completes its execution and publishes data to the topic, the message middleware will proactively call all callback functions subscribed to that topic, thereby accurately and asynchronously forwarding the data to the corresponding downstream tasks.

[0086] For example, you can configure SubTopic and PubTopic for a task, with the task responsible for data generation or processing. When a task starts, you can register a callback function with PubTopic. Each Topic forwards data to the corresponding task via the callback function when it receives data.

[0087] In the above embodiments, by introducing message middleware and callback mechanisms, events and tasks can be decoupled, allowing upstream and downstream tasks to be scheduled independently, thus improving the modularity and maintainability of the system. Furthermore, this asynchronous inter-task communication mode allows upstream tasks to continue execution immediately after generating data, without being blocked and waiting for downstream tasks to process it, thereby improving the overall operational efficiency of the pipeline.

[0088] In one embodiment, loading the model instance corresponding to the task according to the model identifier includes:

[0089] Upon receiving a request to use a model, the system queries the preset model service pool to see if a corresponding model instance exists based on the model identifier carried in the request. If the model does not exist, the system loads the corresponding model file from the preset model library into an independent runtime environment and registers the loaded model instance into the model service pool.

[0090] In this embodiment of the invention, the system can set up a unified model service pool to manage all loaded and running model instances. Before loading any model instance, the model instance to be loaded is registered in the model service pool according to the model identifier.

[0091] When the system receives a task execution request containing a specific model identifier, the system can query whether the model instance corresponding to the model identifier has been registered in the model service pool. If it has not been registered, the system will locate the corresponding model file from the locally stored model library, load it into an independent runtime environment to complete the instantiation, and then register the newly created model instance to the model service pool.

[0092] In some possible implementations, when executing a pipeline through a model instance, if multiple tasks executing simultaneously have the same model identifier, then these tasks share the same model instance.

[0093] In this embodiment of the invention, based on the model identifier, the same loaded and running model instance can be reused by multiple inference tasks. During pipeline execution, when the system detects that multiple tasks simultaneously in an executable state have the same model identifier, these tasks do not need to load independent model instances themselves. Instead, they can share the inference results of the same model instance through the same model identifier, avoiding resource waste caused by loading multiple instances.

[0094] In the above embodiments, by managing model instances uniformly and sharing instances by model identifier across multiple tasks, multiple copies of the same model can be kept in memory at the same time. This not only eliminates the overhead of repeated model loading and initialization, but also provides space for loading more model instances, thereby improving model utilization.

[0095] To further demonstrate the application process of the model task scheduling method of the present invention, an example of a specific edge-side model inference service system will be provided below.

[0096] Figure 3 This is a schematic diagram of the architecture of an edge-side model inference service system provided as an exemplary embodiment of the present invention. Figure 3 As shown, the edge-side model inference service system can include task configuration units, scheduling group configuration units, pipeline configuration units, and execution group configuration units. These configuration units can adopt a modular design and achieve configuration reuse through a referencing mechanism, enabling the system to flexibly adapt to different business scenarios and hardware environments.

[0097] The task configuration unit describes the system's task scheduling and execution process in the task configuration file. The configuration file first defines the basic attributes of each task, including task name, type, user-defined parameters, and each task can be configured with different subscription topics, publishing topics, and scheduling strategies.

[0098] The scheduling group configuration unit can define the scheduling policy, parallelism, and resource allocation method of the scheduling group, supporting multiple scheduling algorithms and core bindings. The pipeline configuration unit can define the data flow relationship between tasks and organize multiple tasks into an execution sequence. The execution group configuration unit can associate pipelines with scheduling groups to form a complete execution unit and can specify the pipeline to be started.

[0099] Figure 4 This is a schematic diagram of a task modular management process provided for an exemplary embodiment of the present invention. For example... Figure 4 As shown, the system's scheduling group configuration unit can store multiple scheduling policies, such as Policy 1, Policy 2, ..., Policy n. This unit also has a resource checking function to check the scheduling policies assigned to the scheduling group according to preset rules. For any task, its task type can be determined. Looping tasks, short tasks, and reentrant tasks within event tasks are all executable. Looping tasks can be executed through looping task threads, reentrant tasks can be placed in the corresponding thread pool according to the scheduling group, and short tasks can be executed directly. Non-reentrant tasks within event tasks need to be placed in a long-term queue, waiting for their upstream tasks to complete before execution. Furthermore, the system can perform resource checks, actively checking the task status in the pipeline. If a task is executable, it is processed normally in sequence; if it is not executable or the task is completed, it can be set to a blocked state, waiting for other services to call or control it.

[0100] For example, based on the above configuration units, the system can load and run tasks related to model inference, and this process can include the following six stages:

[0101] The first phase is the task initialization phase. The system first reads and parses the preset configuration file, extracting all task definition information, including task name, type, associated model identifier, subscribed topic, published topic, and scheduling parameters. Then, based on the task type or function identifier, the system dynamically loads the corresponding plugin library or algorithm module to ensure the scalability of the task logic. Next, the system creates an independent runtime environment for each task instance, isolating its memory space and execution state. Finally, according to the configuration, the system initializes the dedicated computing resources, model references, and runtime parameters for each task, preparing for subsequent scheduling and execution.

[0102] The second phase is the task scheduling configuration phase. After initialization, the system can assign tasks to predefined scheduling groups based on task type or user-specified policies. Each scheduling group corresponds to a scheduling policy (such as priority queues or time-slice round-robin). The system determines the execution order of tasks within each scheduling group according to its policy and further configures the binding relationship between tasks and specific CPU cores or computing units to improve locality and reduce switching overhead. Simultaneously, the precise priority, timeout, and other runtime parameters of each task are set to form an executable scheduling unit.

[0103] The third stage is the inter-task communication establishment stage. Based on the configured communication topics, the system establishes a unidirectional data transmission channel for each pair of upstream and downstream tasks on the message middleware or memory bus. The publish-subscribe relationship for tasks is configured, allowing downstream tasks to subscribe to the output topics of upstream tasks. Necessary synchronization mechanisms between tasks are initialized, and buffer sizes and queue capacities are set for each data channel to balance throughput and memory usage.

[0104] The fourth stage is the pipeline generation stage, where the system organizes logically related tasks into one or more execution pipelines. Based on task dependencies and data flow, the strict order of tasks within the pipeline is determined, forming a directed acyclic graph structure. Simultaneously, pipeline-level resource management is performed, such as allocating shared memory areas or I / O bandwidth. Finally, parallelism parameters are configured for each pipeline, specifying the maximum number of tasks that can be activated simultaneously, thereby achieving optimal concurrency control and resource utilization.

[0105] The fifth stage is the pipeline startup and operation phase. The system starts the designated execution pipeline and, according to the strategies of each scheduling group, submits tasks to the corresponding computing cores for execution in sequence. Tasks receive input data (which may come from external sensors, upstream tasks, or internal generation), call loaded model instances or processing logic to perform calculations, and publish the output results to the output topic or pass them to downstream tasks. At this point, the system enters a stable operating state and continues to process the data stream.

[0106] The sixth stage is the monitoring and management stage. During pipeline operation, the system continuously tracks the execution status, resource usage, and data queue status of each task. It dynamically manages data flow between tasks, ensuring data is correctly and timely distributed to subscribers according to the topic. Based on system load and policy requirements, it maintains the dynamic allocation and rebalancing of computing and memory resources. Through regular checks and adaptive adjustments, it ensures that all tasks always run continuously and reliably according to the predetermined scheduling strategy, and executes predetermined recovery or degradation operations in the event of anomalies.

[0107] Figure 5 This is a schematic diagram of a model lifecycle management process provided for an exemplary embodiment of the present invention. For example... Figure 5 As shown, the edge-side model inference service system can uniformly manage the lifecycle of models related to inference tasks through a model service pool. The management process can include the following three stages:

[0108] The first phase is the model loading phase. Upon system startup, the system actively scans the model library directory stored on the client side to identify all available model files and their metadata. When an application initiates its first inference request for a particular model, the system queries the model service pool to check if the model has been loaded into memory. If not, the system reads the corresponding model file from the model library and loads it completely into memory. Subsequently, the system creates an independent runtime environment for the model and initializes the computational resources required by the model. After loading and initialization are complete, the model instance is officially registered in the model service pool and can be shared and called by other applications (provided the model identifier is the same).

[0109] The second stage is the model execution stage. After the model instance is successfully loaded and registered, multiple applications (such as...) Figure 5 Applications 1, 2, and 3 can concurrently access the same model instance. The system uses a reference counting mechanism to accurately track the number of applications currently using the model to manage its lifecycle. When an application sends an inference request, the system places the request in the task queue associated with the model instance and allocates computing resources according to the scheduling policy. The model instance processes the requests in the queue in a pipelined order and returns the inference results to the corresponding application upon completion. Throughout the entire operation, the system can continuously monitor the status of the model instance to ensure stable and reliable service.

[0110] The third phase is the model unloading phase. When the last application using the model actively releases the reference (or the connection is closed), the reference count drops to zero, and the system triggers the model unloading process. The system first waits for all pending requests for the model instance to complete, avoiding forced interruptions that could lead to data errors or resource leaks. Then, it sequentially cleans up the memory resources and other related resources occupied by the model. Next, it removes the model's registration information from the model service pool, making it no longer available for request. Finally, it completely releases all system resources associated with the model, completing the unloading process, and returns these resources to the system for use by other models.

[0111] like Figure 5 As shown, the system can pause or stop a model before it is unloaded. Model pause can be used to respond to short-term events. In the paused state, the model instance remains in the model service pool, its registration information and reference count remain unchanged, but its execution context is temporarily suspended. The system saves the model's current computation state to memory and immediately stops allocating new computation requests. In the stopped state, the model instance is isolated from the service pool, and the system no longer accepts new inference requests for that model instance, but the model itself is not unloaded.

[0112] Throughout the entire lifecycle of the model, the model service employs a sharing mechanism to ensure that multiple applications can efficiently reuse the same model instance, while reference counting and resource management mechanisms guarantee the stable operation of the system.

[0113] Figure 6 This is a schematic diagram of a model task scheduling device provided as an exemplary embodiment of the present invention. (See diagram below.) Figure 4 As shown, the model task scheduling device 600 may include:

[0114] The acquisition module 601 is used to acquire the task's configuration information, which includes the task type and model identifier.

[0115] The allocation module 602 is used to allocate multiple tasks to at least one scheduling group according to the task type, and the tasks in each scheduling group adopt the same scheduling strategy.

[0116] The generation module 603 is used to generate a pipeline containing multiple tasks based on the scheduling strategy of the scheduling group and the data flow between different tasks.

[0117] Loading module 604 is used to load the model instance corresponding to the task according to the model identifier, and execute the pipeline through the model instance.

[0118] In some possible implementations, the generation module 603 can also be used to: determine the scheduling priority of each task in the scheduling group as specified by the scheduling policy of the scheduling group; determine the dependencies between tasks based on the data flow between different tasks; determine the execution order of multiple tasks based on the scheduling priority and dependencies, and generate a pipeline according to the execution order.

[0119] In some possible implementations, the generation module 603 can also be used to: obtain the parallelism parameters and resource allocation method pre-set for the scheduling group, the parallelism parameters including the number of tasks allowed to be executed concurrently within the scheduling group, and the resource allocation method including the computing cores bound to each scheduling group; and generate a pipeline by combining the execution order, parallelism parameters, and resource allocation method.

[0120] In some possible implementations, the allocation module 602 can also be used to: set the communication topic of a task, enabling downstream tasks to subscribe to data published by upstream tasks in a preset message middleware; and register a corresponding callback function with the message middleware based on the task's communication topic. The callback function is used to forward the data received by the message middleware to the corresponding task.

[0121] In some possible implementations, the loading module 604 can also be used to: respond to a received request for the use of a model, query the preset model service pool for a corresponding model instance based on the model identifier carried in the request; if it does not exist, load the corresponding model file from the preset model library into an independent runtime environment, and register the loaded model instance into the model service pool.

[0122] In some possible implementations, the loading module 604 can also be used to: when executing a pipeline through a model instance, if multiple tasks with the same model identifier are executed simultaneously, the multiple tasks with the same model identifier share the same model instance.

[0123] The model task scheduling device provided in this embodiment is used to execute the technical solutions in any of the aforementioned method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0124] It should be understood that the above-described device embodiments are merely illustrative, and the device of the present invention can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0125] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of the present invention can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0126] Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present invention. For example... Figure 7 As shown, the electronic device 70 includes:

[0127] Processor 71, memory 72, and communication interface 73;

[0128] The memory 72 is used to store the executable instructions of the processor 71; the executable instructions can be instructions that the computer can execute.

[0129] The processor 71 is configured to execute the technical solutions in any of the foregoing method embodiments by executing executable instructions.

[0130] Optionally, the memory 72 can be either standalone or integrated with the processor 71.

[0131] Optionally, when the memory 72 is a device independent of the processor 71, the electronic device 70 may further include:

[0132] Bus 74, memory 72 and communication interface 73 are connected to processor 71 through bus 74 and complete communication with each other. Communication interface 73 is used to communicate with other devices.

[0133] Optionally, the communication interface 73 can be implemented using a transceiver. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write databases, and read-only databases). The memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk drive.

[0134] Bus 74 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0135] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0136] The electronic device is used to execute the technical solutions in any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0137] This invention also provides a readable storage medium, which can be a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the technical solution provided in any of the foregoing method embodiments.

[0138] This invention also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the technical solutions provided in any of the foregoing method embodiments.

[0139] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0140] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0141] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0142] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A model task scheduling method, characterized in that, include: Obtain the task's configuration information, which includes the task type and model identifier; According to the task type, multiple tasks are assigned to at least one scheduling group, and the tasks in each scheduling group adopt the same scheduling strategy. Based on the scheduling strategy of the scheduling group and the data flow between different tasks, a pipeline containing multiple tasks is generated. Load the model instance corresponding to the task according to the model identifier, and execute the pipeline through the model instance.

2. The method according to claim 1, characterized in that, The generation of a pipeline containing multiple tasks based on the scheduling strategy of the scheduling group and the data flow between different tasks includes: For each task in a scheduling group, determine the scheduling priority of the task as specified by the scheduling policy of the scheduling group; Determine the dependencies between tasks based on the data flow between different tasks; Based on the scheduling priority and the dependencies, the execution order of the multiple tasks is determined, and a pipeline is generated according to the execution order.

3. The method according to claim 2, characterized in that, The generation of the pipeline according to the execution order includes: Obtain the parallelism parameters and resource allocation method pre-set for the scheduling group. The parallelism parameters include the number of tasks allowed to be executed concurrently within the scheduling group, and the resource allocation method includes the computing cores bound to each scheduling group. A pipeline is generated by combining the execution order, the parallelism parameter, and the resource allocation method.

4. The method according to any one of claims 1 to 3, characterized in that, The dependencies include downstream tasks in the data flow depending on upstream tasks, and before executing the pipeline, it also includes: Set the communication topic of the task so that the downstream task can subscribe to the data published by the upstream task in the preset message middleware; Based on the communication topic of the task, a corresponding callback function is registered with the message middleware. The callback function is used to forward the data received by the message middleware to the corresponding task.

5. The method according to any one of claims 1 to 3, characterized in that, The step of loading the model instance corresponding to the task according to the model identifier includes: Upon receiving a request to use a model, the system queries the preset model service pool to see if a corresponding model instance exists, based on the model identifier carried in the request. If the model does not exist, the corresponding model file is loaded from the preset model library into the independent runtime environment, and the loaded model instance is registered to the model service pool.

6. The method according to any one of claims 1 to 3, characterized in that, When the pipeline is executed through the model instance, if multiple tasks with the same model identifier are executed simultaneously, the multiple tasks with the same model identifier share the same model instance.

7. A model task scheduling device, characterized in that, include: The acquisition module is used to acquire the configuration information of the task, which includes the task type and model identifier; The allocation module is used to allocate multiple tasks to at least one scheduling group according to the task type, wherein the tasks in each scheduling group adopt the same scheduling strategy. The generation module is used to generate a pipeline containing multiple tasks based on the scheduling strategy of the scheduling group and the data flow between different tasks. The loading module is used to load the model instance corresponding to the task according to the model identifier, and execute the pipeline through the model instance.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images