Task scheduling method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to an industrial automation technology, and provides a task scheduling method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining description information of a target algorithm, and an execution demand and a data processing mode for completing a target task; determining a target node according to the execution demand; determining a target driving type according to the node type of the target node and the data processing mode, wherein the target driving type represents an algorithm operation mode matched with the data processing mode; and generating a target instruction according to the description information and the target driving type, and issuing the target instruction to the target node, so that the target node constructs a running environment based on the target instruction and executes a target algorithm in the running environment according to the target driving type to complete the target task. According to the invention, smooth switching of the same set of algorithm logic among different operation environments can be realized.

Description

Technical Field

[0001] This invention relates to the field of industrial automation technology, and more specifically, to a task scheduling method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In the digital transformation of industry, the computing environment exhibits extremely complex heterogeneity: there are both single-machine servers at the edge and cloud clusters or big data clusters; the data sources are also diverse, including Kafka streams, S3 (Simple Storage Service) protocol files, private protocol databases, etc.

[0003] In industrial settings such as power, chemical, and manufacturing, algorithm applications are often built in a "siloed" manner, primarily following two patterns: Pattern A (Domain-Specific Workflow Engine): Examples include Kubeflow for machine learning and Drools for rules. In this pattern, processes are defined by writing a domain-specific DSL (Domain-Specific Language), directly calling underlying containers or Java classes. Pattern B (Hard-Coded Integration): In industrial software development, rule logic, traditional machine learning models, or large models are hard-coded into the application to meet specific production line requirements. Data sources (such as database connection strings and file paths) are typically configured in code or static configuration files.

[0004] Regardless of the mode mentioned above, algorithm development is a customized development for a specific operating environment. The same set of algorithm logic cannot be smoothly switched between different operating environments such as "single machine testing" and "cluster production". Summary of the Invention

[0005] The purpose of this invention is to provide a task scheduling method, apparatus, electronic device, and computer-readable storage medium that can achieve smooth switching of the same set of algorithm logic between different operating environments.

[0006] The embodiments of the present invention can be implemented as follows: In a first aspect, the present invention provides a task scheduling method, the method comprising: Obtain the description information of the target algorithm, as well as the execution requirements and data processing mode for completing the target task; Determine the target node based on the execution requirements; The target driver type is determined based on the node type of the target node and the data processing mode, and the target driver type represents the algorithm running mode that matches the data processing mode; Based on the description information and the target driver type, a target instruction is generated and sent to the target node, so that the target node can build a runtime environment based on the target instruction and execute the target algorithm in the runtime environment according to the target driver type to complete the target task.

[0007] In an optional implementation, the execution requirements include resource requirements and operational requirements, and the step of determining the target node based on the execution requirements includes: Candidate nodes that meet the resource requirements are selected from the preset node pool; Calculate the execution cost score for each candidate node based on the stated operational requirements; The candidate node with the lowest execution cost score is selected as the target node.

[0008] In an optional implementation, the runtime requirements include a logical identifier of the data source that the target algorithm needs to access, and the step of calculating the execution cost score for each candidate node based on the runtime requirements includes: Calculate the resource load cost of each candidate node based on the resource usage information of each candidate node; The environmental affinity of each candidate node is calculated based on the running algorithm of each candidate node and the target algorithm. Based on the logical identifier, the transmission time for each candidate node to access the data source and the startup time for starting the target algorithm are estimated. Calculate the latency cost of each candidate node based on its transmission time and startup time. Calculate the execution cost score for each candidate node based on its resource load cost, environmental affinity, and latency cost.

[0009] In an optional implementation, the step of calculating the environmental affinity of each candidate node based on the running algorithm of each candidate node and the target algorithm includes: For any candidate node, obtain the running environment and data type of the algorithm running on the candidate node; Calculate the first degree of matching between the operating environment and the operating requirements; Calculate the second matching degree between the processed data type and the data type of the data source; Based on the first matching degree and the second matching degree, the environmental affinity of the candidate nodes is calculated, and finally the environmental affinity of each candidate node is obtained.

[0010] In an optional implementation, the logical identifier of the data source is the Uniform Resource Name (URN) of the data source, and the step of estimating the transmission time for each candidate node to access the data source and the startup time of the target algorithm based on the logical identifier includes: For any candidate node, obtain the network address of the candidate node; Based on the network address and the URN, estimate the transmission time of the candidate node; Obtain the image information of the most recently run algorithm of the candidate node; Based on the image information, the startup time of the candidate nodes is estimated, and finally the transmission time and startup time of each candidate node are estimated.

[0011] In an optional implementation, the description information includes the download address of the algorithm package of the target algorithm, the execution requirements include the logical identifier of the data source that the target algorithm needs to access, and the method further includes: Download the algorithm package of the target algorithm from the provided download address; Download the driver package according to the target driver type; Build a runtime environment based on the algorithm package and the driver package; Convert the logical identifier into the corresponding physical address; The input data required by the target algorithm is read from the physical address to execute the target algorithm in the runtime environment according to the target driver type.

[0012] In an optional implementation, the target node further runs a monitoring process for monitoring the operating environment and a logical heartbeat message for reporting the execution status of the target algorithm. Following the step of reading the input data required by the target algorithm from the physical address to execute the target algorithm in the operating environment according to the target driver type, the process includes: Determine whether the preset self-healing conditions are met based on the physical heartbeat message and the logical heartbeat message sent by the monitoring process. When the preset self-healing conditions are met, the preset self-healing strategy is executed to perform self-healing, so as to re-execute the target algorithm and complete the target task.

[0013] In a second aspect, the present invention provides a task scheduling device, the device comprising: The acquisition module is used to acquire the description information of the target algorithm, as well as the execution requirements and data processing mode for completing the target task; The determination module is used to determine the target node based on the execution requirements; The determining module is further configured to determine the target driving type based on the node type of the target node and the data processing mode, wherein the target driving type represents the algorithm running mode that matches the data processing mode; The generation module is used to generate target instructions based on the description information and the target driver type, and send them to the target node so that the target node can build a runtime environment based on the target instructions and execute the target algorithm in the runtime environment according to the target driver type to complete the target task.

[0014] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory being used to store a program, and the processor being used to implement the task scheduling method as described in the first aspect above when executing the program.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the task scheduling method described in the first aspect above.

[0016] Compared with the prior art, the present invention has the following beneficial effects: This invention decouples the algorithm's description information from the execution requirements and data processing mode of the target task. It determines the target node based on the execution requirements, and then determines the target driver type of the algorithm's running mode that matches the data processing mode based on the node type and data processing mode of the target node. Finally, it generates target instructions based on the description information and the target driver type, and sends them to the target node. This allows the target node to build a running environment based on the target instructions and execute the target algorithm within that environment according to the target driver type, thus completing the target task. By decoupling the algorithm's implementation logic from its running environment, and determining the target algorithm's running environment based on the actual requirements of the target task, it achieves smooth switching of the same algorithm logic between different running environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an example diagram of the architecture provided in this embodiment.

[0019] Figure 2 The code implementation example of the algorithm definition provided in this embodiment.

[0020] Figure 3This example provides a code implementation example for the data definition provided in this embodiment.

[0021] Figure 4 This is a code implementation example of the compute driver definition provided in this embodiment.

[0022] Figure 5 This embodiment provides a standardized JSON code implementation example.

[0023] Figure 6 This is a flowchart illustrating the task scheduling method provided in this embodiment.

[0024] Figure 7 This is a block diagram of the task scheduling device provided in this embodiment.

[0025] Figure 8 This is a block diagram of the electronic device provided in this embodiment.

[0026] Icons: 10-Electronic device; 11-Processor; 12-Memory; 13-Bus; 100-Task scheduler; 110-Acquisition module; 120-Determination module; 130-Generation module; 140-Running module; 150-Self-healing module. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0029] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0030] In the description of this invention, it should be noted that if terms such as "upper," "lower," "inner," or "outer" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0031] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0032] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0033] For algorithm applications in industrial settings, in addition to the aforementioned drawback of strong coupling between the algorithm and the environment, which prevents a smooth switch between "single-machine testing" and "cluster production" without additional code modifications, the inventors discovered the following further shortcomings: Multi-paradigm fragmentation: rule engines, machine learning, and large models are usually managed by three completely independent systems with different interface standards, making it difficult to perform unified task orchestration and resource reuse.

[0034] Hard-coded data connections: Physical data sources (such as IP addresses, ports, and accounts) are often hard-coded in the configuration or code. Once the infrastructure changes, all related algorithm tasks need to be reconfigured or reissued, making it impossible to deploy on a large scale in industrial fields.

[0035] Lack of industrial-grade state self-healing: Current scheduling engines are mostly stateless or only record task-level states. In the event of network fluctuations or power outages and restarts common in industrial settings, long-running tasks (such as model training or long-term series analysis) often need to start from scratch and cannot be resumed from the breakpoint through state replay.

[0036] In view of this, this embodiment provides a task scheduling method, apparatus, electronic device, and computer-readable storage medium to overcome the above-mentioned defects, which will be described in detail below.

[0037] Please refer to Figure 1 , Figure 1 This is an example diagram of the architecture provided in this embodiment. Figure 1 It is an adaptive scheduling architecture that completely separates the control and execution parts of an algorithm. The control part corresponds to... Figure 1 The control plane (L1) and execution part correspond to Figure 1 The execution plane (L2) in the middle.

[0038] Layer L1 primarily maintains the logical definitions and resource contracts used to control the algorithm. The core entities maintained by Layer L1 mainly include algorithm entities, data entities, and computation driver entities. Algorithm entities mainly include the logic code related to the actual business logic and input / output I / O contracts (e.g., the algorithm's input / output variable names), but do not need to include the code that performs the algorithm's logical function or any environment-dependent code. For a more intuitive illustration of the definition description of algorithm entities, please refer to [link / reference]. Figure 2 , Figure 2 The code implementation example of the algorithm definition provided in this embodiment. Figure 2 In the algorithm definition, the basic information of the algorithm is described, including the type (kind), metadata, details (spec), IO contract (ioContract), and business configuration (config).

[0039] Data entities primarily consist of a unique logical identifier for data access and its corresponding physical link. The physical link is invisible to algorithm developers; they only reference the logical identifier. This eliminates the dependency between the algorithm implementation and the physical address of the data source, enabling smooth switching of the same algorithm logic across different heterogeneous data sources. For a more intuitive illustration of the data entity description, please refer to [link / reference]. Figure 3 , Figure 3 This example provides a code implementation example for the data definition provided in this embodiment. Figure 3 In this context, the data definition describes the basic information of the data, including the type (kind), metadata, labels, details (spec), access mode (accessMode), and physical link information. Figure 3 The logical identifier in it is implemented as urn (Uniform Resource Name).

[0040] The compute driver entity defines the framework stack "how" an algorithm runs. It's middleware that connects the static algorithm and data definitions of the L1 layer with the dynamic, actual nodes running the algorithm. Types of compute driver entities include, but are not limited to, Spark Driver (based on the Spark framework), RAY_WORKER (based on the RAY framework), LITE_PYTHON_ENGINE (based on the LITE framework), and TRITON_SERVER (based on the TRITON framework). For a more intuitive description of compute driver entities, please refer to [link to documentation / reference]. Figure 4 , Figure 4 This is a code implementation example of the compute driver definition provided in this embodiment. Figure 4 In the definition, the compute driver is used to describe the basic information of the compute driver, including the type (kind), metadata, and detailed information (spec). Figure 4 In this context, `urn` is the logical identifier of the computing driver. This logical identifier is associated with the physical address of the computing driver's package. During actual runtime, the corresponding driver package can be downloaded based on this logical identifier. For the same algorithm, different driver packages correspond to the different running modes of the algorithm in different runtime environments, thereby achieving a smooth switching of the same algorithm across different runtime environments.

[0041] As an implementation, L1 comprises four parts: a task orchestration engine, an intelligent scheduler, a task dispatcher, and task state management. The task orchestration engine receives user-defined algorithmic business flows (which can be represented by a directed acyclic graph, DAG), uses the Kahn algorithm to parse the user-submitted DAG, calculates the topological sort, and determines the task execution order. Then, based on the algorithm required by the task, it retrieves the corresponding metadata and transforms heterogeneous (including, but not limited to, heterogeneous data sources and runtime environments) business intents into a platform-independent unified specification (Unified Spec). Please refer to [reference needed]. Figure 5 , Figure 5 This embodiment provides a unified and standardized implementation of JSON code. Figure 5 In this process, the standardized task description includes a task identifier (taskId), control information (control), resource information (resource), runtime information (payload), and configuration information (config). At this stage, through "logical encapsulation," the algorithm is parsed, and metadata services are queried to replace algorithm-related physical dependencies (IP addresses, ports, file paths) with logical identifiers (e.g., URNs). By introducing logical identifiers and a standardized approach into the task orchestration engine, the upper-layer business logic is decoupled from the lower-layer physical infrastructure (S3, Kafka, local files), avoiding the environment dependencies caused by hard-coding specific database connection strings or file paths in algorithms, as is common in traditional industrial software. A unified contract (e.g., a JSON contract containing artifacts, configurations, inputs / outputs) smooths out the differences in the operating methods of rule engines, machine learning models, and large language models, making them all standardized "executable units" for the intelligent scheduler. Therefore, on the one hand, algorithm developers do not need to worry about whether the algorithm will run on an edge gateway or a cloud cluster, completely eliminating environment dependencies. On the other hand, dynamic configuration updates can be achieved by dynamically loading configuration forms, reducing the communication costs of cross-departmental collaboration between development and operations.

[0042] The intelligent scheduler determines the execution nodes and methods for standardized tasks. This serverless approach eliminates user concerns about whether tasks run on edge gateways or cloud clusters; the intelligent scheduler makes the decision automatically. It avoids overkill (using a large tool to run simple rules on a cluster) or underutilization (using a small device to run a large model), achieving adaptive deployment of "rules at the edge, models trained in the cloud," thus optimizing cost and timeliness. It also facilitates resource scaling by triggering expansion strategies when resources are insufficient, enabling elastic resource scaling.

[0043] The task dispatcher is responsible for translating the intelligent scheduler's logical decisions into physical network requests. It manages connections, performs authentication and encryption, and ensures instruction delivery. Therefore, the intelligent scheduler doesn't need to concern itself with the specific underlying communication methods, such as HTTP, gRPC, or MQTT. The task dispatcher encapsulates the communication details, ensuring reliable instruction delivery even with network instability. The task dispatcher encapsulates the tasks generated by the task orchestration engine and the intelligent scheduler's logical decisions. To prevent sensitive information from being leaked in plaintext during network transmission, sensitive configurations (such as database passwords) can be encrypted or tokenized during task distribution. For example, a temporary secure access token (SAS / Token) can be generated and included in the instruction to ensure subsequent runtime has the authority to retrieve code and data without transmitting plaintext passwords over the network, adding an extra layer of security. As an implementation method, the task dispatcher can also employ a callback mechanism, releasing resources immediately after task distribution. This asynchronous, non-blocking approach significantly improves throughput in high-concurrency scenarios. Asynchronous delivery can use Fire-and-Forget in conjunction with a callback mechanism to call the execution node's interface. Additionally, register a callback to wait for the execution node to report "startup successful" or "startup failed".

[0044] Task status management is used to monitor the execution of computing drivers and algorithms in order to detect problems in a timely manner and automatically achieve self-healing when problems are detected.

[0045] At the L2 layer, the runtime environment is dynamically loaded according to the actual task requirements, and the corresponding algorithm is executed in the runtime environment to complete the task. Specifically, at runtime, "driver + algorithm package + data connection context" can be automatically assembled according to the task specification (TaskSpec) corresponding to the task requirements, so as to realize "write once, run anywhere" for algorithm logic between heterogeneous computing power runtime environment (single machine / cluster) and heterogeneous data source.

[0046] Please continue to refer to Figure 1The execution process of the NodeAgent daemon in L2 corresponds to the computing driver, that is, the construction of the runtime environment. The Runtime module corresponds to the actual execution of the algorithm, including the actual running of the algorithm package and the driver package.

[0047] The NodeAgent daemon is a resident daemon deployed on each compute node. It does not contain business logic and dynamically "assembles" the runtime environment based on received instructions. As an implementation method, in order to shield the differences in the underlying operating system: regardless of whether the underlying system is a Linux physical machine, a K8s container Pod, or a Windows industrial control machine, the NodeAgent daemon shields the differences downwards and provides a unified interface using the SPI (Service Provider Interface) mechanism. The NodeAgent daemon mainly includes the following processes: (1) After receiving the instructions distributed by the task dispatcher, the HTTP Listener parses the instructions and requests resources; (2) The resource loader downloads the algorithm package and driver package to the local isolation sandbox; (3) The SPI adapter is instantiated and the startup command is constructed. Figure 1 The SPI adaptation layer in the middle includes the Provider Manager module, which is used to manage various Providers: such as Spark Provider, Python Provider, Docker Provider, etc. The SPI instantiation process includes: the NodeAgent daemon reads the compute driver type, and for the Lite / Local type, instantiates LocalProcessProvider and builds Shell command to directly start the process, which is a low-overhead method. For the Spark type, instantiates SparkClusterProvider and automatically assembles spark-submit --class ... --master yarn ... command; (4) execute the start command, and at the same time obtain the process PID and start monitoring.

[0048] The NodeAgent daemon also includes a process watchdog module to monitor the running processes and their survival status.

[0049] The Runtime module is a sandbox for actually executing algorithms. Figure 1 In this module, the main components include the compute driver and the data access SDK (corresponding to...). Figure 1 The DataStore & Config SDK standard development toolkit in the document). As an implementation method, it mainly includes the following processes: (1) The computing driver process starts, corresponding to Figure 1(1) The CALC_DRIVER in the package executes the driver / launcher; (2) Loads the user-written Python / Java algorithm code in the algorithm package (e.g., Figure 1 (3) The SDK intercepts the request, reads the local context configuration in the local mount file, and finds the URN corresponding to "wave_in". The SDK automatically instantiates the URN according to the URN protocol header. For example, if the URN is of type kafka, then KafkaConsumer is automatically instantiated. If it is of type file, then FileReader is instantiated. Thus, in this process, the user only knows that he / she needs to read data, and does not need to care whether the underlying external data source is Kafka, S3, or other types.

[0050] It should be noted that L1 and L2 processing can be performed on the same device or on different devices.

[0051] It should also be noted that, Figure 1 The L1 task dispatcher actively pushes the payload to the L2 agent via HTTP POST. This is just one implementation method; in fact, a subscription / pull pattern can also be used. The agent acts as a subscriber, connecting to a global message middleware (such as MQTT, RabbitMQ, or Redis queues). The L1 publishes the TaskSpec to a specific topic, and idle agents actively fetch tasks based on their own resource status and preset rules. This prevents others from claiming a different architecture by changing "push" to "pull." This model can also achieve decoupled scheduling in unstable, weak network industrial environments (such as mobile devices and mines).

[0052] Furthermore, Figure 1The method of dynamically downloading the Driver and Algo packages via the SPI mechanism and starting the process on the host machine can also be replaced by the following alternatives: (1) Dynamic container image building: The scheduler no longer issues TaskSpec, but triggers an image builder service to package the algorithm package Algo and the driver package Driver into a new Docker image, and then distributes it to K8s for execution. (2) WebAssembly (WASM) sandbox: The algorithm is compiled into WASM format and executed in a lightweight WASM runtime (such as Wasmtime) to achieve more granular isolation than processes and faster startup speed than Docker. (3) Serverless FaaS function: The algorithm package Algo is directly mapped to the function function of the cloud service provider (such as AWS Lambda), and the cloud platform handles elastic scaling. This more comprehensively covers all mainstream environment isolation methods other than "downloading and executing packages".

[0053] also, Figure 1 The algorithm code in the data access SDK calls specific interfaces (such as sdk.getInput()) to obtain data. Alternatively, the following methods can be used: (1) Sidecar injection: Start a proxy process next to the algorithm container. The algorithm uses standard SQL or protocol to access the local port (localhost). The sidecar transparently forwards the request to the real Kafka or database according to the configuration issued by L1. (2) Virtual file system (FUSE): Mount the remote data source (Kafka / S3) as a local virtual folder. The algorithm code operates on / mnt / data / wave_in as if it were a local file. The underlying physical conversion is completed by the driver. This prevents others from bypassing the dependency on the SDK through "non-intrusive proxy" or "virtual driver" technology.

[0054] in addition, Figure 1 The dual-path verification based on PID physical heartbeat and SDK progress logical heartbeat can also adopt the following alternative schemes: (1) Resource fingerprint comparison: Monitor the CPU / memory utilization curve of the process. If a long-term "plateau line" or "extremely low duty cycle" appears, it is determined to be a deadlock. (2) External probes: The L1 scheduler or third-party node periodically sends pathfinding messages (Ping / HealthCheck) to the executor, and the status is determined based on the response delay. Thus, it is ensured that any comprehensive judgment scheme that combines "process-level survival" and "business-level activity" is within the protection scope of this embodiment.

[0055] exist Figure 1 Based on the existing architecture, this embodiment provides a task scheduling method. Please refer to... Figure 6 , Figure 6 This is a flowchart illustrating the task scheduling method provided in this embodiment. The method includes the following steps: Step S101: Obtain the description information of the target algorithm and the execution requirements and data processing mode for completing the target task.

[0056] In this embodiment, the target algorithm can be an algorithm that has been pre-implemented by the developer and is capable of completing the target task, serving as an independent logical function. The description information of the target algorithm describes its basic attributes, including, but not limited to, algorithm identifier, functional description information, programming language, code storage path, and input / output variable names. The description information does not include the specific code implementation or environment dependencies of the algorithm. For specific examples, please refer to the previous section. Figure 2 The algorithm definition in the text needs to be explained. Figure 2 This is just one specific implementation example of the algorithm definition. In fact, those skilled in the art can add or remove descriptive items based on actual needs.

[0057] In this embodiment, a target task can be decomposed from the DAG graph of the algorithm business flow input by the user. Executing multiple target tasks sequentially can realize complex control flows in industrial software. The execution requirements of the target task can be input by the user through the interface, read from a pre-written requirement file, or obtained through a pre-set execution requirement template. Execution requirements include, but are not limited to, resource requirements and runtime requirements. Resource requirements may include the minimum number of processor cores, minimum memory size, minimum number of GPUs, etc. Runtime requirements may include logical identifiers of the input data source and logical identifiers of the storage destination of the output results, etc.

[0058] In this embodiment, data processing modes include, but are not limited to, streaming, batch, and large model processing (LLM). Streaming mode is typically used for streaming data, such as rule extraction. This mode prioritizes low processing latency when the data volume is relatively small and has high requirements for processing efficiency. Batch processing mode is used for batch processing of large amounts of data, such as cleaning large amounts of historical data. This mode prioritizes data throughput and has high requirements for parallel computing capabilities. Large model processing mode is typically used for attribution analysis of data, involving inference analysis using large models. This mode prioritizes data processing capabilities.

[0059] Step S102: Determine the target node based on execution requirements.

[0060] In this embodiment, the target node is the node that meets the execution requirements. The target node can be a single-machine server, or one or more nodes in a cloud cluster or big data cluster. For different target tasks, the target algorithm may be the same, but the execution requirements may be different, and therefore, the determined target node will also be different.

[0061] Step S103: Determine the target driving type based on the node type and data processing mode of the target node. The target driving type represents the algorithm operation mode that matches the data processing mode.

[0062] In this embodiment, node types include, but are not limited to, single-machine types, cluster node types, and types with GPU computing power. Driver type refers to the type of computing driver entity, including, but not limited to, LITE_PYTHON_ENGINE (a computing driver based on the LITE framework), SPARK_DRIVER (a computing driver based on the Spark framework), RAY_WORKER (a computing driver based on the RAY framework), and TRITON_SERVER (a computing driver based on the TRITON framework). Each node has at least one supported driver type. The target driver type is the driver type among all the driver types supported by the target node that matches the data processing mode. For ease of implementation, after determining the target driver type, the target node and the target driver type can be bound. Subsequently, when the target node receives the target instruction, it directly creates the runtime environment according to the bound target driver type.

[0063] As one implementation method, if the target node is a single-machine type and the data processing mode is streaming mode, then the target driver type is LITE_PYTHON_ENGINE; if the target node is a cloud-based Kubernetes cluster and the data processing mode is batch processing mode, then the target driver type is SPARK_DRIVER or RAY_WORKER; if the target node is a GPU-enabled instance type and the data processing mode is large-scale model processing mode, then the target driver type is TRITON_SERVER.

[0064] Step S104: Generate target instructions based on description information and target driver type, and send them to target nodes so that target nodes can build a runtime environment based on target instructions and execute the target algorithm in the runtime environment according to the target driver type to complete the target task.

[0065] In this embodiment, the description information is the static information of the target algorithm, while the target driver type and target node are dynamically determined according to the execution requirements of the target task. This decouples the implementation logic of the algorithm from its running environment. The running environment of the target algorithm is determined according to the actual requirements of the target task, and the same set of algorithm logic is smoothly switched between different running environments.

[0066] In an optional implementation, to better match the target node with execution requirements, fully utilize resources, and avoid resource waste, this embodiment also provides a method for determining the target node: First, candidate nodes that meet the resource requirements are selected from the preset node pool.

[0067] In this embodiment, the preset node pool includes multiple pre-registered nodes that are in normal operating condition and can be accessed. To improve filtering efficiency, pre-filtering rules can be set according to the data processing mode, and the nodes in the preset node pool can be pre-filtered according to the pre-filtering rules to obtain preliminary selected nodes. Then, candidate nodes are selected from the preliminary selected nodes.

[0068] As one implementation method, the pre-filtering rule can be: if the data processing mode is large model processing mode, then the nodes labeled GPU_AVAILABLE in the preset node pool are selected as the initial nodes; if the data processing mode is streaming mode and the target task is sensitive to latency, then the nodes in the preset node pool with a network hop count less than the preset hop count are selected as the initial nodes.

[0069] As an implementation method, in order to quickly remove nodes that cannot physically run the target algorithm from the initial selection nodes and obtain candidate nodes that meet the requirements of the target task, the following two aspects can be used for filtering: (1) Tag matching: retrieve the metadata tags of the initial selection nodes, and determine whether they meet the hardware architecture consistency (e.g., by matching the arch tag, determine whether the hardware architecture is x86 or ARM64, etc.), whether they are specific hardware accelerators (e.g., if the minimum number of GPUs in the resource requirements is greater than 0, then retain the nodes with GPU_AVAILABLE or NPU_COMPUTE tags in the initial selection nodes), and whether the operating system and environment match (e.g., determine whether the initial selection nodes meet the operating system requirements in the resource requirements based on the operating system tags such as Linux and Windows, and determine whether the initial selection nodes meet the database environment requirements in the resource requirements based on the pre-installed underlying database environment); (2) Engine compatibility verification: filter out the nodes that can carry the corresponding computing driver according to the type in the description information of the target algorithm. For example, Spark tasks can only be distributed to nodes with SparkRuntime installed, and cannot be distributed to edge gateways that only support lightweight Python runtime.

[0070] Secondly, the execution cost score for each candidate node is calculated based on operational requirements.

[0071] In this embodiment, the execution cost score is an estimated score for the candidate node to execute the target algorithm based on the real-time running status and running requirements of the candidate node. The lower the execution cost score, the higher the probability of being selected as the target node. This ensures the execution efficiency of the target algorithm and ultimately guarantees the timeliness of the target task response.

[0072] Finally, the candidate node with the lowest execution cost score is selected as the target node.

[0073] In this embodiment, if there are multiple candidate nodes with the lowest execution cost scores, the target node can be selected from them according to a preset optimization strategy. The preset optimization strategy can be a random strategy, the strategy with the longest current fault-free runtime, the strategy with the highest algorithm startup success rate, etc.

[0074] In an optional implementation, to improve the rationality of the execution cost score and enable it to more accurately reflect the real-time operating status of candidate nodes, this embodiment provides an implementation method for comprehensively calculating the execution cost score from multiple dimensions: First, based on the resource usage information of each candidate node, calculate the resource load cost of each candidate node.

[0075] In this embodiment, the resource load cost reflects the current real-time load level of the candidate node. To prevent a single node from overheating and causing an accumulation of industrial tasks, a candidate node with load balancing is preferred as the target node. As one implementation, resource usage information may include CPU utilization and memory utilization. The resource load cost R(N) can be expressed by the formula R(N) = α. UsageCPU+β UsageMem is calculated, where UsageCPU is the CPU utilization of candidate node N, UsageMem is the memory utilization of candidate node N, α and β are the weights of UsageCPU and UsageMem respectively, and α+β=1. For industrial tasks with different resource sensitivities, the values ​​of α and β can be adjusted according to actual needs. For example, for computationally intensive industrial tasks, α can be appropriately increased, and for memory-intensive industrial tasks, β can be appropriately increased.

[0076] Secondly, based on the running algorithm and target algorithm of each candidate node, the environmental affinity of each candidate node is calculated.

[0077] In this embodiment, the running algorithm can be at least one of the following: algorithms that have been run within a preset historical time period for the candidate node, algorithms that are currently running, or algorithms that are currently waiting to be run. The preset historical time period can be set as needed, for example, the preset historical time period is the most recent 5 minutes.

[0078] In this embodiment, environment affinity characterizes the degree of matching between the hardware parameters and runtime environment of the candidate node and the target algorithm. The higher the matching degree, the lower the overhead of preparing the runtime environment. By using environment affinity, we can evaluate the hardware parameters such as memory of the candidate node and the "soft state" of the runtime environment, and determine whether the target algorithm can achieve "in-situ reuse" and "seamless integration" with the existing runtime environment on the candidate node.

[0079] Third, based on the logical identifier, estimate the transmission time for each candidate node to access the data source and the startup time for the target algorithm.

[0080] In this embodiment, the logical identifier is a unique identifier that logically represents the data source that the target algorithm needs to access. The logical identifier can be specified by the user for the target algorithm through the interface. Each logical identifier corresponds to a real physical address, through which the corresponding data source can be accessed. Since the focus is on the logical identifier rather than the physical address before the algorithm is actually run, when the physical address changes, it is only necessary to update the mapping relationship between the logical identifier and the physical address to ensure that the correct data source can be read when the algorithm is actually run, thereby achieving seamless migration of multiple heterogeneous data sources.

[0081] In this embodiment, the logical identifier can be a URN or an alias / symbolic link, such as a logical path created in distributed coordination software (e.g., Zookeeper or Consul). It can also be a metadata tag / label: that is, assigning a set of characteristic tags (e.g., type:vibration, env:prod) to the data source. Alternatively, it can be a virtual UUID: a randomly generated UUID can be used to map a physical connection in a global lookup table.

[0082] In this embodiment, transmission time refers to the network transmission overhead incurred when a candidate node accesses the data source, which is the round-trip delay between the candidate node and the data source. Startup time is the estimated startup time of the candidate node before running the target algorithm, including, but not limited to, file mounting time, time to download the algorithm package and driver package, and warm-up time for CPU cache and memory page tables.

[0083] Fourth, calculate the latency cost of each candidate node based on its transmission time and startup time.

[0084] In this embodiment, the latency cost can be the direct sum of the transmission time and the startup time, or, depending on the actual scenario, corresponding weights can be set for the transmission time and the startup time, and then the weighted sum of the two can be calculated to obtain the latency cost.

[0085] Finally, the execution cost score of each candidate node is calculated based on its resource load cost, environmental affinity, and latency cost.

[0086] In this embodiment, the execution cost score can be calculated using the following formula: S(N) = wlat L(N)+wres R(N)+wsta C(N), where wlat, wres, and wsta are the resource load weight, environmental affinity weight, and time delay weight, respectively, and L(N), R(N), and C(N) are the resource load cost, environmental affinity cost, and time delay cost of candidate node N, respectively.

[0087] As one implementation method, different values ​​can be set for wlat, wres, and wsta according to the needs of the actual scenario. For example, for real-time alarm scenarios, based on the principle that industrial site safety is paramount and response speed is more important than resource conservation, the principle of selecting the node with the closest physical distance and lowest latency regardless of cost is adopted, setting wlat=0.7, wres=0.2, and wsta=0.1. For offline analysis scenarios, based on the principle that offline computing prioritizes throughput and cost, the principle of selecting the cluster center node with abundant resources for processing is adopted, setting wlat=0.1, wres=0.5, and wsta=0.4.

[0088] In an optional implementation, to make the environmental affinity of candidate nodes more accurately reflect whether the candidate node's operating environment matches the target algorithm, this embodiment provides an implementation method for calculating the environmental affinity of candidate nodes for any candidate node: First, obtain the runtime environment and data types of the candidate node's algorithm. In this embodiment, the runtime environment includes, but is not limited to, multiple dimensions such as computing driver packages, algorithm packages, and the language compilation environment for program execution. For example, the runtime environment includes a Python 3.8 language compilation environment, the TensorRT 8.6 algorithm package, and an industry-specific driver package. The data type processed is the same as the data type processed by the running algorithm. For example, if the data processed by the running algorithm is vibration sensor data from Line 1, then the matching degree between the two is high if the running algorithm and the target algorithm process the same data.

[0089] Secondly, calculate the first degree of matching between the operating environment and the operating requirements; In this embodiment, when the operating environment includes multiple dimensions, each dimension is matched with the corresponding dimension in the operating requirements to obtain the matching degree of each dimension. Then, the first matching degree is finally calculated based on the matching degrees of each dimension. Alternatively, each dimension can be used as the algorithm feature of the running algorithm, and the corresponding dimension in the operating requirements can be used as the algorithm feature of the target algorithm. The first matching degree is obtained by comparing the algorithm features of the two.

[0090] Third, calculate the second degree of matching between the data type being processed and the data type of the data source; In this embodiment, the second matching degree can be achieved through either precise matching or fuzzy matching. For example, when precise matching is used, if both the data type processed by the algorithm and the data type of the data source are data from the vibration sensor on line 1, the matching degree is considered to be 100%; otherwise, the matching degree is considered to be 0. When fuzzy matching is used, if the data type processed by the algorithm is data from the vibration sensor on line 1 and the data type of the data source is data from the vibration sensor on line 2, the matching degree is considered to be 80%. If the data type processed by the algorithm is data from the vibration sensor on line 1 and the data type of the data source is data from the temperature sensor on line 1, the matching degree is considered to be 0.

[0091] Finally, based on the first matching degree and the second matching degree, the environmental affinity of the candidate nodes is calculated, and the environmental affinity of each candidate node is finally obtained.

[0092] In this embodiment, weights can be set for the first matching degree and the second matching degree, and the environmental affinity can be obtained by weighted summation of the first matching degree and the second matching degree according to the set weights. Multiple candidate nodes can calculate their respective environmental affinity in parallel, or the environmental affinity of each candidate node can be calculated serially.

[0093] By calculating environment affinity, when a candidate node already has an algorithm of the same type as the target algorithm, that candidate node can be prioritized as the target node. In this case, when running the target algorithm, there is no need to reinitialize the computation context; instead, the existing runtime environment of the target node can be directly reused. This mechanism avoids the processes of image layer reassembly, library file decompression, and complex runtime environment reset, reducing the switching time for similar tasks from "seconds" to "milliseconds".

[0094] In optional implementations, the impact on data source access latency mainly includes latency caused by network transmission and startup warm-up time. To more reasonably evaluate these two parts of time, this embodiment provides an implementation method for estimating the transmission time of each candidate node accessing the data source and the startup time of the target algorithm when the logical identifier of the data source is URN: First, for any candidate node, obtain the network address of the candidate node; Secondly, based on the network address and URN, the transmission time of the candidate nodes is estimated; In this embodiment, the physical address of the data source is obtained according to the URN. By comparing the network address and the physical address of the data source, the transmission time of the candidate node can be estimated. For example, the path prefix matching degree of the network address and the physical address of the data source can be compared. The higher the path prefix matching degree, the less transmission time the candidate node needs to access the data source. Generally speaking, the smaller the local area network, the shorter the transmission time of data within the same local area network.

[0095] Alternatively, you can send a ping command to the data source to estimate the transmission time.

[0096] Third, obtain the mirror information of the most recently run algorithm of the candidate node; In this embodiment, the image information includes, but is not limited to, the program language interpreter associated with the most recently run algorithm, the cached information of common dependency packages in memory, the local storage information of the layers of the most recently run algorithm, and the local cached information of the image file.

[0097] Finally, the startup time of the candidate nodes is estimated based on the mirror information, and the transmission time and startup time of each candidate node are ultimately estimated.

[0098] In this embodiment, addressing the pain points of large fluctuations in industrial network environments and large image packet sizes, a specific implementation method is provided to evaluate the startup status of candidate nodes based on image information, and to quantify the startup time based on the evaluation results: the startup status includes three types: hot, warm, and cold. When all layers of the recently run algorithm image are local, and the recently run algorithm and the target algorithm are of the same type, it means that the CPU cache and memory page table are partially hot, resulting in minimal startup latency. In this case, the candidate node's startup status is hot. When the recently run algorithm and the target algorithm are of the same type, and the algorithm image file exists on the local disk (Containerd cache), but there are no active processes, it means that only file system mounting is required, without network download, and startup can usually be completed within 1-3 seconds. In this case, the candidate node's startup status is warm. When there are no image layers related to the target algorithm locally, it means that the current situation is a weak network or isolated industrial environment, with a delivery delay of minutes. In this case, the candidate node's startup status is cold. Prioritize candidate nodes in a "hot" or "warm" state as target nodes to reduce the time spent downloading images from object storage and initializing the environment. Combined with... Figure 1As a specific implementation method, the intelligent scheduler will maintain a global "Image Cache Manifest" for each node to determine the startup state of the node based on the node's image cache manifest.

[0099] In an optional implementation, after determining the target driver type and target node, this embodiment also provides a processing method to enable the target algorithm to run: First, download the algorithm package of the target algorithm from the download address; In this embodiment, the download address of the algorithm package of the target algorithm can be obtained from the description information, and the algorithm package can be downloaded according to the download address.

[0100] Secondly, download the driver package according to the target driver type; In this embodiment, when determining the target driver type, the target node can bind the target driver type to the target node. Here, the driver package can be downloaded directly from the corresponding address based on the bound target driver type.

[0101] It should be noted that the download of the algorithm package and the driver package can be performed in parallel or sequentially. This embodiment does not limit the order in which they are downloaded.

[0102] Third, build the runtime environment based on the algorithm package and driver package; In this embodiment, as one implementation method, the SPI mechanism can be used to dynamically construct the runtime environment. See the foregoing for details. Figure 1 Instantiation of the SPI adapter in the code.

[0103] Fourth, convert the logical identifier into the corresponding physical address; In this embodiment, to better adapt to different types of heterogeneous storage resources, a common format for the physical address URI of the data source can be defined at the underlying level. An example format definition is: [scheme]: / / [credentials]@[endpoints] / [namespace] / [entity]?[parameters]. Here, [scheme] represents the storage protocol (e.g., Kafka), [credentials] represents authentication information (e.g., username, password), [endpoints] represents the cluster access point address, [namespace] represents the logical isolation zone (e.g., database name), [entity] represents the specific data object (e.g., table name or Topic), and [parameters] represents connection control parameters (e.g., timeout settings). For multi-node clusters (e.g., Kafka or Cassandra), comma separation can be supported in the endpoints section (e.g., node1:port, node2:port).

[0104] In actual runtime, the SDK can parse the passed logical URN and map it to a URI containing specific physical link information. For example, the URI might be: (kafka: / / admin:pass@192.xx.1.xx:9092,192.xx.1.xx:9092 / order_events?acks=all), thus completing the data reading.

[0105] Finally, the input data required by the target algorithm is read from the physical address to execute the target algorithm in the runtime environment according to the target driver type.

[0106] In this embodiment, as one implementation method, the SDK mechanism can be used to request the input data required by the target algorithm. That is, the SDK receives the access request for the source data, and the access request carries the URN of the source data. The SDK automatically instantiates a data reading instance according to the URN protocol header. For example, if the URN is: urn:ronds:kafka:topic_01, the instantiated data reading instance is KafkaConsumer, and the read data is returned to the target algorithm based on the KafkaConsumer instance.

[0107] In an optional implementation, to promptly detect issues where the process is alive but logically deadlocked during the execution of the target algorithm, and to further improve the automation of the target task execution, this embodiment also provides a dual-path heartbeat and self-healing processing method: First, determine whether the preset self-healing conditions are met based on the physical and logical heartbeat messages sent by the monitoring process. In this embodiment, the target node also runs a monitoring process for monitoring the operating environment. If the process PID of the running algorithm is not detected, the monitoring process attempts to restart the algorithm package. The monitoring process can be found in [link to monitoring process documentation]. Figure 1 The NodeAgent daemon shown in the diagram periodically reports logical heartbeat information to the target node, thus informing it of the execution status of the target algorithm. As one implementation, the target node can report logical heartbeat information every 10 seconds via the SDK. If no logical heartbeat information is received for more than one minute, it can be considered "zombie" and requires rescheduling.

[0108] Secondly, when the preset self-healing conditions are met, the preset self-healing strategy is executed to perform self-healing, so as to re-execute the target algorithm and complete the target task.

[0109] In this embodiment, the determination of a preset self-healing condition and the implementation of the corresponding preset self-healing strategy can be found in Table 1.

[0110] Table 1

[0111] To more intuitively illustrate the entire task scheduling process described above, this embodiment uses a centrifuge vibration analysis algorithm from a factory as an example. The algorithm logic is as follows: read high-frequency waveform data -> extract feature indicators -> calculate degradation trend -> send early warning signal. Its processing includes the following stages: Phase 1: Algorithm Development (Local Standalone Environment) The developers wrote Python code to implement the algorithm logic, but instead of specifying `kafka:9092`, they used `sdk.get_input("raw_wave")`. During local testing, the developers mapped `raw_wave` to a local CSV file using a local configuration file. This decoupled the algorithm logic from the data source format during the development phase.

[0112] Phase Two: Task Orchestration and Scheduling (corresponding to...) Figure 1 (L1 processing in the middle) User operation: Drag and drop the algorithm on the L1 interface and point the input to the production environment URN: urn:ronds:kafka:line1_sensor_01 (data source and runtime configuration information).

[0113] Intelligent scheduling decision: The intelligent scheduler discovered that the algorithm needs to process "real-time streaming data" with high latency requirements. The intelligent scheduler scanned the entire computing power and found that there was an edge computing node, Edge-001, with sufficient resources next to the production line.

[0114] Decision result: Assign the task to Edge-001 and specify the use of Lite-Python-Driver (because the data volume is moderate and a Spark cluster is not required).

[0115] Phase Three: Task Distribution and Assignment (corresponding to...) Figure 1 (L2 processing in) Distribution: L1 will distribute the Unified Spec containing the above decisions to the Edge-001 Agent.

[0116] Agent assembly: Upon receiving the instruction, the Agent identifies the need for Lite-Python-Driver through the SPI mechanism.

[0117] The Agent downloads the algorithm script and the lightweight driver package from S3.

[0118] The Agent starts the driver process and injects the physical address resolved by the URN (which is mapped to the Kafka address in the production environment at this time).

[0119] Phase 4: Operation and Self-Healing (During Operation) Execution: The algorithm runs on Edge-001. If the Kafka connection is interrupted due to network fluctuations halfway through the algorithm's execution, the SDK's logical heartbeat will disappear.

[0120] Self-healing: When the L1 monitoring center detects a loss of logical heartbeat, it immediately triggers the self-healing strategy, issuing instructions to the Agent to restart the driver process or switch to the backup data source URN. The entire process requires no manual intervention.

[0121] To perform the corresponding steps in the above embodiments and various possible implementations, an implementation of the task scheduling device 100 is given below. Please refer to... Figure 7 , Figure 7 This is a block diagram of the task scheduling device provided in this embodiment. It should be noted that the basic principle and technical effects of the task scheduling device 100 provided by the present invention are the same as those of the corresponding embodiments described above. For the sake of brevity, some parts of this embodiment are not mentioned.

[0122] The task scheduling device 100 includes an acquisition module 110, a determination module 120, and a generation module 130.

[0123] The acquisition module 110 is used to acquire the description information of the target algorithm and the execution requirements and data processing mode for completing the target task.

[0124] The determination module 120 is used to determine the target node based on execution requirements.

[0125] The determination module 120 is also used to determine the target driving type based on the node type and data processing mode of the target node. The target driving type represents the algorithm operation mode that matches the data processing mode.

[0126] The generation module 130 is used to generate target instructions based on the description information and target driver type, and send them to the target node so that the target node can build a runtime environment based on the target instructions and execute the target algorithm in the runtime environment according to the target driver type to complete the target task.

[0127] In an optional implementation, the execution requirements include resource requirements and operational requirements, and the determination module 120 is specifically used for: Select candidate nodes that meet the resource requirements from the preset node pool; Calculate the execution cost score for each candidate node based on operational requirements; The candidate node with the lowest execution cost score is selected as the target node.

[0128] In an optional implementation, the runtime requirements include logical identifiers of the data sources required by the target algorithm. Specifically, when determining the execution cost score of each candidate node based on the runtime requirements, the determining module 120 is further configured to: Calculate the resource load cost of each candidate node based on its resource usage information. Calculate the environmental affinity of each candidate node based on its running algorithm and target algorithm. Based on the logical identifier, estimate the transmission time for each candidate node to access the data source and the startup time of the target algorithm. Calculate the latency cost of each candidate node based on its transmission time and startup time. Calculate the execution cost score for each candidate node based on its resource load cost, environmental affinity, and latency cost.

[0129] In an optional implementation, the determining module 120 is specifically used to calculate the environmental affinity of each candidate node based on the running algorithm and the target algorithm of each candidate node, and is further used to: For any candidate node, obtain the runtime environment and data types processed by the algorithm running on the candidate node; Calculate the first degree of match between the runtime environment and runtime requirements; Calculate the second degree of matching between the data type being processed and the data type of the data source; Based on the first and second matching degrees, the environmental affinity of the candidate nodes is calculated, and finally the environmental affinity of each candidate node is obtained.

[0130] In an optional implementation, the logical identifier of the data source is the Uniform Resource Name (URN) of the data source. The determining module 120 is specifically used to estimate the transmission time for each candidate node to access the data source and the startup time of the target algorithm based on the logical identifier. For any candidate node, obtain the network address of the candidate node; Based on the network address and URN, estimate the transmission time of the candidate nodes; Obtain the image information of the most recently run algorithm of the candidate node; Based on the mirror information, the startup time of the candidate nodes is estimated, and finally the transmission time and startup time of each candidate node are estimated.

[0131] In an optional implementation, the description information includes the download address of the algorithm package of the target algorithm, the execution requirements include the logical identifier of the data source that the target algorithm needs to access, and the task scheduling device 100 further includes a running module 140, which is used for: Download the algorithm package for the target algorithm from the download address; Download the driver package according to the target driver type; Build the runtime environment based on the algorithm package and driver package; Convert the logical identifier into the corresponding physical address; Read the input data required by the target algorithm from the physical address to execute the target algorithm in the runtime environment according to the target driver type.

[0132] In an optional implementation, the target node also runs a monitoring process for monitoring the operating environment and a logical heartbeat message for reporting the execution status of the target algorithm. The task scheduling device 100 also includes a self-healing module 150, which is used for: Determine whether the preset self-healing conditions are met based on the physical and logical heartbeat messages sent by the monitoring process. When the preset self-healing conditions are met, the preset self-healing strategy is executed to perform self-healing, so as to re-execute the target algorithm and complete the target task.

[0133] This invention also provides a block diagram of an electronic device 10, which implements the task scheduling method of the aforementioned embodiments. Please refer to... Figure 8 , Figure 8 This is a block diagram of the electronic device 10 provided in this embodiment. The electronic device 10 includes a processor 11, a memory 12 and a bus 13. The processor 11 and the memory 12 are connected through the bus 13.

[0134] The processor 11 can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the task scheduling method in the above embodiments can be completed by the integrated logic circuits in the hardware of the processor 11 or by instructions in software form. The processor 11 can be a general-purpose processor, including a CPU (Central Processing Unit), NP (Network Processor), GPU (Graphics Processing Unit), etc.; it can also be a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Logic Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0135] The memory 12 is used to store the program that implements the task scheduling method. The program may be a software function module stored in the memory 12 in the form of software or firmware or embedded in the OS (Operating System) of the electronic device 10.

[0136] After receiving the execution instruction, the processor 11 executes the program to implement the task scheduling method of the aforementioned embodiment.

[0137] In summary, the present invention provides a task scheduling method, apparatus, electronic device, and computer-readable storage medium. The method includes: obtaining the description information of the target algorithm and the execution requirements and data processing mode for completing the target task; determining the target node according to the execution requirements; determining the target driver type according to the node type and data processing mode of the target node, wherein the target driver type represents the algorithm running mode that matches the data processing mode; generating target instructions according to the description information and target driver type, and sending them to the target node so that the target node can build a running environment based on the target instructions and execute the target algorithm in the running environment according to the target driver type to complete the target task. Compared with the prior art, the present invention has at least the following advantages: (1) by decoupling the implementation logic of the algorithm from its running environment, the running environment of the target algorithm is determined according to the actual requirements of the target task, and the same set of algorithm logic is smoothly switched between different running environments; (2) a "Uniform Resource Name (URN)" is introduced in L1 to replace the physical connection, and a "CalcDriver" is introduced in L2 as middleware. The algorithm code only references the logical URN through the Data Access SDK and does not contain any environmental dependencies; at runtime, the Agent dynamically loads the driver adapted to the current node (such as SparkClusterProvider or LocalProcessProvider) through the SPI mechanism. This two-layer decoupling mechanism allows the same algorithm artifact to be seamlessly migrated between heterogeneous computing power (edge ​​gateway / private cloud / public cloud) without any modification. The system automatically completes the "logical-physical" mapping at runtime, which significantly reduces the complexity of cross-environment deployment and improves the efficiency of algorithm delivery; (3) L1 is equipped with an "intelligent scheduler", which dynamically scores and routes tasks through a three-stage decision pipeline (pre-filtering, resource matching, decision output), combined with the control.mode (Batch / Stream / Service) and resource profile in the Unified Spec of the task specification, realizing a serverless-like (computation mode-agnostic) scheduling experience. It can automatically route real-time rules to edge Lite nodes and heavy LLM inference to cloud GPU nodes according to SLA constraints (such as latency sensitivity and throughput requirements). This not only unifies the heterogeneous computing paradigm, but also avoids the idleness or overload of hardware resources through the elastic scaling strategy, achieving the optimal balance between cost and performance; (4) Using Data Access SDK in conjunction with L1 layer metadata parsing, data connection Late Binding is realized.The establishment of physical connections is delayed until the moment the Runtime starts and calls the SDK interface, and authentication is performed through the temporary security access token (SAS) generated by the L1 task dispatcher. Thus, on the one hand, the physical changes of the data source are completely transparent to the running algorithm, and no downtime maintenance is required, which enhances business continuity; on the other hand, it avoids the plaintext exposure of sensitive configuration information in the network and code, and builds a security barrier from the control plane to the execution plane; (5) A dual-path monitoring mechanism combining the "physical heartbeat (PID monitoring)" at the L2 Agent level and the "logical heartbeat (business progress reporting)" at the Runtime SDK level is established, which can accurately distinguish between "resource crash" and "logical deadlock". Once an anomaly occurs, the L1 control plane can accurately execute retry strategies (such as hot migration to a healthy node) or trigger alarms according to the anomaly type, which greatly improves the reliability and self-healing capability of industrial algorithm operation in unattended environments; (6) The Agent of the L2 layer adopts the SPI (Service Provider Interface) design, which abstracts "how to start a task" into a pluggable Provider interface. When it is necessary to support a new computing framework, only the corresponding Provider implementation class needs to be added, without modifying the upper-layer scheduling logic or the lower-layer algorithm code. This highly cohesive and loosely coupled design enables the system to evolve at low cost and remain compatible with new computing engines that will emerge in the future.

[0138] The above descriptions are merely various embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A task scheduling method, characterized in that, The method includes: Obtain the description information of the target algorithm, as well as the execution requirements and data processing mode for completing the target task; Determine the target node based on the execution requirements; The target driver type is determined based on the node type of the target node and the data processing mode, and the target driver type represents the algorithm running mode that matches the data processing mode; Based on the description information and the target driver type, a target instruction is generated and sent to the target node, so that the target node can build a runtime environment based on the target instruction and execute the target algorithm in the runtime environment according to the target driver type to complete the target task.

2. The method according to claim 1, characterized in that, The execution requirements include resource requirements and operational requirements, and the step of determining the target node based on the execution requirements includes: Candidate nodes that meet the resource requirements are selected from the preset node pool; Calculate the execution cost score for each candidate node based on the stated operational requirements; The candidate node with the lowest execution cost score is selected as the target node.

3. The method according to claim 2, characterized in that, The operational requirements include logical identifiers of the data sources that the target algorithm needs to access, and the step of calculating the execution cost score for each candidate node based on the operational requirements includes: Calculate the resource load cost of each candidate node based on the resource usage information of each candidate node; The environmental affinity of each candidate node is calculated based on the running algorithm of each candidate node and the target algorithm. Based on the logical identifier, the transmission time for each candidate node to access the data source and the startup time for starting the target algorithm are estimated. Calculate the latency cost of each candidate node based on its transmission time and startup time. Calculate the execution cost score for each candidate node based on its resource load cost, environmental affinity, and latency cost.

4. The method according to claim 3, characterized in that, The step of calculating the environmental affinity of each candidate node based on the running algorithm of each candidate node and the target algorithm includes: For any candidate node, obtain the running environment and data type of the algorithm running on the candidate node; Calculate the first degree of matching between the operating environment and the operating requirements; Calculate the second matching degree between the processed data type and the data type of the data source; Based on the first matching degree and the second matching degree, the environmental affinity of the node is calculated, and finally the environmental affinity of each candidate node is obtained.

5. The method according to claim 3, characterized in that, The logical identifier of the data source is the Uniform Resource Name (URN) of the data source. The step of estimating the transmission time for each candidate node to access the data source and the startup time of the target algorithm based on the logical identifier includes: For any candidate node, obtain the network address of the candidate node; Based on the network address and the URN, estimate the transmission time of the candidate node; Obtain the image information of the most recently run algorithm of the candidate node; Based on the image information, the startup time of the candidate nodes is estimated, and finally the transmission time and startup time of each candidate node are estimated.

6. The method according to claim 1, characterized in that, The description information includes the download address of the algorithm package of the target algorithm, the execution requirements include the logical identifier of the data source that the target algorithm needs to access, and the method further includes: Download the algorithm package of the target algorithm from the provided download address; Download the driver package according to the target driver type; Build a runtime environment based on the algorithm package and the driver package; Convert the logical identifier into the corresponding physical address; The input data required by the target algorithm is read from the physical address to execute the target algorithm in the runtime environment according to the target driver type.

7. The method according to claim 6, characterized in that, The target node also runs a monitoring process for monitoring the operating environment and a logical heartbeat message for reporting the execution status of the target algorithm. Following the step of reading the input data required by the target algorithm from the physical address to execute the target algorithm in the operating environment according to the target driver type, the process includes: Determine whether the preset self-healing conditions are met based on the physical heartbeat message and the logical heartbeat message sent by the monitoring process. When the preset self-healing conditions are met, the preset self-healing strategy is executed to perform self-healing, so as to re-execute the target algorithm and complete the target task.

8. A task scheduling device, characterized in that, The device includes: The acquisition module is used to acquire the description information of the target algorithm, as well as the execution requirements and data processing mode for completing the target task; The determination module is used to determine the target node based on the execution requirements; The determining module is further configured to determine the target driving type based on the node type of the target node and the data processing mode, wherein the target driving type represents the algorithm running mode that matches the data processing mode; The generation module is used to generate target instructions based on the description information and the target driver type, and send them to the target node so that the target node can build a runtime environment based on the target instructions and execute the target algorithm in the runtime environment according to the target driver type to complete the target task.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store a program, and the processor being used to implement the task scheduling method as described in any one of claims 1-7 when executing the program.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the task scheduling method as described in any one of claims 1-7.

Images