A fault-tolerant scheduling method and device for an OODA flow operating system

Abstract

The application provides a fault-tolerant scheduling method and device for an OODAflow operating system, which comprises the following steps: receiving an intelligent computing task and splitting the intelligent computing task into single tasks, encapsulating the single tasks and an environment health degree monitoring process into self-contained containers, and waiting for a computing device to schedule; deploying the single tasks encapsulated into the self-contained containers to corresponding computing nodes; in the process of executing the tasks on the computing nodes, the environment health degree monitoring process monitors the health condition of a computing device to which the computing node belongs in real time, if the health condition has a fault, the computing node exits the task and saves a checkpoint file; if there is no fault, the execution result of the task is written back to a shared storage area. The fault-tolerant scheduling system provided by the application realizes a fault-tolerant mechanism and efficient fault-tolerant scheduling for the OODAflow operating system task, and guarantees the real-time performance and reliability of the OODAflow operating system in processing the computing task.

Description

Technical Field

[0001] The present invention relates to the fields of computer technology, computer service technology, natural language processing technology, and image perception technology, and in particular to a fault-tolerant scheduling method and device for an OODAflow operating system. Background Art

[0002] OODA (Organizational Optimization) is the first principle of behavioral intelligence analysis. This model, based on the cycle of perception, judgment, decision-making, and execution, emphasizes the flexible scheduling of low-coupling data flows and control decoupling for intelligent computing tasks. With the rapid development of intelligent computing, more and more intelligent computing businesses are adopting the OODA paradigm to build task processes. A typical example is the rapidly developing field of embodied intelligence.

[0003] Embodied intelligence is a new type of intelligent computing service that combines large language models with robots. It aims to enable robots to understand natural language, autonomously determine how to implement commands, perceive their surroundings, and execute correct behaviors. Its task scenarios are typical of complex computing scenarios that combine control computing with intelligent computing. In end-to-end services, from understanding natural language to executing correct actions, multiple computing stages are involved. Therefore, researchers in the field of embodied intelligence are gradually integrating end-to-end embodied intelligence tasks with the OODA paradigm, decoupling intelligent computing from motion control within the end-to-end task flow. Furthermore, they are decoupling environmental perception from motion control, and finally, further decoupling natural language processing from action command generation. This creates an OODA-like loop of "perceiving external information - natural language processing - generating action commands - and responding to commands."

[0004] The development of embodied intelligence is just the tip of the iceberg of OODA's increasingly broad and profound impact on the field of intelligent computing. With the emergence of intelligent computing businesses such as large language models and multimodal intelligence, whose parameter quantities and data volumes are growing at an exponential rate, OODA's problem analysis paradigm of decoupling complex tasks will inevitably be more widely used in the field of intelligent computing.

[0005] The OODAflow operating system is an operating system that deeply integrates the OODA paradigm of intelligent analysis. It divides the overall process of intelligent computing tasks into four sub-processes: O (perception), O (judgment), D (decision-making), and A (execution), and executes them sequentially. The four sub-processes are loosely coupled, and the amount of data transmitted between them is small, providing greater schedulability to optimize the execution efficiency of intelligent computing tasks. The OODAflow operating system is often used in edge scenarios that emphasize real-time performance, such as controlling important mobile equipment or deploying critical services. The real-time tasks it solves are mostly recognition tasks, where intelligent computing plays a large role. The OODAflow operating system, which controls compute nodes to run intelligent computing tasks, typically runs on computing devices such as CPUs and neural network accelerators, which can be divided into at least one computing node.

[0006] However, as a newly proposed computing system, the OODAflow operating system has not yet been designed with fault-tolerant functions. If a computing device failure occurs while the OODAflow operating system is running a computing task, the running task will be abnormally interrupted, which will in turn affect the subsequent calculations of OODAflow. This has a very large impact in real-time tasks, and corresponding fault-tolerant measures are needed to deal with this situation.

[0007] The OODAflow operating system does not have a stable and reliable fault-tolerant scheduling system. When an abnormality occurs in a certain link of the OODA intelligent computing task, the task queue will be blocked and the subsequent process cannot proceed normally. This has a great impact on occasions with high real-time requirements. Summary of the Invention

[0008] In view of the shortcomings of existing technologies, such as Figure 5 As shown, the present invention proposes a fault-tolerant scheduling method for the OODAflow operating system, which includes:

[0009] The initial step is to receive the intelligent computing task and split it into four single tasks: perception, judgment, decision-making, and execution. The single tasks and the environmental health monitoring process are encapsulated into a self-sustaining container, waiting for scheduling by the computing device; the computing node is a CPU, GPU or neural network accelerator; the intelligent computing task can be an image classification task, semantic segmentation task, text recognition task, public opinion monitoring task, etc.

[0010] In the execution step, the control node scheduler in the computing device deploys the single task encapsulated into a self-sustaining container to the corresponding computing node based on the performance of each computing node; the computing node determines whether the received single task has a checkpoint file to be executed. If so, the task corresponding to the checkpoint file is executed first; if not, the current task is executed;

[0011] Monitoring step: During the execution of tasks by computing nodes, the environmental health monitoring process monitors the health of the computing devices of the current computing node in real time. If there is a health failure, the current computing node exits the task and saves the checkpoint file. If there is no failure, the task execution result is written back to the shared storage area, and a task completion signal is sent to the control node. The corresponding execution results can be image classification results, semantic segmentation results, text recognition results, public opinion monitoring results, etc.

[0012] In the end step, the control node determines whether the task is successfully executed based on the task completion signal. If the task is successfully executed, the process ends; otherwise, it returns to the initial step to reschedule the task.

[0013] The fault-tolerant scheduling method for the OODAflow operating system, wherein the computing node includes: perception computing resources, judgment computing resources, decision computing resources and execution computing resources; the computing nodes and management nodes together constitute a computing cluster interconnected by a local area network built by a local virtual bridge.

[0014] The fault-tolerant scheduling method for the OODAflow operating system, wherein the monitoring step includes: embedding checkpoint generation code in the single task executed by the computing node, and when the single task is interrupted, the checkpoint generation code cooperates with the environmental health monitoring process of the computing node to generate a checkpoint file;

[0015] The checkpoint file includes: a metadata file of the computational graph structure, an index file of the mapping relationship between variable names and variable data locations, and a data file of model weights and variable data.

[0016] like Figure 6 As shown, the present invention also proposes a fault-tolerant scheduling device for the OODAflow operating system, which includes:

[0017] The initial module receives intelligent computing tasks and splits them into four single tasks: perception, judgment, decision-making, and execution. It then encapsulates the single tasks and the environmental health monitoring process into a self-sustaining container, waiting for scheduling by computing devices. The computing node is a CPU, GPU, or neural network accelerator.

[0018] The execution module, in which the control node scheduler in the computing device deploys the single task, which has been encapsulated into a self-sustaining container, to the corresponding computing node based on the performance of each computing node. The computing node determines whether the received single task has a checkpoint file to be executed. If so, it prioritizes executing the task corresponding to the checkpoint file; otherwise, it executes the current task.

[0019] The monitoring module monitors the health of the computing devices of the current computing node in real time while the computing node is executing a task. If there is a health failure, the current computing node exits the task and saves the checkpoint file. If there is no failure, the task execution result is written back to the shared storage area and a task completion signal is sent to the control node.

[0020] In the end module, the control node determines whether the task is successfully executed based on the task completion signal. If the task is successfully executed, the process ends; otherwise, it returns to the initial module to reschedule the task.

[0021] The fault-tolerant scheduling device for the OODAflow operating system, wherein the computing node includes: perception computing resources, judgment computing resources, decision computing resources and execution computing resources; the computing nodes and management nodes together constitute a computing cluster interconnected by a local area network built by a local virtual bridge.

[0022] The fault-tolerant scheduling device for the OODAflow operating system, wherein the monitoring module includes: embedding checkpoint generation code in the single task executed by the computing node, and when the single task is interrupted, the checkpoint generation code cooperates with the environmental health monitoring process of the computing node to generate a checkpoint file;

[0023] The checkpoint file includes: a metadata file of the computational graph structure, an index file of the mapping relationship between variable names and variable data locations, and a data file of model weights and variable data.

[0024] The present invention also proposes an electronic device, which includes the fault-tolerant scheduling device for the OODAflow operating system.

[0025] The electronic device is connected to an information display device, which is used to display the execution result using display parameters and attributes set by the user or through an artificial intelligence model.

[0026] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the fault-tolerant scheduling method for the OODAflow operating system are implemented.

[0027] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the fault-tolerant scheduling method for the OODAflow operating system are implemented.

[0028] From the above scheme, it can be seen that the advantages of the present invention are:

[0029] The fault-tolerant scheduling system for the OODAflow operating system implements the fault-tolerant mechanism and efficient fault-tolerant scheduling of the OODAflow operating system tasks, ensuring the real-time and reliability of the OODAflow operating system in processing computing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is the overall flow chart of this application;

[0031] Figure 2 This is a block diagram of the hardware structure of the present invention;

[0032] Figure 3 This is a diagram of the task pipeline of the OODAflow operating system;

[0033] Figure 4 This is a flow chart of fault-tolerant scheduling of the present invention;

[0034] Figure 5 Flow chart of the method of the present invention;

[0035] Figure 6 This is a module diagram of the device of the present invention;

[0036] Figure 7 This is a schematic structural diagram of a first electronic device of the present invention;

[0037] Figure 8 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0038] Figure 9 This is a schematic structural diagram of a second electronic device according to the present invention.

[0039] Reference numerals:

[0040] A-First electronic device;

[0041] B-Fault-tolerant scheduling device for OODAflow operating system;

[0042] C-data acquisition equipment;

[0043] D-information display device;

[0044] 1000- second electronic device;

[0045] Ⅰ-computing unit;

[0046] II-ROM;

[0047] III-RAM;

[0048] IV-bus;

[0049] V-interface;

[0050] VI-input unit;

[0051] VII-output unit;

[0052] VIII-Storage medium;

[0053] IX-Communication unit. DETAILED DESCRIPTION

[0054] It should be noted that, in this application, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0055] Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0056] The processor described in the present invention is the control center of an electronic device and can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0057] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0058] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smartphones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0059] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0060] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0061] The above embodiments can be implemented in whole or in part through software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0062] It should also be understood that the term "and / or" in this document simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " in this document generally indicates an "or" relationship between the related objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0063] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0064] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0065] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0066] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0067] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0068] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0069] In order to achieve the above technical effects, the present invention includes the following key technical points:

[0070] Key Point 1: The device state consistency maintenance system, which consists of containerized tasks, compute nodes, and control nodes, provides a three-pronged approach. When a failure occurs, the containerized task's fault-tolerance program takes the first-priority response, saving the current state for subsequent recovery. The compute node's device health monitoring process takes the second-priority response, obtaining the node's device state and sending it to the control node. The control node's scheduling process takes the third-priority response, obtaining the current status of the abnormal node and applying this information to subsequent fault-tolerance scheduling.

[0071] Key point 2: Checkpoint recovery fault tolerance built with shared storage and container task environment variable tags. By configuring a background monitoring process and environment variables for each container task instance, the task itself can immediately respond to a failure, triggering task interruption and creation of a unique checkpoint file for fault-tolerant scheduling. The file is saved in the shared storage area and named with a unique environment variable. This ensures that the checkpoint file can be correctly indexed in subsequent fault-tolerant operations, guaranteeing the data relevance of the OODA task.

[0072] Key Point 3: In the OODAflow operating system, multiple virtual bridges are used to build an internal LAN for data transmission. This multi-bridge internal LAN not only implements the fault-tolerant scheduling system but also disperses bandwidth pressure, reduces I / O congestion, and improves the real-time performance of the OODAflow operating system's fault-tolerant scheduling function.

[0073] Specifically, in the present invention, the custom service program of the control plane can uniformly monitor the resource usage of all computing devices of the OODAflow operating system, and schedule computing task instances in real time to achieve the optimal overall machine computing efficiency; at the same time, the background service is automatically run in each computing task instance to monitor the failure of the computing device of this node. Once a failure occurs, the computing task instance will automatically execute a "suicide" operation, retain the current checkpoint file and exit abnormally. The custom daemon process of the control plane will sense the failure of the node computing device and automatically delete all computing task instances deployed on the node. If the number of computing task instances does not match the preset number, it will be captured by the control node service and a new computing task instance will be created. The newly created instance will receive the checkpoint file where the last task failed to execute smoothly, and continue execution from the checkpoint to ensure the real-time performance of the system and avoid task congestion.

[0074] While researching OODAflow's fault-tolerant scheduling capabilities, the inventors discovered a key flaw in existing technologies: the control node's inability to detect computing device failures in real time. This limitation results in the computing task often being interrupted by the time the control node finally detects a computing device failure, preventing effective fault-tolerant scheduling. This in turn causes subsequent OODA tasks to be blocked, severely impacting the system's real-time performance.

[0075] After extensive research, this paper proposes splitting the OODA closed-loop task into four separate subtasks (observation, orientation, decision, and action). Each OODA task round is tagged with a unique environment variable key to facilitate subsequent scheduling. This allows the control program to accurately specify y subprocesses within x rounds of tasks. Each subtask is encapsulated as a fully functional, self-sustaining task in the form of a container, which includes a running environment health monitoring process.

[0076] The key is that these health monitoring processes, along with the compute node health checkers and the control node system scheduler, work together to maintain a "device state consistency." This concept was developed to address the issue of real-time response to compute node device failures.

[0077] Specifically, when a compute node device fails, the self-sustaining task in the container first takes emergency measures and saves the current checkpoint file for subsequent fault tolerance. Next, the compute node's health monitoring program is triggered, recording the compute node's abnormality information and feeding it back to the control node's task scheduler. This information is used in the scheduling of the next new OODA task.

[0078] Because the checkpoint file has been saved and the relevant failure information has been synchronized to the control node scheduler, the control node can immediately create a new container-based self-sustaining task on another intact compute node and resume execution from the last saved checkpoint file. This ensures the continuity of OODA tasks and the real-time performance of the system.

[0079] Through this "device state consistency" approach, the inventors have achieved real-time response to fault handling. This not only solves the monitoring lag problem in the prior art, but also significantly improves the system's fault tolerance and the real-time performance of task execution in the event of a computing node failure. This innovative approach provides a reliable technical guarantee for the application of the OODAflow operating system in complex computing environments, and ensures the stable operation of the system under high-load and high-risk environments. It should be noted that as long as the task group meets the "no control-related, data-related" requirement, the monitoring method of the present invention can be used. The unique design of the monitoring method of the present invention is to perform high-real-time fault-tolerant scheduling of tasks that can be split (and then optimized for pipelines) split.

[0080] To illustrate the above-mentioned features and effects of the present invention more clearly and easily, the following embodiments are specifically described below with reference to the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are for illustrative purposes only. The scope of protection of the present invention is not limited to the disclosed embodiments; the present invention is defined by the appended claims.

[0081] like Figure 1 As shown, the present invention includes:

[0082] Step 1: Receive the task. The system receives the task and splits the single-round OODA task into four individual tasks: Observe, Orient, Decide, and Action. Each task is assigned unique environment variables for that round to maintain data relevance during multi-round OODA task scheduling. The individual tasks and the environmental health monitoring process are packaged into a self-sustaining container awaiting scheduling.

[0083] Step 2: The control node scheduler determines whether there are enough idle devices to calculate the current round of OODA tasks based on the current computing device status returned by each computing node.

[0084] If there are enough idle devices, proceed to step 3.

[0085] If there are not enough idle devices, the task spins waiting to be deployed.

[0086] Step 3: Deploy the task to a specific computing node.

[0087] Step 4: After the task is deployed to the computing node, determine whether there is a checkpoint file to be executed.

[0088] If a checkpoint file exists, proceed to step 5.

[0089] If there is no checkpoint file, go to step 6.

[0090] Step 5: Execute the checkpoint file first.

[0091] Step 6: Execute the current task.

[0092] Step 7: During the task execution, the background fault monitoring process monitors the health of the computing devices in the current operating environment in real time.

[0093] If the computing device health condition is faulty, go to step 8.

[0094] If there is no fault, proceed to step 9.

[0095] Step 8: Exit the task and save the checkpoint file.

[0096] Step 9: Write the execution results back to the shared storage area for subsequent use by the control node.

[0097] Step 10: Determine if the task was successfully executed. The control program does not know whether step 9 has been written back to the storage area (it does not monitor the storage area). After step 9 is completed, a signal is sent to the control program to tell it that the task has been completed. At this time, step 10 determines that the task was successfully executed.

[0098] If the task is successfully executed, the process ends.

[0099] If the task is not successfully executed, return to step 1 and reschedule the task.

[0100] Through the above steps, the fault-tolerant scheduling process of the OODAflow operating system is completed, and efficient scheduling and execution of tasks are achieved, ensuring that the system can continue to operate in the event of a failure.

[0101] The computing resource management mode in the OODAflow operating system is that a single set of OODA components is used as a computing node (one perception computing resource + one judgment computing resource + one decision computing resource + one execution computing resource). All computing resources of the whole machine are divided into N sets (in this example Figure 2 There are two sets in the figure). These are two computing nodes, which together with the host management node constitute a computing cluster interconnected by a local area network built by a local virtual bridge. The fault tolerance function of the OODA computing task is achieved by adding various functional components in the cluster.

[0102] like Figure 2 As shown in Figure 1, the control node task scheduler plays a key role in the OODAflow operating system. By controlling the computing resources and task completion status of each computing node in real time, it realizes the real-time scheduling of task flows to achieve the effect of pipeline acceleration. Figure 2 Detailed description of each part:

[0103] Compute node services:

[0104] The Compute Node Service and the Control Node Service are responsible for creating self-sustaining OODA computing task instance programs in the form of containers. These programs will deploy OODA computing tasks to specific compute nodes according to the requirements of the scheduler.

[0105] For example, when a new OODA task is assigned to a compute node, the compute node service creates a container instance to run the task and ensures that the task is executed according to the predetermined process.

[0106] Control node service:

[0107] Control node services are similar to compute node services in that they are responsible for creating and managing OODA compute task instance programs, but they run on control nodes.

[0108] The control node service communicates with each computing node through the network interface to coordinate the allocation and execution of tasks.

[0109] These services also monitor the progress of tasks and log relevant information when tasks complete or fail.

[0110] Compute node health monitoring process:

[0111] The compute node health monitoring process runs on each compute node to determine whether each compute node is functioning properly.

[0112] They monitor the health status of computing nodes in real time, including CPU load, memory usage, network status, computing card failures, etc.

[0113] If any anomalies are detected, such as hardware failures or performance bottlenecks, the health monitoring process immediately records these anomalies and sends the information to the control node task scheduler.

[0114] Control node scheduling process:

[0115] The control node scheduling process is the core scheduling program running on the control node and is responsible for the fault-tolerant scheduling of the entire computing system.

[0116] It will integrate the health information of each computing node to determine whether a computing node has failed.

[0117] After detecting a computing node failure, the control node scheduling process will immediately trigger the fault-tolerant scheduling mechanism to reallocate tasks on the failed node to other healthy computing nodes.

[0118] It is also responsible for optimizing the distribution of tasks to maximize system performance and resource utilization.

[0119] Static IP routing:

[0120] Each compute node communicates with the control node task scheduler via static IP routing.

[0121] Static IP routing ensures stable and reliable network connections between nodes, supporting efficient data transmission and task scheduling.

[0122] This part of the network architecture provides basic communication guarantees for the system, enabling the control node to accurately assign tasks to each computing node and receive health monitoring data from the computing nodes.

[0123] Shared storage area:

[0124] The shared storage area is used to store intermediate results, checkpoint files, and final results of tasks.

[0125] All computing nodes and control nodes can access the shared storage area, ensuring that task data can be shared and reused in the system.

[0126] This design not only improves data availability and consistency, but also supports task fault tolerance and recovery capabilities.

[0127] Through this mechanism, the OODAflow operating system efficiently manages and schedules computing tasks, ensuring timely system response and adjustment when computing node failures occur. The task scheduler achieves high performance and reliability through real-time monitoring and flexible scheduling. In this way, the OODAflow operating system not only solves the problems of fault detection and resource utilization in task scheduling, but also significantly improves the system's real-time performance and task execution efficiency.

[0128] The checkpoint generation code is embedded in the tasks deployed on the compute nodes. When the task is unexpectedly interrupted due to hardware anomalies, this code will work with the compute node's health monitoring process to generate the current checkpoint file. The specific save format is as follows:

[0129] Checkpoint file details:

[0130] Metadata file (.meta): saves the structure of the computational graph;

[0131] Index file (.index): stores the mapping between variable names and variable data locations;

[0132] Data file (.data-xxxxx-of-xxxxx): holds the actual model weights and variable data;

[0133] Metadata file (MetaGraphDef): The file extension is usually .meta. This file stores the structure of the computational graph, including definitions of nodes, variables, operations, etc. It includes the metadata of the computational graph (graph structure and operations) and set definitions (for example, variable sets);

[0134] Index file: The file extension is usually .index. This file stores the mapping between variable names and variable data in the data file. This includes the mapping between variable names and variable data locations, as well as variable data block information.

[0135] Data files: The file extension is usually .data-????-of-????, where ??? represents the file number and the total number of files. These files store the actual weights and variable data of the model. They include parameter data such as model weights and biases, as well as other variable data.

[0136] like Figure 3 、 4 As shown in the figure, the OODAflow operating system schedules and executes tasks during operation and handles failures. The specific process is as follows:

[0137] T0 moment:

[0138] Start executing the perception task of OODA closed loop 1.

[0139] Start executing the judgment task of OODA closed loop 2.

[0140] T1 moment:

[0141] The perception tasks of OODA loop 3 and OODA loop 4 also begin to be executed.

[0142] T2 moment:

[0143] When node 1 is executing the judgment task of OODA closed loop 2, the computing device fails.

[0144] The OODAflow operating system will catch this exception and interrupt the OODA intelligent computing task that has already started on node 1.

[0145] The current computational state is saved as a checkpoint file in the shared storage space. Specifically, the judgment results of OODA loop 2 and the perception results of OODA loop 4 are saved as checkpoints.

[0146] Meanwhile, Node 2 continues to operate normally without failure, so the tasks of OODA loop 1 and OODA loop 3 continue to execute.

[0147] Scheduling priority:

[0148] In the OODAflow operating system, the scheduling priority rules are as follows:

[0149] Tasks that have already started have the highest priority.

[0150] In addition, the smaller the OODA loop number, the higher the priority.

[0151] T3 moment:

[0152] After node 2 determines that the task is completed, the computing device is released.

[0153] The OODAflow operating system immediately reads the checkpoint file for the perception results of OODA loop 4 and sends it to the judgment device of node 2 for calculation, in order to complete the task of OODA loop 4 as quickly as possible. That is, at time T3, node 2 completes its own judgment task and takes over the judgment task of node 1, at which point node 1's judgment task begins execution.

[0154] T4 moment:

[0155] After node 2 completes its decision-making task, the computing device is released.

[0156] The OODAflow operating system immediately reads the checkpoint file of the judgment result of OODA closed loop 2 and sends it to the decision device of node 2 for calculation, so as to complete the task of OODA closed loop 2 as soon as possible.

[0157] Subsequent scheduling:

[0158] The subsequent scheduling process of OODA loop 5 and OODA loop 6 is completed using a staggered pipeline to maximize real-time performance and performance release.

[0159] Through the above steps, the OODAflow operating system can not only monitor and respond to faults, but also give priority to high-priority tasks after resources are released, thereby improving the system's real-time fault response and task execution efficiency.

[0160] This new technical solution effectively solves the fault monitoring and resource occupation problems in the OODAflow operating system and improves the performance and real-time performance of the system.

[0161] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0162] like Figure 6 As shown, the present invention also proposes a fault-tolerant scheduling device for the OODAflow operating system, which includes:

[0163] The initial module receives intelligent computing tasks and splits them into four single tasks: perception, judgment, decision-making, and execution. It then encapsulates the single tasks and the environmental health monitoring process into a self-sustaining container, waiting for scheduling by computing devices. The computing node is a CPU, GPU, or neural network accelerator.

[0164] The execution module, in which the control node scheduler in the computing device deploys the single task, which has been encapsulated into a self-sustaining container, to the corresponding computing node based on the performance of each computing node. The computing node determines whether the received single task has a checkpoint file to be executed. If so, it prioritizes executing the task corresponding to the checkpoint file; otherwise, it executes the current task.

[0165] The monitoring module monitors the health of the computing devices of the current computing node in real time while the computing node is executing a task. If there is a health failure, the current computing node exits the task and saves the checkpoint file. If there is no failure, the task execution result is written back to the shared storage area and a task completion signal is sent to the control node.

[0166] In the end module, the control node determines whether the task is successfully executed based on the task completion signal. If the task is successfully executed, the process ends; otherwise, it returns to the initial module to reschedule the task.

[0167] The fault-tolerant scheduling device for the OODAflow operating system, wherein the computing node includes: perception computing resources, judgment computing resources, decision computing resources and execution computing resources; the computing nodes and management nodes together constitute a computing cluster interconnected by a local area network built by a local virtual bridge.

[0168] The fault-tolerant scheduling device for the OODAflow operating system, wherein the monitoring module includes: embedding checkpoint generation code in the single task executed by the computing node, and when the single task is interrupted, the checkpoint generation code cooperates with the environmental health monitoring process of the computing node to generate a checkpoint file;

[0169] The checkpoint file includes: a metadata file of the computational graph structure, an index file of the mapping relationship between variable names and variable data locations, and a data file of model weights and variable data.

[0170] like Figure 7 As shown, the present invention further proposes a first electronic device A in another embodiment, which includes the fault-tolerant scheduling device for the OODAflow operating system.

[0171] like Figure 8 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect and obtain intelligent computing tasks, such as videos to be identified and classified, and the information display device D is used to display the execution results obtained by the analysis of the present invention, such as video classification results.

[0172] The information display device D can organize and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll, etc. The user is presented with the key information specified by the user, and the user can understand this information more promptly without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present the user with rich and necessary key information.

[0173] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the fault-tolerant scheduling method for the OODAflow operating system provided by the above methods.

[0174] The present invention also proposes a storage medium VIII in another embodiment, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the fault-tolerant scheduling method for the OODAflow operating system are implemented. It should be understood that the storage medium in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0175] Figure 9 A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0176] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. Various programs and data required for the operation of the device 1000 can also be stored in the RAM III. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0177] Multiple components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard and mouse; an output unit VII, such as various types of displays and speakers; a storage medium VIII, such as a magnetic disk and optical disk; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0178] Computing unit I can be various general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. Computing unit I performs the various methods and processes described above, such as method steps S1-S4. For example, in some embodiments, the method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the method described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0179] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A fault-tolerant scheduling method for an OODAflow operating system, characterized in that: include: In the initial step, the intelligent computing task is received and split into four single tasks: perception, judgment, decision-making, and execution. The single tasks and the environmental health monitoring process are encapsulated into a self-sustaining container, waiting for scheduling by the computing device; the computing node is a CPU, GPU, or neural network accelerator; In the execution step, the control node scheduler in the computing device deploys the single task packaged into a self-sustaining container to the corresponding computing node according to the performance of each computing node; The computing node determines whether the single task it receives has a checkpoint file to be executed. If so, it will prioritize executing the task corresponding to the checkpoint file. If not, it will execute the current task. Monitoring step: When a computing node is executing a task, the environmental health monitoring process monitors the health of the computing devices of the current computing node in real time. If there is a health problem, the current computing node exits the task and saves the checkpoint file; If there is no fault, the task execution result is written back to the shared storage area and a task completion signal is sent to the control node; In the end step, the control node determines whether the task is successfully executed based on the task completion signal. If the task is successfully executed, the process ends; otherwise, it returns to the initial step to reschedule the task.

2. The fault-tolerant scheduling method for the OODAflow operating system according to claim 1, wherein: The computing node includes: perception computing resources, judgment computing resources, decision computing resources and execution computing resources; the computing nodes and management nodes together constitute a computing cluster interconnected by a local area network built by a local virtual bridge.

3. The fault-tolerant scheduling method for the OODAflow operating system according to claim 1, wherein: The monitoring step includes: embedding a checkpoint generation code in the single task executed by the computing node, and when the single task is interrupted, the checkpoint generation code cooperates with the environmental health monitoring process of the computing node to generate a checkpoint file; The checkpoint file includes: a metadata file of the computational graph structure, an index file of the mapping relationship between variable names and variable data locations, and a data file of model weights and variable data.

4. A fault-tolerant scheduling device for an OODAflow operating system, characterized in that: include: The initial module receives intelligent computing tasks and splits them into four single tasks: perception, judgment, decision-making, and execution. It then encapsulates the single tasks and the environmental health monitoring process into a self-sustaining container, waiting for scheduling by computing devices. The computing nodes are CPUs, GPUs, or neural network accelerators. An execution module, in which a control node scheduler in the computing device deploys a single task packaged into a self-sustaining container to a corresponding computing node according to the performance of each computing node; The computing node determines whether the single task it receives has a checkpoint file to be executed. If so, it will prioritize executing the task corresponding to the checkpoint file. If not, it will execute the current task. Monitoring module: When a computing node is executing a task, the environmental health monitoring process monitors the health of the computing devices belonging to the current computing node in real time. If there is a health problem, the current computing node exits the task and saves the checkpoint file; If there is no fault, the task execution result is written back to the shared storage area and a task completion signal is sent to the control node; In the end module, the control node determines whether the task is successfully executed based on the task completion signal. If the task is successfully executed, the process ends; otherwise, it returns to the initial module to reschedule the task.

5. The fault-tolerant scheduling device for the OODAflow operating system according to claim 4, characterized in that: The computing node includes: perception computing resources, judgment computing resources, decision computing resources and execution computing resources; the computing nodes and management nodes together constitute a computing cluster interconnected by a local area network built by a local virtual bridge.

6. The fault-tolerant scheduling device for the OODAflow operating system according to claim 4, characterized in that: The monitoring module includes: embedding checkpoint generation code in the single task executed by the computing node, and when the single task is interrupted, the checkpoint generation code cooperates with the environmental health monitoring process of the computing node to generate a checkpoint file; The checkpoint file includes: a metadata file of the computational graph structure, an index file of the mapping relationship between variable names and variable data locations, and a data file of model weights and variable data.

7. An electronic device, characterized in that: A fault-tolerant scheduling device for an OODAflow operating system comprising the method according to any one of claims 4 to 6.

8. The electronic device according to claim 7, wherein: The electronic device is connected to an information display device, which is used to display the execution result using display parameters and attributes set by the user or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the fault-tolerant scheduling method for an OODAflow operating system as recited in any one of claims 1 to 3.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the fault-tolerant scheduling method for the OODAflow operating system described in any one of claims 1 to 3 are implemented.

Images