Fault-tolerant scheduling method and device for OODAflow operating system

Abstract

The invention provides a fault-tolerant scheduling method and device for an OODAflow operating system, and the method comprises the steps: receiving an intelligent calculation task, splitting the intelligent calculation task into single tasks, packaging the single tasks and an environment health degree monitoring process into a self-sustaining container, and waiting for the scheduling of calculation equipment; deploying the single task packaged into the self-sustaining container to a corresponding computing node; in the process that the computing node executes the task, the environment health degree monitoring process monitors the health condition of the computing equipment to which the current computing node belongs in real time, and if the health condition has a fault, the current computing node quits the task and stores a checkpoint file; and if no fault exists, writing the execution result of the task back to the shared storage area. According to the fault-tolerant scheduling system provided by the invention, a fault-tolerant mechanism and efficient fault-tolerant scheduling of the OODAflow operating system tasks are realized, and the real-time performance and the reliability of the OODAflow operating system for processing the computing tasks are ensured.

Description

Technical Field

[0001] The present invention relates to the fields of computer technology, computer service technology, natural language processing technology, and image perception technology, and in particular to a fault-tolerant scheduling method and device for an OODAflow operating system. Background Art

[0002] OODA is the first principle of behavioral intelligent analysis: this model is based on the cycle of perception, judgment, decision-making and execution, and emphasizes the flexible scheduling of low-coupling division of intelligent computing task data flow and control decoupling. With the booming development of intelligent computing, more and more intelligent computing businesses use the OODA paradigm to build task processes. A typical example is the rapidly developing field of embodied intelligence in recent years.

[0003] Embodied intelligence is a new type of intelligent computing service that combines large language models with robots, striving to enable robots to understand natural language, autonomously determine how to implement instructions, perceive the surrounding environment and perform correct behaviors. Its task scenario is a typical complex computing scenario that combines control computing with intelligent computing. In end-to-end services, from understanding natural language to making correct actions, it involves the division of multiple computing links. Therefore, researchers in the field of embodied intelligence are gradually combining end-to-end embodied intelligence tasks with the OODA paradigm, decoupling the intelligent computing of the end-to-end task flow from motion control, and further decoupling environmental perception from motion control. Finally, natural language processing and action command generation are further decoupled to form an OODA-like loop of "external information perception-natural language processing-action command generation-response actions according to commands".

[0004] The development of embodied intelligence is just the tip of the iceberg of OODA's increasingly broad and profound impact on the field of intelligent computing. With the emergence of intelligent computing businesses such as large language models and multimodal intelligence, whose parameter quantities and data volumes are growing at an exponential rate, the OODA problem analysis paradigm that decouples complex tasks will inevitably be more widely used in the field of intelligent computing.

[0005] The OODAflow operating system is an operating system that deeply integrates the OODA paradigm of intelligent analysis. It divides the overall process of intelligent computing tasks into four sub-processes: O (perception), O (judgment), D (decision-making), and A (execution) and executes them sequentially. The four sub-processes are loosely coupled with each other, and the amount of data transmission between the processes is small, providing a larger schedulable space to optimize the execution efficiency of intelligent computing tasks. The OODAflow operating system is often used in application scenarios that emphasize real-time performance at the edge, such as the control of important mobile equipment at the edge or the deployment of important services; the real-time tasks solved are mostly recognition tasks with a relatively large proportion of intelligent computing. The OODAflow operating system that controls computing nodes to run intelligent computing tasks generally runs on computing devices such as CPUs and neural network accelerators, and the computing devices can be divided into at least one computing node.

[0006] As a newly proposed computing system, the OODAflow operating system has not yet been designed with fault-tolerant functions. If a computing device failure occurs when the OODAflow operating system is running a computing task, the running task will be abnormally interrupted, which will in turn affect the subsequent calculations of OODAflow. This has a very large impact in real-time tasks, and corresponding fault-tolerant measures are needed to deal with this situation.

[0007] The OODAflow operating system does not have a stable and reliable fault-tolerant scheduling system. When an abnormality occurs in a certain link of the OODA intelligent computing task, the task queue will be blocked and the subsequent process cannot proceed normally. This has a great impact on occasions with high real-time requirements. Summary of the invention

[0008] In view of the shortcomings of existing technologies, such as Figure 5 As shown, the present invention proposes a fault-tolerant scheduling method for an OODAflow operating system, which includes:

[0009] The initial step is to receive the intelligent computing task and split it into four single tasks: perception, judgment, decision-making, and execution. The single task and the environmental health monitoring process are encapsulated into a self-sustaining container, waiting for the computing device to schedule. The computing node is a CPU, GPU, or neural network accelerator. The intelligent computing task can be an image classification task, a semantic segmentation task, a text recognition task, a public opinion monitoring task, etc.

[0010] In the execution step, the control node scheduler in the computing device deploys the single task encapsulated into a self-sustaining container to the corresponding computing node according to the performance of each computing node; the computing node determines whether the received single task has a checkpoint file to be executed, and if so, the task corresponding to the checkpoint file is executed first, and if not, the current task is executed;

[0011] Monitoring step: During the task execution process of the computing node, the environmental health monitoring process monitors the health status of the computing device to which the current computing node belongs in real time. If there is a health failure, the current computing node exits the task and saves the checkpoint file; if there is no failure, the execution result of the task is written back to the shared storage area, and a task completion signal is sent to the control node; the corresponding execution result can be image classification results, semantic segmentation results, text recognition results, public opinion monitoring results, etc.

[0012] In the end step, the control node determines whether the task is successfully executed based on the task completion signal. If the task is successfully executed, the process ends, otherwise it returns to the initial step to reschedule the task.

[0013] The fault-tolerant scheduling method for the OODAflow operating system, wherein the computing node includes: perception computing resources, judgment computing resources, decision computing resources and execution computing resources; the computing nodes and management nodes together constitute a computing cluster interconnected by a local area network built by a local virtual bridge.

[0014] The fault-tolerant scheduling method for the OODAflow operating system, wherein the monitoring step includes: embedding a checkpoint generation code in the single task executed by the computing node, and when the single task is interrupted, the checkpoint generation code cooperates with the environmental health monitoring process of the computing node to generate a checkpoint file;

[0015] The checkpoint file includes: a metadata file of the computational graph structure, an index file of the mapping relationship between variable names and variable data locations, and a data file of model weights and variable data.

[0016] like Figure 6 As shown, the present invention also proposes a fault-tolerant scheduling device for the OODAflow operating system, which includes:

[0017] The initial module receives intelligent computing tasks and splits them into four single tasks: perception, judgment, decision-making, and execution. It encapsulates the single tasks and the environmental health monitoring process into a self-sustaining container, waiting for the computing device to schedule. The computing node is a CPU, GPU, or neural network accelerator.

[0018] An execution module, in which the control node scheduler in the computing device deploys the single task encapsulated into a self-sustaining container to the corresponding computing node according to the performance of each computing node; the computing node determines whether the received single task has a checkpoint file to be executed, and if so, the task corresponding to the checkpoint file is executed first, otherwise the current task is executed;

[0019] Monitoring module: During the task execution process of the computing node, the environmental health monitoring process monitors the health status of the computing device to which the current computing node belongs in real time. If there is a health failure, the current computing node exits the task and saves the checkpoint file; if there is no failure, the task execution result is written back to the shared storage area, and a task completion signal is sent to the control node;

[0020] In the end module, the control node determines whether the task is successfully executed based on the task completion signal. If the task is successfully executed, the process ends, otherwise it returns to the initial module to reschedule the task.

[0021] The fault-tolerant scheduling device for the OODAflow operating system, wherein the computing node includes: perception computing resources, judgment computing resources, decision computing resources and execution computing resources; the computing nodes and management nodes together constitute a computing cluster interconnected by a local area network built by a local virtual bridge.

[0022] The fault-tolerant scheduling device for the OODAflow operating system, wherein the monitoring module includes: embedding a checkpoint generation code in the single task executed by the computing node, and when the single task is interrupted, the checkpoint generation code cooperates with the environmental health monitoring process of the computing node to generate a checkpoint file;

[0023] The checkpoint file includes: a metadata file of the computational graph structure, an index file of the mapping relationship between variable names and variable data locations, and a data file of model weights and variable data.

[0024] The present invention also proposes an electronic device, which includes the fault-tolerant scheduling device for the OODAflow operating system.

[0025] The electronic device is connected to an information display device, and the information display device is used to display the execution result according to display parameters and attributes set by the user or through an artificial intelligence model.

[0026] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the fault-tolerant scheduling method for the OODAflow operating system are implemented.

[0027] The present invention also proposes a computer program product, including a computer program, wherein the computer program implements the steps of the fault-tolerant scheduling method for the OODAflow operating system when executed by a processor.

[0028] It can be seen from the above scheme that the advantages of the present invention are:

[0029] The fault-tolerant scheduling system for the OODAflow operating system implements the fault-tolerant mechanism and efficient fault-tolerant scheduling of the OODAflow operating system tasks, ensuring the real-time and reliability of the OODAflow operating system in processing computing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is the overall flow chart of this application;

[0031] Figure 2 This is a hardware structure diagram of the present invention;

[0032] Figure 3 This is a task pipeline diagram of the OODAflow operating system;

[0033] Figure 4 This is a flow chart of fault-tolerant scheduling of the present invention;

[0034] Figure 5 is a flow chart of the method of the present invention;

[0035] Figure 6 It is a module diagram of the device of the present invention;

[0036] Figure 7 This is a schematic diagram of the structure of a first electronic device of the present invention;

[0037] Figure 8 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0038] Fig. 9 It is a schematic diagram of the structure of a second electronic device of the present invention.

[0039] Reference numerals:

[0040] A-First electronic device;

[0041] B-Fault-tolerant scheduling device for OODAflow operating system;

[0042] C-data acquisition equipment;

[0043] D-information display device;

[0044] 1000 - second electronic device;

[0045] Ⅰ-computational unit;

[0046] II-ROM;

[0047] III-RAM;

[0048] IV-bus;

[0049] V-interface;

[0050] VI - input unit;

[0051] VII-output unit;

[0052] VIII- Storage medium;

[0053] Ⅸ-Communication unit. DETAILED DESCRIPTION

[0054] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0055] Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0056] The processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0057] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0058] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0059] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0060] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0061] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0062] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0063] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0064] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0065] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0066] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0067] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0068] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0069] In order to achieve the above technical effects, the present invention includes the following key technical points:

[0070] Key point 1: The device status consistency maintenance system composed of containerized tasks, computing nodes, and control nodes; when a fault occurs, the fault-tolerant program of the containerized task itself responds with the first priority and saves the current status for subsequent recovery; the device health monitoring process of the computing node responds with the second priority, obtains the device status of the node and sends it to the control node; the control node scheduling process responds with the third priority, obtains the current status of the abnormal node, and applies this information to the subsequent fault-tolerant scheduling process;

[0071] Key point 2: Checkpoint recovery fault tolerance function built with shared storage area and container task environment variable labels. By configuring the background monitoring process and environment variables for each container task instance, the task itself can respond immediately when a fault occurs and trigger the task interruption of fault-tolerant scheduling and create a unique checkpoint file. The file is saved in the shared storage area and named with a unique environment variable name to ensure that the checkpoint file can be correctly indexed in subsequent fault tolerance and guarantee the correlation of the previous and next data of the OODA task.

[0072] Key point 3: In the OODAflow operating system, an internal LAN is established with multi-virtual bridge technology for data transmission. The multi-bridge internal LAN disperses bandwidth pressure while realizing the fault-tolerant scheduling system function, reduces IO congestion, and improves the real-time performance of the fault-tolerant scheduling function of the OODAflow operating system.

[0073] Specifically, in the present invention, the custom service program of the control plane can uniformly monitor the resource usage of all computing devices of the OODAflow operating system, and schedule computing task instances in real time to achieve the optimal overall machine computing efficiency; at the same time, the background service is automatically run in each computing task instance to monitor the failure of the computing device of this node. Once a failure occurs, the computing task instance will automatically execute a "suicide" operation, retain the current checkpoint file and exit abnormally. The custom daemon process of the control plane will sense the node computing device failure and automatically delete all computing task instances deployed on the node. If the number of computing task instances does not match the preset number, it will be captured by the control node service and a new computing task instance will be created. The newly created instance will receive the checkpoint file that failed to execute smoothly in the last task, and continue to execute from the checkpoint to ensure the real-time performance of the system and avoid task congestion.

[0074] When the inventors were studying the fault-tolerant scheduling function of OODAflow, they found that there was a key defect in the prior art, that is, the control node could not monitor the failure of the computing device in real time. This limitation resulted in the computing task being abnormally interrupted when the control node finally detected the failure of the computing device, and it was impossible to achieve effective fault-tolerant scheduling, which in turn caused the subsequent blocking of the OODA task, seriously affecting the real-time performance of the system.

[0075] After in-depth research, the present invention proposes to split the OODA closed-loop task into four separate subtasks (observation, orientation, decision, and action), and to label each round of OODA tasks with a unique environmental variable tag key to facilitate subsequent scheduling, so that the control program can accurately specify y sub-processes of x rounds of tasks. Each subtask is encapsulated as a fully functional self-sustaining task in the form of a container, and the running environment health monitoring process is added to these containers.

[0076] The key is that these health monitoring processes, together with the health detection program of the computing node and the system scheduler of the control node, maintain a "device state consistency". This concept is proposed to solve the problem of real-time response when the computing node device fails.

[0077] Specifically, when a computing node device fails, the self-sustaining task in the container first takes emergency measures to save the current checkpoint file for subsequent fault tolerance mechanism use. Next, the health monitoring program of the computing node will be triggered, record the abnormal information of the computing node, and feed it back to the task scheduler of the control node. This information will be used in the scheduling process of the next new OODA task.

[0078] Since the checkpoint file has been saved and the relevant fault information has been synchronized to the control node scheduler, the control node can immediately create a new self-sustaining task in the form of a container on another intact computing node and continue to execute from the last saved checkpoint file. In this way, the continuity of the OODA task and the real-time performance of the system can be guaranteed.

[0079] Through this "device state consistency" approach, the inventors have achieved real-time response to fault handling. This not only solves the monitoring lag problem in the prior art, but also significantly improves the system's fault tolerance and the real-time performance of task execution in the event of a computing node failure. This innovative approach provides a reliable technical guarantee for the application of the OODAflow operating system in complex computing environments, and ensures the stable operation of the system under high load and high-risk environments. It should be noted that as long as the task group meets the "no control-related, data-related" requirement, the monitoring method of the present invention can be used. The unique design of the monitoring method of the present invention is to perform high real-time fault-tolerant scheduling of tasks that can be split (pipeline optimized after splitting).

[0080] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0081] like Figure 1 As shown, the present invention includes:

[0082] Step 1: Receive tasks. The system receives tasks and splits a single-round OODA task into four single tasks: Observe, Orient, Decide, and Action. It also assigns unique environmental variables to each single task to maintain data correlation during multi-round OODA task scheduling. At the same time, it encapsulates the single task and the environmental health monitoring process into a self-sustaining container waiting for scheduling.

[0083] Step 2: The control node scheduler determines whether there are enough idle devices to calculate the current round of OODA tasks based on the current computing device status returned by each computing node.

[0084] If there are enough idle devices, go to step 3.

[0085] If there are not enough idle devices, the task spins and waits for deployment.

[0086] Step 3: Deploy the task to a specific computing node.

[0087] Step 4: After the task is deployed to the computing node, determine whether there is a checkpoint file to be executed.

[0088] If there is a checkpoint file, go to step 5.

[0089] If there is no checkpoint file, go to step 6.

[0090] Step 5: Execute the checkpoint file first.

[0091] Step 6: Execute the current task.

[0092] Step 7: During the task execution, the background fault monitoring process monitors the health status of the computing devices in the current operating environment in real time.

[0093] If the computing device health condition is faulty, go to step 8.

[0094] If there is no fault, proceed to step 9.

[0095] Step 8: Exit the task and save the checkpoint file.

[0096] Step 9: Write the execution results back to the shared storage area for subsequent use by the control node.

[0097] Step 10: Determine whether the task is successfully executed. The control program does not know whether step 9 has been written back to the storage area (it does not monitor the storage area). After step 9 is completed, a signal will be given to the control program to tell the control program that it has completed the task. At this time, step 10 determines that the task is successfully executed.

[0098] If the task is successfully executed, the process ends.

[0099] If the task is not successfully executed, return to step 1 to reschedule the task.

[0100] Through the above steps, the fault-tolerant scheduling process of the OODAflow operating system is completed, efficient scheduling and execution of tasks are achieved, and the system can continue to run in the event of a failure.

[0101] The computing resource management mode in the OODAflow operating system is that a single set of OODA components is used as a computing node (one perception computing resource + one judgment computing resource + one decision computing resource + one execution computing resource). All computing resources of the whole machine are divided into N sets (in this example Figure 2 There are two sets in the figure), which are two computing nodes, and together with the host management node, they constitute a computing cluster interconnected by a local area network built by a local virtual bridge. The fault tolerance function of OODA computing tasks is realized by adding various functional components in the cluster.

[0102] like Figure 2 As shown in Figure 1, the control node task scheduler plays a key role in the OODAflow operating system. It controls the computing resources and task completion status of each computing node in real time, and implements real-time scheduling of task flows to achieve the effect of pipeline acceleration. Figure 2 Detailed description of each part:

[0103] Compute node services:

[0104] The compute node service and control node service are responsible for creating self-sustaining OODA computing task instance programs in the form of containers. These programs will deploy OODA computing tasks in specific compute nodes according to the requirements of the scheduler.

[0105] For example, when a new OODA task is assigned to a compute node, the compute node service creates a container instance to run the task and ensures that the task is executed according to the predetermined process.

[0106] Control node services:

[0107] Control node services are similar to compute node services in that they are responsible for creating and managing OODA compute task instance programs, but they run on control nodes.

[0108] The control node service communicates with each computing node through the network interface to coordinate the allocation and execution of tasks.

[0109] These services also monitor the progress of tasks and log relevant information when tasks complete or fail.

[0110] Compute node health monitoring process:

[0111] The compute node health monitoring process runs on each compute node to determine whether each compute node is functioning properly.

[0112] They monitor the health status of computing nodes in real time, including CPU load, memory usage, network status, computing card failures, etc.

[0113] If any anomalies are detected, such as hardware failures or performance bottlenecks, the health monitoring process immediately records these anomalies and sends the information to the controller node task scheduler.

[0114] Control node scheduling process:

[0115] The control node scheduling process is the core scheduling program running on the control node and is responsible for the fault-tolerant scheduling of the entire computing system.

[0116] It will integrate the health information of each computing node to determine whether a computing node has failed.

[0117] After detecting a computing node failure, the control node scheduling process will immediately trigger the fault-tolerant scheduling mechanism to reallocate tasks on the failed node to other healthy computing nodes.

[0118] It is also responsible for optimizing the distribution of tasks to maximize the performance and resource utilization of the system.

[0119] Static IP routing:

[0120] Each computing node communicates with the control node task scheduler through static IP routing.

[0121] Static IP routing ensures stable and reliable network connections between nodes, supporting efficient data transmission and task scheduling.

[0122] This part of the network architecture provides basic communication guarantees for the system, enabling the control node to accurately assign tasks to each computing node and receive health monitoring data from the computing nodes.

[0123] Shared storage area:

[0124] The shared storage area is used to store the intermediate results, checkpoint files, and final results of tasks.

[0125] All computing nodes and control nodes can access the shared storage area, ensuring that task data can be shared and reused in the system.

[0126] This design not only improves data availability and consistency, but also supports task fault tolerance and recovery capabilities.

[0127] Through the above mechanism, the OODAflow operating system can efficiently manage and schedule computing tasks, ensuring that the system can respond and adjust in time when a computing node fails. The task scheduler achieves high performance and high reliability of the system through real-time monitoring and flexible scheduling. In this way, the OODAflow operating system not only solves the problems of fault monitoring and resource occupation in task scheduling, but also significantly improves the real-time performance and task execution efficiency of the system.

[0128] The checkpoint generation code is embedded in the task deployed on the computing node. When the task is unexpectedly interrupted due to hardware abnormalities, this part of the code will work with the health monitoring process of the computing node to generate the current checkpoint file. The specific saving format is as follows:

[0129] Checkpoint file details:

[0130] Metadata file (.meta): saves the structure of the computational graph;

[0131] Index file (.index): stores the mapping relationship between variable names and variable data locations;

[0132] Data file (.data-xxxxx-of-xxxxx): holds the actual model weights and variable data;

[0133] Metadata file (MetaGraphDef): The file extension is usually .meta. This file saves the structure of the computational graph, including definitions of nodes, variables, operations, etc. It includes the metadata of the computational graph (graph structure and operations), set definitions (for example, variable sets);

[0134] Index file: The file extension is usually .index. This file saves the mapping relationship between variable names and variable data in the data file. It includes the mapping relationship between variable names and variable data locations, and variable data block information;

[0135] Data files: The file extension is usually .data-???-of-???, where ??? represents the file number and the total number of files. These files store the actual weights and variable data of the model. They include parameter data such as model weights and biases, and other variable data;

[0136] like Figure 3 , 4 As shown in the figure, the OODAflow operating system schedules and executes tasks during operation and handles failures when they occur. The specific process is as follows:

[0137] T0 time:

[0138] Start executing the perception task of OODA loop 1.

[0139] Start executing the judgment task of OODA closed loop 2.

[0140] T1 moment:

[0141] The perception tasks of OODA loop 3 and OODA loop 4 also begin to be executed.

[0142] T2 moment:

[0143] When node 1 is executing the judgment task of OODA closed loop 2, the computing device fails.

[0144] The OODAflow operating system will capture this exception and interrupt the OODA intelligent computing task that has already started on node 1.

[0145] The current computing state is saved as a checkpoint file and stored in the shared storage space. Specifically, the judgment results of OODA loop 2 and the perception results of OODA loop 4 are saved in the form of checkpoints.

[0146] Meanwhile, Node 2 continues to operate normally without failure, so the tasks of OODA Loop 1 and OODA Loop 3 continue to execute.

[0147] Scheduling priority:

[0148] In the OODAflow operating system, the rules for scheduling priorities are as follows:

[0149] Tasks that have already started have the highest priority.

[0150] In addition, the smaller the OODA loop number, the higher the priority.

[0151] T3 moment:

[0152] After node 2 determines that the task is completed, the computing device is released.

[0153] The OODAflow operating system immediately reads the perception result checkpoint file of OODA closed loop 4 and sends it to the judgment device of node 2 for calculation, in order to complete the task of OODA closed loop 4 as soon as possible. That is, at time T3, node 2 completes its own judgment task and takes over the judgment task of node 1, at which time the judgment task of node 1 begins to execute.

[0154] T4 moment:

[0155] After node 2 completes its decision-making task, the computing device is released.

[0156] The OODAflow operating system immediately reads the judgment result checkpoint file of OODA closed loop 2 and sends it to the decision device of node 2 for calculation, in order to complete the task of OODA closed loop 2 as soon as possible.

[0157] Subsequent scheduling:

[0158] The subsequent scheduling process of OODA loop 5 and OODA loop 6 is completed in a staggered pipeline manner to maximize real-time performance and performance release.

[0159] Through the above steps, the OODAflow operating system can not only monitor and respond to faults, but also give priority to high-priority tasks after resources are released, improving the system's real-time fault response and task execution efficiency.

[0160] This new technical solution effectively solves the fault monitoring and resource occupation problems in the OODAflow operating system and improves the performance and real-time performance of the system.

[0161] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0162] like Figure 6 As shown, the present invention also proposes a fault-tolerant scheduling device for the OODAflow operating system, which includes:

[0163] The initial module receives intelligent computing tasks and splits them into four single tasks: perception, judgment, decision-making, and execution. It encapsulates the single tasks and the environmental health monitoring process into a self-sustaining container, waiting for the computing device to schedule. The computing node is a CPU, GPU, or neural network accelerator.

[0164] An execution module, in which the control node scheduler in the computing device deploys the single task encapsulated into a self-sustaining container to the corresponding computing node according to the performance of each computing node; the computing node determines whether the received single task has a checkpoint file to be executed, and if so, the task corresponding to the checkpoint file is executed first, otherwise the current task is executed;

[0165] Monitoring module: During the task execution process of the computing node, the environmental health monitoring process monitors the health status of the computing device to which the current computing node belongs in real time. If there is a health failure, the current computing node exits the task and saves the checkpoint file; if there is no failure, the task execution result is written back to the shared storage area, and a task completion signal is sent to the control node;

[0166] In the end module, the control node determines whether the task is successfully executed based on the task completion signal. If the task is successfully executed, the process ends, otherwise it returns to the initial module to reschedule the task.

[0167] The fault-tolerant scheduling device for the OODAflow operating system, wherein the computing node includes: perception computing resources, judgment computing resources, decision computing resources and execution computing resources; the computing nodes and management nodes together constitute a computing cluster interconnected by a local area network built by a local virtual bridge.

[0168] The fault-tolerant scheduling device for the OODAflow operating system, wherein the monitoring module includes: embedding a checkpoint generation code in the single task executed by the computing node, and when the single task is interrupted, the checkpoint generation code cooperates with the environmental health monitoring process of the computing node to generate a checkpoint file;

[0169] The checkpoint file includes: a metadata file of the computational graph structure, an index file of the mapping relationship between variable names and variable data locations, and a data file of model weights and variable data.

[0170] like Figure 7 As shown, the present invention further proposes a first electronic device A in another embodiment, which includes the fault-tolerant scheduling device for the OODAflow operating system.

[0171] like Figure 8 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect and obtain intelligent computing tasks, such as videos to be identified and classified, and the information display device D is used to display the execution results obtained by the analysis of the present invention, such as video classification results.

[0172] The information display device D can sort and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll and play, etc. The user is presented with the key information specified by the user, and the user can understand the information more promptly without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present rich and necessary key information to the user.

[0173] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the fault-tolerant scheduling method for the OODAflow operating system provided by the above methods.

[0174] The present invention also proposes a storage medium VIII in another embodiment, on which a computer program is stored, and when the computer program is executed by the processor, the steps of the fault-tolerant scheduling method for the OODAflow operating system are implemented. It should be understood that the storage medium in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0175] Fig. 9 A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same or different from the first electronic device A.

[0176] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. In RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0177] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard, a mouse, etc.; an output unit VII, such as various types of displays, speakers, etc.; a storage medium VIII, such as a disk, an optical disk, etc.; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0178] The computing unit I may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S4. For example, in some embodiments, the method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM II and / or a communication unit IX. When the computer program is loaded into RAM III and executed by the computing unit I, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit I may be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0179] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A fault-tolerant scheduling method for an OODAflow operating system, characterized in that: include: The initial step is to receive the intelligent computing task and split it into four single tasks: perception, judgment, decision-making, and execution. The single tasks and the environmental health monitoring process are encapsulated into a self-sustaining container, waiting for the computing device to schedule. The computing node is a CPU, GPU, or neural network accelerator. In the execution step, the control node scheduler in the computing device deploys the single task packaged into a self-sustaining container to the corresponding computing node according to the performance of each computing node; The computing node determines whether the single task it receives has a checkpoint file to be executed. If so, it will prioritize executing the task corresponding to the checkpoint file. If not, it will execute the current task. Monitoring step: During the task execution of the computing node, the environmental health monitoring process monitors the health status of the computing device of the current computing node in real time. If there is a health failure, the current computing node exits the task and saves the checkpoint file; If there is no fault, the task execution result is written back to the shared storage area, and a task completion signal is sent to the control node; In the end step, the control node determines whether the task is successfully executed based on the task completion signal. If the task is successfully executed, the process ends, otherwise it returns to the initial step to reschedule the task.

2. The fault-tolerant scheduling method for the OODAflow operating system according to claim 1, characterized in that: The computing node includes: perception computing resources, judgment computing resources, decision computing resources and execution computing resources; the computing nodes and management nodes together constitute a computing cluster interconnected by a local area network built by a local virtual bridge.

3. The fault-tolerant scheduling method for the OODAflow operating system according to claim 1, characterized in that: The monitoring step includes: embedding a checkpoint generation code in the single task executed by the computing node, and when the single task is interrupted, the checkpoint generation code cooperates with the environmental health monitoring process of the computing node to generate a checkpoint file; The checkpoint file includes: a metadata file of the computational graph structure, an index file of the mapping relationship between variable names and variable data locations, and a data file of model weights and variable data.

4. A fault-tolerant scheduling device for an OODAflow operating system, characterized in that: include: The initial module receives intelligent computing tasks and splits them into four single tasks: perception, judgment, decision-making, and execution. It encapsulates the single tasks and the environmental health monitoring process into a self-sustaining container, waiting for the computing device to schedule. The computing node is a CPU, GPU, or neural network accelerator. An execution module, in which a control node scheduler in the computing device deploys a single task packaged into a self-sustaining container to a corresponding computing node according to the performance of each computing node; The computing node determines whether the single task it receives has a checkpoint file to be executed. If so, it will prioritize executing the task corresponding to the checkpoint file. If not, it will execute the current task. Monitoring module: During the execution of tasks by computing nodes, the environmental health monitoring process monitors the health of computing devices of the current computing node in real time. If there is a health problem, the current computing node exits the task and saves the checkpoint file. If there is no fault, the task execution result is written back to the shared storage area, and a task completion signal is sent to the control node; In the end module, the control node determines whether the task is successfully executed based on the task completion signal. If the task is successfully executed, the process ends, otherwise it returns to the initial module to reschedule the task.

5. The fault-tolerant scheduling device for the OODAflow operating system according to claim 4, characterized in that: The computing node includes: perception computing resources, judgment computing resources, decision computing resources and execution computing resources; the computing nodes and management nodes together constitute a computing cluster interconnected by a local area network built by a local virtual bridge.

6. The fault-tolerant scheduling device for the OODAflow operating system according to claim 4, characterized in that: The monitoring module includes: embedding a checkpoint generation code in the single task executed by the computing node, and when the single task is interrupted, the checkpoint generation code cooperates with the environmental health monitoring process of the computing node to generate a checkpoint file; The checkpoint file includes: a metadata file of the computational graph structure, an index file of the mapping relationship between variable names and variable data locations, and a data file of model weights and variable data.

7. An electronic device, characterized in that: It includes a fault-tolerant scheduling device for an OODAflow operating system as described in claims 4-6.

8. The electronic device according to claim 1, wherein: The electronic device is connected to an information display device, which is used to display the execution result according to display parameters and attributes set by a user or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the fault-tolerant scheduling method for an OODAflow operating system as described in any one of claims 1 to 3.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the fault-tolerant scheduling method for the OODAflow operating system described in any one of claims 1 to 3 are implemented.

Images