Multiple system control system and cycle management method

Abstract

To maintain consistency of data between a main system and a sub-system, without using a dedicated OS or HW.SOLUTION: A multiple execution base unit 12A of a reader node of a multiple system control system 100 matches processing contents of each node by consensus building between the plurality of nodes related to a task woken up in a next cycle and a wake-up factor of the task, and multiple execution base units 12B and 12C of a follower node match a start-up request factor with a wake-up factor in a main system node, when the start-up factor of the task in the reader node and a start-up request factor which is a start factor in own node are not matched.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a multi-system control system and a cycle management method.

Background Art

[0002] Conventionally, a multi-system control system capable of switching a slave node to a new master node when a failure occurs in the master node is known. Patent Document 1 discloses a dual-computer system that enables synchronized operation of both the master and slave nodes by using a synchronization task group control function that matches the execution order of task groups implemented in both the master and slave systems. According to the technique described in Patent Document 1, since both the master and slave nodes operate synchronously, it is possible to perform the switching in a state where consistency is guaranteed even when switching the slave node to a new master node at the time of failure.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in recent years, there has been an increasing demand for cloud migration of multi-system control systems for the purpose of asset lightening, BCP (Business Continuity Plan) compliance, etc. However, when a multi-system control system is configured using a dedicated OS (Operating System) or dedicated HW (Hard Ware), it is not possible to directly migrate the multi-system control system to a cloud environment.

[0005] In the dual computer system described in Patent Document 1, although dedicated OS, HW, etc. are not used, when switching from a slave computer to a master computer, if there is a task in the middle of processing, it waits for the processing to end, and when the processing ends, control is performed to stop the operation of the task. Then, the information stored in the memory of the master computer is copied to the slave computer, and after the copy is completed, the duplex operation is started. That is, when switching from the slave computer to the master computer, the real-time performance in the duplex computer system is impaired.

[0006] On the other hand, according to Raft, which is a consensus algorithm for managing replicated logs, it is possible to construct a highly reliable multi-system control system that maintains real-time performance and ensures data consistency between the master and slave systems. However, Raft is an algorithm that maintains data consistency between the master and slave systems through request-response type interactions. Therefore, for example, it cannot be simply applied to a system that handles tasks driven by internal events generated at each node based on a timer or the like.

[0007] The present invention has been made in consideration of the above situation. An object of the present invention is to ensure data consistency between the master and slave systems without using a dedicated OS or HW, etc. in a multi-system control system that handles tasks driven by internal events.

Means for Solving the Problems

[0008] A multi-system control system according to one aspect of the present invention is a multi-system control system including a plurality of nodes assigned to either one main node and a plurality of slave nodes. Each node includes a periodic control unit that periodically executes one or more tasks that perform the same state transition and generate the same output when given the same input, and a multi-execution infrastructure unit that controls synchronization with other nodes. The multi-execution infrastructure unit of the main node unifies the processing contents at each node by forming an agreement among the plurality of nodes regarding the tasks to be woken up in the next cycle and the wake-up factors of the tasks. Further, when the wake-up factor of the task in the master node and the wake-up request factor, which is the wake-up factor in its own node, do not match, the multi-execution infrastructure unit of the slave node makes the wake-up request factor match the wake-up factor in the master node.

Advantages of the Invention

[0009] According to at least one aspect of the present invention, in a multi-system control system that handles tasks driven by internal events, it becomes possible to maintain data consistency between the master and slave systems without using a dedicated OS or HW, etc. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each figure, the same components are denoted by the same reference numerals. The following description and drawings are examples for explaining the present invention, and for the sake of clarity of explanation, omissions and simplifications are made as appropriate. The present invention can be implemented in various other forms. Unless otherwise particularly limited, each component may be singular or plural.

[0012] <First Embodiment> [Configuration of Multisystem Control System] First, with reference to FIG. 1, the configuration of the multisystem control system according to the present embodiment will be described. FIG. 1 is a block diagram showing a functional configuration example of a multisystem control system 100 according to the first embodiment.

[0013] As shown in FIG. 1, the multisystem control system 100 includes a server cluster 10 and a plurality of external devices 2. The server cluster 10 includes computers 1A to 1C. Each of the computers 1A to 1C is communicably connected to a plurality of external devices 2 via a network Nt.

[0014] In the example shown in FIG. 1, computer 1A is selected as a reader node (an example of a master node), and computers 1B and 1C are each appointed as follower nodes (an example of slave nodes). The computer 1A selected as the reader node performs data transmission and reception processing with the external device 2, and transmits the data received from the external device 2 to the follower nodes. In the following description, when it is not necessary to distinguish between the computers 1A to 1C, these are collectively referred to as "computer 1". Also, in the following description, computer 1 may be referred to as a "node".

[0015] The external device 2 is constituted by, for example, a controlled device or the like that is the object of control by the computers 1A to 1C. Note that the external device 2 may be constituted by a client device or the like that transmits a request message to the computer 1.

[0016] Computers 1A to 1C each include a periodic control application 11A (hereinafter referred to as the "periodic control app") to a periodic control app 11C, and a multi-execution base unit 12A to a multi-execution base unit 12C. In the following description, when there is no need to distinguish between the periodic control apps 11A to 11C, they are collectively referred to as the "periodic control app 11". Also, when there is no need to distinguish between the multi-execution base units 12A to 12C, they are collectively referred to as the "multi-execution base unit 12".

[0017] The periodic control app 11 (an example of a periodic control unit) sequentially executes a group of tasks implemented in the app at regular intervals. Each task included in the task group performs various processes by referring to, for example, a received message from the external device 2 or its own app state, updates the app state based on the result of the process, and returns the process result to the external device 2. Also, the task periodically generates a control output based on sensor data received from the external device 2 and transmits the generated control output to the external device 2.

[0018] The periodic control apps 11A to 11C process the same group of tasks in the same execution order. The task group is composed of a plurality of tasks classified according to the length of the processing time of the tasks, etc., and the task indicates each unit obtained by dividing the internal processing executed by the computer 1 for each functional unit.

[0019] The multi-execution base unit 12 controls the synchronization process between each computer 1, which is necessary for each computer 1 to behave deterministically, in addition to managing each computer in the server cluster 10. "Deterministic" behavior refers to the behavior where, when the same input sequence is given to a state machine (not shown) in the same app state, the same state transition is performed and the same output is produced. The state machine indicates the logic of the application that switches (performs a state transition) the content of the subsequent process depending on the app state.

[0020] By having each state machine of computers 1A to 1C in server cluster 10 perform the same task in the same state, even if the leader node stops due to a failure, the follower node can become the new leader node and continue the processing.

[0021] The leader node sends a regular heartbeat to all follower nodes to maintain its authority. If a follower node does not receive a communication within a time called the election timeout, the follower node assumes that the leader node has stopped and starts an election to select a new leader node. Among the follower nodes as candidates, the node that has received votes from more than half of the computers 1 in the entire server cluster 10, that is, the node that has won the election, is selected as the new leader node. The election can be realized by the Raft mechanism described above. The configurations of the periodic control application 11 and the multi-execution infrastructure unit 12 will be described in detail with reference to FIG. 2 below.

[0022] [Functional Configuration of Computer] Next, the functional configuration of computer 1 will be described. FIG. 2 is a block diagram showing an example of the functional configuration of computer 1.

[0023] [Periodic Control Application] As shown in FIG. 2, the periodic control application 11 of computer 1 includes tasks T1 to Tn (n is a natural number of 2 or more) and an application state information storage unit 111. In the following description, when it is not necessary to distinguish tasks T1 to Tn from each other, these are collectively referred to as "task T".

[0024] Task T has the following three types: (1) Periodically driven task: A task executed every period (2) Event-driven task: A task executed in the next period when a specific event occurs (3) Reception-driven task: A task executed in the next period when a message addressed to the task is received from an external device

[0025] A plurality of tasks T scheduled to be executed in a certain period are arranged in a ring type in the order of their execution. The configuration of task T will be described in detail with reference to FIG. 4 described later.

[0026] The application state information storage unit 111 is a storage unit that stores information on the internal state of the application referred to by task T during processing, and is composed of, for example, a table on shared memory, an in-memory DB (Database) such as a KVS (Key-Value Store), etc.

[0027] (Multi-execution infrastructure part) The multi-execution infrastructure part 12 includes a task execution unit 121, a task management table 122, a wake-up request list 123, a cluster management unit 124, an event log storage unit 125, and a snapshot storage unit 126.

[0028] The task execution unit 121 sequentially executes the tasks to be woken up, which have reached an agreement among nodes before the start of the period, in a predetermined order for each period. The agreement among nodes is formed when the leader node requests the follower nodes to append to the event log and receives a notification of successful append to the event log from more than half of the follower nodes. The event log will be described in detail with reference to FIG. 5 described later.

[0029] The tasks to be woken up are the tasks that are the targets to be woken up and have not received a wake-up request in the event-driven tasks in (2) above. The details of the tasks to be woken up and the corresponding contents at each node of the leader node and the follower nodes when a wake-up request is received will be described in detail with reference to FIG. 8.

[0030] In addition, the task execution unit 121 provides an API (Application Programming Interface) for information necessary for task execution. The API includes waiting for an event without a timeout, waiting for an event with a timeout, issuing an event, receiving a message, obtaining the current time, and so on. The call processing of the event waiting API with a timeout by the task execution unit 121 will be described in detail with reference to FIGS. 18 and 19 described later. Also, the waiting processing for the occurrence of an event or a timeout after the event waiting API call by the task execution unit 121 will be described in detail with reference to FIG. 20 described later.

[0031] The task management table 122 is a table that manages the tasks to be executed (awakened) in the cycle and the execution order of the tasks. The configuration of the task management table 122 will be described in detail with reference to FIG. 9 described later.

[0032] The wake-up request list 123 is a list in which the tasks to be awakened after the next cycle and the wake-up requests for the tasks are associated. The wake-up request list 123 will be described in detail with reference to FIG. 8 described later.

[0033] The wake-up request list 123 is a list in which the wake-up requests generated at each node are once written. Among the information written in the wake-up request list 123, at the arrival of the start timing of the cycle, the information regarding the tasks determined to wake up in that cycle is deleted from the wake-up request list 123 and transferred to the task management table 122.

[0034] In the computer 1 selected as the leader node, the cluster management unit 124 manages each node in the server cluster 10 that performs multi-execution and performs various synchronization processes between the nodes.

[0035] Specifically, when the cluster management unit 124 of the leader node receives a request (message) from the external device 2, it sequentially adds the content (command) of the request to the event log in the event log storage unit 125. Further, the cluster management unit 124 of the leader node forms a consensus with other follower nodes regarding the content of the added event log (hereinafter also referred to as "log entry"), and applies (inputs) the content of the log entry for which the consensus has been formed to a state machine (not shown).

[0036] The cluster management unit 124 of the leader node applies the content of the log entry for which the consensus has been formed to the state machine not only when receiving a message, but also when other events that require consensus formation occur, for example, when a cycle start request event (to be described later) occurs.

[0037] When the cluster management unit 124 of the follower node receives a request to replicate the event log from the leader node, it replicates the log entry of the event log to the event log in its own event log storage unit 125, and notifies the leader node of the completion of the replication. Thereafter, the cluster management unit 124 of the follower node applies the replicated event log to its own state machine. The functions of the cluster management unit 124 can be realized by using the above-described Raft technology or the like.

[0038] The event log storage unit 125 is a storage unit that stores the log of events (hereinafter referred to as the event log) for which consensus has been formed among nodes.

[0039] The snapshot storage unit 126 is a storage unit that stores a snapshot of the application state at a certain point in time when the content of the event log has been sequentially applied to the state machine. After the application state at that point in time is written to the snapshot, all the logs up to that point are discarded. By performing such processing, the capacity of the event log stored in the event log storage unit 125 can be reduced.

[0040] When a newly started follower node is added to the cluster as a recovery process from a failure, the cluster management unit 124 of the follower node restores the content written in the snapshot as an application state, and sequentially applies the content of the event log after the snapshot creation time to the state machine. By performing such processing, the state of the follower node newly added to the cluster finally becomes the same as the state of the old leader node. The function of the snapshot can be realized by the function of the snapshot in Raft.

[0041] [Example of computer hardware configuration] Next, the hardware configuration of the device for realizing the functions of the multi-system control system 100 according to the present embodiment will be described with reference to FIG. 3.

[0042] FIG. 3 is a block diagram showing an example of the hardware configuration of each computer 1 constituting the multi-system control system 100. The computer 200 shown in FIG. 3 is hardware used as a so-called computer.

[0043] The computer 200 includes a control unit 210, a non-volatile storage 220, a display unit 230, an operation input unit 240, and a communication I / F (Interface) 250, which are respectively connected to a bus B.

[0044] The control unit 210 includes a CPU (Central Processing Unit) 211, a ROM (Read Only Memory) 212, and a RAM (Random Access Memory) 213.

[0045] The CPU 211 reads the program code of the software for realizing each function according to the present embodiment from the ROM 212, expands it in the RAM 213, and executes it. Variables, parameters, etc. generated during the arithmetic processing are temporarily written in the RAM 213.

[0046] The functions of the periodic control application 11 and the multi-execution infrastructure unit 12 of the computer 1 of the multi-system control system 100 are realized by the CPU 211 reading the corresponding program code from the ROM 212, expanding it in the RAM 213, and executing it.

[0047] Note that the control unit 210 may include a processing device such as an MPU (Micro-Processing Unit) instead of the CPU 211. Alternatively, the control unit 210 may include a dedicated circuit (for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), etc.) that performs specific processing.

[0048] As the non-volatile storage 220, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flexible disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a non-volatile memory card, etc. can be used. In this non-volatile storage 220, in addition to the OS (Operating System) and various parameters, a program for operating the computer 200 and the like are recorded. Note that the program may be stored in the ROM 212.

[0049] The program is stored in the form of a computer-readable program code, and the CPU 211 sequentially executes operations according to the program code. That is, the ROM 212 or the non-volatile storage 220 is used as an example of a computer-readable non-transitory recording medium storing a program executed by a computer.

[0050] The display unit 230 is, for example, a monitor composed of an LCD (Liquid Crystal Display) or the like, and displays the results of processing performed by the computer 200 and the like. The operation input unit 240 is composed of, for example, a keyboard, a mouse, a touch sensor, etc., generates an operation signal according to an operation by the user, and supplies it to the CPU 211. Note that the display unit 230 and the operation input unit 240 may be integrally configured as a touch panel. Further, the computer 200 may be configured not to include the display unit 230 and the operation input unit 240.

[0051] For the communication I / F 250, for example, a NIC (Network Interface Card) or the like is used, and various data can be transmitted and received between the external device 2 via a network or a communication line.

[0052] [Configuration of Task Group] Next, the configuration of the task group that constitutes the periodic control application 11 will be described. FIG. 4 is a diagram showing a configuration example of the task group. The task group shown on the left side of FIG. 4 is a task group scheduled to be executed in a certain period, and includes tasks T1 to T5. It is determined that tasks T1 to T5 are executed in the order of task T1, task T2, task T3, task T4, and task T5. Among this task group, tasks T2 to T4 are event-driven tasks, and it is assumed that a wake-up request is given only to task T3 among tasks T2 to T4. In this case, as shown on the right side of FIG. 4, the periodic control application 11 executes the tasks included in the task group in the order of task T1, task T3, and task T5.

[0053] [Events and Event Logs] Next, the events and event logs that occur in the computer 1 according to this embodiment will be described. Events include events that require consensus formation between nodes and events that do not require consensus formation. Events that require consensus formation include a cycle start request event for determining the content of tasks to be executed in the next cycle and a message reception event that occurs when a message processed by a task is received from the external device 2. On the other hand, events that do not require consensus formation include "wait events" and "timeouts" that drive "event-driven tasks". Here, the types of events that require consensus formation will be described.

[0054] FIG. 5 is a diagram showing an example of the types of events that require consensus formation. As shown in FIG. 5, the types of events that require consensus formation between nodes include a cycle start request event and a message reception event.

[0055] (Cycle start request event) The cycle start request event is an event that occurs after the end of a certain cycle and before the start of the next cycle, and occurs only at the leader node. By reaching a consensus on the content of this event between nodes, the processing content of the next cycle is determined. Here, with reference to FIG. 6, the configuration of the cycle start request event will be described. FIG. 6 is a diagram showing an example of the configuration of the cycle start request event.

[0056] As shown in FIG. 6, the cycle start request event includes information on "cycle number", "virtual time", and "wake-up task list". The "cycle number" is a number that identifies the cycle and is indicated by a number with a # such as "#10". The "virtual time" is the virtual current time used by tasks in each cycle. Since there may be a deviation in the local time at each node, if each task is executed with reference to the local time of each node, the processing content will not be uniquely determined. Therefore, in this embodiment, a consensus is formed between nodes at the start of each cycle regarding the virtual current time used in each cycle.

[0057] The wake-up task list is a list that defines the content of the tasks to be woken up in the next cycle. In the wake-up task list, the task identifier (such as T1) of the event-driven task is associated with the information on the wake-up factor of the task (such as "E", "TO"). The wake-up factor "E" indicates "waiting for an event to occur", and "TO" indicates "time-out occurred".

[0058] As shown in FIG. 6, for the cycle start request event, the cycle number is "10", and it is shown that the virtual time at which the cycle with the cycle number "10" starts is "2:03:05.5". Further, for the cycle start request event, the task identifier "T3" task that wakes up when the wait event occurs (E) (the wake-up factor is "E"), and the task identifier "T5" task that wakes up when the timeout occurs (TO) (the wake-up factor is "TO") are defined by the wake-up task list.

[0059] The cycle control application 11 (see FIG. 1) of each node starts the processing of the next cycle when the current time (virtual time) defined in the cycle start request event becomes the virtual time defined in the cycle start request. In the next cycle that is started, the node wakes up the event-driven tasks in the wake-up task list in a predetermined order (in the example shown in FIG. 6, in the order of "T3", "T5"). Note that in this embodiment, in the follower node, there may be a situation where the task cannot be woken up when the start time of the cycle arrives, and the task is executed later. In this case, the follower node that is executing later executes the task in the wake-up task list after the start time of the cycle has passed.

[0060] (Message reception event) Returning to FIG. 5, the description will continue. The message reception event is an event that occurs when a task receives a message from the external device 2. The received message needs to be passed to the task that is the transmission destination of the message, but the cluster management unit 124 of the leader node performs the process of passing the received message to the task in the next cycle instead of the cycle in which the message reception event occurs.

[0061] The message reception event includes "destination information" and "message body". The "destination information" is information indicating the destination of the message. The "message body" indicates the body of the message.

[0062] When a message reception event occurs, after the current (at that time) periodic processing is completed and after reaching an agreement among nodes regarding the periodic start request event of the next period, before starting the next period, the cluster management unit 124 of the leader node sends the message received in the message reception event to the task of a predetermined destination. The message to be sent to the task of the predetermined destination is the message received by the message reception event that occurred between the periodic start request event of the current period and the periodic start request event of the next period.

[0063] Events related to wake-up requests can occur in the same way in all nodes, but the addition of events to the event log is performed only by the cluster management unit 124 of the leader node. Note that the targets for adding to the event log are only the periodic start request and the message reception event. The cluster management unit 124 of the leader node duplicates the event log for the cluster management unit 124 of the follower node. Then, when each cluster management unit 124 of the leader node and the follower node applies the agreed-upon periodic start request event to the state machine, the information of the wake-up task list included in the event entry is reflected in the task management table.

[0064] Each cluster management unit 124 of the leader node and the follower node performs the application of the agreed-upon log entry to the state machine at an appropriate timing from after the completion of the periodic processing indicated by the periodic start request in the most recent past to before the start of the next period.

[0065] Next, the configuration of the event log will be described. FIG. 7 is a diagram showing a configuration example of the event log. In the event log shown in FIG. 7, the left end is the oldest and the right end is the newest. In the event log shown in FIG. 7, the logs of "RX#29", "Cyc#10 WU T5", "RX#30", "RX#31", "Cyc#11 WU T2,T3", and "RX#32" are entered (stored) in order from the oldest.

[0066] In the event log shown in FIG. 7, it is shown that task T5 was instructed to wake up (WU: WakeUp) by a cycle start request event (Cyc) targeting the cycle of cycle number #10. Also, in the cycle of cycle number #10, the received message of "RX#29" received in the previous cycle is also processed and passed to the receiving target task.

[0067] Also, it is shown that task T2 and task T3 were instructed to wake up by a cycle start request event (Cyc) targeting the cycle of cycle number #11. Also, in the cycle of cycle number #11, the received messages of "RX#30" and "RX#31" received in the previous cycle (the cycle of cycle number #10) are also processed and passed to the receiving target tasks.

[0068] The received message of "RX#32" received after the occurrence of the cycle start request event (Cyc) targeting the cycle of cycle number #11 is not passed to the target task during the execution of the cycle of cycle number #11, even if it is agreed upon in the cycle of cycle number #11, and is passed in the next cycle (the cycle of cycle number #12).

[0069] The tasks to be executed (tasks to be woken up, received messages to be read) in the cycle in which the message reception event occurs are already determined at the time when the cycle start request event for that cycle is created. Therefore, if an operation not defined in the cycle start request event, such as passing the received message with RX#32, is performed in the cycle of cycle number #11, the behavior may change between the nodes that sent and received the message. Therefore, in the present embodiment, in order to prevent such a situation from occurring, when a received message not defined in the cycle start request event is received in that cycle, the cluster management unit 124 passes the received message to the target task in the next cycle and later.

[0070] [Type of wake-up request] Next, the types of wake-up requests will be explained. A wake-up request is issued to an "event-driven task", and when a wake-up request is issued, the task execution unit 121 executes the task in a predetermined order in the next period. Since the execution time of a task changes due to disturbances in each node, for example, the timing of a wake-up request that occurs upon completion of an analysis task that runs in parallel with a periodic task may differ depending on the node. However, even if the timing of the occurrence differs, it is expected that a wake-up request for the same task will eventually occur in all nodes.

[0071] Fig. 8 is a diagram showing examples of types of wake-up requests. As shown in Fig. 8, the causes of a wake-up request, that is, wake-up causes, include "waiting for an event without a timeout", "waiting for an event with a timeout", and "unconditional timer".

[0072] "Waiting for an event without timeout" indicates a state of waiting for an event (waiting event) that is expected to occur eventually. The trigger for the generation of a wake-up request for the wake-up cause is the occurrence of the waiting event. In response to the occurrence of the waiting event, the leader node operates to follow its own wake-up request.

[0073] In the case of a follower node, when a wake-up request occurs only in the leader node, the follower node waits for a wake-up request to occur in the follower node, and after the wake-up request occurs, executes the task associated with the wake-up request following the leader node. On the other hand, when a wake-up request does not occur in the leader node, the follower node postpones the execution of the task that is the target of the wake-up request to the next cycle or later.

[0074] "Waiting for an event with timeout" indicates a waiting state for the occurrence of an event with a timeout that is expected to occur eventually. The trigger for the wake-up request for the wake-up factor is either the occurrence of the waiting event or the occurrence of a timeout. As a response when an event with a timeout occurs, at the leader node, an operation is performed to follow its own wake-up request or wake-up factor (timeout).

[0075] At the follower node, when a timeout occurs at the leader node, it is regarded as if a timeout has occurred in the same way as at the leader node, and processing based on the occurrence of the timeout is performed. On the other hand, when a waiting event occurs at the leader node, at the follower node, waiting for the occurrence of the waiting event is also performed, and after the occurrence of the waiting event, the task associated with the occurred event is executed following the leader node.

[0076] The trigger for the wake-up request of the "unconditional timer" is the arrival of the time set in the timer. As a response when this event occurs, at the leader node, an operation is performed to follow its own wake-up request. At the follower node, an operation is performed to wake up the target task of the wake-up request at a predetermined cycle following the wake-up request that occurred at the leader node.

[0077] [Configuration of the task management table] Next, with reference to FIG. 9, the configuration of the task management table 122 included in the multi-execution base unit 12 will be described. FIG. 9 is a diagram showing a configuration example of the task management table 122. As shown in FIG. 9, the task management table 122 has columns for "Task", "Drive type", "Wake-up flag", "Wake-up factor", "Wake-up request flag", and "Wake-up request factor".

[0078] In the "Task" column, information on the task identifier is stored. FIG. 9 shows an example in which task identifiers "T1" to "T5" are stored. In the item of "Drive Type", information on the type of task (event-driven task) is stored. The "Period" in the item of "Drive Type" indicates a periodic drive task, that is, a task executed every period. "Message Reception" indicates a reception-driven task, that is, a task that is executed in the next period when a message addressed to the task is received from an external device. "Event" indicates an event-driven task, that is, a task that is executed in the next period when a specific event occurs.

[0079] In the item of "Wake-up Flag", a flag indicating whether the task is scheduled to wake up in that period is stored. If the task is a task scheduled to wake up, "1" is stored in the item of "Wake-up Flag". Which task the "1" of the wake-up flag is set for is determined by the wake-up task list that the period start request has. And when an agreement on the period start request event with the wake-up task list set is formed between the leader node and the follower node, the setting content of the wake-up flag is determined.

[0080] In the item of "Wake-up Cause", information on the wake-up cause of the task is stored. The information on the wake-up cause is passed from the task management unit 124 to the task as information when the target task wakes up, and like the wake-up flag, it is information determined by the wake-up task list. In the "Wake-up Cause", there are "Unconditional" (not shown in the figure), "Waiting for Event Occurrence", and "Timeout", etc.

[0081] The "Unconditional" wake-up cause wakes up every period. The task associated with the "Unconditional" wake-up cause is not described in the wake-up task list. For the "Unconditional" wake-up cause, "1" of the wake-up flag is fixedly associated.

[0082] The wake-up cause of "Waiting for Event Occurrence" is that a waiting event occurs. The wake-up cause of "Timeout" is that a predetermined time elapses without a waiting event occurring.

[0083] In the item of "wake-up request flag", a flag indicating the presence or absence of a wake-up request at each node is stored. If a wake-up request has occurred, "1" is stored, and if not, "0" is stored.

[0084] For example, in the "Task T3" of the task management table 122, the value of the "wake-up flag" is "1" and the value of the "wake-up request flag" is "0". This indicates that although a wake-up request has been issued at the leader node, no wake-up request has been issued yet (no waiting event has occurred) at the follower node.

[0085] In the item of "wake-up request factor", information on the factor that caused the wake-up request flag to become "1", that is, the factor that caused a wake-up request at the follower node, is stored.

[0086] [Configuration of Wake-up Request List] Next, with reference to FIG. 10, the wake-up request list 123 will be described. FIG. 10 is a diagram showing a configuration example of the wake-up request list 123. The wake-up request list 123 is a list used to manage tasks to be woken up after the next cycle. In the wake-up request list 123, the task identifier of the task to be woken up and information on the wake-up factor of the task are stored. An entry for a newly generated wake-up request is appended to the end (the right end in the figure) of the wake-up request list 123.

[0087] In the wake-up request list 123 shown in FIG. 10, the wake-up factor "TO (timeout)" is associated with the task with the task identifier "T5" (Task T5). For the subsequent tasks with task identifiers "T3" (Task T3) and "T6" (Task T6), the wake-up factors "E (waiting event occurred)" are respectively associated.

[0088] When the cluster management unit 124 (see FIG. 2) of the leader node and the follower nodes transfers the content of an entry in the wake-up request list 123 to the task management table 122, it deletes the entry to be transferred from the wake-up request list 123. Note that the deletion of the entry in this case may not be performed in order from the entry of the task at the head of the wake-up request list 123.

[0089] For a task in the task management table 122 where the value of the "wake-up flag" is "1", if the entry of the task is in the wake-up request list 123, the cluster management unit 124 deletes the entry from the wake-up request list 123. Then, the cluster management unit 124 sets "1" to the value of the "wake-up request flag" of the task in the task management table 122, and sets the wake-up request factor entered in the wake-up request list 123 to the item of "wake-up request factor".

[0090] [Periodic Management Process] Next, the periodic management process by the multi-system control system 100 according to the present embodiment will be described. The multi-system control process according to the present embodiment, FIG. 11 is a flowchart showing an example of the procedure of the periodic management process.

[0091] First, the cluster management unit 124 of the multi-execution base unit 12 performs next-period preparation processing (step S1). The next-period preparation processing will be described in detail with reference to FIG. 12 below. Next, the cluster management unit 124 executes task management table update processing in accordance with the content of the wake-up tasks for the next period agreed upon by the next-period preparation processing in step S1 (step S2). The task management table update processing will be described in detail with reference to FIG. 13 below.

[0092] Next, the cluster management unit 124 performs a process of transferring the received message to the task to be received (step S3). The received message for which the transfer process is performed in step S3 is a message received by the leader node in the "message reception event" that exists between the periodic start request event of the next cycle and the previous periodic start request event. In step S3, the cluster management unit 124 determines the task to be received based on the destination information such as the port number included in the received message, and transfers the received message to the task.

[0093] Next, the task execution unit 121 waits for the arrival of the periodic start time (step S4). Specifically, the task execution unit 121 waits until the virtual time at which the agreement was formed in step S1 arrives. Alternatively, in the follower node that is following the leader node during execution, the virtual time may have already elapsed, and in this case, the task execution unit 121 immediately proceeds to step S5.

[0094] Next, the task execution unit 121 performs task execution processing (step S5). The task execution processing will be described in detail with reference to FIG. 16 described later.

[0095] (Next cycle preparation process) Next, with reference to FIG. 12, the next cycle preparation process performed in step S1 of FIG. 11 will be described. FIG. 12 is a flowchart showing an example of the procedure of the next cycle preparation process.

[0096] First, the cluster management unit 124 determines whether or not its own node has been elected as the leader node (step S11). If it is determined in step S11 that the node has been elected as the leader node (S11 is YES), the cluster management unit 124 determines the virtual time of the start time of the next cycle and the tasks to be woken up (step S12). For example, if the period is 1 second and the start time of the current cycle is "2:03:04.5", the cluster management unit 124 determines the start time of the next cycle to be "2:03:05.5".

[0097] Next, the cluster management unit 124 generates a periodic start request event (see FIG. 6) based on the content determined in step S12, and adds it to the event log in the event log storage unit 125 (see FIG. 2) (step S13). More specifically, the cluster management unit 124 sets the cycle number to the number of the current cycle number + 1, and sets the virtual time to the cycle start time determined in step S12. Next, the cluster management unit 124 generates a wake-up task list in the format of the information corresponding to the wake-up request list 123, that is, the identifier of the task to be woken up and the wake-up factor, and creates a periodic start request event including the wake-up task list.

[0098] Next, the cluster management unit 124 determines whether the consensus formation of the periodic start request event is completed (step S14). If it is determined in step S14 that the consensus formation is not completed (step S14 is NO), the cluster management unit 124 repeats the determination in step S14. On the other hand, if it is determined that the consensus formation is completed (step S14 is YES), the next cycle preparation process ends and the process proceeds to step S2 in FIG. 11.

[0099] On the other hand, if it is determined in step S11 that the own node has not been selected as the leader node (step S11 is NO), that is, if it is determined that the own node is a follower node, the cluster management unit 124 determines whether the consensus formation of the periodic start request event is completed (step S15). If it is determined that the consensus formation is not completed (step S15 is NO), the cluster management unit 124 returns to the determination in step S11. On the other hand, if it is determined that the consensus formation is completed (step S15 is YES), the next cycle preparation process ends and the process proceeds to step S2 in FIG. 11.

[0100] Note that the determination as to whether the own node has been selected as the leader node in step S11 is periodically performed. And in step S15, if it can be confirmed that the own node has been selected as the leader node before it is determined that the consensus formation is completed, at that time, the cluster management unit 124 issues a periodic start request event again as the leader node.

[0101] Also, in the follower node, when the tasks of the leader node are being executed subsequently, it is also conceivable that the cycle start request for the cycles after the next cycle has already been agreed upon at the start of the self-cycle preparation process. In this case, the follower node immediately ends the next-cycle preparation process.

[0102] (Task management table update process) Next, the task management table update process executed in step S2 of FIG. 11 will be described. FIG. 13 is a flowchart showing an example of the procedure of the task management table update process.

[0103] First, the cluster management unit 124 initializes the task management table 122 for the next cycle (step S21). Specifically, the cluster management unit 124 arranges the tasks to be executed in the next cycle in the execution order, and sets the "wake-up flag" and "wake-up request flag" associated with each task to "0". Further, the cluster management unit 124 sets the "wake-up factor" and "wake-up request factor" to "unspecified (blank)".

[0104] Next, the cluster management unit 124 sets "1" to the wake-up flag of the reception target task of the received message (step S22). The received message processed in step S22 is the message received in the message reception event that occurred between the cycle start request event of the current cycle and the cycle start request event of the previous cycle.

[0105] Next, the cluster management unit 124 reflects the content of the wake-up task list included in the cycle start request event in the task management table 122 (step S23). Specifically, for the tasks included in the wake-up task list included in the cycle start request event of the next cycle, the cluster management unit 124 sets "1" to the "wake-up flag" of the task management table 122. Then, the cluster management unit 124 transcribes the content described in the "wake-up factor" of the wake-up task list to the "wake-up factor" of the task management table 122.

[0106] Next, the cluster management unit 124 of each node reflects the content of the wake-up request list 123 of its own node in the task management table 122 (step S24). Specifically, for a task whose "wake-up flag" value in the task management table 122 is "1", if the entry of the task is in the wake-up request list 123, the cluster management unit 124 deletes the entry from the wake-up request list 123. Then, the cluster management unit 124 sets the "wake-up reason" of the deleted entry in the "wake-up request reason" in the task management table 122. After the processing of step S24, the task management table update process ends and the process moves to step S3 in FIG. 11.

[0107] (Message reception processing) Next, the message reception processing when a message is transmitted from the external device 2 (see FIG. 2) at an arbitrary timing will be described. FIG. 14 is a flowchart showing an example of the procedure of the message reception processing.

[0108] First, the cluster management unit 124 waits for the reception of a message (step S31). The message waited for reception in step S31 is a message transmitted from the external device 2 at an arbitrary timing.

[0109] Also, the message waited for reception in step S31 is finally transferred to a predetermined task at the start of the next cycle and is a message that the task processes as input. Conversely, messages that the cycle control application 11 (see FIG. 2) does not reflect in a state machine (not shown), for example, messages transmitted and received on a protocol such as Raft, are not included in the messages waited for reception in step S31.

[0110] After receiving the message that has been waiting in step S31, the cluster management unit 124 determines whether or not the own node has been selected as the leader node (step S32). If it is determined that the own node has been selected as the leader node (S32 is YES), the cluster management unit 124 adds an entry including the entire received message as a message reception event to the event log in the event log storage unit 125 (step S34). Then, the cluster management unit 124 starts consensus formation among nodes regarding the entry and returns to step S31.

[0111] In step S32, if it is determined that the own node has not been selected as the leader node, that is, it is a follower node (S32 is NO), the cluster management unit 124 of the follower node transfers the received message to the leader node (step S33). After the process of step S33, the cluster management unit 124 returns the process to step S31.

[0112] Normally, the leader node receives messages from the external device 2, but there may be cases where a follower node receives the message for some reason. In this case, the follower node transfers the message to the leader node. Note that the message transferred from the follower node to the leader node in step S33 is the target of the message waiting to be received in step S31.

[0113] (Wake-up request reception process) Next, the wake-up request reception process will be described. The wake-up request reception process is a process that operates asynchronously when a wake-up request occurs. The waiting for a wake-up request is always performed. FIG. 15 is a flowchart showing an example of the procedure of the wake-up request reception process.

[0114] First, the cluster management unit 124 waits for a wake-up request for a task to occur within its own node (step S41). Next, the cluster management unit 124 determines whether its own node is a follower node (step S42). That is, it determines whether it has not been selected as the leader node. When it is determined that its own node is a follower node (step S42 is YES), the cluster management unit 124 determines whether the task to be woken up is waiting for the occurrence of a wake-up request (step S43). More specifically, for the row of the task in the task management table 122, if the wake-up flag is "1" and the wake-up request flag is "0", it is determined that it is waiting for the occurrence of a wake-up request.

[0115] When it is determined that it is waiting for the occurrence of a wake-up request (step S43 is YES), the cluster management unit 124 updates the task management table 122 by setting "1" in the "wake-up request flag" of the task management table 122. (step S44). After the process of step S44, the cluster management unit 124 returns the process to step S41.

[0116] Note that when follow-up execution is being performed in the follower node, the execution of the task may be temporarily stopped due to waiting for an event to occur. In this case, the cluster management unit 124 of the follower node does not append an entry to the wake-up request list 123 and directly updates the task management table 122. That is, for the task, "1" is set in the "wake-up request flag" of the task management table 122, and the wake-up factor is set in the "wake-up request factor".

[0117] On the other hand, when step S42 is determined to be NO, that is, when it is determined that its own node is the leader node, or when in step S43, it is determined that the task to be woken up is not in the status of waiting for the occurrence of a wake-up request (step S43 is NO), the process of step S45 is performed.

[0118] In step S45, the cluster management unit 124 performs a process of adding an entry to the wake-up request list 123. The entry to be added to the wake-up request list 123 is composed of the identifier of the task targeted by the wake-up request and the wake-up factor. The cluster management unit 124 adds this entry to the end of the wake-up request list 123. After the process of step S45, the cluster management unit 124 returns the process to step S41.

[0119] (Task execution process) Next, the task execution process performed in step S5 of FIG. 11 will be described. FIG. 16 is a flowchart showing an example of the procedure of the task execution process.

[0120] The task execution process shown in FIG. 16 is carried out in order from the top row of the task management table 122 (see FIG. 9) and ends when the bottom row is reached. That is, the task execution process is processed in the predefined execution order of the tasks.

[0121] First, the cluster management unit 124 determines whether the value of the "wake-up flag" in the task management table 122 is "1" (step S51). If the value of the "wake-up flag" is not "1", that is, if it is "0" (step S51 is NO), the cluster management unit 124 does not cause the task execution unit 121 to execute the task in this cycle. That is, the task execution process in this cycle ends.

[0122] On the other hand, when it is determined that the value of the wake-up flag is "1" (step S51 is YES), the cluster management unit 124 determines whether the task targeted by the task execution process is an event-driven task (step S52). When it is determined that the task is an event-driven task (step S52 is YES), the cluster management unit 124 determines whether the value of the "wake-up request flag" in the task management table 122 is "1" (step S53).

[0123] In the leader node, when the task execution process is performed, the value of the "wake-up request flag" should be "1". Therefore, the determination in step S53 is YES. On the other hand, in the follower node, the determination in step S53 may be NO.

[0124] If it is determined in step S53 that the value of the "wake-up request flag" is not "1", that is, it is "0" (step S53 is NO), the cluster management unit 124 of the follower node waits for the value of the "wake-up request flag" to be set to "1" (step S54). When a wake-up request is made for the task, "1" is also set in the corresponding wake-up request flag of the task management table 122. Therefore, the cluster management unit 124 of the follower node holds off on task execution until then, that is, until the wake-up request flag becomes "1".

[0125] If the determination in step S53 is YES, or after the process of step S54, the cluster management unit 124 determines whether the content of the "wake-up cause" in the task management table 122 matches the content of the "wake-up request cause" (step S55). As described above, the "wake-up cause" is determined by the leader node, and the "wake-up request cause" is the cause based on the actual wake-up request that occurred at the node. Therefore, in the leader node, it is assumed that the "wake-up cause" and the "wake-up request cause" match.

[0126] On the other hand, in the follower node waiting for the occurrence of a timed-out event, there is a possibility that the "wake-up cause" and the "wake-up request cause" do not match. If they do not match, that is, if the determination in step S55 is NO, the cluster management unit 124 performs a wake-up cause matching process (step S56). The wake-up cause matching process will be described in detail with reference to FIG. 17 below.

[0127] If the determination in step S52 is NO, if the determination in step S55 is YES, or after the process of step S56, the task execution unit 121 executes a task with a wake-up factor (step S57). When the determination in step S52 is NO, it means that the task is a task other than an event-driven task, that is, a periodic drive task or a reception drive task.

[0128] In step S57, the task execution unit 121 passes the wake-up factor information to the task, and the task executes the process deterministically according to the received wake-up factor. After the process of step S57 or when the determination in step S51 is NO and the task execution process for all tasks in the task management table 122 is completed, the task execution process ends.

[0129] (Wake-up factor matching process) Next, the wake-up factor matching process executed in step S56 of FIG. 16 will be described. FIG. 17 is a flowchart showing an example of the procedure of the wake-up factor matching process. As described above, this flow is executed at the follower node when the wake-up factor determined by the leader does not match the wake-up request factor generated at the follower.

[0130] First, the cluster management unit 124 determines whether the "wake-up factor" in the task management table 122 is "waiting for event occurrence" (step S61). When a timeout occurs at the leader node and a waiting event occurs at the follower node, the determination in step S61 is NO.

[0131] If the determination in step S61 is NO, a waiting event has occurred in the follower node, but the cluster management unit 124 of the follower node assumes that a timeout has occurred in its own node in the same way as in the leader node (step S62). Specifically, the task execution unit 121 of the follower node sets the "wake-up cause" in the task management table 122 to "timeout occurred" and passes this wake-up cause to the task for execution. By performing this process, the processing results of the tasks can be made consistent between the leader node and the follower node.

[0132] On the other hand, if a waiting event has occurred in the leader node and a timeout has occurred in the follower node, the determination in step S61 in the follower node is YES. If the determination in step S61 is YES, the task execution unit 121 of the follower node assumes that no timeout has occurred in its own node and waits indefinitely for the occurrence of the waiting event (step S63). During the execution of the process in step S63, the execution of the task in the follower node is temporarily suspended. However, by performing the process in step S63, the processing results of the tasks can be made consistent between the leader node and the follower node. After the process in step S62 or step S63, the wake-up cause consistency process ends.

[0133] In the above-described embodiment, when wake-up requests for the next cycle are issued only at some nodes due to accidental factors, or when a discrepancy in wake-up factors occurs between a node at which a waiting event occurred near the timeout and a node that timed out, a process is performed to match the wake-up request factor, which is the wake-up factor of the task in the follower node, with the wake-up factor of the leader node. Since the execution order of the periodic tasks is predetermined, by making the tasks to be woken up in the next cycle consistent among the nodes as described above, each node can execute the same tasks in the same order and with the same inputs (including the wake-up factors) in the next cycle. Therefore, according to the present embodiment, in a multi-system control system that handles tasks driven by internal events, consistency in behavior between the main system and the slave system, that is, consistency in the application state and the content transmitted to the external device, can be maintained.

[0134] Further, according to the present embodiment, since a state in which consistency between the leader node and the follower node is maintained can be maintained, even when the leader node stops due to a failure, it is possible to realize switching of the follower node to a new leader node without stopping the multi-system control system 100.

[0135] Further, according to the present embodiment, since data consistency between the main system and the slave system of the multi-system control system can be maintained without using a dedicated OS or HW, etc., it is also possible to achieve cloudification of the multi-system control system.

[0136] [Event Waiting API Call Processing] Next, the call processing of the event waiting API with timeout by the task execution unit 121 of the multi-execution base unit 12 will be described. FIG. 18 is a flowchart showing an example of the procedure of the event waiting API call processing with timeout. The event waiting API call processing with timeout starts when the task calls the API provided by the multi-execution base unit 12.

[0137] First, the task execution unit 121 checks, in order from the younger cycle number, whether the task that is the call source of the API is present in the wake-up task list within the consensus-formed cycle start request event after the next cycle of its own node (step S71). If it is determined that the task that is the call source is included (step S71 is YES), the process ends without setting a timer (not shown). If the task that is the call source is included in the wake-up task list within the consensus-formed cycle start request event after the next cycle, it means that the own node is a follower node that is performing catch-up execution, and it means that it has already been determined based on what factor and in which cycle the task wakes up. Therefore, there is no need to set a timer.

[0138] On the other hand, if it is determined that the task that is the call source is not included (step S71 is NO), the task execution unit 121 sets a timer (step S72). This is because when the task that is the call source is not included in the cycle start request event, it is not determined when the event-waiting state times out or when the waiting event occurs. When step S71 is determined to be YES, or after the process of step S72, the API call process for waiting for an event with a timeout ends.

[0139] FIG. 19 is a diagram showing an example of an opportunity for generating a wake-up request for a task. As shown in FIG. 19, when an event corresponding to the wake-up factor of "waiting for an event with a timeout" occurs, or when a timer associated with the wake-up factor of "unconditional timer" fires, the task execution unit 121 (of the multiple execution base unit 12) wakes up (executes) the task. That is, a wake-up request is issued.

[0140] [Processing for waiting for an event or timeout to occur] Next, the processing for waiting for an event or timeout to occur after the API call for waiting for an event by the task execution unit 121 will be described. FIG. 20 is a flowchart showing an example of the procedure for the processing for waiting for an event or timeout to occur. The processing for waiting for an event or timeout to occur starts when a task calls the event-waiting API provided by the multiple execution base unit 12.

[0141] First, the task execution unit 121 waits for the occurrence of a predetermined event or the firing of a set timer (step S81). After the waiting in step S81, a predetermined event occurs or the timer fires. Next, the task execution unit 121 determines whether the "wake-up flag" associated with the calling task in the task management table 122 is "1" and the "wake-up request flag" is "0" (step S82). If the node is a follower node and is in the process of trailing and executing the leader node, step S82 results in a YES determination.

[0142] On the other hand, when step S82 results in a YES determination, that is, when the node is a follower node in the process of trailing and executing, the task execution unit 121 determines whether the wake-up factors match (step S84). Specifically, the task execution unit 121 checks whether the "wake-up factor" on the task management table 122 matches the factor of the current wake-up request (wake-up factor), that is, whether it matches "waiting event occurrence" or "timeout".

[0143] When it is determined that the wake-up factors match (step S84 is YES), the task execution unit 121 sets "1" in the "wake-up request flag" corresponding to the task in the task management table 122 (step S85). If it was waiting for "1" to be set in the "wake-up request flag" in step S54 of FIG. 15, after the process of step S85, the task execution unit 121 resumes the execution of the waiting task.

[0144] In step S84, if it is determined that the wake-up factors do not match (step S84 is NO), the task execution unit 121 determines whether the "wake-up factor" on the task management table 122, that is, the wake-up factor determined by the leader node, is "timeout" (step S86). If it is "timeout" (step S86 is YES), the task execution unit 121 performs the process of step S85. That is, it sets "1" in the "wake-up request flag" of the task management table 122. As a result, the task will wake up in the next cycle with the wake-up factor = "timeout".

[0145] On the other hand, if the "wake-up factor" on the task management table 122 is not "timeout" (NO in step S86), that is, if it is "waiting for event occurrence", after canceling the timer, it returns to step S81 and waits for the occurrence of the waiting event.

[0146] If the determination in step S82 is NO, that is, if the node is not a follower node that is being executed later, the wake-up request for the task is a wake-up request for after the next cycle. Therefore, the task execution unit 121 registers the wake-up factor of the task in the wake-up request list 123 (step S83). After the process of step S83 or step S85, the process of waiting for the occurrence of an event or timeout ends.

[0147] [Synchronization process between leader node and follower node] Next, the synchronization process between the leader node and the follower node will be described. In the present embodiment, the objects synchronized by the synchronization process are the execution order of tasks, wake-up factors, input (received) messages, and virtual times referred to by tasks. Since each task is assumed to behave deterministically, if these matters are synchronized between the master and slave systems (leader node and follower node), the behaviors of each system can also be made consistent. And by making the behaviors of each system consistent, the consistency when switching the slave system to the master system is also guaranteed.

[0148] FIG. 21 is a sequence diagram showing an example of the procedure of synchronization processing between a leader node and a follower node. The leader node shown in FIG. 21 is the computer 1A shown in FIG. 1, and the follower node is either the computer 1B or the computer 1C.

[0149] First, the cluster management unit 124 of the leader node receives a message from the external device 2 (see FIG. 1) (step S101). Next, the cluster management unit 124 adds the message received in step S101 to the event log (denoted as "log" in the figure) in the event log storage unit 125 (step S102). Then, the cluster management unit 124 of the leader node replicates the added event log to the event log of the follower node (step S103). Next, the cluster management unit 124 of the leader node detects the occurrence of a wake-up request for task T1 (step S104). After the process of step S104, it is assumed that the period N has ended.

[0150] Next, the cluster management unit 124 generates a cycle start request event (Cyc(N + 1, T N+1 , [T1])) that includes information about the wake-up request that occurred in the cycle N, that is, the task T1 based on the wake-up request detected in step S104 (step S105). The content of the cycle start request event Cyc is, in order from the left, the cycle number (N + 1), the virtual time (T N+1 ) and the wake-up task list ([T1]).

[0151] Next, the cluster management unit 124 of the leader node adds the cycle start request event generated in step S105 to the event log (step S106). Next, the cluster management unit 124 of the leader node replicates the added event log to the event log of the follower node (step S107).

[0152] Next, from the follower node, for each event log replicated in steps S103 and S107, a replication OK (completion) is notified (step S108). Next, from the cluster management unit 124 of the leader node, a consensus formation is notified (step S109).

[0153] Next, at the leader node, the event log ([Rx, Cyc]) is applied to the state machine (step S110), and the message received in step S101 is transferred to a task (step S111). Also, at the follower node, the event log ([Rx, Cyc]) is applied to the state machine (step S112), and the message received by the leader node in step S101 is transferred to a task (step S113).

[0154] And at the arrival of the virtual time (T N+1 ) defined in the cycle start request event that occurred in step S105, an instruction to wake up the head task of task T5 is issued from the cluster management unit 124 to the task execution unit 121 (step S114). Similarly, also at the follower node, at the arrival of the virtual time (T N+1 ) an instruction to wake up the head task of task T5 is issued from the cluster management unit 124 to the task execution unit 121 (step S115). After the processing of steps S114 and S115, cycle N+1 starts.

[0155] According to this embodiment, the event log of the cycle start request event (Cyc ) that occurred at the leader node, and the event log of the received message (RX) are also replicated to the follower node. Thereby, the execution order of tasks, wake-up factors, virtual times referenced by tasks, and each piece of information of the received message defined within the cycle start request event are synchronized between the leader node and the follower node. As a result, the behaviors in each system of the leader node and the follower node are also made to match, so the consistency when switching the follower node to the leader node is also guaranteed.

[0156] [Synchronization of Wake-up Factors between Leader Node and Follower Node] Next, the period for waking up event-driven tasks and the synchronization process of wake-up factors between the leader node and the follower node according to this embodiment will be described. When generating a period start request event for the next period, the leader node determines which event-driven task to wake up with which factor for the next period, and each node wakes up a predetermined task in a predetermined order and with a predetermined factor in the next period according to the determination. However, there may be cases where the presence or absence of a task wake-up request and the wake-up factors differ between nodes, and in order to unify the behavior of each node, it is necessary to synchronize the presence or absence of wake-up and the wake-up factors of each task between nodes. Cases where differences occur in the wake-up requests and wake-up factors between the leader node and the follower node include the following cases (a), (b), and (c).

[0157] (a) A case where a wait event has occurred and a wake-up request has been issued at the leader node, but no wake-up request has been issued at the follower node, or a wake-up request has been issued due to a timeout (b) A case where a timeout has occurred and a wake-up request has been issued at the leader node, but no wake-up request has been issued at the follower node, or a wake-up request has been issued due to the occurrence of a wait event (c) A case where a wake-up request has been issued at a certain follower node due to the occurrence of a wait event or a timeout, but no wake-up request has been issued at the leader node

[0158] When the wake-up factor inconsistency of the above (a) occurs, as shown in step S63 of FIG. 17, by performing control to wait for the occurrence of a wait event at the follower node, the inconsistency in the presence or absence of wake-up and the wake-up factor can be eliminated. After the wait event occurs, the follower node executes the tasks that have been executed at the leader node in a catch-up manner.

[0159] On the other hand, when the wake-up factor inconsistency in (b) above occurs, as shown in step S62 of FIG. 17, control is performed such that a timeout is considered to have occurred also in the follower node, whereby the wake-up factor inconsistency can be resolved.

[0160] In the case of (c) above, since no wake-up request for the task is issued at the leader node, the task will not be woken up in the next cycle, and at this point, it is not necessary to match the wake-up factors. Thereafter, when a wake-up request is issued for the task at the leader node, it is determined whether the presence or absence of the wake-up request and the factors match between nodes, or whether it falls under either (a) or (b) above.

[0161] FIG. 22 is a diagram showing an example of a method for resolving the inconsistency in the presence or absence of a wake-up request (the inconsistency in (a) above) that occurs while waiting for a wake-up request for a timeout-free event.

[0162] On the left side of FIG. 22, the execution status of the tasks at the leader node is shown in chronological order from top to bottom, and on the right side, the execution status of the tasks at the follower node is shown in chronological order from top to bottom. "t3" to "t5" in the figure indicate the virtual time t in cycles n (n = 3, 4, 5). At the leader node and the follower node, the tasks executed in cycle n are tasks T1 to T5 shown in FIG. 4. In the example shown in FIG. 22, task T3 is an event-driven task that waits for the completion of the processing of the analysis task shown in the figure, and when the processing of the analysis task is completed, a wake-up request is issued at "wake-up event occurs". Note that this analysis task operates in parallel with the periodic task, receives a unique input from one of the periodic tasks and is driven, performs a unique process based on the input, and generates a unique output upon completion of the process. Then, the woken-up task T3 reads its output (processing result) as an input and performs a unique process. Therefore, a series of processes including the analysis task do not cause a difference in the behavior of the periodic tasks between nodes.

[0163] In the phase shown at the uppermost stage of FIG. 22, it is assumed that the processing timing Tm1 at each of the leader node and the follower node is at the position indicated by the thick solid line, that is, immediately before the virtual time "t3".

[0164] In the leader node, it is assumed that the execution of the analysis task that outputs the analysis result to be read by the periodic task T3 starting at the virtual time "t3" is completed at the processing timing Tm1 before the virtual time "t3". And at the time of the processing timing Tm1, a wake-up request for the tasks (task T1 to task T5) to be executed in the period starting at the virtual time "t3" is also generated.

[0165] As a result, in the next period starting at the virtual time "t4" shown in the middle of the figure, the leader node completes the processing from task T1 to task T5, and the processing timing Tm2r is at the end time of task T5.

[0166] On the other hand, in the follower node, it is assumed that the analysis task that outputs the analysis result to be read by task T3 is still being executed at the time of the processing timing Tm1 shown in the upper part of the figure. As a result, in the follower node, even when the processing timing Tm1 arrives, the wake-up request for task T3 for the next period starting at the virtual time "t3" has not yet been issued.

[0167] In this case, in the next period starting at the virtual time "t4" shown in the middle of the figure, in the follower node, the execution of task T3 is suspended without being executed. As a result, the processing timing Tm2f in the follower node remains stopped at the time before task T3.

[0168] When the execution of the analysis task that outputs the analysis result to be read in task T3 is completed (the lower part of the figure), the execution of task T3 in the follower node is resumed (postponed execution). Also, at the time when the execution of task T5 in the follower node is completed, since the virtual time "t5" at which the execution of the next cycle starts has already passed, the follower node executes tasks T1, T2, T3... that should be executed in the cycle of virtual time "t5" in a postponed manner. After the periodic processing is completed, there is idle time until the start of the next cycle, so the follower node that is executing tasks in a postponed manner should eventually catch up with the start of the cycle of the leader node.

[0169] According to the present embodiment, by performing such control, even when a wake-up request for a certain task does not occur in the follower node, the execution status of tasks between the leader node and the follower node can be made consistent. FIG. 23 below shows the procedure for resolving the inconsistency in wake-up factors between the leader node and the follower node when the same inconsistency as in (a) above occurs.

[0170] FIG. 23 is a sequence diagram showing an example of the procedure for resolving the inconsistency in wake-up factors, which is performed when a wait event occurs in the leader node and a timeout occurs in some of the follower nodes in waiting for an event with a timeout. The leader node shown in FIG. 23 is the computer 1A shown in FIG. 1, the follower node B is the computer 1B, and the follower node C is the computer 1C.

[0171] First, in the leader node, follower node A, and follower node B, an API for waiting for an event with a timeout (denoted as "TO" in the figure) is called from task T1 (steps S121A, S121B, S121C). Next, a wake-up request (T1, E) based on the occurrence of a wait event occurs in follower node C (step S122C), and subsequently, a wake-up request (T1, E) based on the occurrence of a wait event occurs in the leader node (step S122A).

[0172] Next, in follower node B, a "wake-up request (T1, TO)" based on the occurrence of a timeout is generated (step S122B). That is, in follower node B, no waiting event has occurred, which means that a timeout has occurred.

[0173] Next, the leader node adds an event log of the cycle start request event (Cyc) for the next cycle (N + 1) and replicates it to follower nodes B and C (steps S123B, step S123C). Next, a notification of the completion of replication (OK) of the event log is received from follower node C (step S124C). The leader node recognizes that the Cyc (cycle start request) event has been replicated and completed at more than half of the nodes including its own node, and notifies follower node B and follower node C of the commit (completion of consensus formation) of the event (steps S125B, step S125C).

[0174] Next, when applying the consensus-formed Cyc event to each node, each follower node performs unification so as to wake up task T1 due to the wake-up factor (occurrence of a waiting event) of task T1 determined by the leader. That is, in follower node B, the timeout that occurred for the wake-up request of task T1 is ignored, and the event waiting for task T1 is continued (step S126B). Next, a notification of the completion of replication (OK) of the event log added in step S123A is sent from follower node B to the leader node (step S127B) (since the response for the completion of log replication is asynchronous, it does not necessarily occur at this timing). At this point, cycle N ends.

[0175] In the next cycle N + 1, in the leader node and follower node C, according to the "wake-up request (T1, E)" for cycle N + 1, task T1 is woken up due to the wake-up factor "occurrence of a waiting event" (steps S128A, step S128C). On the other hand, in follower node B, since the event that task T1 is waiting for has not occurred yet, task T1 is not woken up at this point.

[0176] After that, when a waiting event for task T1 occurs at follower node B (step S128B), task T1 is woken up (step S129B), and the execution of the periodic task is resumed (postponed execution).

[0177] Next, the process for resolving the wake-up factor inconsistency in (b) described above will be explained. FIG. 24 is a sequence diagram showing an example of the procedure of the wake-up factor inconsistency (the inconsistency in (b) above) resolution process when a timeout occurs at the leader node and a waiting event occurs at the follower node in waiting for an event with a timeout.

[0178] First, the leader node, follower node A, and follower node B call the API for waiting for an event with a timeout from task T1 (steps S131A, S131B, S131C). Next, wake-up requests (T1, E) based on "waiting event occurred" occur at follower nodes B and C (steps S132B, S132C). On the other hand, at the leader node, a "wake-up request (T1, TO)" based on "timeout occurred" occurs (step S132A). That is, it means that a timeout occurred without a waiting event occurring at the leader node. At this point, it is assumed that cycle N has ended.

[0179] Next, the leader node creates a cycle start request event (Cyc) having a wake-up task list including task T1 to be woken up in the next cycle (N + 1), and appends it to the log. As a result, this log entry is replicated to follower nodes B and C (steps S133B, S133C). Next, a notification of completion of replication of the event log (OK) is sent from follower node C (step S134C), and subsequently, a notification of completion of replication of the event log (OK) is sent from follower node B (step S134B).

[0180] The leader node that has detected that replication of the Cyc event has been completed at a majority of nodes including its own node shall consider that the Cyc event has been committed (consensus reached), and notify follower node B and follower node C of the completion of consensus formation in the event log (step S135B, step S135C).

[0181] Next, the next cycle N+1 starts, and each node wakes up task T1 due to the wake-up factor "timeout" according to the wake-up task list included in the Cyc event of cycle N+1. (step S136A, step S136B, step S136C).

[0182] According to the above-described embodiment, when the wake-up factor in a certain task is different between the leader node and the follower node depending on whether it is "wait for event occurrence" or "timeout", a process of aligning the wake-up factor of the follower node with that of the leader node is performed. In this way, by eliminating the inconsistency of the wake-up factors between the leader node and the follower nodes, the behavior between the nodes can be made consistent. Thereby, the consistency when switching the follower node to the leader node is also guaranteed.

[0183] <Second Embodiment> [Example 1] Even when the cycle management process according to the first embodiment described above is performed, problems still remain. The problem is that when leader re-election (leader change) of the follower nodes occurs due to the leader node stopping due to a failure, and the node newly elected as the leader is in the process of catch-up execution, the overall processing of the server cluster 10 will stop until the node catches up with the processing of the latest cycle.

[0184] To solve the above problem, in Example 1 of the second embodiment, each of the leader node and the follower node (cluster management unit 124 thereof) performs a process of sharing the wake-up request list in advance prior to the generation of the cycle start request event.

[0185] In Example 1, before generating a cycle start request event, each of the leader node and the follower node periodically shares (broadcasts) a wake-up request list including cycle number information. Specifically, each node performs pre-sharing of the wake-up request list according to the following first to third rules.

[0186] First rule: Let the current cycle number in the own node be N. Second rule: In the phase where no agreement on the cycle start request event in cycle N has been formed, the current cycle number is N. Third rule: When an agreement on the cycle start request event in cycle N is formed and applied to the event log, delete the entry corresponding to the task described in the wake-up task list included in the cycle start request event from the wake-up request list of the own node. Then, reflect the deleted content in the task management table 122 for cycle N.

[0187] When the wake-up request list is shared, the shared wake-up request list needs to be distinguishable as to which cycle it is for. For example, the wake-up request list for cycle N should not be treated as a wake-up request for cycle N+1. By observing the above first and second rules, such a situation can be prevented.

[0188] Also, in the follower node that is executing the tasks of cycle N retrospectively, when the cycle execution is temporarily stopped due to waiting for a wake-up request for task T and a wake-up request is received for task T. In this case, it is necessary to prevent this wake-up request for task T from being pre-shared as a wake-up request for cycle N+1. By observing the above third rule, such a situation can be prevented.

[0189] Also, in Example 1, when the node that pre-shares the wake-up request list is a follower node that is running in catch-up, and a wake-up request is issued for task T that is waiting for a wake-up request, the cluster management unit 124 of the follower node does not add the wake-up request to its own wake-up request list. Then, the wake-up request is directly reflected in the task management table 122. This process is the process of step S85 in FIG. 20. The procedure of the pre-sharing process of the wake-up request list will be described in detail with reference to FIG. 25 described later.

[0190] Also, in Example 1, when each of the leader node and the follower node generates a cycle start request event, based on the pre-shared wake-up request list, a wake-up task list is created according to the following Rule 4 to Rule 6.

[0191] Rule 4: Select any majority of nodes including the leader node, and find the combination in which the number of tasks with a wake-up request of "waiting for event occurrence" is the largest among all the selected nodes. Rule 5: Enumerate the tasks with a wake-up request of "waiting for event occurrence" in all nodes of the combination as the wake-up task list. The wake-up requests that do not enter the wake-up task list are carried over to the wake-up task list for the next cycle. Rule 6: There may be a case where the wake-up request list for the next cycle is not pre-shared from the follower node that is running in catch-up. In this case, the wake-up request list of the follower node is treated as empty.

[0192] In addition, in the first embodiment, when reselecting a leader node, the cluster management unit 124 of each node also preferentially selects a follower node that is not in the process of catch-up execution as the leader node. By adjusting the timeout period from when a follower node detects the absence (stop) of the leader node until it becomes a candidate for the next leader node, it is possible to prioritize follower nodes that are not in the process of catch-up execution. Specifically, the cluster management unit 124 can make it less likely for a follower node in the process of catch-up execution to be selected as the next leader node by setting a longer timeout period for the follower node in the process of catch-up execution.

[0193] Furthermore, in the first embodiment, in the situation of waiting for an event with a timeout, in view of the fact that a wake-up request should be issued at least at the timeout time, the cluster management unit 124 of each node performs the following first process or second process.

[0194] First process: When a timeout occurs in the leader node, as in the first embodiment, the wake-up factor is set to "timeout" and agreement is formed in the periodic start request event. Second process: When a waiting event has occurred only in the leader node and has not occurred in more than half of the nodes including the leader node, until the timeout occurrence time, the wake-up request in the follower node is carried over to subsequent cycles. And if no waiting event occurs in any of the follower nodes before the timeout occurrence time arrives, as in the first embodiment, the wake-up factor is set to "waiting event occurrence" and agreement is formed in the periodic start request event.

[0195] For a specific example of generating a wake-up task list based on the above fourth to sixth rules, refer to FIG. 25 for a detailed description, and for the procedure of generating a wake-up task list, refer to FIGS. 27 and 28 described later for a detailed description.

[0196] (Pre-shared processing of wake-up request list) Next, with reference to FIG. 25, the pre-sharing process of the wake-up request list will be described. FIG. 25 is a flowchart showing an example of the procedure for the pre-sharing process of the wake-up request list.

[0197] First, each cluster management unit 124 of the leader node and the follower nodes waits for a predetermined time corresponding to the sharing interval when periodically sharing the wake-up request list (step S141). For example, 100 ms or the like can be set as the predetermined time.

[0198] Next, the cluster management unit 124 determines whether the cycle start request event of the next cycle N has been consensus-formed and whether the event has been applied to a state machine (not shown) (step S142). If it has been applied, it means that the cycle start request event of the next cycle N has been transferred from the wake-up request list 123 to the task management table 122. When the cycle start request event of the next cycle N has been consensus-formed and applied (step S142 is YES), the cluster management unit 124 sets "N + 1" in the cycle number t (step S143). On the other hand, when the cycle start request event of the next cycle N has not been consensus-formed or has not been applied (step S142 is NO), the cluster management unit 124 sets "N" in the cycle number t (step S144).

[0199] After the process of step S143 or step S144, the cluster management unit 124 broadcasts the current wake-up request list to other nodes as the wake-up request list for cycle t (step S145). Each wake-up request list includes information such as the task identifier to be woken up, the wake-up factor, and the time when the wake-up request was issued. After the process of step S145, the process returns to step S141.

[0200] (Specific example of the process of determining the wake-up task list from the pre-shared wake-up request list) Next, a specific example of generating the wake-up task list based on the above fourth to sixth rules will be described. FIG. 26 is a diagram showing a specific example of the process of determining the wake-up task list from the pre-shared wake-up request list.

[0201] The table shown in FIG. 26 is composed of items of "Task", "Pre-shared Wake-up Request List", "AND of Wake-up Requests", and "Wake-up Task List".

[0202] The "Task" item stores a task identifier. In the "Pre-shared Wake-up Request List" item, the occurrence status information ("E" or "-") of wake-up requests in the pre-shared wake-up request list by each of the leader node (denoted as "Leader" in the figure), follower node B (denoted as "Follower B" in the figure), and follower node C (denoted as "Follower C" in the figure) is described. "E" in the occurrence status information of wake-up requests indicates the occurrence of a waiting event, and "-" indicates that the wake-up request has not occurred or the pre-sharing of the wake-up request has not been performed.

[0203] For example, from a follower node during backtracking execution, the wake-up request list for the next cycle may not be pre-shared. In this case, the cluster management unit 124 treats the wake-up request list of the follower node as empty ("-") based on the above Rule 6.

[0204] The leader node, follower node B, and follower node C shown in the "Pre-shared Wake-up Request List" item are any majority nodes including the leader node selected according to the above Rule 4. And in the example shown in FIG. 26, in task T1, waiting events have occurred in the leader node and follower node C, and in task T2, waiting events have occurred in follower node B and follower node C. Also, in task T3, waiting events have occurred in all nodes of the leader node, follower node B, and follower node C.

[0205] For each combination of a majority of nodes including the leader node selected in the above Rule 4 in the item of "AND of wake-up requirements", tasks with wake-up requirements due to "occurrence of waiting event" are indicated by "E" in all nodes included in the combination. In FIG. 25, as the combinations, a combination of the leader node and follower node B and a combination of the leader node and follower node C are shown. "AND of wake-up requirements" indicates the logical product (AND) of the "occurrence of waiting event" situations in each combination.

[0206] In task T1, since waiting events have occurred in the leader node and follower node C, the "AND of wake-up requirements" in the leader node and follower node C becomes "E". On the other hand, in the leader node and follower node B, since no waiting event has occurred in the follower node B, the "AND of wake-up requirements" in the leader node and follower node B becomes "-".

[0207] The item of "wake-up task list" shows the tasks selected for the wake-up task list based on the above Rule 5. As shown in FIG. 25, the number of "E"s (in the column direction) is the largest in the combination of the leader and follower C in the column of "AND of wake-up requirements". Therefore, tasks T1, T3, and T4 with wake-up requirements in this combination are described in the wake-up task list.

[0208] (Wake-up task list generation process) Next, the wake-up task list generation process according to Example 1 will be described. FIG. 27 is a flowchart showing an example of the procedure of the wake-up task list generation process. In the first embodiment described above, the cluster management unit 124 of the leader node generates a wake-up task list from its own wake-up request list when generating a cycle start request event in step S13 of the next cycle preparation process shown in FIG. 11. In contrast, in the present Example 1, the cluster management unit 124 performs the process described in FIG. 27, that is, generates a wake-up task list based on the information of the wake-up request list pre-shared from each node.

[0209] First, the cluster management unit 124 of the leader node initializes the wake-up candidate list (not shown) to empty (step S151). The wake-up candidate list is a list managed for each follower node in the leader node that executes this process, and corresponds to the "Leader & Follower B" column and "Leader & Follower C" column in FIG. 26. Next, the cluster management unit 124 of the leader node performs the processes of the subsequent steps S152 to S155 for each event-driven task that is a wake-up target in that cycle.

[0210] In step S152, the cluster management unit 124 determines whether a wake-up request has occurred in the leader node. If it is determined that no wake-up request has been issued in the leader node (NO in step S152), the cluster management unit 124 makes the determination in step S152 for the next event-driven task.

[0211] On the other hand, when it is determined that a wake-up request has been issued at the leader node (step S152 is YES), the cluster management unit 124 determines whether the wake-up factor of the target event-driven task is "waiting for an event with a timeout" and whether the timeout time has elapsed (step S153). If the determination in step S153 is YES, the cluster management unit 124 adds the task to the wake-up task list (step S154). That is, regardless of the status of the follower nodes, the cluster management unit 124 of the leader node adds its own wake-up request and wake-up factor to the wake-up task list of the task. After the process of step S154, the cluster management unit 124 makes the determination of step S152 for the next event-driven task.

[0212] On the other hand, if the determination in step S153 is NO, the cluster management unit 124 of the leader node performs the process of step S155. The process of step S155 is a process based on the fourth rule above. That is, when waking up a task for which a wake-up request has been issued at more than half of the nodes including the leader node in cycle N, it is a process of searching for the follower node with the largest number of wake-up tasks. Here, for simplicity, the number of follower nodes is assumed to be 2. That is, the combinations referred to in the fourth rule above are only the two combinations of "leader & follower B" and "leader & follower C", and the flow is shown as a loop for each of the follower nodes B and C. When there are no more follower nodes to be searched, the process of step S155 ends.

[0213] In step S155, the cluster management unit 124 determines whether the task is in the state of "waiting for an event to occur" in the wake-up request list 123 pre-shared from the follower nodes. If it is in the state of "waiting for an event to occur", the cluster management unit 124 adds the task to the wake-up candidate list.

[0214] Next, the cluster management unit 124 of the leader node adds the content described in the wake-up candidate list with the longest list length, that is, the largest number of registered tasks, to the wake-up task list (step S156). Next, the cluster management unit 124 of the leader node performs carry-over processing for the pre-shared wake-up request (step S157).

[0215] When the presence / absence state of the wake-up request for a certain task differs among nodes, carry-over to the next cycle of the wake-up request may occur. The process of step S157 is executed when carry-over occurs. The carry-over process will be described in detail with reference to FIG. 28 below.

[0216] (Carry-over process) FIG. 28 is a flowchart showing an example of the procedure for carry-over processing of the pre-shared wake-up request from the next cycle N to the successive cycle N+1. The cluster management unit 124 performs the processes of steps S161 to S165 for each follower node for each event-driven task.

[0217] In step S161, the cluster management unit 124 determines whether the task to be determined is already included in the wake-up task list. If it is determined that the task is already included in the wake-up task list (step S161 is YES), since carry-over is not required, the determination moves to the next task. On the other hand, if it is determined that the task is not included in the wake-up task list (step S161 is NO), the cluster management unit 124 determines whether a wake-up request for that task has been issued at one or more nodes (step S162).

[0218] If it is determined that no wake-up request has been issued for the task at any node (step S162 is NO), the process proceeds to the determination for the next task. On the other hand, if it is determined that a wake-up request has been issued for the task at one or more nodes (step S162 is YES), the cluster management unit 124 determines whether an abnormality detection timer (not shown) has been set (step S163). If it is determined that the abnormality detection timer has not been set (NO in step S163), the cluster management unit 124 sets the abnormality detection timer (step S164).

[0219] The abnormality detection timer is a timer set for a task for which carry-over occurs and is set on a task-by-task basis. The abnormality detection timer counts the elapsed time from the time when the wake-up request first occurred in the task of any node. When the abnormality detection timer times out, the cluster management unit 124 determines that the node is an abnormal node. This determination is made in Example 2 described later. Note that the length of time set for the abnormality detection timer may be different for each task content. For example, for a task that waits for an event whose occurrence timing is likely to vary greatly among nodes, adjustments such as increasing the time set for the abnormality detection timer may be made.

[0220] When step S163 is determined to be YES, or after the process of step S164, the cluster management unit 124 posts the wake-up request for the task to the wake-up request list for the pre-shared period N + 1 (step S165).

[0221] Note that the cluster management unit 124 manages the pre-shared wake-up request list for each period. Also, the wake-up request for the task targeted for the carry-over process shown in FIG. 27 is posted to the pre-shared wake-up request list at each follower node. Since the carry-over of the wake-up request at the leader node is executed at the timing when the transfer from the wake-up request list to the task management table 122 is performed, the related process is not executed here.

[0222] After the processing of step S165 for all event-driven tasks is completed, the carry-over process ends, and the wake-up task list generation process (see FIG. 27) also ends.

[0223] In the above-described Example 1, the cluster management unit 124 generates different combinations of the leader node and the follower nodes for more than half of the nodes, and extracts tasks for which wake-up requests have occurred in both the leader node and the follower nodes in the combination from the pre-shared wake-up request list 123. Then, the cluster management unit 124 generates a wake-up task list to be included in the cycle start request event using the information on the tasks and the wake-up requests applied to the tasks. As a result, it is guaranteed that wake-up requests are applied to all the tasks included in the wake-up task list in at least more than half of the nodes. Therefore, according to Example 1, when leader re-election occurs, it is possible to select a follower node that does not enter the catch-up execution state as the new leader. As described above, it is possible to prevent the processing of the entire server cluster 10 from stagnating.

[0224] [Example 2] Even when the cycle management method according to the above-described first embodiment is performed, for example, when a situation occurs in which wake-up requests for specific tasks are not permanently applied to only some nodes due to node-specific abnormalities that occur accidentally, etc., there is a risk that the processing of the entire system will stagnate indefinitely. For example, assume that a deadlock occurs due to a timing bug in the leader node, and an event occurs in which a wake-up request for a specific task is not permanently applied. In this case, although wake-up requests for the task are applied in the follower nodes, the task is not included in the "wake-up task list" determined by the leader node. Therefore, as long as the current leader continues to exist, the remaining follower nodes cannot execute the task either.

[0225] In Example 2, in order to prevent the occurrence of such a situation, in addition to the processing of Example 1, as a first countermeasure, the cluster management unit 124 also pre-shares the generation time of the wake-up request for each task. Then, when setting the anomaly detection timer in the above-mentioned step S164, the ignition time is set as the time obtained by adding a predetermined time to the time when the wake-up request was received earliest. Further, as a second countermeasure, the cluster management unit 124 performs a process of removing a follower node whose number of laps behind exceeds a predetermined number from the cluster, regarding it as an anomaly.

[0226] In the first countermeasure, when the anomaly detection timer described above times out in the cluster management unit 124 of the leader node, the minority nodes are excluded from the server cluster 10 (see FIG. 1) as anomalies. The minority nodes refer to the nodes belonging to the minority group when the nodes are classified into a group with a wake-up request and a group without a wake-up request in the task. When the number of nodes in both groups is the same, the group including the leader node is left, and the nodes in the other group are excluded from the server cluster 10. Note that nodes that are in the process of catch-up execution, that is, nodes for which the wake-up request list for the next cycle has not been pre-shared, are excluded from the targets of the first countermeasure.

[0227] In the second countermeasure, the cluster management unit 124 counts the number of laps behind for each follower node, and when the number of laps behind exceeds a predetermined threshold, the node is regarded as an abnormal node and excluded from the server cluster 10.

[0228] It is assumed that which of the above first countermeasure or second countermeasure is adopted can be switched depending on the case. For example, the first countermeasure is performed when only the leader node does not receive a wake-up request, when only the leader node receives a wake-up request, or when only some of the follower nodes receive a wake-up request. The second countermeasure is performed when only some of the follower nodes do not receive a wake-up request.

[0229] FIG. 29 is a diagram showing an example of an event to be processed according to Example 2. FIG. 29A shows an example where no wake-up request is issued to only the leader node, FIG. 29B shows an example where a wake-up request is issued only to the leader node, and FIG. 29C shows an example where no wake-up request is issued to only some followers.

[0230] The tables shown in FIGS. 29A to 29C are composed of items of "period", "task", "pre-shared wake-up request list", and "event delay".

[0231] In the item of "period", a number indicating the period is stored, and in the item of "task", information of the task identifier is stored. In the item of "pre-shared wake-up request list", the wake-up request list 123 shared among the nodes before the occurrence of the period start request event is stored. The item of "event delay" indicates the elapsed time from the occurrence time of the earliest wake-up request in the task (T1), that is, the time counted by the anomaly detection timer.

[0232] In the example shown in FIG. 29, it is assumed that the timeout time of the anomaly detection timer is "2 seconds" and one period is 1 second. The information of the delay time stored in "event delay" is managed in units of tasks. In FIG. 29, for simplicity of explanation, only the information regarding task T1 is shown. Also, in FIG. 29, for simplicity of explanation, only an example where the wake-up factor is "waiting for a timeout-free event" is shown. The numbers shown in parentheses in the item of "pre-shared wake-up request list" indicate the number of laps behind.

[0233] In the example shown in FIG. 29A, the event delay, that is, the count time of the anomaly detection timer, becomes "2.5" at cycle N = 4, exceeding the "2 seconds" of the timeout time. Since the only node that has not received a wake-up request at this point is the leader node, the leader node is excluded from the server cluster 10 by the above first countermeasure.

[0234] In the example shown in FIG. 29B, the event delay time is "2.5" at a period N = 3, exceeding the "2 seconds" of the timeout time. At this point, the node to which the wake-up request is applied is only the leader node, and in follower node B and follower node C, no wake-up request is applied. That is, since the minority node is the leader node, the leader node is excluded from the server cluster 10.

[0235] In the example shown in FIG. 29C, a cycle delay has occurred in follower node C, and at a period N = 12, the number of cycle delays is "10". If the threshold value set for the number of cycle delays is, for example, "10", follower node C is excluded from the server cluster 10 at this point.

[0236] According to the above-described Example 2, even when a situation occurs in which a wake-up request is not permanently applied to a specific task for only some nodes due to accidental node-specific abnormalities, the node is excluded from the server cluster 10. Therefore, it is possible to prevent the processing of the multi-system control system 100 from being indefinitely stalled.

[0237] Note that the above-described embodiment has described the configuration of the system in detail and specifically for easy understanding of the present invention, and is not necessarily limited to the one having all the configurations described.

[0238] Also, the control lines or information lines indicated by solid lines or arrows in FIGS. 1 to 3 show those considered necessary for explanation, and not necessarily all the control lines and information lines are shown on the product. In reality, it may be considered that almost all the components are interconnected.

[0239] Also, in this specification, the processing steps for describing time-series processing include not only the processing that is performed in time series in the described order, but also the processing that is not necessarily processed in time series, but is executed in parallel or individually (for example, parallel processing or object-based processing).

[0240] Moreover, each component of the multi-system control system according to each embodiment of the present disclosure described above may be implemented on any hardware as long as the respective hardware can transmit and receive information to and from each other via a network. Further, the processing performed by a certain processing unit may be realized by one piece of hardware or may be realized by distributed processing by a plurality of pieces of hardware.

Description of Reference Numerals

[0241] 1, 1A, 1B, 1C... Computers, 2... External device, 10... Server cluster, 11, 11A, 11B, 11C... Periodic control applications, 12, 12A, 12B, 12C... Multi-execution infrastructure units, 100... Multi-system control system, 111... Application state information storage unit, 121... Task execution unit, 122... Task management table, 123... Wake-up request list, 124... Cluster management unit, 125... Event log storage unit, 126... Snapshot storage unit

Claims

1. A multi-system control system including a plurality of nodes assigned to either one main system node or a plurality of slave system nodes, wherein the nodes include: a periodic control unit that periodically executes one or more tasks that perform the same state transition and generate the same output when given the same input; a multi-execution infrastructure unit that controls synchronization with other nodes, and the multi-execution infrastructure unit of the main system node makes the processing contents in each node consistent by forming an agreement among a plurality of nodes regarding the tasks to be woken up in the next cycle and the wake-up factors of the tasks; when the wake-up factor of the task in the main system node and the wake-up request factor, which is the wake-up factor in its own node, of the slave system node do not match, the multi-execution infrastructure unit of the slave system node makes the wake-up request factor match the wake-up factor in the main system node A multi-system control system.

2. The task includes an event-driven task that is executed in the next cycle when a specific event occurs, when the wake-up factor of the event-driven task in the slave system node is waiting for an event with a timeout, and a timeout occurs because a predetermined time has elapsed without the expected waiting event occurring, the multi-execution infrastructure unit of the slave system node deems that a timeout has also occurred in its own node, thereby making the wake-up request factor match the wake-up factor. The multi-system control system according to claim 1.

3. When a waiting event occurs in the main system node at the stage of waiting for the occurrence of the wake-up request for the event-driven task, the multi-execution infrastructure unit of the slave system node also waits for the occurrence of the waiting event in its own node, and when the waiting event occurs, executes the event-driven task retroactively, thereby making the wake-up request factor match the wake-up factor. The multi-system control system according to claim 2.

4. Each of the main system node and the slave system node further includes a task management table that manages information on the wake-up factors of the tasks in the main system node and the presence or absence of the occurrence of the wake-up requests for the tasks, and information on the wake-up request factors of the tasks in the slave system node and the presence or absence of the occurrence of the wake-up requests for the tasks. The multiplexed execution infrastructure unit of the slave node matches the wake-up request factor with the wake-up factor based on the information described in the task management table. The multiplexed control system according to claim 3.

5. Each of the master node and the slave node further includes a wake-up request list indicating correspondence information between a wake-up request generated in its own node and a task woken up based on the wake-up request. The multiplexed execution infrastructure units of the master node and the slave node write the wake-up request generated in their own nodes into the wake-up request list, and at the arrival of the start timing of a period, delete information regarding tasks determined to be woken up in that period from the wake-up request list and transfer it to the task management table. The multiplexed control system according to claim 4.

6. The multiplexed execution infrastructure unit of the master node generates a period start request event including the period number of the next period, the virtual time corresponding to the start time of the next period, and a wake-up task list in which the tasks to be woken up in the next period and the wake-up factors of the tasks are described. The multiplexed execution infrastructure unit updates the information in the wake-up request list based on the content described in the wake-up task list included in the period start request event. The multiplexed control system according to claim 5.

7. The event includes a message reception event that occurs when a message is received from an external device. The multiplexed execution infrastructure unit of the master node transfers a received message corresponding to a message reception event that occurred between the period start request event for the next period and the previous period start request event to the task that is the reception target of the received message. The multiplexed control system according to claim 6.

8. The multiplexed execution infrastructure units of the master node and the slave node broadcast the wake-up request list to other nodes prior to the occurrence of the period start request event in the master node. The multiplexed control system according to claim 6.

9. The multi-execution infrastructure unit of the master node generates different combinations of the master node and the slave nodes, with the number of combinations being more than half of the nodes. The tasks for which wake-up requests have occurred in both the master node and the slave nodes in the combination are extracted from the wake-up request list shared by broadcast, and the wake-up task list to be included in the periodic start request event is generated using the tasks and the information on the wake-up requests associated with the tasks. The multi-system control system according to claim 8.

10. Before the occurrence of the periodic start request event in the master node, the multi-execution infrastructure units of the master node and the slave nodes broadcast the wake-up request list and the information on the occurrence time of the wake-up requests to other nodes. In a certain task, if the elapsed time from the time when the wake-up request occurred earliest is equal to or more than a predetermined time and no wake-up request has been received, the node is excluded from the cluster consisting of the master node and the slave nodes as an abnormal node. The multi-system control system according to claim 9.

11. When the number of laps behind in a node where the execution of the task is behind during follow-up execution by the multi-execution infrastructure units of the master node and the slave nodes reaches a predetermined number or more, the node is excluded from the cluster as an abnormal node. The multi-system control system according to claim 10.

12. A periodic management method by a multi-system control system including a plurality of nodes assigned to either one master node and a plurality of slave nodes, a procedure in which a periodic control unit of the nodes periodically executes one or more tasks that perform the same state transition and generate the same output when given the same input; and a procedure in which a multi-execution infrastructure unit of the nodes performs control related to synchronization with other nodes, wherein the multi-execution infrastructure unit of the master node unifies the processing contents in each node by reaching an agreement among a plurality of nodes regarding the tasks to be woken up in the next period and the wake-up factors of the tasks; and wherein the multi-execution infrastructure unit of the slave node matches the wake-up request factor, which is the wake-up factor in the slave node, with the wake-up factor in the master node when they do not match. Periodic management method.

Images