Fault-tolerant system and data processing method

Abstract

To provide a fault tolerant system and a data processing method that can perform matching of behavior between nodes while suppressing a load due to synchronization, have low delay, and are scalable.SOLUTION: In a fault tolerant system 1 in which a multiplexed plurality of nodes 10 are connected through a communication network 2, an application operating in each node 10 is constituted from one or more tasks 20 that deterministically generate output for input, and one of the plurality of nodes 10 becomes a leader node, and nodes except for the leader node becomes follower nodes. The leader node and the follower nodes are constituted to execute the same task respectively. When the task 20 transmits a message to a transmission destination task having a plurality of message transmission sources, processors of the plurality of nodes 10 form an agreement between the plurality of nodes for a delivery order of a message for the transmission destination task, and transmits the message following the delivery order having the agreement formed.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a multi-processing system that applies the technology of a fault-tolerant system. [Background technology]

[0002] 2. Description of the Related Art A fault-tolerant system is known as a computer system in which the same processing is carried out between multiple computers (nodes) and operations can continue without interruption even if a fault occurs on one of the computers.

[0003] State machine replication is known as one approach to realizing a fault-tolerant system. In state machine replication, the same input is given to each replicated node that has the same state, and processing is performed deterministically using that input and state. As a result, each node produces the same output and is updated to the same state. This means that even if a failure occurs in some nodes, the remaining nodes can continue operations without causing inconsistencies.

[0004] Here, in the case of an application configured to execute a plurality of tasks in parallel, the overall behavior can be made consistent between nodes by making the order of access to the shared state consistent.

[0005] For example, in the data processing method disclosed in Patent Document 1, the leader node transmits the access order to the follower nodes each time it accesses shared data, and the follower nodes wait for an instruction on the access order from the leader node each time they access the shared data and process the data in accordance with that order. In this way, the data processing method of Patent Document 1 makes the behavior of replicated nodes consistent. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2012 / 127652 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the above-mentioned Patent Document 1, determinism is guaranteed by matching the order of access to data shared between tasks between nodes each time an access is made. In other words, in the data processing method of Patent Document 1, communication occurs between the leader node and follower node each time an access request to data shared between tasks occurs, which may increase the load for synchronization depending on the workload.

[0008] The present invention has been made in consideration of the above points, and aims to provide a low-latency, scalable fault-tolerant system and data processing method that can match the behavior between nodes while suppressing the load caused by synchronization. [Means for solving the problem]

[0009] In order to solve such problems, the present invention provides a fault-tolerant system in which multiplexed nodes are connected via a communication network, each of the multiple nodes having a processor, a memory, and a storage area, and an application that runs in each of the nodes by the processor reading a program from the storage area into the memory and executing it is composed of one or more tasks that deterministically generate an output for an input, one of the multiple nodes being a leader node, and the nodes other than the leader node being follower nodes, and the leader node and the follower node are configured to each execute the same task, and each of the tasks has a status indicating the state of the own task. A fault-tolerant system is provided, characterized in that a task is linked one-to-one to a local state, the input of the task includes a message received from outside the system or from another task, and access to the local state linked to the task, the output of the task includes a message sent by the task to outside the system or to another task, and an update of the local state linked to the task, and when the task sends a message to a destination task having multiple message senders, processors of the multiple nodes reach an agreement among the multiple nodes on the order of delivery of messages to the destination task, and send the messages according to the agreed-upon delivery order.

[0010] In order to solve the above problem, the present invention provides a data processing method using a fault-tolerant system in which multiplexed nodes are connected via a communication network, wherein each of the multiple nodes has a processor, a memory, and a storage area, and an application that runs in each of the nodes by the processor reading a program from the storage area into the memory and executing it is made up of one or more tasks that deterministically generate an output for an input, one of the multiple nodes being a leader node and the nodes other than the leader node being follower nodes, and the leader node and the follower nodes are configured to each execute the same task, and each of the tasks is A data processing method is provided in which a task is linked one-to-one to a local state that indicates the state of the task itself, the input of the task includes a message received from outside the system or from another task, and access to the local state linked to the task, the output of the task includes a message that the task sends to outside the system or to another task, and an update of the local state linked to the task, and when the task sends a message to a destination task having multiple message senders, processors of the multiple nodes reach a consensus among the multiple nodes on the order of delivery of messages to the destination task, and send the messages in accordance with the agreed-upon delivery order. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide a low-latency, scalable fault-tolerant system and data processing method that can match the behavior between nodes while suppressing the load caused by synchronization. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of the configuration of a fault-tolerant system (FT system) 1 according to an embodiment of the present invention. [Figure 2] FIG. 1 is an image diagram showing the overall characteristics of processing when an event occurs in the FT system 1. [Figure 3] FIG. 10 is an image diagram illustrating the correlation between nodes. [Figure 4] FIG. 10 is a diagram illustrating the characteristics of a message. [Figure 5] FIG. 10 is an image diagram showing an example of the operation of task 20. [Figure 6] FIG. 10 is a diagram showing an example of a message path management table 35. [Figure 7] FIG. 10 is a diagram for explaining characteristics of consensus building on an event log between nodes. [Figure 8] 10 is a flowchart illustrating an example of a processing procedure of a leader in consensus building of an event log. [Figure 9] 10 is a flowchart illustrating an example of a processing procedure of a follower in consensus building of an event log. [Figure 10] 10 is a flowchart showing an example of a processing procedure for reception processing by an input processing unit 31 of the reader. [Figure 11] 10 is a diagram showing an example of the data structure of an external input type log entry held in the event log 34. FIG. [Figure 12] 10 is a flowchart illustrating an example of a processing procedure for a message request acceptance process. [Figure 13] FIG. 10 illustrates an example of a message delivery type log entry. [Figure 14] FIG. 10 is a model diagram for explaining the relationship between nodes when a message path is added. [Figure 15] 10 is a flowchart illustrating an example of a processing procedure for a message path addition process. [Figure 16] FIG. 10 illustrates an example of a message path addition type log entry. [Figure 17] FIG. 10 is a diagram illustrating another example of a log entry of the message path addition type. [Figure 18] FIG. 10 is a diagram illustrating an example of message path information. [Figure 19] 10 is a flowchart illustrating an example of a processing procedure for message path information reception processing. [Figure 20]FIG. 10 is a diagram showing a specific example of a message path management table 35. [Figure 21] FIG. 10 illustrates an example of a log entry of a message path information type. [Figure 22] FIG. 10 is a sequence diagram showing an example of a processing procedure for adding a message path according to the second improvement method. [Figure 23] 10 is a flowchart showing an example of a processing procedure for message path addition processing according to a second improvement method. [Figure 24] 10 is a flowchart illustrating an example of a processing procedure for message pass information reception processing according to a second improvement method. [Figure 25] FIG. 10 is a conceptual diagram illustrating order assurance of external outputs. [Figure 26] FIG. 10 is a conceptual diagram illustrating a configuration in which external outputs from a plurality of nodes are adjusted and output. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0014] Note that the following description and drawings are examples for explaining the present invention, and have been omitted or simplified as appropriate for clarity of explanation. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. The present invention is not limited to the embodiments, and all application examples consistent with the concept of the present invention are included in the technical scope of the present invention. Those skilled in the art can make various additions and modifications to the present invention within the scope of the present invention. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be plural or singular.

[0015] In the following explanation, various types of information may be described using expressions such as "table," "list," "queue," etc., but the various types of information may also be expressed using data structures other than these. To indicate that it is not dependent on the data structure, "XX table," "XX list," etc. may be referred to as "XX information." When describing the content of each piece of information, expressions such as "identification information," "identifier," "name," "ID," "number," etc. are used, but these are interchangeable.

[0016] In addition, in the following explanation, when describing elements of the same type without distinguishing between them, reference signs or common numbers in reference signs will be used, and when describing elements of the same type with distinction between them, the reference signs of those elements will be used or an ID assigned to those elements will be used instead of the reference signs.

[0017] Furthermore, although the following description may describe processing performed by executing a program, the program is executed by at least one processor (e.g., a CPU) to perform a predetermined process using storage resources (e.g., memory) and / or interface devices (e.g., communication ports) as appropriate, and therefore the processor may be the subject of the processing. Similarly, the subject of the processing performed by executing a program may be a controller, device, system, computer, node, storage system, storage device, server, management computer, client, or host having a processor. The subject of the processing performed by executing a program (e.g., a processor) may include a hardware circuit that performs part or all of the processing. For example, the subject of the processing performed by executing a program may include a hardware circuit that performs encryption and decryption or compression and decompression. The processor operates as a functional unit that realizes a predetermined function by operating in accordance with the program. Apparatuses and systems including a processor are apparatuses and systems that include these functional units.

[0018] A program may be installed on a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server includes a processor (e.g., a CPU) and storage resources, and the storage resources may further store a distribution program and a program to be distributed. The processor of the program distribution server may then execute the distribution program, causing the processor of the program distribution server to distribute the program to be distributed to other computers. In the following description, two or more programs may be realized as one program, and one program may be realized as two or more programs.

[0019] (1) Composition 1 is a block diagram showing an example of the configuration of a fault-tolerant system (FT system) 1 according to one embodiment of the present invention. As shown in FIG. 1, the FT system 1 is a computer system made up of multiple computers (nodes 10) connected via a communication network 2.

[0020] The FT system 1 according to this embodiment is a so-called leader-follower distributed system, in which one of the multiple nodes 10 serves as a leader node (also called the leader), and the nodes other than the leader node serve as follower nodes (also called followers). Specifically, for example, consider the leader and follower in the distributed consensus algorithm RAFT. In the FT system 1, the same input is given to an event-driven application replicated on multiple nodes 10, and the same output is obtained. An event-driven application is made up of tasks 20, which are multiple processing modules that can operate independently. In other words, the FT system 1 is configured so that when an event occurs, the same task 20 is executed on both the leader node and the follower node.

[0021] While the FT system 1 is operating normally, the leader node 10 and follower node 10 do not switch positions, but if, for example, there is a possibility that a failure has occurred in the leader node 10, that node 10 is disconnected from the FT system 1, and one of the follower nodes 10 becomes the new leader, and the FT system 1 continues processing. With this kind of configuration, a scalable FT system 1 is realized.

[0022] The FT system 1 is also connected to one or more terminals 3 via a communications network 2, and transmits and receives data to and from the terminals 3. The communications network 2 is, for example, a LAN (Local Area Network), but is not limited to this. The terminals 3 are computers used by users, and, for example, act as clients to send events to the FT system 1 and receive the results of event processing from the FT system 1. Note that, as shown in Figure 2, which will be described later, the leader node in the FT system 1 is the first to receive an event.

[0023] The node 10 is a computer equipped with a processor such as a CPU (Central Processing Unit), a memory, a storage device (storage area), a communication interface, etc. The storage device stores various programs and data, such as an event processing platform 30 and a task 20, to be executed by the node 10. The processor then reads these various programs from the storage device, expands them in the memory, and executes them.

[0024] The task 20 is an event-driven processing module (program) that can operate independently and that configures an application with a group of multiple tasks on one or more nodes 10. Specifically, the task 20 is, for example, a microservice, a function on FaaS (Function as a Service), or a process on an OS (Operating System).

[0025] In this embodiment, the state of an application is managed by a state (local state) 21 independently for each task 20. The state 21 is accessed exclusively. Each task 20 is associated one-to-one with the state 21 indicating the state of the task itself, and operates in a loosely coupled manner in cooperation with other tasks 20. In other words, each task 20 does not have directly shared application data, and when accessing application data (state 21) managed by another task 20, this is performed exclusively by a unidirectional message (hereinafter simply referred to as a message) requesting processing. Furthermore, each task 20 is configured to behave deterministically. In other words, once the input and current state are determined, the combination of the output and the state after processing is uniquely determined.

[0026] The event processing infrastructure 30 is an operation system that processes events. The event processing infrastructure 30 is capable of multitasking and can execute multiple tasks 20 in parallel. The event processing infrastructure 30 has an input processing unit 31, a message processing unit 32, an output processing unit 33, an event log 34, a message path management table 35, and a message waiting queue 36.

[0027] The input processing unit 31 receives generated events from the outside (terminal 3) or the leader node, and calls an appropriate task to process the events in the order in which they occur.

[0028] The message processing unit 32 receives a message request from a task 20 and sends the message to the target task 20. At this time, the message processing unit 32 manages a message path management table 35 required to guarantee determinism of the entire system and performs inter-node synchronization processing.

[0029] The output processing unit 33 arbitrates the processing results of each node 10 as necessary and finally transmits them to the outside. For arbitration of the processing results, a method of transmitting the result selected by majority vote, a method of transmitting on a first-come, first-served basis, or the like can be adopted.

[0030] The event log 34 is a data structure that manages information for which consensus is to be reached (or for which consensus has been reached) between the nodes 10. Specifically, for example, consider a log in the distributed consensus algorithm RAFT. The event log 34 is composed of a collection of log entries. A log entry is a unit of information for which consensus is to be reached, and is first added to the log by the leader, who then replicates it to the followers. When the leader receives reports from the majority of followers that replication of the log entries is complete, the log entries are committed, starting with the oldest log entry. The leader then notifies the followers how many log entries have been committed. Each node 10 applies the committed log entries, starting with the oldest. How they are applied depends on the type of log entry.

[0031] The message path management table 35 is data for managing message paths. A message path is a route connecting a message sender and a message destination. Details will be described later, but the message path management table 35 is registered based on message path information received from each node, including the node itself.

[0032] A message waiting queue 36 is provided for each message path. When a message request occurs, the message request is queued in order in the message waiting queue 36. If the message waiting queue 36 is not in a blocked state, the message request is dequeued from the top of the message waiting queue 36 at a predetermined timing and sent to the message destination.

[0033] (2) Overview of event processing Figure 2 is an image diagram showing the overall processing characteristics when an event occurs in the FT system 1. The external input 38 shown in Figure 2 transfers the event to the external input or follower nodes 10B, 10C, and forms a consensus between nodes regarding the externally input event. The external input 38 specifically corresponds to the input processing unit 31 shown in Figure 1. The external output 39 outputs the processing results in node 10 to the leader node 10A or to the outside. The external output 39 specifically corresponds to the output processing unit 33 shown in Figure 1.

[0034] 2, in the FT system 1 according to this embodiment, the external input 38 of the leader node 10A first receives an event, then forwards the received event to the external inputs 38 of the follower nodes 10B and 10C, and consensus is reached between the nodes. Then, each of the nodes 10A, 10B, and 10C calls the appropriate task 20 to process the received event in the order determined by the leader node 10A.

[0035] In Figure 2, each task 20 processes received messages sequentially and deterministically. For example, if task A receives messages Ma1 and Ma2 in that order, it first processes message Ma1 and sends the resulting message Mb1 to task B, then processes message Ma2 and sends the resulting message Mb2 to task B. In this case, because task B has only a single message sender, messages Mb1 and Mb2 are processed sequentially and deterministically in that order at every node. In this way, in the FT system 1, determinism is maintained for single-path message delivery without synchronization.

[0036] In addition, as shown by the dashed line from task D to task B in Figure 2, when a message path is added from one task to another for the first time, a consensus is reached between the nodes about the presence of the message. As a result, if multiple message senders appear on the path, a consensus is reached on the message processing order thereafter.

[0037] In addition, when messages are delivered from multiple paths, such as when task C operates upon receiving a message from task B or D in Figure 2, the order of message processing is agreed upon between nodes. This ensures that the behavior of a task with multiple paths (for example, task C) is consistent between nodes.

[0038] After the processing of the task 20 has been completed in each of the nodes 10A, 10B, and 10C, the external output 39 performs arbitration such as majority voting as necessary, and outputs the processing results to the outside (terminal 3).

[0039] (3) Guaranteeing determinism of task behavior FIG. 3 is an image diagram explaining the correlation between nodes. In FIG. 3, a leader node (leader) 10A is shown in the left half of the drawing, and one follower node (follower) 10B is shown in the right half of the drawing, showing the correlation between them. As will be described in detail later, various methods can be used for external output, so the output processing unit 33 is shown without distinguishing between the leader 10A and the follower 10B. As shown in FIG. 3, three types of consensus are formed between nodes: external input, message path addition, and message delivery order (task invocation order).

[0040] Fig. 4 is a diagram for explaining the characteristics of message delivery. As shown in Fig. 4, information exchange and linkage operations are performed between tasks and between external input / output and tasks using messages 101. Note that messages handled in this embodiment are unidirectional, and for example, a response to a message from task A to task B is treated as a message from task B to task A.

[0041] Messages flowing on one message path do not change order on that path. Each message 101 is assigned a sequence number indicating the order of transmission for each message path. In this explanation, the sequence numbers of messages within the same message path (message sequence numbers) are represented as #1, #2, ... in the order of transmission. Message sequence numbers are managed in the message path management table 35.

[0042] For example, as shown in Figure 4, on a message path going from task B to task C, if task B sends messages #1 and #2 in that order, task C will receive the messages in the order #1 and #2. On the other hand, for messages on a message path that have the same sender but different destinations, such as message "B → External #2" going from task B to external output and message "B → C #1" going from task B to task C, the order in which the messages are sent does not matter which message arrives at the destination first.

[0043] As mentioned above, the task 20 is an event-driven task that operates upon receiving a message 101. Each task 20 behaves deterministically and outputs a unique processing result for each input. Here, the input to the task 20 is the contents of the message 101 received from the external input 38 or another task 20, or the contents of its own local state 21. The output from the task 20 is the message 101 sent to another task 20 or the external output 39, or updates to its own local state 21. Each task 20 processes one message 101 at a time. When multiple messages 101 are received, the messages 101 are processed sequentially one by one. The local state 21 is independent for each task 20 and is not directly accessed by other tasks 20. When a task 20 wants to access the local state 21 of another task 20, it does so by issuing a message 101 to the other task 20 requesting access.

[0044] Fig. 5 is an image diagram showing an example of the operation of task 20. Fig. 5(A) shows an example of a configuration when there are multiple message senders, and Fig. 5(B) shows an example of a configuration when there is a single message sender. With reference to Fig. 5, the principle by which the determinism of applications (specifically, task operations) by each node 10 is guaranteed in this embodiment will be described.

[0045] In FIG. 5A, task C performs processing based on the contents of messages 101A and 101B from tasks A and B and the contents of local state 21C, updates local state 21C, and outputs the processing results to external output 39. At this time, if consensus is not reached between nodes 10 on the message delivery order (processing order), differences may occur in the output contents of task C and the final local state 21C between when message 101A from task A is processed first and when message 101B from task B is processed first. Specifically, for example, when message 101A from task A is processed first, task C updates local state 21C of task C with message 101A, and the contents of the updated local state 21C are used as input in processing message 101B from task B.

[0046] In order to solve the above problems and guarantee determinism of the processing results in the entire system (in other words, to obtain the same processing results among the nodes 10), in this embodiment, in addition to each task behaving deterministically, for a task 20 having multiple message senders, an agreement is formed among the nodes 10 as to the order in which messages are to be dispatched. Details of the agreement formation will be described later.

[0047] On the other hand, when there is only a single message source (task A) like task C in Figure 5(B), as mentioned above, the processing order in which message 101A is dispatched in task C is uniquely determined. Specifically, message 101A is dispatched in task C in the processing order of #3 and #4. Therefore, in the case of a single path with only a single message source, consensus building as in the case of Figure 5(A) is not necessary.

[0048] Fig. 6 is a diagram showing an example of the message path management table 35. The message path management table 110 shown in Fig. 6 is an example of the message path management table 35 shown in Fig. 1, and manages, for each message path, the sequence numbers of messages that have been dispatched in that message path. The message path management table 110 is configured to have the following items: message path 111, status 112, and dispatched message sequence number 113.

[0049] The message path 111 indicates the source and destination of the message path managed by the record.

[0050] Status 112 indicates whether each message path is in a "blocked state." Specifically, for example, if the message path of the record is in a "blocked state," status 112 is set to "blocked." Details will be described later in the explanation of the processing by the message processing unit, but when a second message path is added when a first message path exists for a message destination task, the first message path is set to a "blocked state," and dispatching of messages on the first message path is suspended until consensus is reached on the second message path. The blocked state of this first message path is released when consensus is reached on the "Message Path Added" log entry regarding the addition of the second message path and this log entry is applied.

[0051] The dispatched message sequence number 113 indicates, by message sequence number, up to which message has been dispatched in the message path of the record. The message sequence number is counted for each message path, and is incremented by one each time a message occurs on that message path, and that value is assigned to the message. As a result, the message path and message sequence number can be used to uniquely identify each message.

[0052] If there is no pair of a message destination and a message source in the message path management table 110, the message path is treated as a new message path, and a new record is added and registered in the message path management table 110. The new message path is added to the message path management table 110 after consensus is reached on the addition of the new message path. In other words, even if a message to the new message path is generated again during consensus, it is treated as a new message path.

[0053] Furthermore, while it is expected that during actual operation of the FT system 1, each message path will be newly registered in the message path management table 110 each time a message is generated, as a variant, a certain number of message paths may be registered in advance. That is, the contents of the message path management table constructed during system testing may be output to a file, and during actual operation, the contents of the output file may be loaded into the message path management table 110, so that actual operation can begin with a certain number of message paths constructed. In this way, the load on the message path registration process during actual operation can be reduced.

[0054] (4) Consensus building process The following describes how consensus is formed between the nodes 10 in the FT system 1 according to this embodiment.

[0055] In the FT system 1, when various consensuses are formed between the nodes 10, the event log 34 (or simply the log) manages the process and results of the consensus. RAFT is a known consensus protocol, and the FT system 1 according to this embodiment basically uses RAFT to form consensus between the nodes 10. However, for the sake of simplicity, RAFT is expressed in a simplified manner in the description of this embodiment, and, for example, the RAFT leader re-election protocol is not described. Applying a log entry in RAFT means reflecting the contents of the log entry in a state machine. The processing content of applying a log entry in this embodiment differs depending on the type of log entry.

[0056] The types of log entries handled in this embodiment can be roughly divided into three types: "external input," "message delivery," and "message path addition."

[0057] The purpose of the "External Input" log entry is to ensure that external events received by the leader are replicated to followers and dispatched in the same order at each node. The specific operation when applying the "External Input" log entry is that the message processing unit 32 dispatches the contents of the external input event to the appropriate task for processing it, depending on the contents of the event. An example of the data structure of the external input log entry is shown in FIG. 11.

[0058] The purpose of the "Message Delivery" log entry is to ensure that nodes agree on the order in which messages are dispatched when there are multiple message senders. By applying this log entry, each message is uniquely identified by the combination of the message path and the message sequence number. The specific operation when the "Message Delivery" log entry is applied is for the message processing unit 32 to dispatch the message. A specific example of a message delivery log entry is shown in FIG. 13.

[0059] The purpose of the "Message Path Added" log entry is to ensure that nodes agree on the existence of message paths. In particular, when a second message path is added when a first message path already exists, the nodes also agree on the message sequence numbers up to which messages should be dispatched on the first message path before the second message path is added. Specifically, when the "Message Path Added" log entry is applied, the message processor 32 adds the message path to the message path management table 35 (the message path management table 110 in FIG. 5). Furthermore, when a first message path is added, the message processor 32 dispatches messages in the message waiting queue 36 of the message path in order. Furthermore, when a second message path is added, the message processor 32 dispatches messages in the message waiting queue 36 of the first message path in order up to the agreed-upon message sequence number, and then releases the "blocked state" of the first message path before adding the second message path to the message path management table 35 (the message path management table 110 in FIG. 5). Specific examples of the message path added log entry are shown in FIGS. 17 and 18.

[0060] 7 is a diagram for explaining the characteristics of consensus building on an event log between nodes, showing an example of the progress of log entries in different nodes 10 (leader, follower 1, follower 2).

[0061] Common to the above three types of log entries, consensus on the event log 34 between nodes in this embodiment is achieved as follows in the case of FIG.

[0062] In Figure 7, follower 1 has completed adding log entries up to #5, and follower 2 has completed adding log entries up to #6. At this time, if the leader receives notifications from followers 1 and 2 that each entry has been completed, the leader will commit the log entries up to #6 since the majority of nodes, including the leader's own node, have completed adding entries up to #6, and will notify followers 1 and 2 of how many entries have been committed. Then, each node will apply any committed but unapplied log entries in order.

[0063] The detailed procedure for forming a consensus on the event log 34 will be described with reference to FIGS.

[0064] Fig. 8 is a flowchart showing an example of a processing procedure of a leader in consensus building on an event log. Fig. 9 is a flowchart showing an example of a processing procedure of a follower in consensus building on an event log. Note that the processing shown in Figs. 8 and 9 is executed by the input processing unit 31 if the log entry is of the "external input" type, and by the message processing unit 32 if the log entry is of the "message delivery" or "message path addition" type, but for simplicity in the following explanation, it will be described as being executed by the message processing unit 32.

[0065] 8, first, the message processing unit 32 of the leader adds a log entry to the log (event log 34) (step S11), and then the message processing unit 32 of the leader requests the follower to add a log entry (step S12).

[0066] 9, when the follower's message processing unit 32 receives the request in step S12 in FIG. 8 (step S21), it adds the log entry to the log (event log 34) in the order of transmission on the message path (step S22), and then notifies the leader that the log entry has been successfully added (step S23).

[0067] Next, in Fig. 8, the message processing unit 32 of the leader determines whether or not it has received notifications of successful addition in step S23 from a majority of the followers of the multiple followers that make up the FT system 1 (step S13). In step S13, it determines whether or not the majority of nodes, including itself, have successfully added the message. If notifications of successful addition have been received from a majority of the followers (YES in step S13), the process proceeds to step S14. On the other hand, if notifications of successful addition have not been received from a majority of the followers (NO in step S13), the leader ends the processing in Fig. 8 and waits until it receives notifications of successful addition from a majority of the followers.

[0068] In step S14, the leader's message processing unit 32 commits the log entries for which a majority of notifications were received in step S13, in order from oldest to newest. Next, the leader's message processing unit 32 notifies the followers of up to which log entries (up to which log entry serial numbers) they have committed (step S15). Finally, the leader's message processing unit 32 applies the committed but unapplied log entries in order from oldest to newest (step S16), and ends the consensus building process.

[0069] Meanwhile, in the follower, after notifying the leader of the success of adding the log entry in step S23, the message processing unit 32 determines whether or not the commit notification of step S15 has been received from the leader (step S24). If the commit notification has been received (YES in step S24), the follower applies the committed but unapplied log entries in order from the oldest (step S25), and ends the consensus building process. On the other hand, if the commit notification has not been received (NO in step S24), the follower ends the process without performing any special processing.

[0070] 8 and 9 are flowcharts in which processing is performed sequentially for ease of explanation, but this embodiment is not limited to this, and the consensus building processing may be performed asynchronously, for example, by inserting processing to wait for various messages from other nodes. Also, in the following explanation, when "adding a log entry to an event log" is indicated, it is assumed that the consensus building processing shown in Figures 8 and 9 is operating in the background (individual explanations will be omitted).

[0071] (5) Input processing unit 31 The following describes in detail the processing performed by the input processing unit 31. More specifically, the reception processing performed by the input processing unit 31 in response to an externally input event will be described.

[0072] 10 is a flowchart showing an example of a procedure for reception processing by the reader's input processing unit 31. The processing shown in FIG. 10 is executed by the reader's input processing unit 31, for example, periodically.

[0073] 10, the input processing unit 31 of the reader determines whether or not an external input event has been received (step S31). If an external input event has not been received (NO in step S31), the input processing unit 31 repeats the process of step S31 in the next cycle. If an external input event has been received (YES in step S31), the input processing unit 31 adds a log entry of the received external input event to the log (event log 34) (step S32).

[0074] 11 is a diagram showing an example of the data configuration of an external input type log entry stored in the event log 34. As shown in Fig. 11, an external input type log entry 120 is configured with a type 121 and an external input 122. The type 121 indicates the type of log entry, and the external input 122 indicates the content of the external input. In other words, in an external input type log entry, an event received from outside is stored in the event log 34 as is.

[0075] In step S32, the input processing unit 31 adds a log entry of the external input event to the event log 34, and as a result, the consensus building process is performed as described with reference to Figures 8 and 9, and the content of the external input event is transferred from the leader to the follower. Therefore, the follower does not have a process for directly receiving the external input event as shown in Figure 10.

[0076] By performing the above-described receiving process, the input processing unit 31 ensures that the same events are dispatched in the same order to any node 10.

[0077] Thereafter, the input processing unit 31 processes the external input event by sending a message to an appropriate task 20 that processes the event, depending on the content of the event. Specifically, the input processing unit 31 sends a message request to the message processing unit 22, requesting an operation of the task 20, and the message processing unit 22, upon receiving this message request, sends a message to the task 20 by performing a message request reception process shown in FIG. 12, which will be described later.

[0078] As will be described later, for messages between tasks, consensus is reached to ensure that the dispatch order of the messages is consistent between nodes (consensus on message delivery order). However, messages regarding external input events are not subject to consensus on message delivery order because the dispatch order between nodes is consistent at the time the messages are sent.

[0079] (6) Message processing unit 32 The various processes performed by the message processing unit 32 will be described in detail below.

[0080] (6-1) Message request acceptance process 12 is a flowchart showing an example of a processing procedure for message request acceptance processing, which is executed when the message processing unit 22 accepts a message request from the task 20.

[0081] 12, first, the message processing unit 22 determines whether the received message request is a request for message delivery on a new message path (step S41). Specifically, the message processing unit 22 checks whether the combination of the message destination and message source indicated by the message request is registered in the message path management table 110, and can determine that the message path is a new message path if the pair of the message destination and message source does not exist in the message path management table 110.

[0082] If it is determined in step S41 that the message path is a new one (YES in step S41), the message processing unit 22 performs a message path addition process to add a new message path (step S42). Details will be described later with reference to FIG. 15 etc., but in the message path addition process of step S42, a log entry of the "message path added" type is added to the log (event log 34). At this time, as described above, the consensus building process shown in FIGS. 8 and 9 operates in conjunction with the addition of the log entry to the log. Then, after the process of step S42, the message processing unit 22 adds the accepted message request to the message waiting queue 36 (step S43) and ends the process.

[0083] If it is determined in step S41 that the message path is not a new one (NO in step S41), the message processing unit 22 determines whether the message from the accepted message request is a message for a task 20 having multiple message senders (step S44).

[0084] If it is determined in step S44 that the message is for a task 20 having multiple message senders (YES in step S44), the message processing unit 22 adds a log entry of type "message delivery" to the log (event log 34) (step S45). After that, since the message cannot be executed until an agreement is reached between the nodes, the process proceeds to the above-mentioned step S43, where the accepted message request is added to the message waiting queue 36 and the process ends.

[0085] If the message is for a task 20 that does not have multiple message senders (i.e., is connected by a single path) in step S44 (NO in step S44), the message processing unit 22 checks whether the message path is in a blocked state (step S46). If the message path is in a blocked state (YES in step S46), the message cannot be executed until consensus is reached between the nodes, so the process proceeds to the above-mentioned step S43, adds the accepted message request to the message waiting queue 36, and terminates the process. On the other hand, if the message path is not in a blocked state (NO in step S46), consensus is not required, so the message processing unit 22 executes the message in accordance with the accepted message request (step S47) and terminates the process.

[0086] As described above, according to the message request acceptance process shown in Fig. 12, in the case of a message through a new message path or a message to a task with multiple message senders, the message is not executed until consensus is reached among the nodes, and the message request is temporarily held in the message waiting queue 36. Furthermore, for requests to an existing single message path, the message can be executed without consensus unless the message path is blocked. Furthermore, while a second message path is being added to the destination task (while consensus is being reached among the nodes), messages through the first message path are blocked.

[0087] Fig. 13 is a diagram showing a specific example of a message delivery type log entry. As shown in Fig. 13, a message delivery type log entry 130 is composed of a type 131 and a message 132. Type 131 indicates the type of information contained in the log entry, including a message path and a message sequence number. Message 132 indicates the content of each type of information indicated in type 131.

[0088] For messages to a message destination with multiple message senders, a "message delivery" type log entry such as log entry 130 shown in Fig. 13 is added to the log (event log 34), and the nodes reach a consensus on the order in which to dispatch the messages. Specifically, when log entry 130 shown in Fig. 13 is applied, the message with message sequence number "3" on the message path "A → C" from task A to task C is dispatched to task C.

[0089] (6-2) Message path addition process The following describes in detail the process when a new message path is added.

[0090] As mentioned above, in the FT system 1, for messages to a message destination that has multiple message senders, the message dispatch order must be consistent between the nodes 10. However, it is sometimes not easy to know in advance whether or not multiple message senders exist. Therefore, in this embodiment, a new message path can be added when a task 20 based on the occurrence of an event is executed. Then, to prevent inconsistencies from occurring at this time, a consensus is reached so that each node 10 adds a new message path in the same state (with the same sequence numbers for the messages to be dispatched).

[0091] Figure 14 is a model diagram for explaining the relationship between nodes when adding a message path. With reference to Figure 14, the necessity of consensus building when adding a new message path will be explained in more detail.

[0092] Figure 14 illustrates a case where a new message path "D → B" is added when an existing message path "A → B" exists. Note that the colored messages 101 in Figure 14 represent messages that have already been dispatched. Specifically, in the case of Figure 14, node 1 (leader) has already dispatched messages up to #5, and node 2 (follower 1) has already dispatched messages up to #6.

[0093] In Figure 14, until a new message path "D → B" is added, there is only one message path "A → B" with Task B as the message destination, so messages on "A → B" are not synchronized between nodes. Therefore, when you try to add the message path "D → B," there may be a discrepancy between nodes as to how many messages have already been dispatched.

[0094] To explain this in more detail using the case of Figure 14, suppose each node adds a new message path "D → B" and dispatches messages at its own timing without reaching a consensus among the nodes. After message #5 on "A → B" is dispatched at node 1, a new message path "D → B" is added, message #1 on "D → B" is dispatched, and then messages #6 and #7 on "A → B" are dispatched in that order. In this case, the event log 34 of node 1 (the leader) records log entries in the following order: "A → B #5," "D → B Added," "D → B #1," "A → B #6," and "A → B #7." The addition of the new message path and the messages immediately before it (the first and second messages above) may be recorded as a single inseparable log entry.

[0095] Meanwhile, at node 2, since message #6 on "A → B" has already been dispatched before the new message path "D → B" is added, after message #6 on "A → B" is dispatched, message #1 on "D → B" is dispatched, and then message #7 on "A → B" is dispatched. At this time, the event log 34 of node 2 (follower) records log entries in the following order: "A → B #6," "D → B added," "D → B #1," and "A → B #7." As in the case of node 1, the addition of the new message path and the messages immediately before it (the first and second messages above) may be recorded as a single inseparable log entry.

[0096] Comparing the processing results of the nodes 1 and 2 described above reveals that different log entries are recorded in the event log 34, and that synchronization between the nodes 1 and 2 cannot be maintained.

[0097] Therefore, in this embodiment, after adding the message path "D → B", it is necessary to make the nodes agree on the order in which the messages on "A → B" and the messages on "D → B" should be dispatched. In other words, it is necessary to reach a consensus among the nodes that "after messages on the message path A → B have been dispatched up to number X, add the message path D → B."

[0098] Specifically, as a protocol unique to this embodiment that is not specified in RAFT, when a new message path is added, each node broadcasts message path information 102 related to the message path addition to other nodes, as shown on the right side of FIG. 14. For example, the message path information 102 (node ​​1, A → B, 5) shown in FIG. 14 indicates that a new message path was added at node 1 when message #5 was dispatched on the message path A → B. Next, based on the message path information 102 broadcast from each node, the leader determines the maximum message sequence number (e.g., #6) of messages dispatched on the existing message path A → B at each node. Based on the above process, each node reaches a consensus on adding the message path D → B after the maximum message sequence number (#6) of messages dispatched on A → B, thereby achieving synchronization among all nodes.

[0099] The process of adding a message path (message path addition process), which serves as a trigger for the above-mentioned consensus formation, will be described in detail below.

[0100] Fig. 15 is a flowchart showing an example of the processing procedure for message path addition processing. The message path addition processing shown in Fig. 15 corresponds to step S42 in Fig. 12, and is executed by the message processing unit 22 of each node 10 when it becomes necessary to add a new message path due to the acceptance of a message transmission request.

[0101] According to FIG. 15, first, the message processing unit 22 determines whether or not the message path to be newly added (hereinafter, referred to as a new message path) is the second path for the message destination (step S51).

[0102] If the new message path is the second message path in step S51 (YES in step S51), the message processing unit 22 sets the first message path to a blocked state (step S52). Next, the message processing unit 22 broadcasts the message path information of the existing message path (see FIGS. 14 and 18) to other nodes (step S53), and ends the message path addition process.

[0103] In addition, in each node that receives the broadcast in step S53, the message path information receiving process shown in Figure 19, which will be described later, is performed, and the received message path information is registered in the message path management table 35, and the leader node determines the dispatch order to be consistent between nodes.

[0104] If the new message path is not the second message path in step S51, that is, if it is the first or third or subsequent message path (NO in step S51), the message processing unit 22 checks whether the own node is the leader (step S54). If the own node is the leader (YES in step S54), the message processing unit 22 adds a log entry of type "message path added" to the log (event log 34) (step S55), and ends the message path addition process. If the own node is a follower (NO in step S54), the message processing unit 22 ends the message path addition process without performing any special process.

[0105] Fig. 16 is a diagram showing an example of a log entry of the message path addition type. Log entry 140 shown in Fig. 16 is an example of a log entry that is added to the log (event log 34) in step S55 of Fig. 15 when the new message path is the first or third or subsequent message path.

[0106] Log entry 140 is data managed in units of the addition of a new message path, and includes existing message path 141 indicating an existing message path, added message path 142 indicating a new message path, and dispatched sequence number 143 indicating the sequence number of a message already dispatched on the existing message path when the new message path is added.

[0107] In the case of log entry 140 in Figure 16, the new message path is not the second message path for the message destination, so the process of broadcasting message path information for the existing message path (step S53 in Figure 15) is not performed. Furthermore, the message path information addition process in Figure 19, which is executed at each node in response to step S53 in Figure 15, is not executed, so there is no need to refer to the dispatch sequence number (step S64 in Figure 19 is not performed). Therefore, in log entry 140 in Figure 16, it is not necessary to specify values ​​for existing message path 141 and dispatched sequence number 143, and "A → B" is specified for added message path 142, indicating that the new message path is a message path from task A to task B.

[0108] Fig. 17 is a diagram showing another example of a log entry of the message path addition type. Log entry 150 shown in Fig. 17 is an example of a log entry that is added to the log (event log 34) in step S64 of Fig. 19, which will be described later, when the new message path is the first or third or subsequent message path.

[0109] Log entry 150, like log entry 140 in Figure 16, is data managed in units of the addition of a new message path, and includes existing message path 151 indicating an existing message path, added message path 152 indicating a new message path, and dispatched sequence number 153 indicating the sequence number of a message already dispatched on the existing message path when the new message path was added.

[0110] Specifically, when log entry 150 in Figure 17 is applied, after messages up to message sequence number "5" are dispatched on the existing message path "A → B", the message path "D → B" is added to the message path management table 35.

[0111] Figure 18 is a diagram showing an example of message path information. Message path information is data for sharing between nodes when a second message path to a certain message destination is added, indicating how far the message has been dispatched on the first message path. The message path information is also used as a hint for the leader to determine when to add a message path.

[0112] The message path information 160 shown in FIG. 18 includes a node ID 161, an existing message path 162, an additional message path 163, and a dispatched sequence number 164. The node ID 161 indicates the identifier of the node 10 that is the sender of the message path information 160. The existing message path 162 indicates the sender and destination of the first existing message path. The additional message path 163 indicates the sender and destination of the message path to be added (i.e., the second message path that has the same message destination as the existing message path). The dispatched sequence number 164 indicates the maximum value of the dispatched message sequence number in the first message path indicated in the existing message path 162.

[0113] Fig. 19 is a flowchart showing an example of the processing procedure of message pass information reception processing. The message pass information reception processing shown in Fig. 19 is processing executed by the message processing unit 22 in each node 10 that receives the message pass information when the message pass information of the existing message path is broadcast in step S53 of Fig. 15. That is, the message pass information reception processing is executed when the new message path is the second path for the message destination.

[0114] 19, first, the message processing unit 22 registers the received message path information in the message path management table 35 (step S61). A specific example of the message path management table 35 is shown in FIG. 20, which will be described later.

[0115] Next, the message processing unit 22 checks whether the node itself is the leader (step S62). If the node itself is the leader (YES in step S62), the process proceeds to step S63, and if the node itself is not the leader, i.e., is a follower (NO in step S62), the message path information receiving process ends.

[0116] In step S63, the message processing unit 22 of the leader determines whether or not message path information regarding the addition of the new message path has been received from all other nodes. If the message path information has been received from all other nodes (YES in step S63), the process proceeds to step S64, and if the message path information has not been received from all other nodes (NO in step S63), the message path information receiving process ends.

[0117] In step S64, the message processing unit 22 of the leader determines the maximum value of the dispatched message sequence number on the existing message paths based on the message path information from all nodes including the self-node, and then adds a log entry of the "Message path addition" type to the log (event log 34) with the determined maximum message sequence number as the dispatched sequence number, and ends the message path information reception process.

[0118] Fig. 20 is a diagram showing a specific example of the message path management table 35. The message path management table 170 shown in Fig. 20 has a message path 171 and a dispatched message sequence number 172. The message path 171 indicates the source and destination of an existing message path, and the dispatched message sequence number 172 indicates the sequence number of a dispatched message on the existing message path.

[0119] Here, in the message path management table 170 of Fig. 20, the dispatched message sequence number 172 in the message path 171 of "A → B" is "5, 8". This means that, for example, in receiving message path information from another node (step S61 of Fig. 19), message path information indicating that the dispatched sequence number on the message path "A → B" is "8" has been received in addition to message path information indicating that the dispatched sequence number on the message path "A → B" is "5" (for example, message path information 160 from node ID "2" shown in Fig. 18). If such a message path management table 170 is the message path management table 35 of the leader node 10A, in step S64 of Fig. 19, the message processing unit 22 of the leader determines that the maximum value of the dispatched message sequence numbers on the message path "A → B" is "8". Therefore, in step S64, the message processing unit 22 of the leader will add a log entry of the "Message path added" type to the log (event log 34) with the existing message path 151 set to "A → B", the additional message path 152 set to "D → B", and the dispatched sequence number 153 set to "8".

[0120] The message path addition process of FIG. 15 and the message path information reception process of FIG. 19 described above have the following features.

[0121] If the new message path is the second message path for the task or the like at the message destination, a log entry of type "Message path added" is added to the log in the message path information reception process (step S64 in FIG. 19), and a consensus is reached between the nodes on the addition of the message path. At this time, the dispatch of the message to the message destination is put on hold until the consensus on the addition of the new message path is completed (step S52 in FIG. 15).

[0122] Furthermore, when consensus on the message path addition is completed and the log entry is committed and applied, the message path is added to the message path management table 35 (step S55 in FIG. 15, step S61 in FIG. 19).

[0123] If the new message path is the first message path, dispatching of messages on the first message path is suspended until the message path is added to the message path management table 35.

[0124] If the new message path is the third or subsequent message path, messages to the message destination that occur after the new message path is added will be added to the log after the "Message path added" type log entry that corresponds to the addition of the new message path. As a result, dispatching will be put on hold until the addition of the new message path is complete.

[0125] (6-3) Improved message waiting time when adding a message path As mentioned above, when adding a second message path, dispatching of messages on the first message path is temporarily suspended until consensus is reached between the nodes on the "Message Path Added" type log entry and the message path is added to the message path management table. At this time, processing in the FT system 1 stagnates for the time that dispatching is suspended (message waiting time).

[0126] In order to suppress such processing stagnation, for example, at the stage of system testing, the content of the message path management table 35 can be constructed to a certain extent and the content can be backed up, and the content can be restored during actual operation. According to this method, the occurrence frequency of dynamic message path addition can be suppressed to a certain extent, but it is more preferable if further suppression can be achieved.

[0127] Therefore, in this embodiment, as a method for further improving the message waiting time, for example, the following two improvement methods can be adopted.

[0128] (6-3-1) The first improvement method The first improvement method is that when a node receives message path information regarding the addition of the second message path from another node, prior to adding the message path information to the message path management table 35 (step S61 in FIG. 19), the messages on the first message path are dispatched up to the maximum value of the dispatched message sequence numbers described in the message path information.

[0129] The first improvement method is suitable for a situation where there is little increase or decrease in nodes, and can suppress the delay due to holding of dispatching to the maximum extent.

[0130] However, the first improvement method has the following risks in a situation where the frequency of node increase or decrease is high. If the node that reported the maximum value M of the dispatched message sequence number in the message path information fails and stops before transmitting the message path information to the leader, the leader uses N < M as the dispatched message sequence number and adds a log entry of the "message path addition" type to the log. At this time, if there is already a node that has dispatched up to M, that node cannot apply the log entry as instructed by the leader and needs to be excluded from the cluster. Therefore, the higher the frequency of node increase or decrease, the higher the risk of being unable to maintain the multiplicity of node 10 in the FT system 1.

[0131] (6-3-2) Second improvement method The second improvement method, in reaching a consensus on adding a message path, does not reach a consensus among the nodes after the leader receives message path information regarding the addition of a second message path from all nodes as in the method described above, but rather reaches a consensus on adding a message path among the nodes based on a consensus protocol such as RAFT each time the maximum value of the message completion sequence number is updated while the leader is receiving message path information regarding the addition of a second message path from all nodes, and dispatches messages until a consensus is reached. Below, the configuration and processing for realizing the second improvement method are described in detail.

[0132] When the second improvement method is adopted, the FT system 1 introduces a new type of log entry, a "message path information" type log entry.

[0133] The purpose of the "Message Path Information" log entry is to reach an agreement among nodes when adding a second message path to a certain message destination that "a new message path will be added after dispatching at least N messages with the same sequence number already dispatched on the existing message path." Since consensus can be reached if there is a response from the majority of nodes, by using such a "Message Path Information" log entry, it is possible to preemptively dispatch up to N message requests before the leader receives message path information from all followers. The specific operation when applying the "Message Path Information" log entry is to dispatch message requests that are stuck on an existing message path with the "Blocked State" set, in order, up to the message request that corresponds to the dispatched message sequence number indicated in the "Message Path Information" log entry.

[0134] Fig. 21 is a diagram showing an example of a log entry of the message path information type. As shown in Fig. 21, a log entry 180 of the message path information type is configured to have a type 181 and message path information 182. The type 181 indicates the type of information the log entry has, namely, a node ID, an existing message path, an additional message path, and a dispatched sequence number. The message path information 182 indicates the content of each type of information indicated in the type 181.

[0135] Fig. 22 is a sequence diagram showing an example of a processing procedure for adding a message path by the second improvement method. In Fig. 22, one leader (node ​​ID = 1) and two followers (node ​​ID = 2, 3) are shown as a simplified node configuration for the purpose of explanation. Furthermore, the processing of each step described below is executed by the message processing unit 32 of the corresponding node.

[0136] 22, log entries 191 and 193 are examples of log entries of the "message path information" type added to the log (event log 34) in steps S71 and S73, and log entry 195 is an example of a log entry of the "message path addition" type added to the log (event log 34) in step S75. Also, message path information 192 and 194 are examples of message path information broadcast in steps S72 and S74. Message path information 192 and 194 are notifications broadcast from one node to other nodes using a protocol specific to this embodiment that is not specified in RAFT.

[0137] According to FIG. 22, first, node 1, which is the leader node, appends log entry 191 of its own node's "message path information" to the log (step S71). As a result of this appending to the log, a copy of the log is broadcast to follower nodes (nodes 2 and 3) according to the RAFT protocol. Then, each node applies log entry 191 as soon as it has completed committing it. As a result, each node dispatches message requests that were blocked on message path "A → B" up to message sequence number "5," specifically.

[0138] After that, it is assumed that the node 3, which is a follower node, broadcasts message path information 192 indicating that the message sequence number "6" has been dispatched (step S72).

[0139] In this case, node 1, which has received message path information 192, finds that the dispatched message sequence number on the message path "A → B" in message path information 192 is "6," which is greater than the dispatched message sequence number "5" indicated in log entry 191 added to the log in step S71. Therefore, node 1 adds to the log a log entry 193 of the "message path information" type indicating the contents of message path information 192 (step S73). With this addition, the dispatched message sequence number in the log entry of the "message path information" type is updated to "6." Then, similar to the addition to the log in step S71, node 1 broadcasts a copy of the log to follower nodes (nodes 2 and 3) in accordance with the RAFT protocol, and each node applies it as soon as it completes the commit.

[0140] Then, suppose that node 2, which is a follower node, broadcasts message path information 194 indicating that message sequence number "4" has been dispatched (step S74). In this case, node 1, which has received message path information 194, does not perform any special processing because the dispatched message sequence number on the message path "A → B" in message path information 194 is "4," which is smaller than the maximum value of the dispatched message sequence number up to that point, which is "6."

[0141] Then, when node 1, which is the leader node, receives the message path information from all the nodes, it generates a log entry of type "add message path" based on the log entry of type "message path information" that was finally added to the log, and adds this to the log (event log 34) (step S75). Specifically, it adds to the log a log entry 195 of type "add message path" in which the dispatched message sequence number on the message path "A → B" is set to "6" and the added message path is set to "D → B". Note that the application processing after this log addition is the same as the processing described in FIG. 12.

[0142] As described above, in the second improvement method, when adding a message path, a consensus is reached using the log entry of the "message path information," so the processing procedures for the message path addition process shown in Figure 15 and the message path information reception process shown in Figure 19 are partially modified as follows:

[0143] Fig. 23 is a flowchart showing an example of the processing procedure for message path addition processing according to the second improvement method. The flowchart in Fig. 23 will be explained mainly focusing on the differences from the message path addition processing shown in Fig. 15, and an explanation of the common processing will be omitted.

[0144] 23, after setting the first message path to a blocked state in step S52, the message processing unit 22 checks whether the node itself is the leader (step S81). If the node itself is the leader (YES in step S81), the message processing unit 22 adds a log entry of the "message path information" of the node itself to the log (step S82). The processing of step S82 is realized by adding a log in the RAFT protocol. As explained in step S71 of FIG. 22, as a result of adding the log, a copy of the log is broadcast to followers in accordance with the RAFT protocol. On the other hand, if the node itself is not the leader, that is, if the node itself is a follower (NO in step S81), the message processing unit 22 broadcasts message path information of the existing message path to other nodes by message transmission specific to this embodiment, which is independent of the RAFT protocol, as explained in steps S72 and S74 of FIG. 22 (step S53).

[0145] Fig. 24 is a flowchart showing an example of the processing procedure for message pass information reception processing according to the second improvement method. The flowchart in Fig. 24 will be explained mainly focusing on the differences from the message pass information reception processing shown in Fig. 19, and explanations of common processing will be omitted.

[0146] According to Figure 24, the message processing unit 22 of the node that received the message path information registers the message path information in the message path management table 35 (step S61), and if the node is the leader node (YES in step S62), it determines whether message path information regarding the addition of a new message path has been received from all other nodes (step S63).

[0147] Then, in step S63, if the leader has not received message path information regarding the addition of a new message path from all other nodes (NO in step S63), the message processing unit 22 of the leader checks whether the maximum value of the dispatched message sequence number has been updated by registering the message path information in the message path management table 35 in step S61 (step S91).

[0148] If the maximum value of the dispatched message sequence number has been updated in step S91 (YES in step S91), the message processing unit 22 of the leader adds a log entry of the "message path information" type to the log based on the message path information received in step S61 (step S92), and ends the message path information reception process. The process of step S92 corresponds to the process of step S71 in Fig. 22. That is, for the dispatched message sequence number on an existing message path, if the "maximum value" including that of its own node among the message path information currently received from other nodes has been updated, the message processing unit 22 of the leader adds a log entry of the "message path information" type to the log even if it is in the middle of receiving message path information from all nodes.

[0149] On the other hand, if the maximum value of the dispatched message sequence number has not been updated in step S91 (NO in step S91), the message processing unit 22 of the leader ends the message pass information receiving process without performing any special process. This processing procedure corresponds to the case where the message pass information 194 is broadcast from the follower (node ​​2) in step S74 of Figure 22, and the leader (node ​​1) that receives this message pass information 194 does not perform any special process.

[0150] To summarize the above explanation, the second improvement method for adding a message path has the following features when compared with the method for adding a message path explained in (6-2).

[0151] The difference between the message path addition process in the second improvement method and that in (6-2) is that a log entry of type "message path information" is introduced. Then, instead of broadcasting the message path information to follower nodes, the leader node appends a log entry of type "message path information" to the log. As a result, a copy of the log is broadcast to follower nodes according to the RAFT protocol.

[0152] In addition, when a leader node receives message path information from a follower node, if the maximum value of the dispatched message sequence number in an existing message path is updated, the leader node adds a log entry of type "message path information" corresponding to the received message path to the log.

[0153] In addition, when a log entry of the "message path information" type is committed and each node applies the contents of the log entry in response to this commit, message requests that are blocked on the existing message path are dispatched up to the dispatched message sequence number indicated in the log entry (message path information).

[0154] On the other hand, the second improvement method has in common with the message path addition process of (6-2), for example, that the follower broadcasts message path information to other nodes. The reason for this configuration is that if the leader stops due to a failure or other reason during the message path addition process, and the follower who becomes the new leader continues the message path addition process, the information broadcast from each follower is required. Also, when the leader receives message path information from all nodes, it adds a log entry of type "Message path addition" to the log. Furthermore, the process for applying this "Message path addition" type log entry is also common to the message path addition process of (6-2).

[0155] By incorporating the above-described features, the second improvement method can safely dispatch some of the blocked message requests before the leader has finished receiving message passing information from all nodes. "Safely" here means that even if some nodes leave during processing due to a failure or other reason, the situation will not arise in which synchronization cannot be maintained between nodes, as explained with reference to FIG. 14. The second improvement method is particularly effective when the FT system 1 is configured such that there is a node that is slower to notify message passing information than other nodes, and node departures are more likely to occur.

[0156] (7) Output processing unit 33 The processing by the output processing unit 33 will be described in detail below.

[0157] The output processing unit 33 receives an external output request as a processing result of each task 20, determines the content to be externally output in accordance with a predetermined policy, and transmits the determined content to the outside (terminal 3 such as a client).

[0158] As described above, in this embodiment, it is guaranteed that the task group of each node 10 behaves deterministically. Therefore, when focusing on a certain task 20, basically, external output requests with the same content are sent from all nodes 10 in the same order.

[0159] On the other hand, for external output requests from different tasks 20, there is no guarantee that the order of requests will match between nodes, and the order in which the external outputs are actually made is not determined deterministically either (see Figure 25(A)). However, it is generally assumed that the contents of these external outputs are independent of each other, and that the order of the external outputs does not affect the results of the system. If the order of the external output requests from different tasks 20 is important, it is possible to make the output order consistent between nodes by placing a common task (task C in Figure 25(B)) that relays the contents of each external output (see Figure 25(B)).

[0160] 25A and 25B are conceptual diagrams for explaining the order guarantee of external outputs. In both of Figs. 25A and 25B, external output requests from different tasks 20A and 20B (tasks A and B) are sent to an external output 39. The external output 39 can be read as the output processing unit 33.

[0161] In the configuration of FIG. 25(A), task A sends message 101A of external output requests "A → External #2" and "A → External #3" to the external output 39, and task B sends message 101B of external output request "B → External #4" to the external output 39. Here, for external output requests from the same task A, it is guaranteed that "A → External #3" will be output to the outside after "A → External #2." On the other hand, for external output requests from different tasks A and B, there is no guarantee that "A → External #2" or "B → External #4" will be output to the outside first, and the order in which they arrive at the external output 39 (output processing unit 33) may differ between nodes.

[0162] The configuration in Fig. 25(B) is the same as the configuration in Fig. 25(A), except that a common task C is added to aggregate and relay the contents of each external output. As described above, by having task C receive external output requests from different tasks A and B, the order in which the external outputs from task C to external output 39 (output processing unit 33) can be made consistent between nodes.

[0163] The "predetermined policy" that the output processing unit 33 follows when determining the content to be output to the outside may be, for example, a first-come-first-served policy or a majority vote policy.

[0164] In the first-come-first-served policy, the first external output request received from each replica node (each node 10) is output externally, and the rest are discarded. In the majority voting policy, a majority vote is carried out on external output requests that are thought to be identical and received from each replica node (each node 10), and the most likely content is selected and output externally, and the rest are discarded.

[0165] Furthermore, in the FT system 1 of this embodiment, a dedicated external output device may be provided to output the processing results at the node 10 to an external device (e.g., terminal 3), but by configuring it as shown in Figure 26 below, the external output of each node 10 can be adjusted and output to an external device even without a dedicated external output device.

[0166] Fig. 26 is an image diagram illustrating a configuration for adjusting and outputting external outputs from multiple nodes. In Fig. 26, when task A of each node transmits an external output request, it broadcasts it to the external output 39 (output processing unit 33) of each node. Then, the external output 39 (output processing unit 33) compares the contents of the external output requests received from task A of each node, and transmits the one that is determined to be correct, for example, by majority vote, to an external device (terminal 3). In the case of Fig. 26, as shown in the enlarged view, the contents of the external output requests from node 1 and node 2 are "α," which is the majority, and therefore "α" is output to the outside.

[0167] Furthermore, in Figure 26, if an idempotent messaging protocol is used for communication between the external output 39 and the external device (terminal 3), duplicate messages can be discarded by the external device. For example, a sequence number can be assigned to the transmitted message for each session, and the receiving side can receive the messages one by one in the order of the sequence number. In this way, even if a node fails and stops before an external transmission, the external device can receive the processing results without delay. [Explanation of symbols]

[0168] 1 Fault-tolerant (FT) system 2. Communication Network 3. Terminal 10 nodes 10A Leader node (leader) 10B, 10C Follower node (follower) 20 Tasks 21 State (Local State) 30 Event Processing Platform 31 Input processing section 32 Message processing section 33 Output Processing Section 34 Event Log 35 Message path management table 36 Message Waiting Queue 38 External Input 39 External Output

Claims

1. A fault-tolerant system in which multiplexed nodes are connected via a communication network, each of the plurality of nodes has a processor, a memory, and a storage area; An application that runs by the processor in each of the nodes reading a program from the storage area into the memory and executing it is composed of one or more tasks that deterministically generate an output for an input, one of the plurality of nodes is a leader node, and the nodes other than the leader node are follower nodes, and the leader node and the follower nodes are configured to perform the same task, Each task is associated with a local state that indicates the state of the task itself, The task's inputs include messages received from outside the system or from other tasks, and access to the local state associated with the task; The outputs of the task include messages that the task sends to the outside of the system or to other tasks, and updates to the local state associated with the task; When the task sends a message to a destination task having a plurality of message senders, the processors of the plurality of nodes reach a consensus among the plurality of nodes on the order of delivery of the messages to the destination task, and deliver and process the messages in accordance with the agreed-upon delivery order. A fault-tolerant system characterized by:

2. When the task sends a message to a destination task having a plurality of message senders, the sending of the message to the destination task is temporarily suspended until an agreement is reached among the plurality of nodes regarding the order of delivery of the messages to the destination task.

2. The fault-tolerant system of claim 1.

3. When a second message path is newly added from a third task different from one or more first tasks to a second task for which an existing first message path has been generated from the first task, the processor performs recognition among the plurality of nodes regarding the addition of the second message path.

3. The fault-tolerant system according to claim 2.

4. If the second message path to be added is a second message path for the second task, The processor forms a consensus among the plurality of nodes as to how many messages should be dispatched on the first message path before adding the second message path, and temporarily suspends transmission of messages on the first message path until the consensus is reached.

4. The fault-tolerant system according to claim 3.

5. When the second message path to be added is the second message path for the second task, The processor of the leader node determines the maximum value of the sequence number of messages already dispatched on the first message path in each node, and forms a consensus among the plurality of nodes to add the second message path after the message with the maximum sequence number has been dispatched on the first message path.

5. The fault-tolerant system according to claim 4.

6. When the processor of each of the nodes intends to add a second message path for the second task as the second message path, the processor of each of the nodes broadcasts message path information regarding the addition of the second message path to other nodes; the message path information indicates a sequence number of a message that has been dispatched on the first message path at the node; The processor of the other node that has received the message path information dispatches messages on the first message path in its own node up to the maximum sequence number indicated in the message path information, prior to forming an agreement among the plurality of nodes as to what number of messages should be dispatched on the first message path before adding the second message path.

5. The fault-tolerant system according to claim 4.

7. When the processor of the follower node intends to add a second message path for the second task as the second message path, the processor of the follower node broadcasts message path information regarding the addition of the second message path to other nodes; While the leader node is receiving message path information regarding the addition of the second message path from all nodes, at a point in time when the maximum value of the sequence numbers of the messages already dispatched on the first message path indicated in the message path information received by the leader node from any node exceeds the maximum value of the sequence numbers of the messages already dispatched on the first message path in the leader node, a consensus is formed among the nodes to dispatch messages on the first message path in the own node up to the maximum value of the sequence numbers indicated in the message path information.

5. The fault-tolerant system according to claim 4.

8. A data processing method using a fault-tolerant system in which multiplexed nodes are connected via a communication network, comprising: each of the plurality of nodes has a processor, a memory, and a storage area; An application that runs by the processor in each of the nodes reading a program from the storage area into the memory and executing it is composed of one or more tasks that deterministically generate an output for an input, one of the plurality of nodes is a leader node, and the nodes other than the leader node are follower nodes, and the leader node and the follower nodes are configured to perform the same task, Each task is associated with a local state that indicates the state of the task itself, The task's inputs include messages received from outside the system or from other tasks, and access to the local state associated with the task; The outputs of the task include messages that the task sends to the outside of the system or to other tasks, and updates to the local state associated with the task; When the task sends a message to a destination task having multiple message senders, the processors of the multiple nodes reach a consensus among the multiple nodes on the order of delivery of the messages to the destination task, and transmit the messages according to the agreed-upon delivery order. A data processing method comprising:

Images