Fault-tolerant distributed computing system based on dynamic reconfiguration

Abstract

A fault tolerant distributed computing system includes a communication link and a plurality of nodes in electronic communication with each other through the communication link. Each node executes at least one node-specific application, including a backup database that stores a backup copy corresponding to one of the node-specific applications executed by one of the remaining nodes as part of the distributed computing system, and includes backup computing capabilities, each node executing at least one node-specific application, including a backup database that stores a backup copy corresponding to one of the node-specific applications executed by one of the remaining nodes as part of the distributed computing system. The backup computing power is sufficient to execute at least one backup copy of one of the node-specific applications stored in the backup database. In response to determining that the particular node is non-runnable, the remaining nodes execute all spare copies of the one or more node-specific applications previously executed by the particular node that is now non-runnable.

Description

Technical Field

[0001] The present invention relates to a fault-tolerant distributed computing system comprising a plurality of nodes in electronic communication with each other, wherein the nodes are dynamically reconfigured in response to detecting that one or more nodes are inoperable. Background Art

[0002] A distributed computing system consists of multiple computing devices, also called nodes, that send and receive information over communication links. The communication links can be implemented in hardware, such as, for example, a bus-based communication network, or wirelessly. Dynamic reconfiguration refers to a change in the set of processes being executed by a node. Fault tolerance refers to the ability of a system to continue operating in the presence of faults that are included as part of the fault model. The fault model defines the number and types of faults that can be tolerated that may affect the system.

[0003] In a specific embodiment of the distributed computing system, a 10Base-T network communication protocol is used. In this embodiment, the distributed computing system includes multiple electronic control units (ECUs) that communicate electronically with each other via twisted pair cables. During operation, the master node can execute multiple mission-critical applications, while one or more slave nodes execute one or more non-mission-critical applications, which are referred to as slave operations. During operation, a state transfer occurs when the master node transfers data belonging to a mission-critical application to the slave node. If the master node becomes inoperative due to a failure, one of the slave nodes can execute the mission-critical application previously executed by the master node. However, due to limited computing power, the slave node no longer executes the secondary application. One method to alleviate this problem involves including fully redundant slave nodes, which introduces additional complexity and components to the communication system.

[0004] Therefore, while distributed computing systems achieve their intended purposes, there is a need in the art for an improved approach for dynamic reconfiguration management. Summary of the invention

[0005] According to several aspects, a fault-tolerant distributed computing system is disclosed, which includes a communication link and a plurality of nodes in electronic communication with each other through the communication link. Each node executes at least one node-specific application and includes a backup database that stores a backup copy corresponding to one of a plurality of node-specific applications executed by one of a plurality of remaining nodes that are part of the distributed computing system, and includes backup computing capacity sufficient to execute at least one backup copy of one of the plurality of node-specific applications stored in the backup database. The plurality of nodes execute instructions to: monitor the communication link for data transmitted by the particular node during a unique transmission opportunity corresponding to the particular node by the remaining nodes that are part of the distributed computing system; observe silence during the unique transmission opportunity of the particular node by the plurality of remaining nodes that are part of the distributed computing system; determine that the particular node is inoperable in response to observing silence during the unique transmission opportunity of the particular node; and in response to determining that the particular node is inoperable, the remaining nodes execute all backup copies of one or more node-specific applications previously executed by the now inoperable particular node.

[0006] In another aspect, each node-specific application executed by a specific node that is part of a distributed computing system is stored in a standby database corresponding to one of a plurality of remaining nodes that are part of the distributed computing system.

[0007] In yet another aspect, the number of node-specific applications executed by each node is equal to the number of remaining nodes that are part of the distributed computing system.

[0008] In one aspect, each of the remaining nodes that are part of the distributed computing system stores one of the node-specific applications executed by the particular node in a corresponding standby database.

[0009] In another aspect, the remaining nodes that are part of the distributed computing system store more than one node-specific applications executed by a specific node in a corresponding standby database.

[0010] In yet another aspect, the data transmitted by the particular node includes one of: state information, application data, or an activity code.

[0011] In one aspect, the communication link is a bus.

[0012] In another aspect, the communication link is implemented based on one of: a 10Base-T network communication protocol, a switched Ethernet, a controller area network (CAN) protocol, and an automotive communication protocol based on International Organization for Standardization (ISO) 17458-1 to 17458-5.

[0013] In yet another aspect, the communication link is implemented wirelessly.

[0014] In one aspect, the communication link is implemented based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards.

[0015] In another aspect, the communication link is implemented based on a network communication protocol, wherein state information sent by any node that is part of a distributed computing system is visible to the remaining nodes, and periodic, guaranteed transmission opportunities are provided to all nodes that are part of a fault-tolerant distributed computing system.

[0016] In yet another aspect, the plurality of nodes represent electronic control modules (ECUs) that control one or more systems that are part of a vehicle.

[0017] In one aspect, a node represents one of: a smart sensor and a smart actuator.

[0018] In another aspect, a method for dynamically reconfiguring a plurality of nodes that are in electronic communication with each other via a communication link is disclosed, the plurality of nodes being part of a fault-tolerant distributed computing system. The method includes monitoring a communication link for data transmitted by a particular node during a unique transmission opportunity corresponding to the particular node by the remaining nodes that are part of the distributed computing system, wherein each node executes at least one node-specific application and includes a standby database that stores a standby copy corresponding to one of the node-specific applications executed by one of the remaining nodes that are part of the distributed computing system. The method includes observing silence by the remaining nodes that are part of the distributed computing system during the unique transmission opportunity of the particular node. The method includes, in response to observing silence during the unique transmission opportunity of the particular node, determining that the particular node is inoperable. The method includes, in response to determining that the particular node is inoperable, executing all standby copies of one or more node-specific applications previously executed by the now inoperable particular node by the remaining nodes of the respective parts of the distributed computing system, wherein each node includes a backup computing capacity sufficient to execute at least one standby copy of one of the node-specific applications stored in the standby database.

[0019] In another aspect, a fault-tolerant distributed computing system for a vehicle is disclosed, comprising a communication link implemented as a bus and a plurality of nodes in electronic communication with each other through the communication link, wherein the plurality of nodes represent ECUs that control one or more systems as part of the vehicle, and wherein each node executes at least one node-specific application and includes a backup database that stores backup copies corresponding to one of the node-specific applications executed by one of the remaining nodes that are part of the distributed computing system, and includes backup computing capacity sufficient to execute at least one backup copy of one of the node-specific applications stored in the backup database. The plurality of nodes execute instructions to: monitor the communication link for data transmitted by the particular node during a unique transmission opportunity corresponding to the particular node by the remaining nodes that are part of the distributed computing system; observe silence during the unique transmission opportunity of the particular node by the remaining nodes that are part of the distributed computing system; determine that the particular node is inoperable in response to observing silence during the unique transmission opportunity of the particular node; and execute all backup copies of one or more node-specific applications previously executed by the now inoperable particular node by the remaining nodes that are each part of the distributed computing system in response to determining that the particular node is inoperable.

[0020] In yet another aspect, each node-specific application executed by a specific node that is part of the distributed computing system is stored in a standby database corresponding to one of the remaining nodes that are part of the distributed computing system.

[0021] In one aspect, the number of node-specific applications executed by each node is equal to the number of remaining nodes that are part of the distributed computing system.

[0022] In another aspect, each of the remaining nodes that are part of the distributed computing system stores one of the node-specific applications executed by the particular node in a corresponding standby database.

[0023] In yet another aspect, remaining nodes that are part of the distributed computing system store more than one node-specific applications executed by a particular node in a corresponding standby database.

[0024] In one aspect, the data transmitted by the particular node includes one of: state information, application data, or an activity code.

[0025] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0027] Figure 1 is a schematic diagram of a vehicle including the disclosed distributed computing system according to an exemplary embodiment, the distributed computing system including a primary node and a plurality of secondary nodes; and

[0028] Figure 2 is a process flow diagram illustrating a method for dynamically reconfiguring a plurality of nodes as part of a fault-tolerant distributed computing system according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] The following description is merely exemplary in nature and is not intended to limit the present invention, application, or uses.

[0030] Reference Figure 1 , a schematic diagram of an exemplary fault-tolerant distributed computing system 10 as part of a vehicle 12 is shown. The distributed computing system 10 includes a plurality of electronic communication nodes 20 that communicate electronically with each other via communication links 22. The communication links 22 are implemented in hardware, such as a bus communication system or wirelessly. Figure 1 In the non-limiting embodiment shown, the distributed computing system 10 is implemented as part of a vehicle, such as, but not limited to, a car, truck, sport utility vehicle, van, or RV. However, it should be understood that Figure 1 The invention is merely exemplary in nature, and the disclosed distributed computing system 10 is not limited to vehicles. In fact, the distributed computing system 10 can be implemented in various other applications, such as, for example, industrial automation control systems and the Internet of Things (IoT).

[0031] In one non-limiting embodiment, the communication link 22 is a bus implemented based on the 10Base-T network communication protocol. In another embodiment, the network communication protocol is based on a switched Ethernet, a controller area network (CAN) protocol, or an automotive communication protocol based on the International Organization for Standardization (ISO) 17458-1 to 17458-5. In another embodiment, the communication link 22 is implemented wirelessly based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. It should be understood that the distributed computing system 10 is implemented based on any other type of network communication protocol, wherein the status information sent by any node 20 that is part of the distributed computing system 10 is visible to the other nodes, and periodic, guaranteed transmission opportunities are provided to all nodes 20 that are part of the distributed computing system 10.

[0032] Each of the nodes 20 represents a computing device including one or more processors and memory. In one non-limiting embodiment, the nodes 20 represent an electronic control module (ECU) that controls one or more systems that are part of the vehicle 12. In one embodiment, the nodes 20 may represent devices such as smart sensors and smart actuators. Figure 1 In the example shown, there are five nodes 20 as part of the system, each of which is assigned a unique node identifier or node ID. The nodes 20 include node identifiers 0, 1, 2, 3, and 4 (node ​​ID = 0, node ID = 1, node ID = 2, node ID = 3, node ID = 4).

[0033] like Figure 1 As shown, each node 20 as part of the distributed computing system 10 executes at least one node-specific application 40. In the example shown, each node 20 executes four node-specific applications 40, denoted as Ai,j, where i represents a node identifier, j represents an application identifier, and the node identifier is not equal to the application identifier, or i≠j. In the example shown, a node 20 assigned a node identifier of 0 executes four applications A0,1, A0,2, A0,3, A0,4, a node 20 assigned a node identifier of 1 executes four applications A1,0, A1,2, A1,3, A1,4, a node 20 assigned a node identifier of 2 executes four applications A2,0, A2,1, A2,3, A2,4, a node 20 assigned a node identifier of 3 executes four applications A3,0, A3,1, A3,2, A3,3, A3,4, and a node assigned a node identifier of 4 executes four applications A4,0, A4,1, A4,2, A4,3.

[0034] Each node 20 that is part of the distributed computing system 10 also includes a backup database 42, wherein the backup database 42 stores a copy or backup copy 44 corresponding to one of the node-specific applications 40 executed by one of the remaining nodes 20 (as part of the distributed computing system 10). Each node-specific application 40 executed by a node 20 (as part of the distributed computing system 10) is stored in the backup database 42 corresponding to one of the remaining nodes 20 (as part of the distributed computing system 10). Therefore, each node-specific application 40 executed by one of the nodes 20 is saved in the backup database 42 of one of the remaining nodes 20 (as part of the distributed computing system 10).

[0035] In a non-limiting embodiment, such as Figure 1In the example shown, the number of node-specific applications 40 executed by each node 20 is equal to the number of remaining nodes 20 (as part of the distributed computing system 10). For example, the node 20 assigned with node identifier 0 executes four node-specific applications 40, and there are four remaining nodes 20 as part of the distributed computing system 10. However, as described above, Figure 1 This is merely exemplary in nature, and the nodes 20 are not limited to executing a number of node-specific applications 40 equal to the number of remaining nodes 20 that are part of the distributed computing system 10 .

[0036] exist Figure 1 In the example shown, the standby database 42 stores a standby copy 44 corresponding to one of the node-specific applications 40 for each remaining node 20 (as part of the distributed computing system 10), so the number of standby copies 44 stored in the standby database 42 is equal to the number of remaining nodes 20 (as part of the distributed computing system 10). For example, the node 20 assigned node identifier 0 stores the standby copies 44 of the node-specific application 40 corresponding to the nodes 20 assigned node identifier 1 (A1,0), node identifier 2 (A2,0), node identifier 3 (A3,0), and node identifier 4 (A4,0), and the number of standby copies 44 stored in the standby database 42 is 4, which is equal to the number of remaining nodes 20. However, it should be understood that in some embodiments, the standby database 42 can also store more than one standby copy 44 of the node-specific application 40 corresponding to each remaining node 20 (as part of the distributed computing system 10).

[0037] Each node 20 (as part of the distributed computing system 10) includes backup computing capacity C, where the backup computing capacity is sufficient to execute at least one backup copy 44 of one of the node-specific applications 40 stored in the backup database 42 of the corresponding node 20. In some embodiments, the backup computing capacity C may be sufficient to execute more than one backup copy 44 stored in the backup database 42. For example, in one embodiment, the computing capacity C of the node 20 assigned with the node identifier 0 is sufficient to execute two of the backup copies 44, each copy corresponding to one of the node-specific applications 40 stored in the backup database 42.

[0038] During operation, each node 20 transmits data during the only transmission opportunity provided to each node 20 of the distributed computing system 10. The data may include one of the following: state information, application data, or activity code. Activity codes are sent when neither state information nor application data is sent. During the only transmission opportunity corresponding to a particular node 20 (as part of the distributed computing system 10), the remaining nodes 20 (as part of the distributed computing system 10) monitor the communication link 22 for data transmitted by the particular node 20.

[0039] It should be understood that if a particular node 20 experiences a failure that renders the particular node 20 inoperable, the particular node 20 will miss transmitting data during its only transmission opportunity. Therefore, the remaining nodes 20 (as part of the distributed computing system 10) observe silence during the only transmission opportunity of the particular node 20. In response to observing silence during the only transmission opportunity of the particular node 20, the remaining nodes 20 (as part of the distributed computing system 10) determine that the particular node 20 is inoperable and, therefore, can no longer execute the corresponding node-specific application 40. As described below, in response to determining that the particular node 20 is inoperable, the remaining nodes 20 (as part of the distributed computing system 10) are dynamically reconfigured to execute the node-specific application 40 that is no longer executed by the particular node 20.

[0040] It should be understood that in some cases, a particular node 20 may recover from a failure that caused the particular node 20 to become inoperable and may resume transmitting data during its only transmission opportunity. However, in one embodiment, the remaining nodes 20 that are part of the distributed computing system 10 will ignore the data transmitted by the particular node 20 after the particular node 20 resumes transmitting data during its only transmission opportunity. Alternatively, in another embodiment, the remaining nodes 20 are dynamically reconfigured to recover to the original configuration, under which each node 20 (as part of the distributed computing system 10) continues to execute its node-specific application 40.

[0041] Dynamically reconfiguring the remaining nodes 20 as part of the distributed computing system 10 will now be described. In response to determining that the particular node 20 is inoperable, the remaining nodes 20 (as respective parts of the distributed computing system 10) execute all node-specific applications 40 previously executed by the now inoperable particular node 20. As described above, each node-specific application 40 executed by the particular node 20 (as part of the distributed computing system 10) is stored as a standby copy 44 in a standby database 42 that corresponds to one of the remaining nodes 20 (as part of the distributed computing system 10). Therefore, all node-specific applications previously executed by the particular node 20 are accessible and available to the remaining nodes 20 (as part of the distributed computing system 10).

[0042] exist Figure 1 In the example shown, each of the remaining nodes 20 (as part of the distributed computing system 10) stores one of the node-specific applications 40 executed by the specific node 20 in a corresponding standby database 42. Therefore, each of the remaining nodes 20 executes one of the node-specific applications 40 that was previously executed by the specific node 20 that is now non-operational. Specifically, for example, if the node 20 assigned with node identifier 0 is not operational, the node 20 assigned with node identifier 1 executes node-specific application A0,1, the node 20 assigned with node identifier 2 executes node-specific application A0,2, the node 20 assigned with node identifier 3 executes node-specific application A0,3, and the node 20 assigned with node identifier 4 executes node-specific application A0,4. Therefore, in Figure 1 In the illustrated embodiment, the distributed computing system 10 tolerates a failure that renders one of the nodes 20 inoperative by utilizing the backup computing capacity C available in each of the remaining nodes 20. In other words, Figure 1 The illustrated embodiment shows an example where the standby replicas 44 are evenly distributed among all nodes 20 , where each node 20 is responsible for executing a single node-specific application 40 that was previously executed by another node 20 .

[0043] although Figure 1Each of the remaining nodes 20 (as part of the distributed computing system 10) is shown as storing one of the node-specific applications 40 executed by the particular node 20 in a corresponding standby database 42, but it should be understood that this configuration is merely exemplary in nature. For example, in another embodiment, the remaining nodes 20 as part of the distributed computing system 10 store standby copies 44 of multiple node-specific applications 40 executed by the particular node 20 in a corresponding standby database 42. For example, a node 20 assigned a node identifier 1 may store standby copies 44 of multiple node-specific applications 40 executed by a node 20 assigned a node identifier 0. It should also be understood that although Figure 1 The example shown in FIG. 1 dynamically reconfigures the remaining nodes 20 (as part of the distributed computing system 10) in response to determining that a single node 20 becomes inoperable, but in some embodiments, the distributed computing system 10 also has the ability to dynamically reconfigure the remaining nodes 20 in response to more than one node 20 becoming inoperable. In this embodiment, the remaining nodes 20 (as part of the distributed computing system 10) will have sufficient backup computing capacity C to execute the node-specific applications 40 executed by the two inoperable nodes 20 and save the corresponding backup copies 44 in their corresponding backup databases 42.

[0044] Figure 2 is a process flow diagram illustrating a method 200 for dynamically reconfiguring a plurality of nodes 20 as part of a fault-tolerant distributed computing system 10. Figure 1 and Figure 2 , the method 200 may begin at block 202. In block 202, the remaining nodes 20 (as part of the distributed computing system 10) monitor the communication link 22 for data transmitted by the particular node 20 during the particular node's 20 only transmission opportunity. The method 200 may then proceed to decision block 204.

[0045] In decision block 204, the remaining nodes 20 (as part of the distributed computing system 10) either receive data during the only transmission opportunity corresponding to the particular node 20, or alternatively observe silence. In response to receiving data during the only transmission opportunity corresponding to the particular node 20, the method 200 may terminate. However, in response to the remaining nodes 20 (as part of the distributed computing system 10) observing silence during the only transmission opportunity of the particular node 20, the method 200 proceeds to block 206.

[0046] In block 206, in response to observing silence during the only transmission opportunity of a particular node 20, the remaining nodes 20 determine that the particular node is inoperable.

[0047] In block 208, in response to determining that the particular node 20 is inoperable, the remaining nodes 20 (as part of the distributed computing system 10) execute all backup copies 44 of the node-specific applications 40 previously executed by the now inoperable particular node 20. As described above, Figure 1 The illustrated embodiment is merely one example of how the node specific application 40 may be executed by the remaining nodes 20. The method 200 may then terminate.

[0048] Referring generally to the accompanying drawings, the disclosed distributed computing system provides a variety of technical effects and benefits. Specifically, the distributed computing system provides a method for tolerating a failure that causes one of the nodes that are part of the system to become inoperable without requiring the provision of fully redundant nodes. In some embodiments, the distributed computing system tolerates a failure that renders one of the nodes inoperable by utilizing the backup computing power available in each of the remaining nodes, and thus effectively utilizes a relatively small amount of backup computing power distributed across the remaining nodes that are part of the distributed computing system.

[0049] A controller may refer to an electronic circuit, a combinational logic circuit, a field programmable gate array (FPGA), a processor (shared, dedicated, or group) that executes code, or a combination of some or all of the above, or as part of them, such as in a system on a chip. In addition, the controller may be microprocessor-based, such as a computer having at least one processor, memory (RAM and / or ROM), and associated input and output buses. The processor may operate under the control of an operating system residing in the memory. The operating system may manage computer resources so that computer program code embodied as one or more computer software applications (such as applications residing in the memory) may have instructions executed by the processor. In an alternative embodiment, the processor may execute the application directly, in which case the operating system may be omitted.

[0050] The description of the present invention is merely exemplary in nature, and variations that do not depart from the gist of the present invention are intended to be within the scope of the present invention. Such variations should not be regarded as a departure from the spirit and scope of the present invention.

Claims

1. A fault-tolerant distributed computing system, comprising: Communication links; as well as a plurality of nodes in electronic communication with each other via the communication link, wherein each node executes at least one node-specific application and includes a backup database storing a backup copy of one of the one or more node-specific applications, the one node-specific application being executed by one of the remaining nodes of the plurality of remaining nodes as part of the distributed computing system, and each of the nodes including backup computing capacity sufficient to execute at least one backup copy of the one node-specific application stored in the backup database, and wherein the plurality of nodes execute instructions to: monitoring, by remaining nodes that are part of the distributed computing system, the communication link for data transmitted by the particular node during a unique transmission opportunity corresponding to the particular node; observing silence by the remaining nodes that are part of the distributed computing system during the only transmission opportunity of the particular node; In response to observing silence during the only transmission opportunity for the particular node, determining that the particular node is inoperative; and In response to determining that the particular node is not operational, the remaining nodes that are respective parts of the distributed computing system execute all of the standby copies of the one or more node-specific applications that were previously executed by the particular node that is currently not operational.

2. The fault-tolerant distributed computing system according to claim 1, wherein: Each node-specific application executed by the specific node that is part of the computing system is stored in the standby database corresponding to one of the remaining nodes that are part of the distributed computing system.

3. The fault-tolerant distributed computing system according to claim 1, wherein: The number of node-specific applications executed by each node is equal to the number of remaining nodes that are part of the distributed computing system.

4. The fault-tolerant distributed computing system according to claim 1, wherein: Each of the remaining nodes that are part of the distributed computing system stores one of the node-specific applications executed by the particular node in a corresponding standby database.

5. The fault-tolerant distributed computing system according to claim 1, wherein: The remaining nodes as part of the distributed computing system store one or more node-specific applications executed by the specific node in a corresponding standby database.

6. The fault-tolerant distributed computing system according to claim 1, wherein: The data transmitted by the specific node includes one of the following: status information, application data, or activity code.

7. The fault-tolerant distributed computing system according to claim 1, wherein: The communication link is a bus.

8. The fault-tolerant distributed computing system according to claim 6, wherein: The communication link is implemented based on one of the following protocols: 10Base-T network communication protocol, switched Ethernet, controller area network CAN protocol, and automotive communication protocol based on International Organization for Standardization ISO17458-1 to 17458-5.

9. The fault-tolerant distributed computing system according to claim 1, wherein: The communication link is implemented wirelessly.

10. The fault-tolerant distributed computing system according to claim 9, wherein: The communication link is implemented based on the Institute of Electrical and Electronics Engineers IEEE 802.11 family of standards.

Images