Fault processing method of multi-device system

Abstract

The invention discloses a fault processing method of a multi-device system, the multi-device system comprises a plurality of devices, a plurality of edge controllers and a cloud, and the fault processing method comprises the following steps: the cloud regularly trains and updates fault prediction models of the edge controllers and the devices based on historical data; after the latest fault prediction model is obtained, the cloud end predicts the fault levels of the edge controller and the equipment thereof at regular intervals; the fault level comprises at least one of a normal state, a quality reduction state, a quality early warning state and a fault state; and when the fault level is a quality reduction state or a quality early warning state, the cloud and the edge controller verify the fault through a corresponding heartbeat communication mechanism, and if the corresponding fault is verified, the cloud restarts and recovers a control program of a corresponding device in the edge controller. According to the invention, software faults of multiple devices can be timely handled.

Description

Technical Field

[0001] The present invention relates to the technical field of digital bases, and in particular to a fault handling method for a multi-device system. Background Art

[0002] With the rapid development of industrial digitalization, big data and cloud computing have provided strong support for various complex projects.

[0003] Currently, multi-device systems use edge controllers to manage large groups of devices, and then the devices can be centrally managed through the edge controllers in the cloud.

[0004] Taking smart buildings as an example, smart buildings have become an important means to achieve modern management and energy conservation. Thousands of devices in buildings are managed by multiple edge controllers, which are then connected to the cloud for unified management. This is the basic architecture of existing smart buildings.

[0005] In order to facilitate unified management of the cloud, it is currently managed through the digital base. The digital base is an integrated infrastructure framework designed to uniformly manage various engineering projects in smart buildings. It realizes intelligent management of the entire life cycle of data by connecting devices to a unified management platform. For example, edge sensors obtain real-time data inside the building, such as temperature, humidity, energy consumption, etc., from sensors and other devices. The data is transmitted to the cloud through IoT communication protocols (such as MQTT, CoAP, HTTP). The cloud is based on cloud computing or edge computing to process large amounts of real-time data and store historical data. The cloud generates optimization plans or abnormal alarms through machine learning or big data analysis technology.

[0006] It can be seen that the edge controller in the prior art undertakes an extremely important task in this process. The current fault detection of the IoT system in the prior art mainly detects the failure of the equipment. When the equipment fails, the edge controller reports to the cloud, and the cloud notifies the operation and maintenance personnel to replace the equipment or repair the equipment.

[0007] But in fact, the more common faults in the IoT system are not hardware faults of the device itself. Since the edge controller controls many devices, all devices of an edge controller suddenly access the edge controller concurrently within a period of time, which will cause the edge controller program to freeze and network congestion. These are common faults in the IoT. At present, when these faults need to be maintained, users need to discover the problems and then send operation and maintenance personnel to deal with them before they can be solved. This leads to serious lags and also makes the stability of the existing IoT poor.

[0008] Therefore, a method for handling failures in a multi-device system that are not device hardware failures is a technical problem to be solved. Summary of the invention

[0009] In order to solve the technical problem that non-hardware failures of equipment in the prior art are also handled manually, the present invention proposes a failure handling method for a multi-equipment system.

[0010] The fault handling method of the multi-device system of the present invention includes a plurality of edge controllers for controlling a plurality of the devices and a cloud for managing the plurality of edge controllers. Specifically, the fault handling method includes:

[0011] The cloud regularly trains and updates the fault prediction model of the edge controller and its devices based on historical data;

[0012] After obtaining the latest fault prediction model, the cloud regularly predicts the fault level of the edge controller and its equipment; the fault level includes: at least one of normal state, quality degradation state, quality warning state and fault state;

[0013] When the fault level is in a quality degradation state or a quality warning state, the cloud and the edge controller verify the fault through the corresponding heartbeat communication mechanism. If the corresponding fault is verified to exist, the cloud restarts and recovers the control program of the corresponding device in the edge controller.

[0014] Furthermore, when the fault level is a fault state, the cloud restarts and recovers the control program of the corresponding device in the edge controller.

[0015] Furthermore, when the fault level is in a quality degradation state, the corresponding edge controller periodically generates longitudinal heartbeat information and reports to the cloud. If the cloud does not receive longitudinal heartbeat information less than a preset number of times within a preset time, a restart self-recovery script program is sent to the corresponding edge controller.

[0016] Further, when the fault level is a quality warning state, the cloud sends a lateral heartbeat control instruction to at least one adjacent edge controller of the corresponding edge controller;

[0017] The adjacent edge controller periodically generates horizontal heartbeat information and sends it to the corresponding edge controller. Based on the reply information of the corresponding edge controller, it generates vertical heartbeat information and reports it to the cloud. When the number of replies of the corresponding edge controller is less than the preset number within the preset time, the cloud sends a restart self-recovery script program to the corresponding edge controller or to the corresponding edge controller through the adjacent edge controller.

[0018] Furthermore, the horizontal heartbeat information is sent in a broadcasting manner.

[0019] Furthermore, the longitudinal heartbeat information includes the communication time, the unique identifier of the corresponding edge controller, the system load of the corresponding edge controller, and the network bandwidth of the device.

[0020] Furthermore, the edge controller manages the control program of the device by creating a container corresponding to the device one by one.

[0021] Furthermore, the restart self-recovery script program creates an image container of the corresponding device in the corresponding edge controller, and configures the control program in the image container according to the latest data of the device to adapt to the running status of the device.

[0022] Furthermore, the historical data stored in the cloud is time series data, which includes collection time, processing performance data of the edge controller, and network performance data of the device and the edge server.

[0023] Furthermore, the processing performance data of the device includes at least one of a response speed of the device, a processing time of the device, and a stability of the device.

[0024] Furthermore, the network performance data of the edge server and the cloud corresponding to the device includes: at least one of network fluctuation data and network congestion data.

[0025] The present invention can detect software failures of equipment in a timely manner and perform timely recovery. Moreover, the present invention further verifies partial network failures so as to recover the failures in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention is described in detail below with reference to the embodiments and accompanying drawings, wherein:

[0027] Figure 1 It is a system architecture diagram of an embodiment of the present invention.

[0028] Figure 2 is a flow chart of an embodiment of the present invention.

[0029] Figure 3 It is a schematic diagram of the processing flow of an embodiment of the present invention.

[0030] Figure 4 is a detailed flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] Thus, a feature indicated in this specification will be used to illustrate one of the features of an embodiment of the present invention, rather than implying that each embodiment of the present invention must have the described feature. In addition, it should be noted that this specification describes many features. Although some features can be combined together to illustrate possible system designs, these features can also be used in other combinations that are not explicitly described. Thus, unless otherwise stated, the described combinations are not intended to be limiting.

[0033] like Figure 1 As shown, the multi-device system of the present invention is usually a three-layer architecture, with the bottom layer being the device layer, the middle layer being the edge controller layer, and the upper layer being the cloud. The edge controller can use a digital base.

[0034] Multiple edge controllers in the edge controller layer manage a batch of devices respectively. The edge controllers store the control programs of the corresponding devices, and the cloud manages all edge controllers of the system.

[0035] like Figure 2 As shown, the multi-device fault handling method of the present invention includes the following steps.

[0036] The cloud regularly trains and updates the fault prediction models of edge controllers and their devices based on historical data;

[0037] After obtaining the latest fault prediction model, the cloud regularly predicts the fault level of the edge controller and its equipment; the fault level includes at least one of normal state, quality degradation state, quality warning state and fault state.

[0038] When the fault level is in a quality degradation state or a quality warning state, the cloud and edge controller verify the fault through the corresponding heartbeat communication mechanism. If the corresponding fault is verified to exist, the cloud restarts and recovers the control program of the corresponding device in the edge controller.

[0039] The present invention lists four common possible situations. One is the normal state, which means that the edge controller and the device are fault-free and no processing is required. One is the quality degradation state, which is usually some minor faults, such as a relatively slight increase in load, decrease in flow, increase in delay, packet loss, etc. on the edge controller. One is the quality warning state, which is a relatively serious load increase, decrease in flow, increase in delay, packet loss, etc. Another is the fault state, which means that the edge controller has crashed at this moment, or the corresponding control program has been stuck.

[0040] Through model training, the present invention not only distinguishes the normal state and the fault state, but also separately extracts two fault states, one is the quality degradation state, and the other is the quality warning state. Although these two states are states where the fault has not yet occurred compared to the fault state, if they are not handled in time, they are likely to cause the edge controller program to get stuck or crash. The present invention further verifies these two faults through a heartbeat mechanism, and restarts and recovers in time to ensure the stability of the multi-device system operation.

[0041] In a specific embodiment, when the fault level is a fault state, the cloud restarts and recovers the control program of the corresponding device in the edge controller.

[0042] The fault status of the model can be located on specific devices and edge controllers, so the control program of the faulty device can be restarted and restored. The normal operation of the device can be restored automatically and in a timely manner by detecting the fault status.

[0043] In one embodiment, when the fault level is in a quality degradation state, the corresponding edge controller periodically generates longitudinal heartbeat information and reports to the cloud. If the cloud does not receive longitudinal heartbeat information less than a preset number of times within a preset time, a restart self-recovery script program is sent to the corresponding edge controller.

[0044] When the quality is degraded, it means that there is a certain increase in load, decrease in traffic, increase in latency or packet loss between the device and the edge controller. After the fault prediction model predicts the possible situation, the edge controller begins to actively report to the cloud, and observes the load and network status of the edge controller through the longitudinal heartbeat. Once there is a risk of instability, it will immediately restart the control program of the corresponding device, so that the load of the edge controller can be reduced accordingly, or reduce sudden concurrent conditions, etc.

[0045] In a specific embodiment, when the fault level is a quality warning state, the following measures are taken.

[0046] The cloud sends a lateral heartbeat control instruction to at least one adjacent edge controller of the corresponding edge controller;

[0047] The adjacent edge controller periodically generates horizontal heartbeat information and sends it to the corresponding edge controller. Based on the reply information of the corresponding edge controller, it generates vertical heartbeat information and reports it to the cloud.

[0048] When the corresponding edge controller replies less than the preset number of times within the preset time, the cloud sends a restart self-recovery script program to the corresponding edge controller or to the corresponding edge controller through the adjacent edge controller.

[0049] The quality warning state predicts that the device has a serious fault, that is, the edge controller is under a large load, causing the control program to freeze, or the communication between the device and the edge controller is not smooth due to network congestion. At this time, the cloud selects a neighboring edge controller to send a horizontal heartbeat control command to the corresponding edge controller. The corresponding edge controller passively reports. Once it is found that the corresponding edge controller does not respond in time, it will promptly send a restart self-recovery script program to the corresponding edge controller through the neighboring edge controller, so that the corresponding edge controller can restart the corresponding control program in time to ensure the stability of the device operation.

[0050] In one embodiment, the horizontal heartbeat information is sent in a broadcast manner.

[0051] When multiple edge controllers and their devices are in a quality warning state, the cloud only needs to select an adjacent edge controller with the smallest load to send a horizontal heartbeat message by broadcasting, so as to collect the reply information of multiple corresponding edge controllers on the mobile phone, and then report it to the cloud in a centralized manner. In addition, for the edge controller that predicts the quality warning state, the horizontal heartbeat message is a passive mode. It does not need to be controlled by the general controller to actively report regularly like the vertical heartbeat message. When the corresponding edge controller receives the horizontal heartbeat message, the communication module of the corresponding edge controller only needs to reply the corresponding data packet to the adjacent edge controller. It can also be executed when the control program of the device crashes or the edge controller is under a large load. Once it is found that the execution also has unstable signs, it will be restarted immediately to ensure the stability of the device operation.

[0052] In the quality warning state, the longitudinal heartbeat information sent by the adjacent edge controller contains data such as the communication time, the unique identifier of the corresponding edge controller, the system load of the corresponding edge controller, and the network bandwidth with the device.

[0053] When the quality is degraded, the vertical heartbeat information reported by the corresponding edge controller is relatively simple, including only the communication time and the unique identifier of the corresponding edge controller.

[0054] The unique identifier of the edge controller referred to in the present invention may be the ID of the edge controller, or the MAC address of the edge controller, etc.

[0055] In a specific embodiment, the edge controller of the present invention manages the control program of the device by creating a container corresponding to the device one by one.

[0056] Managing device control programs through containers has the following advantages.

[0057] The container corresponding to the device shares the operating system kernel of the edge controller and does not need to run a complete operating system instance, so it starts faster and uses fewer resources. Containers can usually be started in seconds, while virtual machines take minutes. The container corresponding to each device runs in an independent environment and is isolated from each other, ensuring high availability and stability of the application. Even if a container fails, it will not affect the operation of other containers, improving the overall reliability of the system. In other words, when the control program of a device is restarted, it will not affect other devices.

[0058] In one embodiment, the restart self-recovery script program creates an image container of the corresponding device in the corresponding edge controller, and configures the control program in the image container according to the latest data of the device to adapt to the operating status of the device.

[0059] The latest data of the device referred to in the present invention may be the latest configuration file of the stored control program, or the operation log file of the device, etc. Through these files, after the image container restarts the control program, the control program can continue the original control of the device. Taking the air conditioner as an example, the opening degree of the electronic expansion valve, target temperature, target humidity, etc. between the restarts of the air conditioner will not all return to the default value state due to the restart, but can continue the previous control on the basis of the restart, thereby ensuring the stability of the system operation.

[0060] In one embodiment, the historical data stored in the cloud is time series data, which includes collection time, processing performance data of the edge controller, and network performance data of the device and the edge server.

[0061] The processing performance data of the device includes: at least one of the response speed of the device, the processing time of the device, and the stability of the device.

[0062] The network performance data of the edge server and the cloud corresponding to the device includes: at least one of network fluctuation data and network congestion data.

[0063] By collecting these historical data as training data, the fault prediction model between the edge controller and the device is trained, so that the fault prediction model can predict the upcoming network and software-related failures between the edge controller and the device, and restart in time to ensure the stability of system operation.

[0064] Taking the Internet of Things system as an example, in addition to collecting business data of devices, the edge server also collects network fluctuation data and network congestion data between the edge server and the device, the device's response speed, device processing time, device stability, etc.

[0065] like Figure 3 , Figure 4As shown in the figure, these data are unified into a piece of data with the collection time as the timestamp in the time series format. Each piece of time series data is classified according to different edge controllers. The time series data format collected by each edge controller (edge ​​gateway node 1 to edge gateway node N) is a long time series, that is, multi-variable input and single variable output, and the output value is the current detection device fault level. For example, the time series data collected by edge server 1 at time t is defined as The data sequence collected by edge server N is Then the length of the time series formed by all n time series data of length h collected by the cloud at time t is

[0066]

[0067] The total length is n*h.

[0068] The input of the fault prediction model in the cloud uses the sliding window method to construct a time series sample. The length is set to d as the size of the sliding window. Assuming that the sliding step is 1, the current cloud time is X t ={x t , x t-1 ,...,x t-d+1}, the time sample of the next moment is X t+1 ={x t+1 , x t , .., x t-d+2}, the fault level label values ​​output are y t and t+1 .

[0069] In a specific embodiment, the fault prediction model of the present invention adopts a three-layer GRU neural network model, which integrates previous network feature information through a fully connected layer to improve the learning ability of the network. Finally, the output of the fully connected layer is used as the input of the softmax classifier to obtain the final fault detection result. In order to prevent overfitting when training the network, Dropout regularization is used to discard some repeated information generated during the training process.

[0070] After that, the historical data collected in the cloud is continuously updated, and the fault prediction model of the present invention can be further optimized and trained. In order to reduce the error between the output result and the real network, the back propagation algorithm can be used to iteratively update the network parameters. And the network weights are optimized layer by layer by gradient descent until the model converges.

[0071] The trained fault prediction model can predict the fault level between the device (device node 1 to device node N) and the edge controller (edge ​​gateway node 1 to edge gateway node N). If the network congestion or the control program of a device is stuck due to excessive data volume between a certain device and the edge controller, it can be restarted in a targeted and timely manner to avoid congestion or stuck. It should be noted that each edge gateway node in the figure corresponds to a device node, which is only a simple example of the present invention. In other embodiments, usually one edge gateway node is connected to multiple device nodes, that is, the edge gateway node and the device node may not have a one-to-one correspondence.

[0072] When the fault level of the device is predicted, the restart self-recovery script program will be pre-migrated and loaded to the edge controller so that the automatic recovery mechanism can be started in time. The self-restart self-recovery script program can automatically restart the edge controller, or restart the control program of the device in the loaded container. This mechanism needs to combine the results of calling the fault prediction model in the cloud to judge the abnormal situation of the device and locate the edge controller where the device is located. After the device fault is determined, the edge controller will start the search strategy to retrieve the available information of the device for restarting and recovering the control program of the device. After that, the image is pulled and the container is created. The image warehouse retrieves the latest controller application image, and uses the image to create a container or batch create containers in the appropriate node area, that is, the restart strategy. Finally, the container information is loaded and recovered. According to the log information of the controller application, the historical data or operation status information is reloaded and configured to the new container, and then it is immediately put online for use.

[0073] The present invention can host the control programs of multiple devices in containers, realize timely detection of faults, timely start of control programs, and ensure that the multi-device system provides stable services, which is of great significance for the high reliability discovery of the Internet of Things system.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A fault handling method for a multi-device system, wherein the multi-device system includes a plurality of edge controllers for controlling a plurality of the devices, and a cloud for managing the plurality of edge controllers, characterized in that: The fault handling method comprises: The cloud regularly trains and updates the fault prediction model of the edge controller and its devices based on historical data; After obtaining the latest fault prediction model, the cloud regularly predicts the fault level of the edge controller and its equipment; the fault level includes: at least one of normal state, quality degradation state, quality warning state and fault state; When the fault level is in a quality degradation state or a quality warning state, the cloud and the edge controller verify the fault through the corresponding heartbeat communication mechanism. If the corresponding fault is verified to exist, the cloud restarts and recovers the control program of the corresponding device in the edge controller.

2. The fault handling method of a multi-device system according to claim 1, characterized in that: When the fault level is a fault state, the cloud restarts and recovers the control program of the corresponding device in the edge controller.

3. The method for troubleshooting a multi-device system according to claim 1, wherein: When the fault level is in a quality degradation state, the corresponding edge controller periodically generates longitudinal heartbeat information and reports to the cloud. If the cloud does not receive longitudinal heartbeat information less than a preset number of times within a preset time, a restart self-recovery script program is sent to the corresponding edge controller.

4. The method for troubleshooting a multi-device system according to claim 1, wherein: When the fault level is a quality warning state, the cloud sends a lateral heartbeat control instruction to at least one adjacent edge controller of the corresponding edge controller; The adjacent edge controller periodically generates horizontal heartbeat information and sends it to the corresponding edge controller. Based on the reply information of the corresponding edge controller, it generates vertical heartbeat information and reports it to the cloud. When the number of replies of the corresponding edge controller is less than the preset number within the preset time, the cloud sends a restart self-recovery script program to the corresponding edge controller or to the corresponding edge controller through the adjacent edge controller.

5. The method for handling a fault in a multi-device system according to claim 4, wherein: The horizontal heartbeat information is sent in a broadcasting manner.

6. The method for handling a fault in a multi-device system according to claim 4, wherein: The longitudinal heartbeat information includes the communication time, the unique identifier of the corresponding edge controller, the system load of the corresponding edge controller, and the network bandwidth of the device.

7. The method for handling a fault in a multi-device system according to claim 1, wherein: The edge controller manages the control program of the device by creating a container corresponding to the device one by one.

8. The method for handling a fault in a multi-device system according to claim 7, wherein: The restart self-recovery script program creates a mirror container of the corresponding device in the corresponding edge controller, and configures the control program in the mirror container according to the latest data of the device to adapt to the running status of the device.

9. The method for handling a fault in a multi-device system according to claim 1, wherein: The historical data stored in the cloud is time series data, which includes collection time, processing performance data of the edge controller, and network performance data of the device and the edge server.

10. The method for handling a fault in a multi-device system according to claim 9, wherein: The processing performance data of the device includes at least one of a response speed of the device, a processing time of the device, and a stability of the device.

11. The method for handling a fault in a multi-device system according to claim 9, wherein: The network performance data of the edge server and the cloud corresponding to the device includes: at least one of network fluctuation data and network congestion data.

Images