A highly reliable data acquisition method for industrial Internet of Things

Abstract

The present invention discloses a highly reliable data collection method for industrial Internet of Things. In the face of downstream systems, by monitoring data forwarding behavior, abnormal data is stored locally, and when communication is normal, the stored data is resent to the downstream system; in the face of data perception, by establishing multiple service instances, adding a heartbeat mechanism and a task dynamic allocation algorithm based on the number of instances, when the instance is interrupted, tasks can be reallocated to ensure that the data collection function continues to operate normally. The advantages are: by performing local storage when data forwarding is abnormal and automatically reissuing when communication is restored, the degree of manual intervention in the data collection process is reduced, and the ability to cope with network interruptions during data collection is improved. During data collection, a protocol-based distributed service construction and task allocation method is adopted to make full use of collection resources. When an instance is abnormal, task reallocation can be automatically realized. The system has high availability and ensures the reliability of the data collection function.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial Internet of Things, and in particular to a high-reliability data acquisition method for industrial Internet of Things. Background Art

[0002] The network connection in the internal industrial network or computer room of an enterprise is often interrupted, such as power outages in the workshop, network signal fluctuations in some areas, etc. The occurrence of the above situations may cause data transmission failure, loss of connection between distributed service instances, abnormal processing of business logic programs, etc., which brings challenges to the reliability of industrial Internet of Things data collection and affects the enterprise's business decision-making and analysis based on data.

[0003] To address the above issues, the current common method is to manually re-record network anomaly monitoring logs and data to ensure that data is as continuous as possible after a network anomaly occurs. However, the accuracy and timeliness of data re-recording vary from person to person, and if the amount of data is large, the re-recording operation is almost impossible to complete.

[0004] As enterprises' digital awareness increases, the amount of data collected by the Industrial Internet of Things is also increasing continuously, and it is common to see daily data in the tens of millions. If the existing methods of abnormal log monitoring, manual re-entry and backup systems are still used, it will be difficult to ensure the real-time, continuity and reliability of the data, which will affect the company's business decisions. Summary of the invention

[0005] The object of the present invention is to provide a high-reliability data acquisition method for industrial Internet of Things, thereby solving the above-mentioned problems existing in the prior art.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A high-reliability data collection method for the industrial Internet of Things. Facing the downstream system, it monitors the data forwarding behavior and stores the abnormal data locally. When the communication is normal, the stored data is resent to the downstream system. Facing the data perception level, by establishing multiple service instances, adding a heartbeat mechanism and a task dynamic allocation algorithm based on the number of instances, tasks can be reallocated when an instance is interrupted, ensuring that the data collection function continues to operate normally.

[0008] Preferably, reissuing data to the downstream system specifically includes the following steps:

[0009] A1. Capture exceptions in the data forwarding program. When data exceptions are captured, it is considered that data transmission has failed.

[0010] A2. Set a unified network status mark and use the heartbeat method to request the downstream system once every N seconds. When data is abnormal, the network status mark is set to abnormal first. Before forwarding data, the data forwarding program first determines the status of the network status mark. When the network status mark is found to be abnormal, the abnormal data will be written to the local database. When the network communication of the downstream system is detected to be normal, the network status mark is set to normal, and the abnormal data stored in the local database is sent to the downstream system.

[0011] A3. The scheduler reads the local database every M seconds. When data is found and the network is normal, it forwards the data in batches, querying X items in each batch, recording the forwarded data ID, and deleting the data from the local database after the data is successfully forwarded.

[0012] Preferably, the local database is stored in a single file and data is written in an insert mode; the database storage design is based on the application scenario of the industrial Internet of Things, using fields such as path, attribute, collection value and timestamp; the path represents a unique mark; the attribute corresponds to the collection field name; the collection value is the actual read value, and the timestamp is the time when the data is stored.

[0013] Preferably, the thread occupancy time for limiting the forwarding behavior cannot exceed K seconds. Once the time is exceeded, the data forwarding is considered abnormal.

[0014] Preferably, the task allocation for the data perception layer specifically includes the following steps:

[0015] B1. Each protocol builds a separate collection system. The collection system adopts a distributed deployment solution. Each protocol uses a unified handle to identify the service instance. When the service is started, all protocol services register the service instance in the registration center. The registration center can obtain the server IP and process port number of the service instance and distinguish the service instances. After registration, each instance will decide its own collection task.

[0016] B2. Obtain the instance list of all protocol deployments from the registration center according to the unified handle, and obtain the position of the current instance in the instance list according to the server IP and process port number; all protocol collection items are uniformly configured in the database, and a unique digital tag is assigned to each item; and based on the task dynamic allocation algorithm, the collection items are assigned to the task list until all instance lists are traversed;

[0017] B3. The heartbeat in the system synchronizes the instance list of the registration center every Y seconds. When an instance is abnormal, the instances under all protocols will reallocate tasks.

[0018] Preferably, the task dynamic allocation algorithm is specifically:

[0019] The position of the current instance in the list is recorded as I, the instance list is recorded as J; the unique digital identifier assigned to the item point is recorded as T;

[0020] Use the instance to perform T modulo J operation on all the collection item points, and compare the result with I. If the two are the same, the collection item points are assigned to the task list until all the instance lists are traversed.

[0021] The beneficial effects of the present invention are as follows: 1. The method of the present invention reduces the degree of manual intervention in the data collection process by performing local storage when data forwarding is abnormal and automatically reissuing when communication is restored, thereby improving the ability to cope with network interruptions during data collection and making the data continuous and reliable. 2. During data collection, the method of the present invention adopts a protocol-based distributed service construction and task allocation method, which can fully utilize collection resources. When an exception occurs in a certain instance, task reallocation can be automatically realized. The system has high availability and also ensures the reliability of the data collection function. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of data resending to a downstream system in an embodiment of the present invention;

[0023] Figure 2 It is a flowchart of task allocation for the data perception layer in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0025] Embodiment 1

[0026] In this embodiment, in order to solve the problems of data loss, discontinuity and difficulty in re-recording caused by network anomalies during data collection. A high-reliability data collection method for industrial Internet of Things is provided, which is mainly used to deal with the problem of data loss caused by network fluctuations in industrial Internet of Things scenarios. When the network is abnormal, the collection gateway will sense it and store the data in the local database of the system; after the network is restored, the collection gateway will resend the locally stored data in segments to the downstream system. At the same time, the collection gateway service adopts a distributed deployment, and the system will automatically assign the collection tasks of the service according to the number of instances and the position in the distributed instance list. And the heartbeat method is used to continuously monitor the health status of each instance. When an instance is interrupted, the remaining instances will reallocate the collection tasks to ensure the high reliability of data collection.

[0027] The present invention improves the reliability of data collection by transforming the data collection gateway. Facing the downstream system, by monitoring the data forwarding behavior, the abnormal data is stored locally, and when the communication is normal, the stored data is resent to the downstream system; facing the data perception level, by establishing multiple service instances, adding a heartbeat mechanism and a task dynamic allocation algorithm based on the number of instances, when the instance is interrupted, the task can be reallocated to ensure the continuous and normal operation of the data collection function.

[0028] 1. Re-issuance of data to downstream systems (see the attached Figure 1 )

[0029] A1. Exception capture is performed on the data forwarding program. When data exceptions are captured, it is considered that data transmission has failed. During data forwarding, once network fluctuations occur, the transmission thread will be blocked for a long time. Therefore, the thread occupancy time for limiting forwarding behavior cannot exceed K seconds (such as 3 seconds). Once this time is exceeded, data forwarding is considered abnormal.

[0030] The system adopts a defensive programming method to capture exceptions in the data forwarding program. When an exception is captured, it is considered that the data transmission has failed and the data is stored in the local database (SQLite database).

[0031] A2. In order to avoid thread blocking caused by a large number of data requests, the system sets a unified network status mark and uses a heartbeat method to request the downstream system every N seconds (such as 5 seconds); when the data is abnormal, the network status mark is first set to abnormal, and the data forwarding program first determines the status of the network status mark before forwarding the data. When the network status mark is found to be abnormal, the abnormal data will be written to the local database; when the network communication of the downstream system is monitored to be normal, the network status mark is set to normal, and the abnormal data stored in the local database is sent to the downstream system; when the network status mark is set to normal, the data will not be stored in the local database, but the collected data will be directly forwarded to the downstream system.

[0032] A3. The scheduler reads the local database every M seconds (e.g. 10 seconds). When data is found and the network is normal, the data is forwarded in batches, with X items (e.g. 1000 items) queried in each batch, and the forwarded data ID is recorded. After the data is successfully forwarded, it is deleted from the local database. This operation will not cause the local database file to accumulate, nor will it have too much impact on the normal forwarding of data, thus improving the reliability of data.

[0033] In this embodiment, the local database uses a single file for storage, which is convenient for writing and querying. The database uses insert mode to write data, and does not use transactions to improve data writing efficiency. The database storage design is based on the application scenario of the industrial Internet of Things, using fields such as path, attribute, collection value and timestamp; the path represents a unique mark (for example, using a factory, workstation, and equipment to form a unique mark path), which can be constructed according to actual applications; the attribute corresponds to the name of the collected field; the collection value is the actual read value, and the timestamp is the time when the data is stored.

[0034] 2. Task allocation for data perception

[0035] When the data collection system is deployed in a distributed manner, a master-slave solution is usually adopted. When a problem occurs in the master system, the backup system is activated. This method will cause a certain waste of resources. In order to ensure availability, multiple computer rooms may be deployed, which is very expensive. According to the characteristics of industrial Internet of Things data collection, the present invention proposes a distributed service construction and task allocation method based on a protocol, see Attachment 1. Figure 2 , the specific process is:

[0036] B1. Each protocol builds a separate collection system. The collection system adopts a distributed deployment solution. Each protocol uses a unified handle to identify the service instance. When the service is started, all protocol services register the service instance in the registration center. The registration center can obtain the server IP and process port number of the service instance and distinguish the service instances. After registration, each instance will decide its own collection task.

[0037] B2. First, according to the unified handle, obtain the instance list of all protocol deployments from the registration center, and obtain the position of the current instance in the instance list according to the server IP and process port number; all protocol collection items are uniformly configured in the database, and a unique digital tag is assigned to each item; and based on the task dynamic allocation algorithm, the collection items are assigned to the task list until all instance lists are traversed;

[0038] The specific task dynamic allocation algorithm is as follows:

[0039] The position of the current instance in the list is recorded as I, and the instance list is recorded as J; the unique digital identifier assigned to the item point is recorded as T; use the instance to perform T modulo J operation on all the collection item points, and compare the result with I. If the two are the same, the collection item point is assigned to the task list until all the instance lists are traversed.

[0040] B3. There is a heartbeat inside the system. The heartbeat synchronizes the instance list of the registration center every Y seconds (such as 2 seconds). When an instance has an abnormality, such as network interruption or system crash, the number of instances will change, and the instances under all protocols will reallocate tasks.

[0041] By adopting the above steps, the resources of the distributed system can be fully utilized, and due to the miniaturization of service instances, a high server configuration is not required, and cross-computer room deployment can be easily achieved. Even cloud services can be used directly, which improves the reliability of data collection.

[0042] Embodiment 2

[0043] In this embodiment, the process of the method of the present invention is explained with reference to specific examples.

[0044] The system collects OPCUA protocol data, including 30 data points, and sends the data points once a second. After data collection, it is forwarded to the downstream system. The collection gateway service instances are deployed in 3 computer rooms and registered to the Nacos service registration center.

[0045] First, start the system and warm it up. After executing the collection task for 30 minutes, disconnect the network of the collection gateway cluster and the downstream system. At the same time, monitor the collection gateway operation log and find that the forwarding is abnormal. Query the SQLite data and find that there are data records, and 10 records are added per second. Since the collection tasks are evenly distributed among the three instances, the number is consistent with the number sent. The local data storage is normal and lasts for 10 minutes. Restore the network connection and continue to monitor the SQLite database. The data gradually decreases. According to the forwarding log, the data forwarding is normal.

[0046] Select a service instance and interrupt the network with the registration center. After a period of time, the other two services detect the change in the number of instances in the registration center and reallocate the collection tasks. The collection tasks of a single instance are increased from 10 data points to 15, and the collection tasks are executed normally. After the network is restored, the interrupted instance is re-registered with the registration center, and the collection tasks are reallocated. The collection tasks of a single instance are adjusted from 15 data points to 10.

[0047] The above two network interruption operations were performed at the same time. According to log analysis, the data that was not successfully forwarded was stored normally locally. The remaining two service instances reallocated the collection tasks and continued to store the collected data in the local SQLite database file for 10 minutes. The interrupted network was restored, the instances reallocated the collection tasks, and the data collection and forwarding functions were performed normally. The SQLite database data was monitored and the data was reissued normally. The entire system was running normally.

[0048] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0049] The present invention discloses a highly reliable data collection method for industrial Internet of Things. The method of the present invention reduces the degree of manual intervention in the data collection process by performing local storage when data forwarding is abnormal and automatically reissuing when communication is restored, thereby improving the ability to cope with network interruptions during data collection and making data continuous and reliable. During data collection, the method of the present invention adopts a protocol-based distributed service construction and task allocation method, which can make full use of collection resources. When an exception occurs in a certain instance, task reallocation can be automatically realized. The system has high availability and also ensures the reliability of the data collection function.

[0050] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A high-reliability data acquisition method for industrial Internet of Things, characterized by: For downstream systems, data forwarding behavior is monitored and abnormal data is stored locally. When communication is normal, the stored data is resent to the downstream system. For data perception, multiple service instances are established, and a heartbeat mechanism and a task dynamic allocation algorithm based on the number of instances are added. When an instance is interrupted, tasks can be reallocated to ensure that the data collection function continues to operate normally. The specific steps for reissuing data to the downstream system include the following: A1. Exception capture is performed on the data forwarding program. When data exception is captured, it is considered that data transmission fails. The thread occupancy time of the forwarding behavior cannot exceed K seconds. Once this time is exceeded, data forwarding is considered abnormal. A2. Set a unified network status mark and use the heartbeat method to request the downstream system once every N seconds. When data is abnormal, the network status mark is set to abnormal first. Before forwarding data, the data forwarding program first determines the status of the network status mark. When the network status mark is found to be abnormal, the abnormal data will be written to the local database. When the network communication of the downstream system is detected to be normal, the network status mark is set to normal, and the abnormal data stored in the local database is sent to the downstream system. A3. The scheduler reads the local database every M seconds. When data is found and the network is normal, it forwards the data in batches, querying X items in each batch, recording the forwarded data ID, and deleting the data from the local database after the data is successfully forwarded. The task allocation for data perception layer includes the following steps: B1. Each protocol builds a separate collection system. The collection system adopts a distributed deployment solution. Each protocol uses a unified handle to identify the service instance. When the service is started, all protocol services register the service instance in the registration center. The registration center can obtain the server IP and process port number of the service instance and distinguish the service instances. After registration, each instance will decide its own collection task. B2. Obtain the instance list of all protocol deployments from the registration center according to the unified handle, and obtain the position of the current instance in the instance list according to the server IP and process port number; all protocol collection items are uniformly configured in the database, and a unique digital tag is assigned to each item; and based on the task dynamic allocation algorithm, the collection items are assigned to the task list until all instance lists are traversed; The task dynamic allocation algorithm is specifically as follows: The position of the current instance in the list is recorded as I, the instance list is recorded as J; the unique digital identifier assigned to the item point is recorded as T; Use the instance to perform T modulo J operation on all the collection item points, and compare the result with I. If the two are the same, assign the collection item points to the task list until all instance lists are traversed; B3. The heartbeat in the system synchronizes the instance list of the registration center every Y seconds. When an instance is abnormal, all instances under the protocol will reallocate tasks.

2. The high-reliability data acquisition method for industrial Internet of Things according to claim 1 is characterized in that: The local database uses a single file for storage and inserts data into it. The database storage design is based on the application scenario of the Industrial Internet of Things and uses fields such as path, attribute, collection value, and timestamp. The path represents a unique tag; the attribute corresponds to the collected field name; The collected value is the actual value read, and the timestamp is the time when the data is stored.

Images