Uninterrupted Internet of Things high throughput service system based on digital clone

Abstract

The invention provides an uninterrupted Internet of Things high throughput service system based on digital cloning, and aims to solve the problem that a traditional Internet of Things service system is prone to service interruption during code upgrading and data maintenance and remarkably improve the throughput performance of the system. According to the system, continuous and efficient management of the Internet of Things terminal equipment is realized through three layers of technologies of network data cloning, service process cloning and data persistent storage cloning. The network data cloning layer monitors and copies a network data packet, the service process cloning layer realizes seamless service through process copying and dynamic switching, and the data persistent storage cloning layer ensures reliable storage and consistency of data by utilizing a virtualization or containerization technology. The system can adapt to large-scale Internet of Things application, guarantees 7 * 24-hour uninterrupted operation, supports millisecond-level operation response, and has high reliability and high expansibility.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things and software architecture, and in particular to a service system that provides continuous operation and real-time control of more than one million terminal devices for Internet of Things devices, and specifically to an uninterrupted Internet of Things high-throughput service system based on digital cloning technology, which aims to provide uninterrupted, high-throughput operation capabilities for large-scale Internet of Things applications. Background Art

[0002] With the popularization of mobile network technology and the improvement of chip processing power, the number of IoT terminal devices has exploded. There are many types of devices connected to the IoT network, ranging from home smart devices to industrial-grade monitoring systems. These terminal devices are usually connected to one or more central servers and centrally managed by the IoT service system deployed on the server. Through the IoT service system, users can monitor the status of each device in real time and remotely perform control operations, such as switching devices on and off, adjusting operating parameters, etc.

[0003] However, the current IoT service system for massive devices faces two major challenges: 1) Uninterrupted operation requirements: The terminal devices managed by the IoT service system usually need to run uninterruptedly 7x24 hours, so the service system must ensure uninterrupted operation (service interruption time cannot exceed 1-10 seconds). This high requirement makes it difficult to ensure uninterrupted service system during function upgrades and maintenance. 2) High throughput requirements: Since the service system needs to provide services to a large number of terminal devices and their users at the same time, the communication frequency of the system is extremely high. The number of operations per second (including requests from terminal devices to servers and instructions from servers to terminal devices) can exceed 1,000 times, and the system must meet millisecond-level latency requirements.

[0004] The existing IoT service system architecture is usually unable to continue to provide services while upgrading the code or maintaining the data, which easily leads to service interruptions or reduced throughput. How to break through these bottlenecks based on the existing IoT network architecture and software architecture to achieve an IoT service system with uninterrupted services and high throughput performance has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to solve the deficiencies in the prior art, the present invention proposes an uninterrupted IoT high-throughput service system based on digital cloning. By copying the key states of the IoT service system (including network data transmission and reception, database storage, log recording, etc.) to generate a group of "clone" processes, the system can still maintain uninterrupted operation during code upgrades and data maintenance, and ensure high throughput performance. Specifically, the technical solution of the present invention mainly includes the following three levels: 1) Network data cloning layer: At the network protocol stack level of the IoT service system, by monitoring the network data transmission and reception ports, the network data of the original IoT service system is cloned and forwarded to the local. The data forwarding protocol uses commonly used inter-process communication means (such as Unix Domain Socket, etc.) to ensure low latency of data forwarding. 2) Service process cloning layer: At the operating system level of the server, the core service process of the IoT service system is copied to generate multiple clone processes. Each clone process processes specific data types, such as terminal device status information, instruction data, etc. This design ensures that even if a clone process stops running for some reason, other processes can continue to process, thereby avoiding overall service interruption. 3) Data persistent storage cloning layer: For state information that needs to be persistently stored (such as log records and database records), multiple persistent storage copies are started through operating system virtualization or containerization technology, and the persistent storage requests of the original IoT service system are copied and distributed to different copies for processing. This mechanism not only enhances the reliability of the system, but also reduces the risk of data loss caused by the failure of a single storage node.

[0006] In summary, the system of the present invention overcomes the service interruption problem that may occur in the traditional IoT service system during code upgrades and data maintenance by introducing a digital cloning mechanism, and at the same time greatly improves the throughput performance of the system. The system has low requirements for operating platforms and code changes, is simple to operate, and has strong adaptability, and is particularly suitable for real-time monitoring and management of large-scale IoT devices.

[0007] The specific implementation of the present invention is as follows: The present invention provides an uninterrupted high-throughput Internet of Things service system based on digital cloning technology, which is implemented through the following main levels of technology:

[0008] Network data cloning layer: At the network protocol stack level of the IoT service system, the present invention monitors the network data transceiver ports by introducing code stub technology in the server's operating system. Through stub technology, the system can capture all inbound and outbound network data packets and copy these data packets to multiple clone processes. These cloned data packets are then forwarded to different service process copies through inter-process communication (such as Unix Domain Socket or shared memory), ensuring low latency and high efficiency of data processing. This layer also supports data cloning and forwarding of multiple network protocols (such as TCP, UDP), and can adapt to the data transmission requirements of different types of IoT devices.

[0009] Service process cloning layer: At the server operating system level, the core service process of the IoT service system is replicated to generate multiple clone processes. Each clone process processes a specific data type, such as terminal device status information, command data, etc. This design ensures that even if a clone process stops running for some reason, other processes can continue to process, thus avoiding overall service interruption.

[0010] Data persistence storage cloning layer: For state information that needs to be persistently stored (such as log records and database records), multiple persistent storage copies are started through operating system virtualization or containerization technology, and the persistent storage requests of the original IoT service system are copied and distributed to different copies for processing. This mechanism not only enhances the reliability of the system, but also reduces the risk of data loss caused by the failure of a single storage node.

[0011] In summary, the system of the present invention overcomes the service interruption problem that may occur in the traditional IoT service system during code upgrades and data maintenance by introducing a digital cloning mechanism, and at the same time greatly improves the throughput performance of the system. The system has low requirements for operating platforms and code changes, is simple to operate, and has strong adaptability, and is particularly suitable for real-time monitoring and management of large-scale IoT devices.

[0012] The present invention provides an uninterrupted Internet of Things high-throughput service system based on digital cloning technology, and the specific implementation of the system is achieved through the following main steps.

[0013] S1 Network data cloning: At the network protocol stack level of the IoT service system, the present invention monitors the network data transceiver ports by introducing code stub technology in the server's operating system. Through stub technology, the system can capture all inbound and outbound network data packets and copy these data packets to multiple cloned processes. These cloned data packets are then forwarded to different service process copies through inter-process communication (such as Unix Domain Socket or shared memory), ensuring low latency and high efficiency of data processing. This layer also supports data cloning and forwarding of multiple network protocols (such as TCP, UDP), and can adapt to the data transmission needs of different types of IoT devices.

[0014] S2 Service process cloning: At the operating system level, using the existing process replication technology, the present invention replicates the core service process of the Internet of Things service system to generate multiple independent clone processes. Each clone process is responsible for processing a specific data type or service request. This design allows different processes to process different data streams to reduce the load of a single process and improve the overall throughput of the system. - Specifically, the service process cloning layer can use the Linux fork() system call or other equivalent operating system functions to generate clone processes. By using inter-process communication (such as pipes, message queues, or shared memory), the clone process can efficiently interact with the network data cloning layer. In the process of processing data, each clone process can further improve data processing efficiency through threading or asynchronous programming technology. When the service process needs to be upgraded or maintained, the system can dynamically create a new clone process and gradually switch the data processing tasks to the new process, thereby achieving seamless service switching and system upgrades.

[0015] S3 data persistent storage clone: ​​- In order to ensure the data persistence and reliability of the IoT service system, the present invention creates multiple copies for the persistent storage system through virtualization or containerization technology. In these copies, data storage requests are replicated and distributed to different copies for processing to ensure data security and consistency. - The implementation of the data persistent storage clone layer can be based on Docker containers or KVM virtual machine technology, running independent database or file system instances in each copy. These instances work in pairs with each process of the service process clone layer, receiving and processing persistent storage requests from the process through the API interface. In order to ensure data consistency, this layer also supports distributed consistency protocols (such as Paxos or Raft) for data synchronization and conflict resolution between multiple copies. - In addition, the present invention can also be combined with distributed file systems (such as Ceph or HDFS) to further enhance the scalability and fault tolerance of the persistent storage system and adapt to the needs of large-scale data storage.

[0016] S4 Seamless service switching and upgrading: In order to ensure the uninterrupted operation of the system when the service system is undergoing code upgrades or maintenance, the present invention proposes a seamless service switching mechanism. This mechanism is based on the aforementioned network data cloning layer and service process cloning layer, and gradually transfers data processing tasks from one process to another by dynamically creating and destroying process copies. - In specific implementation, the system can start a new cloned process in advance and forward new network data to the process. When the new cloned process is ready, the old process will stop receiving new requests, but will continue to process currently ongoing tasks until all tasks are completed. In this way, the system achieves seamless switching and upgrading of services, avoiding the service interruption problem that occurs during upgrades in traditional systems.

[0017] S5 System optimization and expansion: The present invention also provides a variety of system optimization methods to further improve the performance and scalability of the system. For example, through the load balancing algorithm, the system can dynamically allocate the processing tasks of each cloned process to prevent some processes from affecting the overall performance due to overload. At the same time, the system architecture of the present invention allows for flexible increase or decrease of cloned processes and persistent storage copies to meet the deployment requirements of IoT of different scales. Through automated monitoring and management tools, the system can automatically adjust resource configuration at runtime to ensure that the best performance state is always maintained.

[0018] In summary, the uninterrupted IoT high-throughput service system based on digital cloning technology provided by the present invention solves the service interruption problem of the existing IoT service system during code upgrades and data maintenance through three-level implementation methods: network data cloning, service process cloning, and data persistent storage cloning, and significantly improves the system's throughput performance and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The overall architecture diagram of the system designed for the present invention.

[0020] Figure 2 This is a diagram of the network data cloning layer architecture designed for the present invention.

[0021] Figure 3 A diagram of the service process cloning layer architecture designed for the present invention.

[0022] Figure 4 The data persistence cloning layer architecture diagram designed for the present invention.

Claims

1. An uninterrupted high-throughput Internet of Things service system based on digital cloning, characterized in that: The system comprises: Network data cloning layer: used to clone and forward network data to the local computer by monitoring the network data receiving and sending ports at the network protocol stack level of the IoT service system; Service process cloning layer: used to generate multiple clone processes by copying the core service process of the IoT service system at the server operating system level, and distribute the data forwarded by the network data cloning layer to these clone processes to process the corresponding data; Data persistent storage cloning layer: used to use the virtualization or containerization technology of the operating system to start multiple persistent storage copies, copy and distribute the persistent storage requests of the IoT service system to different copies to achieve persistent storage of data.

2. The uninterrupted high-throughput Internet of Things service system based on digital cloning according to claim 1 is characterized in that: The network data cloning layer clones and forwards network data on the same computer device to ensure low latency in data forwarding.

3. The uninterrupted high-throughput Internet of Things service system based on digital cloning according to claim 1 or 2, characterized in that: Each clone process of the service process clone layer is only responsible for processing a specific type of data, and separates and processes different types of data to avoid the blockage of single data processing affecting the overall service performance, reduce system blockage, and improve throughput performance.

4. The uninterrupted Internet of Things high-throughput service system based on digital cloning according to any one of claims 1 to 3, characterized in that: The data persistent storage cloning layer exports the service interface of the persistent storage system through containerization technology for use by the cloning process to support the uninterrupted operation of the Internet of Things service system.

5. The uninterrupted Internet of Things high-throughput service system based on digital cloning according to any one of claims 1 to 4, characterized in that: The system introduces a state replication mechanism within the same operating system to enable multiple copies of the IoT service system to run simultaneously, and can continue to run uninterruptedly when the original service stops.

Images