Internet-of-things data full-amount high-frequency real-time acquisition method and system

Abstract

The invention discloses an Internet of Things data full-amount high-frequency real-time acquisition method and system, and the method comprises the steps: planning a comprehensive architecture comprising a data acquisition node, a log collection node, a service registration center and message middleware, and guaranteeing the efficient cooperation of all assemblies; deploying a log collection tool to each collection node and the key service component, and capturing a system log, an operation log and an error log in real time; eureka is deployed as a service registration center, so that each acquisition node and each service component are registered to the center when being started, and service discovery and management are realized; high-performance message middleware is deployed to serve as a center of data and log transmission, and high-concurrency and low-delay data exchange is supported; deploying a backup center and a log storage system in multiple regions, and realizing remote backup and synchronization of data and logs through a multi-copy mechanism of message middleware; based on the load condition of the message queue, the number of collecting and processing nodes is dynamically adjusted, and elastic expansion and contraction of resources are achieved. It can be seen that full-amount high-frequency real-time collection can be conducted on the Internet of Things data, real-time and accurate collection and efficient processing of physical world information are achieved by integrating Internet of Things equipment, a sensor network, a wireless communication technology and a data processing technology, and key data support is provided for Internet of Things application.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for real-time collection of full-volume, high-frequency, and Internet of Things data. Background Art

[0002] Currently, with the rapid development of IoT technology, the number of devices has exploded, and the amount of data generated has also increased exponentially. Traditional data collection methods often face problems such as high data processing latency, poor system scalability, and high risk of data loss. Especially for scenarios that require real-time processing and analysis, how to achieve high-frequency, low-latency data collection while ensuring data integrity has become a technical problem that needs to be solved urgently. In addition, high system availability and persistent data storage are also key to ensuring service continuity and data security. Summary of the invention

[0003] The present invention provides a method and system for real-time collection of full-volume, high-frequency IoT data. By integrating IoT devices, sensor networks, wireless communication technology and data processing technology, real-time, accurate collection and efficient processing of physical world information can be achieved, providing key data support for IoT applications.

[0004] In order to solve the above technical problems, the first aspect of the present invention discloses a method for real-time collection of full-volume, high-frequency IoT data, the method comprising:

[0005] Plan a comprehensive architecture that includes data collection nodes, log collection nodes, service registration center, and message middleware to ensure efficient collaboration among components;

[0006] Deploy log collection tools to each collection node and key service components to capture system logs, operation logs, and error logs in real time;

[0007] Deploy Eureka as the service registration center (Eureka is one of a series of projects in Netflix's open source microservice framework), so that each collection node and service component registers with the center when starting to achieve service discovery and management;

[0008] Deploy high-performance message middleware as the hub for data and log transmission, supporting high-concurrency and low-latency data exchange;

[0009] Deploy backup centers and log storage systems in multiple regions, and use the multi-copy mechanism of the message middleware to achieve remote backup and synchronization of data and logs;

[0010] Based on the load of the message queue, the number of collection and processing nodes is dynamically adjusted to achieve elastic scaling of resources.

[0011] As an optional implementation, in the first aspect of the present invention, after deploying the log collection tool to each collection node and key service component to capture system logs, operation logs and error logs in real time, the method further includes:

[0012] Log processing and analysis: The collected logs are transmitted to the central log processing system through the message middleware for log parsing, archiving and real-time analysis to provide decision support for system operation and maintenance.

[0013] As an optional implementation, in the first aspect of the present invention, after deploying Eureka as a service registration center and allowing each collection node and service component to register with the center when starting up and realizing service discovery and management, the method further includes:

[0014] Health check and monitoring: Use the health check function of the service registration center to regularly check the status of each service node. Combined with the monitoring suite, it can monitor the system health in real time and promptly warn of potential problems.

[0015] As an optional implementation, in the first aspect of the present invention, as a hub for data and log transmission, after supporting high-concurrency, low-latency data exchange, the method further includes:

[0016] Data flow management, design the topic structure of the message middleware (topic is a collection of messages of a certain type, used to store messages of different types), implement data classification and routing, ensure the decoupling and efficiency of data collection and processing processes, and support data retry and dead letter processing mechanisms to enhance the reliability of data transmission.

[0017] As an optional implementation, in the first aspect of the present invention, after dynamically adjusting the number of acquisition and processing nodes based on the load of the message queue to achieve elastic scaling of resources, the method further includes:

[0018] Performance tuning and monitoring feedback loop, continuously monitor system performance, adjust optimization strategies based on log analysis and monitoring data feedback, including message queue parameters and service configuration.

[0019] The second aspect of the present invention discloses a system for real-time collection of full-volume, high-frequency IoT data, the system comprising:

[0020] Network design module, used to design distributed acquisition networks;

[0021] Log collection module, used to capture system logs, operation logs and error logs;

[0022] The service registration module is used to deploy the service registration center, allowing each collection node and service component to register with the center when starting;

[0023] The message deployment module is used to deploy high-performance message middleware as the hub for data and log transmission;

[0024] The synchronization module is used to deploy backup centers and log storage systems in multiple regions, and to achieve remote backup and synchronization of data and logs through the multi-copy mechanism of the message middleware;

[0025] The adjustment module is used to dynamically adjust the number of collection and processing nodes to achieve elastic scaling of resources.

[0026] The third aspect of the present invention discloses a device for real-time collection of full-volume, high-frequency IoT data, the device comprising the collection system as described above, and further comprising:

[0027] Terminal devices and cloud platform gateways;

[0028] The terminal device is located upstream of the collection system, and the collection system is used to collect data and logs of the terminal device in real time;

[0029] The cloud platform gateway is located downstream of the collection system, and the collection system uploads the collection data, alarm data and activity status data to the cloud platform gateway for backup and synchronization.

[0030] As an optional implementation, in the third aspect of the present invention, the device further includes a local workstation;

[0031] The local workstation and the cloud platform gateway are connected in parallel downstream of the acquisition system;

[0032] When the acquisition system uploads the acquisition data, alarm data and activity status data to the cloud platform gateway for backup and synchronization, it also transmits the acquisition data, alarm data and activity status data to the local workstation for real-time monitoring.

[0033] A fourth aspect of the present invention discloses another device for real-time collection of full-volume, high-frequency IoT data, the device comprising:

[0034] A memory storing executable program code;

[0035] a processor coupled to the memory;

[0036] The processor calls the executable program code stored in the memory to execute the method for real-time collection of full-volume, high-frequency IoT data disclosed in the first aspect of the present invention.

[0037] The fifth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the method for real-time collection of full-amount high-frequency IoT data disclosed in the first aspect of the present invention.

[0038] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0039] In the embodiment of the present invention, a comprehensive architecture including data collection nodes, log collection nodes, service registration center and message middleware is planned to ensure efficient collaboration between various components; log collection tools are deployed to each collection node and key service components to capture system logs, operation logs and error logs in real time; Eureka is deployed as a service registration center to allow each collection node and service component to register with the center at startup to achieve service discovery and management; high-performance message middleware is deployed as the hub of data and log transmission to support high concurrency and low-latency data exchange; backup centers and log storage systems are deployed in multiple regions to achieve remote backup and synchronization of data and logs through the multi-copy mechanism of message middleware; based on the load of the message queue, the number of collection and processing nodes is dynamically adjusted to achieve elastic scaling of resources. It can be seen that the implementation of the present invention can collect the full amount of high-frequency real-time data of the Internet of Things, and realize the real-time, accurate collection and efficient processing of physical world information by integrating Internet of Things devices, sensor networks, wireless communication technologies and data processing technologies, providing key data support for Internet of Things applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the steps of a method for real-time collection of full-volume, high-frequency IoT data disclosed in an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a process for real-time collection of full-volume, high-frequency IoT data disclosed in an embodiment of the present invention;

[0042] Figure 3 It is a dependency graph of alarm triggering rules of a method for real-time collection of full-volume, high-frequency IoT data disclosed in an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the structure of a device for real-time collection of full-volume, high-frequency IoT data disclosed in an embodiment of the present invention;

[0044] Figure 5 It is a structural schematic diagram of another device for real-time collection of full-volume and high-frequency IoT data disclosed in an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of the structure of another device for real-time collection of full-volume, high-frequency IoT data disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0047] The technical solution of the present invention is described in detail below with specific embodiments.

[0048] The present invention discloses a method and system for real-time collection of full-volume, high-frequency IoT data, which can realize real-time, accurate collection and efficient processing of physical world information by integrating IoT devices, sensor networks, wireless communication technology and data processing technology, and provide key data support for IoT applications. The following are detailed descriptions.

[0049] Example 1

[0050] Please also read Figure 1 and Figure 2 , Figure 1 1 is a schematic diagram of the steps of a method for real-time collection of full-volume, high-frequency IoT data disclosed in an embodiment of the present invention. Figure 2 The flowchart of a method for real-time collection of full-volume, high-frequency data of the Internet of Things disclosed in an embodiment of the present invention is shown in FIG. The method for real-time collection of full-volume, high-frequency data of the Internet of Things can be applied to a system for real-time collection of full-volume, high-frequency data of the Internet of Things, wherein the real-time collection of full-volume, high-frequency data of the Internet of Things can be integrated in a local server or a cloud server, which is not limited in the embodiment of the present invention. Figure 1 As shown, the method for real-time collection of full-volume high-frequency IoT data may include the following operations:

[0051] S101. System architecture design and deployment, design of distributed collection network.

[0052] In the embodiment of the present invention, a comprehensive architecture including data collection nodes, log collection nodes, service registration center and message middleware is planned to ensure efficient collaboration among various components.

[0053] S102, log collection mechanism, integrated log collection tool.

[0054] In the embodiment of the present invention, a log collection tool is deployed to each collection node and key service component to capture system logs, operation logs and error logs in real time.

[0055] In an embodiment of the present invention, the log collection tool includes at least one of Elasticsearch (an open source distributed search engine that can be used to implement search, log statistics, analysis, system monitoring and other functions), Logstash (a free and open source server-side data processing pipeline that can collect data from multiple sources, transform data, and then send the data to a user-specified repository), and Kibana (an Elasticsearch-based platform that can perform comprehensive and thorough analysis of any data from any source to achieve observability, security and search).

[0056] For example, when processing and analyzing logs, the collected logs are transmitted to the central log processing system (Elasticsearch) through the message middleware for log parsing, archiving and real-time analysis to provide decision support for system operation and maintenance.

[0057] S103. Service registration and supervision, implementation of service registration center.

[0058] In the embodiment of the present invention, Eureka is deployed as a service registration center, so that each collection node and service component registers with the center when starting up to achieve service discovery and management.

[0059] In the embodiment of the present invention, after deploying Eureka as a service registration center and allowing each collection node and service component to register with the center when starting up and realizing service discovery and management, the method further includes:

[0060] Health check and monitoring: Use the health check function of the service registration center to regularly check the status of each service node. Combined with the Prometheus+Grafana monitoring suite, monitor the system health in real time and warn of potential problems in a timely manner. Prometheus is an open source monitoring software developed by Google. It collects data from remote machines through the HTTP protocol through the exporter installed on the remote machine and stores it in the local time series database. Grafana is an open source data visualization tool that can obtain data from multiple data sources and create, explore and share all data in a beautiful and flexible data panel.

[0061] S104, message middleware integration, message queue deployment.

[0062] In the embodiment of the present invention, high-performance message middleware is deployed as the hub of data and log transmission, supporting high-concurrency and low-latency data exchange.

[0063] For example, the high-performance message middleware of the present invention is Apache Kafka and RabbitMQ, wherein Apache Kafka is a distributed data stream processing platform that can publish, subscribe, store and process data streams in real time; RabbitMQ is a communication method between processes or between different threads of the same process, and the software storage queue is used to process a series of inputs.

[0064] After deploying high-performance message middleware as the hub for data and log transmission to support high-concurrency and low-latency data exchange, the method further includes:

[0065] Data flow management, design the topic structure of the message middleware, implement data classification and routing, ensure the decoupling and efficiency of data collection and processing processes, and support data retry and dead letter processing mechanisms to enhance the reliability of data transmission.

[0066] S105. Establish and optimize the disaster recovery backup system, and synchronize data and logs.

[0067] In the embodiment of the present invention, backup centers and log storage systems are deployed in multiple regions, and off-site backup and synchronization of data and logs are achieved through the multi-copy mechanism of the message middleware.

[0068] S106, high availability and performance optimization, elastic scaling strategy.

[0069] In the embodiment of the present invention, the number of acquisition and processing nodes is dynamically adjusted based on the load of the message queue to achieve elastic scaling of resources.

[0070] It also includes performance tuning and monitoring feedback loops, continuously monitoring system performance, and adjusting optimization strategies based on log analysis and monitoring data feedback, including message queue parameters and service configuration.

[0071] It can be seen that the method for real-time collection of full-volume, high-frequency, and Internet of Things data described in the embodiment of the present invention can achieve real-time, accurate collection and efficient processing of physical world information by integrating Internet of Things devices, sensor networks, wireless communication technologies, and data processing technologies, thereby providing key data support for Internet of Things applications.

[0072] Example 2

[0073] Combination Figure 2 , and the collection method mentioned in Example 1, the embodiment of the present invention discloses a system for real-time collection of full-volume high-frequency IoT data, the system comprising:

[0074] Network design module, used to design distributed acquisition networks;

[0075] Log collection module, used to capture system logs, operation logs and error logs;

[0076] The service registration module is used to deploy the service registration center, allowing each collection node and service component to register with the center when starting;

[0077] The message deployment module is used to deploy high-performance message middleware as the hub for data and log transmission;

[0078] The synchronization module is used to deploy backup centers and log storage systems in multiple regions, and to achieve remote backup and synchronization of data and logs through the multi-copy mechanism of the message middleware;

[0079] The adjustment module is used to dynamically adjust the number of collection and processing nodes to achieve elastic scaling of resources.

[0080] Specifically, the network design module is used to execute step S101 in Example 1, plan a comprehensive architecture including data collection nodes, log collection nodes, service registration center and message middleware, and ensure efficient collaboration among various components; the log collection module is used to execute step S102 in Example 1, deploy log collection tools to each collection node and key service components, capture system logs, operation logs and error logs in real time, and perform log parsing, archiving and real-time analysis to provide decision support for system operation and maintenance; the service registration module is used to execute step S103 in the embodiment, deploy Eureka as a service registration center, allow each collection node and service component to register with the center at startup, realize service discovery and management, and use the health check function of the service registration center to regularly check the status of each service node, monitor the system health status in real time, and promptly warn of potential problems; the message deployment module is used to execute step S103 in Example 1 Step S104, deploy high-performance message middleware as the hub of data and log transmission, support high-concurrency, low-latency data exchange, and design the topic structure of the message middleware to realize data classification and routing, ensure the decoupling and efficiency of data collection and processing processes, and support mechanisms such as data retry and dead letter processing to enhance the reliability of data transmission; the synchronization module is used to execute step S105 in Example 1, deploy backup centers and log storage systems in multiple regions, and realize off-site backup and synchronization of data and logs through the multi-copy mechanism of the message middleware; the adjustment module is used to execute step S106 in Example 1, dynamically adjust the number of collection and processing nodes based on the load of the message queue, realize elastic scaling of resources, and continuously monitor system performance. According to log analysis and monitoring data feedback, continuously adjust the optimization strategy, including message queue parameters, service configuration, etc., to ensure optimal system performance.

[0081] The acquisition system of this embodiment includes the following core components: load balancing, disaster recovery backup, high-availability services and data persistence, among which load balancing is used to evenly distribute traffic to multiple servers to improve the overall response speed and availability of the acquisition system; the disaster recovery backup involved in the synchronization module includes service disaster recovery and data disaster recovery; the adjustment module will ensure the continuity of the system when involving high-availability services; the data persistence involved in the adjustment module includes data persistence on the platform side and data persistence on the acquisition side.

[0082] In the embodiment of the present invention, load balancing adopts the CDN+H5+LVS+nginx+cluster solution, wherein the full name of CDN is Content Delivery Network, which is a distributed network established and covered on the bearer network and composed of edge node server groups distributed in different areas; the full name of H5 is HTML5, which is the latest version of HTML and supports playing video and audio on mobile devices, drawing graphics, using local storage and SQL data, etc.; the full name of LVS is Linux VirtualServer, which is a virtual server technology based on the Linux operating system for achieving load balancing and high availability; nginx is a lightweight Web server / reverse proxy server and email (IMAP / POP3) proxy server, released under the BSD-like protocol.

[0083] When the business volume is unknown, a single nginx is a good choice. It will not make the system start too low or too high, and can quickly separate dynamic and static resources without using CDN. Clusters can effectively reduce the pressure on a single server and make the system highly available.

[0084] In the embodiment of the present invention, data disaster recovery mainly adopts the master-slave backup + master-slave switching + collection side backup solution on the platform side. InfluxDB and mysql databases are mainly used for data persistence in this architecture; influxDB is a distributed time series database based on TSM technology, which supports efficient writing, querying and visualization. InfluxDB can use or not use cluster solutions as needed; mysql is a relational database management system. The relational database stores data in different tables instead of putting all data in a large warehouse, which increases speed and flexibility. mysql adopts the master-slave + mycat (mycat is a database middleware that can realize read-write separation, data sharding, multi-data source integration and other functions) solution for disaster recovery backup. Collection side backup is mainly used for data backup and disaster recovery when the platform side fails or is inaccessible.

[0085] Service disaster recovery adopts keepalived (primary and standby) + virtual IP + cluster solution; keepalived is a lightweight high-availability solution based on Linux, which can achieve high availability of services or networks; this solution can not only quickly switch to the backup service when the main service crashes or is unreachable, but also provide a unified IP address for all lower-level services, so that upper-level applications do not need to care about the details of the lower-level IP, and the cluster can improve the system throughput while improving the system availability.

[0086] In the embodiment of the present invention, high-availability services include stateless services and stateful services; stateless services use DNS and gateway routing as high-availability solutions. The full name of DNS is Domain Name Server, which is a domain name system responsible for resolving domain names into IP addresses; stateful services include collection-side gateway services, data freezing services, and alarm security services.

[0087] In the embodiment of the present invention, the collection side gateway service specifically includes: when the collection system logs in, the gateway service records which service the collection system corresponds to, so as to issue a command to find the corresponding collection system.

[0088] Define the platform-side gateway service to write the gateway corresponding to the collection system into redis every x seconds, and the cache retention time is y.

[0089] Among them, redis is an open source memory storage data structure server that can be used as a database, cache, and message queue agent.

[0090] When an instruction needs to be sent to the collection system, the corresponding gateway service address is obtained from redis to forward the request.

[0091] When the platform-side gateway service receives the collected data, it also needs to push the data to a specific topic based on the cache information stored in redis by the alarm security service; when the platform-side gateway service receives the collected data, it also needs to push the data to a specific topic based on the cache information stored in redis by the monitoring data real-time push service.

[0092] In the embodiment of the present invention, the data freezing service is currently a single-point service, and the above-mentioned high-availability basic rules can be adopted.

[0093] The horizontal expansion of the alarm security service requires solving the problem of data calculation dependency of measurement point data items. All data of the same dependency link must be processed in the same service. Figure 3 , Figure 3 It is an alarm trigger rule dependency graph of a method for real-time collection of full-volume, high-frequency IoT data disclosed in an embodiment of the present invention; Figure 3The green circles are the measurement point data items for which the alarm trigger rules are set, and the white circles are the measurement point data items for which the alarm trigger rules are not set. The arrows indicate the formula parsing dependencies.

[0094] Among them, the data item of measurement point 1 of Company 1 depends on the data items of measurement points 2 and 3. The data item of measurement point 3 depends on the data item of measurement point 4. The data item of measurement point 5 has no dependency. The data item of measurement point 6 of Company 2 depends on the data item of measurement point 4 of Company 1. The data item of measurement point 7 depends on the data item of measurement point 4.

[0095] When the alarm security service starts, it uses the consistent hash algorithm to hash the company id to which the alarm trigger rule belongs (hash is also called hashing. The basic principle is to transform an input of any length into an output of a fixed length through the hash algorithm). Each alarm security service is only responsible for some companies (Note: 1. Hash the company id. 2. Let the total number of virtual groups be X, and take the remainder of the hash value of 1 divided by X. 3. Each alarm service is responsible for N virtual groups, where N <= X). When the company scope responsible for by the alarm security service changes, rehashing is required.

[0096] When the alarm security service loads the alarm trigger rule for model instantiation, it will parse the dependencies according to the measurement point data items targeted by the model. Then, the parsed data is stored in redis, and the expiration time is set to Y. Among them, the cache key k is the unique key composed of the measurement point + data item, and the value is the set of topic names subscribing to the data item of this measurement point (considering measurement point 4 in the figure). The alarm security service regularly updates its related keys at time X (X < Y) and rewrites the topics it subscribes to into the value.

[0097] The alarm security service subscribes to the file change notification. When the measurement point data item is updated or deleted, in addition to changing the associated alarm model, the corresponding cache in redis also needs to be updated.

[0098] In the embodiment of the present invention, the stateful service also includes a monitoring data real-time push service, a notification push service and an IoT system monitoring service. Among them, the horizontal expansion of the monitoring data real-time push service is the same as that of the alarm security service, and it also faces the problem of data calculation dependency of the measurement point data item. The solution is roughly the same as that of the alarm security service. However, unlike the alarm security service, the alarm security service originates from the alarm trigger rule, while the monitoring data real-time push service originates from the user's subscription to the measurement point. However, their abstract logic is consistent, and only the company id hash of the alarm trigger rule needs to be changed to the user's company hash. In addition, when the user requests to allocate the alarm push service, the archive service will perform a consistent hash on the company to which the customer belongs and then return the corresponding monitoring data real-time push service address information after the hash, so as to minimize the need for the data of a measurement point to be distributed to multiple topics. The horizontal expansion of the notification push service is similar to the monitoring data real-time push service, but it does not care about everything about the measurement point, it only cares about who the destination user is. Therefore, it is only necessary to refer to the monitoring data real-time push service, and then change the hash measurement point belonging formula to a direct hash company id. The IoT system detection service is consistent with the data freezing service and is also a single-point service. The above basic high-availability rules can be adopted.

[0099] In the embodiment of the present invention, data persistence includes persistence on the platform side and data persistence on the collection side. Both sides adopt the influxDB+mysql method. Among them, influxDB is mainly used to store monitoring data and statistical data and other data that are rarely updated, and mysql is mainly used to store business and archive data. In addition, both sides have monitoring data storage services, which are responsible for writing the collected or received monitoring data into the influxDB database. After receiving the parsed collected data, the monitoring data storage service on the collection side will push the data to the monitoring data storage service on the platform side.

[0100] As an optional implementation, the network design module of this embodiment can also adopt a customized data collection strategy when planning data collection nodes, allowing users to adjust the frequency and scope of data collection according to specific needs; when backing up data and logs, the synchronization module of this embodiment can use the Caiyun hybrid cloud architecture, combining the advantages of public cloud and private cloud to provide more flexible data storage and processing options; when collecting data, the log collection module uses edge computing integration to perform preliminary processing near the source of data generation, reducing the pressure on the central node and improving the response speed.

[0101] After implementing the collection system provided by the embodiment of the present invention, the efficiency and quality of IoT data collection can be significantly improved. The specific effects include:

[0102] Significantly improve data collection efficiency: Through intelligent load balancing technology, the embodiment of the present invention can dynamically allocate tasks according to the real-time load of each collection node, avoiding overload of a single node, thereby ensuring that the entire system can still maintain efficient operation when facing large-scale, high-frequency data streams. This not only shortens the time from data collection to processing, but also improves the throughput of the overall system.

[0103] Enhanced system stability and reliability: High availability design, including redundant architecture, automatic fault detection and recovery mechanism, ensures that even if some components fail, the system can still seamlessly switch to backup resources to continue running, greatly reducing the risk of service interruption and providing users with a continuous and stable collection service experience.

[0104] Ensure data integrity and security: By implementing data persistence strategies and disaster recovery backup mechanisms, the embodiments of the present invention can ensure that data is reliably stored at the moment of data collection, and achieve real-time synchronous backup of data in different geographical locations or data centers. This not only prevents data loss, but also provides the ability to quickly recover in the event of a catastrophic event, further strengthening data security.

[0105] Improve scalability and flexibility: By utilizing virtualization and containerization technologies, the embodiments of the present invention can quickly expand or shrink capacity according to actual data traffic and business needs, and can dynamically adjust resources without downtime, greatly improving the flexibility of the system and its ability to adapt to future development needs.

[0106] Optimize storage efficiency and cost: Through data compression, sharding and other strategies, the embodiments of the present invention effectively reduce the demand for storage space and the cost of long-term storage of large amounts of data while ensuring data integrity. At the same time, the optimized data retrieval mechanism makes data analysis and application development more efficient and accelerates the conversion of data value.

[0107] Promoting innovation in intelligent applications: Real-time and comprehensive data collection capabilities provide a solid foundation for the in-depth development of IoT applications, making advanced applications such as predictive maintenance, intelligent scheduling, and precision marketing possible, and promoting the deep integration and innovative application of IoT technology in all walks of life.

[0108] Example 3

[0109] See also Figure 1 The embodiment of the present invention discloses a device for real-time collection of full-volume, high-frequency, and Internet of Things data, including the collection system described in Embodiment 2, and further comprising:

[0110] Terminal devices and cloud platform gateways;

[0111] The terminal device is located upstream of the collection system, and the collection system is used to collect data and logs of the terminal device in real time;

[0112] The cloud platform gateway is located downstream of the collection system, and the collection system uploads the collection data, alarm data and activity status data to the cloud platform gateway for backup and synchronization.

[0113] Please also read Figure 2 The collection device of this embodiment also includes a local workstation; the local workstation and the cloud platform gateway are connected in parallel downstream of the collection system; when the collection system uploads the collected data, alarm data and activity status data to the cloud platform gateway for backup and synchronization, it also transmits the collected data, alarm data and activity status data to the local workstation for real-time monitoring.

[0114] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of a device for real-time collection of full-volume, high-frequency IoT data disclosed in an embodiment of the present invention. Figure 4 The described IoT data full-volume high-frequency real-time collection device may include:

[0115] The acquisition module 301 is used to acquire the response information of the acquisition device to the control request when receiving the control request corresponding to the target device controlled by the acquisition device, and is also used to acquire the output information output by the acquisition device to the target device before the acquisition device receives the control request;

[0116] A generating module 302, for generating acquisition parameters of the acquisition device based on the response information of the acquisition device and the output information of the acquisition device, wherein the acquisition parameters of the acquisition device are used to control the operation of the target device;

[0117] The judging module 303 is used to judge whether the acquisition parameters of the acquisition device meet the preset operating conditions;

[0118] The analysis module 304 is used to analyze the acquisition parameters of the acquisition device to obtain abnormal parameters of the acquisition device if it is determined that the acquisition parameters of the acquisition device do not meet the operating conditions. The abnormal parameters of the acquisition device are the parameters in the acquisition parameters that need to be corrected.

[0119] In another optional embodiment, the acquisition module 301 is further used to acquire actual reception information when the target device receives the output information of the acquisition device after the generation module 302 generates the initial acquisition parameters of the acquisition device according to the response information of the acquisition device and the output information of the acquisition device;

[0120] And, if Figure 5 As shown, the device may also include:

[0121] A comparison module 305 is used to compare the output information of the acquisition device and the actual received information of the target device to obtain a comparison result;

[0122] The calibration module 306 is used to calibrate the initial acquisition parameters of the acquisition device according to the comparison result to obtain the calibrated initial acquisition parameters.

[0123] In yet another optional embodiment, the acquisition module 301 is further used to acquire historical acquisition parameters of the acquisition device and the rated service life of the acquisition device;

[0124] The analysis module 304 is further used to analyze the acquisition parameters of the acquisition device and the historical acquisition parameters of the acquisition device to obtain the current life loss of the acquisition device;

[0125] And, if Figure 5 As shown, the device may also include:

[0126] The prediction module 307 is used to predict the remaining service life of the collection device according to the rated service life of the collection device and the current life loss of the collection device.

[0127] In this optional embodiment, as an optional implementation, the analysis module 304 analyzes the acquisition parameters of the acquisition device and the historical acquisition parameters of the acquisition device, and the manner in which the current life loss of the acquisition device is obtained may specifically include:

[0128] Determine the current operation loss parameter of the collection device according to the collection parameter of the collection device and the historical collection parameter of the collection device;

[0129] Obtaining an operation loss coefficient of the collection device, where the operation loss coefficient is used to represent the relationship between the operation loss parameter of the collection device and the life loss of the collection device;

[0130] The current life loss of the collection device is calculated based on the current operating loss parameters of the collection device and the operating loss coefficient of the collection device.

[0131] Example 4

[0132] See also Figure 6 , Figure 6 Schematic diagram of the structure of another device for real-time collection of full-volume, high-frequency IoT data disclosed in an embodiment of the present invention. Figure 6 As shown, the device for real-time collection of full-volume high-frequency IoT data may include:

[0133] A memory 401 storing executable program codes;

[0134] a processor 402 coupled to the memory 401;

[0135] The processor 402 calls the executable program code stored in the memory 401 to execute the steps of the method for real-time collection of full-amount high-frequency IoT data described in the first embodiment of the present invention.

[0136] Example 5

[0137] An embodiment of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the steps of the method for real-time collection of full-amount high-frequency Internet of Things data described in the first embodiment of the present invention.

[0138] Example 6

[0139] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps of the method for real-time collection of full-volume, high-frequency, and Internet of Things data described in Example 1.

[0140] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0141] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (Erasable

[0142] Programmable Read Only Memory, EPROM, One-time Programmable Read Only Memory (EPROM)

[0143] Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0144] The above is a detailed description of an embodiment of a method and system for real-time collection of full-volume, high-frequency IoT data provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for real-time collection of full-volume, high-frequency IoT data, characterized in that: The method comprises: Plan a comprehensive architecture that includes data collection nodes, log collection nodes, service registration center, and message middleware to ensure efficient collaboration among components; Deploy log collection tools to each collection node and key service components to capture system logs, operation logs, and error logs in real time; Deploy Eureka as the service registration center, and let each collection node and service component register with the center when starting to realize service discovery and management; Deploy high-performance message middleware as the hub for data and log transmission, supporting high-concurrency and low-latency data exchange; Deploy backup centers and log storage systems in multiple regions, and use the multi-copy mechanism of the message middleware to achieve remote backup and synchronization of data and logs; Based on the load of the message queue, the number of collection and processing nodes is dynamically adjusted to achieve elastic scaling of resources.

2. The method according to claim 1, characterized in that After deploying the log collection tool to each collection node and key service component to capture system logs, operation logs and error logs in real time, the method further includes: Log processing and analysis: The collected logs are transmitted to the central log processing system through the message middleware for log parsing, archiving and real-time analysis to provide decision support for system operation and maintenance.

3. The method according to claim 1, characterized in that After deploying Eureka as a service registration center and allowing each collection node and service component to register with the center when starting to realize service discovery and management, the method further includes: Health check and monitoring: Use the health check function of the service registration center to regularly check the status of each service node. Combined with the monitoring suite, it can monitor the system health in real time and promptly warn of potential problems.

4. The method according to claim 1, characterized in that: After deploying high-performance message middleware as the hub for data and log transmission to support high-concurrency and low-latency data exchange, the method further includes: Data flow management: Design the topic structure of the message middleware to implement data classification and routing, ensure the decoupling and efficiency of data collection and processing processes, and support data retry and dead letter processing mechanisms to enhance the reliability of data transmission.

5. The method according to claim 2, characterized in that: After dynamically adjusting the number of acquisition and processing nodes based on the load of the message queue to achieve elastic scaling of resources, the method further includes: Performance tuning and monitoring feedback loop: Continuously monitor system performance and adjust optimization strategies, including message queue parameters and service configuration, based on log analysis and monitoring data feedback.

6. A system for real-time collection of full-volume, high-frequency IoT data, characterized in that: The system comprises: Network design module, used to design distributed acquisition networks; Log collection module, used to capture system logs, operation logs and error logs; The service registration module is used to deploy the service registration center, allowing each collection node and service component to register with the center when starting; The message deployment module is used to deploy high-performance message middleware as the hub for data and log transmission; The synchronization module is used to deploy backup centers and log storage systems in multiple regions, and to achieve remote backup and synchronization of data and logs through the multi-copy mechanism of the message middleware; The adjustment module is used to dynamically adjust the number of collection and processing nodes to achieve elastic scaling of resources.

7. A device for real-time collection of full-volume, high-frequency IoT data, characterized in that: The acquisition system according to claim 6 further comprises: Terminal devices and cloud platform gateways; The terminal device is located upstream of the collection system, and the collection system is used to collect data and logs of the terminal device in real time; The cloud platform gateway is located downstream of the collection system, and the collection system uploads the collection data, alarm data and activity status data to the cloud platform gateway for backup and synchronization.

8. The collection device according to claim 7, characterized in that: The apparatus further comprises a local workstation; The local workstation and the cloud platform gateway are connected in parallel downstream of the acquisition system; When the acquisition system uploads the acquisition data, alarm data and activity status data to the cloud platform gateway for backup and synchronization, it also transmits the acquisition data, alarm data and activity status data to the local workstation for real-time monitoring.

9. A device for real-time collection of full-volume, high-frequency IoT data, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the method for real-time collection of full-volume, high-frequency IoT data as described in any one of claims 1-5.

10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, which, when called, are used to execute the method for real-time collection of full-volume, high-frequency, and Internet of Things data as described in any one of claims 1-5.

Images