A method and device for processing node serial data acquisition

Abstract

The present invention provides a method and device for processing data based on node serial acquisition, the method comprising: performing statistics on data to be collected, performing data modeling, and storing the modeled data as a first result in a Mysql database; creating a new edge channel, mounting the modeled device point data under the corresponding channel, and setting the point acquisition frequency; starting the edge channel, and the edge end periodically obtaining the point real-time value as a second result according to the point acquisition frequency; storing or sending the path of the point of the second result to a message queue respectively according to whether the path of the point exists in a database event monitor or a business event monitor; a third-party business platform monitors the message queue to obtain the real-time value of the data. The present invention can realize the requirements of automation, precision, and customization of data information transmission, while also achieving the effects of improving efficiency and saving costs.

Description

Technical Field

[0001] The present invention relates to the field of collected data processing, and in particular to a method and device for processing node serial collected data. Background Art

[0002] The Internet of Things (IoT) refers to the use of various devices and technologies, including information sensors, radio frequency identification (RFID), global positioning systems (GPS), infrared sensors, and laser scanners, to collect real-time data from any object or process that requires monitoring, connection, and interaction. This data includes acoustic, optical, thermal, electrical, mechanical, and chemical information, enabling ubiquitous connectivity between objects and between objects and people, and enabling intelligent perception, identification, and management of objects and processes. With the widespread adoption of IoT applications, the use cases of IoT systems are becoming increasingly complex, and the number of access devices in these systems is increasing. These devices require the configuration of corresponding data collection service components to achieve data collection. Therefore, configuring multiple access devices in an IoT system is a crucial step in data collection, processing, and push.

[0003] In existing technologies, because the communication data collection formats and collection processes of various IoT devices may differ, developers must organize and process data after collecting data from newly connected devices, integrate the data based on the business needs, analyze the data formats of the newly connected devices, and redevelop units to further process the collected data. This approach results in low development efficiency and a high software programming workload during the development of data access units in current IoT systems, resulting in increased development costs and longer delivery times. Summary of the Invention

[0004] The present invention provides a method and device for processing data based on node serial acquisition, including data acquisition, data processing, data storage and data transmission, and adopts a multi-process and multi-tasking functional mode to solve the technical problems in the prior art such as large amount of data information, high requirements for data acquisition timeliness, high requirements for automation, and limited capacity of load balancers and application expansion capabilities.

[0005] A method for processing node serial data acquisition includes the following steps:

[0006] The data to be collected is statistically analyzed and data modeled, and the modeled data is stored as a first result in a MySQL database. The first result includes spatial information, device information, and information about the location of the device.

[0007] Create a new edge channel, mount the modeled device point data to the corresponding channel, and set the point collection frequency;

[0008] Start the edge channel. The edge obtains the real-time value of the point as the second result based on the point collection frequency, and sends it to the main platform by establishing a long connection through Netty. The second result includes the path of the point and the real-time value of the point.

[0009] Get the second result and determine whether the path of the point in the second result exists in the database event monitoring. If so, store the path of the point in the second result and the real-time value of the point in the memory cache database Redis and the real-time historical database Influxdb to achieve multi-database storage;

[0010] Get the second result, determine whether the path of the point in the second result exists in the business event monitoring, if so, get the message queue corresponding to each point, and send the path of the point and the real-time value of the point to the message queue;

[0011] The third-party business platform monitors the message queue to obtain real-time data values.

[0012] Furthermore, the edge end channel refers to a transmission channel established between the main platform and the edge end, and a series of processes for transmission configuration, when the configuration is completed and the edge end is turned on, to send a start transmission signal to the main platform.

[0013] Furthermore, the point acquisition frequency is similar to a timer. According to the time frequency set by the timer, the edge end regularly collects the real-time data of the point.

[0014] A device for processing node serial data acquisition, comprising:

[0015] A first obtaining unit is used to collect statistics of the data to be collected, perform data modeling, and store the modeled data as a first result in a MySQL database, where the first result includes spatial information, device information, and information about the location of the device;

[0016] The second acquisition unit is used to create an edge channel, mount the modeled device point data under the corresponding channel, set the point acquisition frequency, and start the edge channel. The edge obtains the point real-time value as the second result according to the point acquisition frequency, and establishes a long connection through Netty to send it to the main platform. The second result includes the path of the point and the real-time value of the point;

[0017] a database event monitoring unit, configured to obtain the second result obtained by the second obtaining unit, determine whether the path of the point in the second result exists in the database event monitoring, and if so, store the path of the point in the second result and the real-time value of the point in the in-memory database Redis and the real-time historical database Influxdb, thereby realizing multi-database storage;

[0018] a business event monitoring unit, configured to obtain the second result obtained by the second obtaining unit, determine whether the path of the point in the second result exists in the business event monitoring, and if so, obtain the message queue corresponding to each point, and send the path of the point and the real-time value of the point to the message queue;

[0019] A third-party monitoring unit is used to monitor the message queue and obtain real-time data values.

[0020] Furthermore, the edge end channel refers to a transmission channel established between the main platform and the edge end, and a series of processes for transmission configuration, when the configuration is completed and the edge end is turned on, to send a start transmission signal to the main platform.

[0021] Furthermore, the point acquisition frequency is similar to a timer. According to the time frequency set by the timer, the edge end regularly collects the real-time data of the point.

[0022] The present invention is based on a node serial data acquisition processing method and device, which can realize network communication functions such as basic data acquisition, storage, communication reception and transmission, communication mode differentiation, communication process control, underlying protocol regulation, and basic data integration, achieving efficient transmission and full sharing of data information, realizing the requirements of automation, precision, and customization of data information transmission, and also achieving the effects of improving efficiency and saving costs. For example: during the entire process of basic data acquisition, storage, and communication with the business platform, the CPU utilization rate is between 40% and 100%, and the average utilization rate is less than 85%, which meets the performance indicators. The peak memory utilization rate is around 66%, and at the same time, the server also has system resources occupying memory, so the actual memory occupancy rate is less than 66%, less than 85%, which meets the performance indicators. At the same time, during the process of second-level acquisition, storage, and communication with the business platform based on 1 million basic data, the peak rate is close to 12.8% of the total rate, that is, 9Mbps is equivalent to 9000kbps, which fully meets the user's rate requirement (10kbps). BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of a process for processing node serial data acquisition in accordance with an embodiment of the present invention;

[0024] Figure 2 This is a schematic structural diagram of a node-based serial data acquisition and processing device according to an embodiment of the present invention;

[0025] Figure 3 Schematic diagram of the structure of the node serial data acquisition and processing device in the prior art. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0027] Example 1

[0028] Figure 1 FIG. 1 is a flow chart of a method for processing node serial data acquisition in an embodiment of the present invention. Figure 1 As shown, a data collection and processing method in an embodiment of the present application is applied to a server, and the method includes:

[0029] In step 110, statistics are first collected on the data to be collected, and data modeling is performed, and the modeled data is stored in a MySQL database. The stored data is the first result, which includes spatial information, device information, and information about the points under the device.

[0030] The collected data and information must be organized, processed, and stored in a specific manner for effective processing and utilization. Disorganized data is not only inconvenient for storage, transmission, and processing, but also hinders its use. A basic approach is to use a real-time database to store information. A real-time database is both a fundamental method for information storage and an effective form of information organization. It organically combines information organization and storage, and serves as a fundamental method for device information storage. A key characteristic of a real-time database is real-time performance, encompassing both data real-time and transaction real-time performance. Data real-time performance refers to the update cycle of field I / O data. As a real-time database, data real-time performance must be considered. Data real-time performance is generally constrained by field equipment, especially for older systems. Transaction real-time performance refers to the speed at which the database processes transactions. It can be event-triggered or timed. Event-triggered events are scheduled immediately upon their occurrence, allowing for immediate processing but consuming more system resources. Timed triggers, on the other hand, grant scheduling rights within a specified timeframe. A complete real-time database must provide both scheduling methods to ensure system stability and real-time performance.

[0031] Specifically, in the era of the Internet of Things, data collection, processing, and push have wide applications, playing an important role in retail stores, power inspections, scenic area management, production and manufacturing, smart buildings and other fields.

[0032] Retail stores are used for data collection, processing and push in offline stores such as supermarkets, chain stores, counters and stores, helping stores to achieve efficient and convenient goods in and out management, inventory counting, allocation, shopping guide, etc., helping stores to achieve all-round digital operations, optimize processes, improve efficiency and increase sales.

[0033] Power inspection, in the power inspection work, the main applications include data collection, data push, route arrangement, work status supervision, personnel inspection, equipment monitoring, etc., which can help to scientifically manage power equipment and arrange personnel inspections, improve work efficiency, and reduce production and operation costs.

[0034] Scenic area management: Commonly used in ticket checking, ticket verification, passenger flow statistics, and ticket identification, fully electronic operations not only significantly reduce staff workload and improve work efficiency, helping to better cope with peak holiday passenger flow, but also strengthen ticket anti-counterfeiting management. Suitable for amusement parks, tourist attractions, expositions, etc.

[0035] Production and manufacturing is used for warehouse management, process control, equipment management, personnel tracking and other aspects of manufacturing enterprises. Using data as a link, it connects people, objects and equipment throughout the entire production process, helping enterprises achieve efficient, accurate and intelligent business and personnel management.

[0036] Smart buildings can sense indoor lighting conditions through collected data and automatically adjust brightness to save energy. At the same time, they can realize multiple functions such as smart home, smart security, and remote monitoring.

[0037] Step 120: Create a new edge channel, mount the modeled device point data under the corresponding channel, and set the point collection frequency.

[0038] Specifically, the edge channel refers to: a transmission channel established between the main platform and the edge end, and transmission configuration is performed. When the configuration is completed and the edge end is turned on, a series of processes are sent to the main platform to start the transmission signal. Note: The creation of the edge channel should be based on professional relevance, so that it is convenient to view the log output by profession. The professional relevance means: each profession starts an edge channel, and there may be multiple channels. The channel identification can be based on the unique identification of the profession, so that the log information of each profession can be clearly observed. The point acquisition frequency is similar to a timer. According to the time frequency set by the timer, the edge end regularly collects point real-time data.

[0039] Step 130 starts the edge channel. The edge periodically obtains the real-time value of the point based on the point collection frequency as a second result. This result is sent to the main platform via a persistent connection established by Netty. The second result includes the path to the point and the real-time value of the point. Data processing is based on data. Data processing is the process of receiving, storing, transforming, transmitting, and publishing data. Data processing is the process of removing false data from the collected data, refining the data from the coarse to the fine, and integrating the superficial into the underlying structure.

[0040] Data processing is the process of producing valuable, user-friendly secondary information based on raw data. This process increases the value of data. Only through proper data processing can new, effective data be generated to guide decision-making. After data is acquired, it is always chaotic. A series of data processing (such as reorganization) is required to process all point data and generate new, effective information.

[0041] Step 140: Obtain the second result and determine whether the path of the point in the second result exists in the database event monitoring. If so, store the path of the point in the second result and the real-time value of the point in the memory cache database Redis and the real-time historical database Influxdb to achieve multi-database storage;

[0042] Specifically, the data format stored in the database is: the path of the equipment point and the value of the equipment point.

[0043] Furthermore, in-memory caching is a common design pattern: this pattern generally decouples low-speed components from high-speed components by increasing storage space. In other words, wherever decoupling is achieved, in-memory caching is likely to exist. Since decoupling is a common practice in computer system architecture design, in-memory caching is bound to be ubiquitous in IT systems and architectures, such as internal CPU cache, client browser cache, web server cache, application cache, and database cache. Among various commonly used caching technologies, NoSQL distributed caching technology based on in-memory databases has been widely adopted in the current cache database field. The most typical example is Redis, a high-performance distributed in-memory cache database, which is generally used to cache frequently accessed hot data and reduce pressure and load on physical databases.

[0044] Redis supports storing relatively more value types, including string, list, set, and zset. These data types support push / pop, add / remove, intersection, union, difference, and richer operations, and these operations are atomic. On this basis, Redis supports various different sorting methods. To ensure efficiency, data is cached in memory. The difference is that Redis will periodically use the RDB snapshot method to write updated data to disk or use the AOF log method similar to MySQL to write modification operations to additional record files, and on this basis, master-slave synchronization is implemented. Redis supports synchronizing data to multiple slave databases. This feature is very beneficial to improving read performance.

[0045] Furthermore, a key feature of the real-time historical database InfluxDB is real-time performance, encompassing both data and transaction real-time. Data real-time refers to the update cycle of on-site I / O data, and as a real-time database, data real-time performance is essential. InfluxDB is a database service that supports efficient read and write, compressed storage, and real-time computing capabilities for time-series data. In addition to its cost-effective high-performance read and write capabilities, high storage rates, and no system environment dependencies, InfluxDB offers easy deployment, a flexible and powerful schema-less data model, a native HTTP management interface that eliminates plugin configuration and third-party dependencies, powerful SQL-like query statements with a low learning curve and quick adoption, extensive permission management capabilities down to the "table" level, extensive time management capabilities including automatic deletion of expired data and custom deletion of indicator data, low-cost storage, sampled time-series data, and compressed storage, and a rich set of aggregation functions, including support for AVG, SUM, MAX, and MIN.

[0046] Step 150: Obtain the second result, determine whether the path of the point in the second result exists in the business event monitoring, and if so, obtain the message queue corresponding to each point, and send the path of the point and the real-time value of the point to the message queue;

[0047] Finally, the third-party business platform monitors the message queue to obtain real-time data values.

[0048] Specifically, if the path of the point of the second result exists in the monitoring of the business event, the point data will be actively pushed to the message queue. The business platform monitors the message queue, obtains the real-time value of the point data, and performs subsequent business processing.

[0049] Example 2

[0050] like Figure 2As shown, an embodiment of the present invention provides a node serial data acquisition processing device, comprising:

[0051] The first acquisition unit is used to collect statistics of the data to be collected, perform data modeling, and store the modeled data in a MySQL database. The stored data is the first result, which includes spatial information, device information, and information about the points under the device.

[0052] The second acquisition unit is used to create a new edge channel, mount the modeled device point data under the corresponding channel, set the point acquisition frequency, and start the edge channel. The edge obtains the real-time value of the point according to the point acquisition frequency as the second result, and establishes a long connection through Netty and sends it to the main platform. The second result includes the path of the point and the real-time value of the point.

[0053] a database event monitoring unit, configured to obtain the second result obtained by the second obtaining unit, determine whether the path of the point in the second result exists in the database event monitoring, and if so, store the path of the point in the second result and the real-time value of the point in the in-memory database Redis and the real-time historical database Influxdb, thereby realizing multi-database storage;

[0054] The business event monitoring unit is used to obtain the second result obtained by the second obtaining unit, determine whether the path of the point in the second result exists in the business event monitoring, and if so, obtain the message queue corresponding to each point, and send the path of the point and the real-time value of the point to the message queue.

[0055] A third-party monitoring unit is used to monitor the message queue and obtain real-time data values.

[0056] like Figure 3As shown, the prior art node-based serial data collection and processing device provides a bus interface 306. The edge terminal collects real-time data from the point and sends it to the bus interface 306, allowing other interfaces to monitor and modify the data. The processor 302 and memory 304 monitor the bus interface 306, processing and monitoring the real-time data from the point. While this method can obtain real-time data from the point, it cannot achieve flexible point configuration. The present invention provides a method and device for processing data based on node serial data collection. This creates a new edge channel, mounts the modeled device point data to the corresponding channel, sets the point collection frequency, and starts the edge channel. The edge terminal periodically obtains the real-time value of the point based on the point collection frequency, allowing for flexible point configuration. Furthermore, data is sent to the main platform through a persistent connection established by Netty, rather than through a bus interface. Netty provides an asynchronous, network application framework and tools for the rapid development of high-performance, highly reliable network servers and clients. This ensures data stability and real-time performance when the edge terminal sends data to the main platform, verifying the validity of data from the in-memory database Redis and the real-time historical database InfluxDB.

[0057] This invention can be applied in smart agriculture. Value-added smart agriculture is the smart economy in agriculture, or the specific manifestation of the smart economy in agriculture. Smart agriculture is an important component of the smart economy; smart agriculture is a major component of the smart economy and a major way for developing countries to eliminate poverty, achieve latecomer advantages, catch up in economic development, and achieve a catch-up strategy.

[0058] Implementing automatic information detection and control within the agricultural park, using serial data collection and processing methods and devices to collect data from agricultural equipment points. First, collect statistics on the equipment point data to be monitored, such as soil moisture, soil temperature, air temperature, air humidity, light intensity, plant nutrient content, and other point parameters. Store this data in a MySQL database. Create edge channels related to smart agriculture, mount the modeled equipment point data to the corresponding channels, set the point collection frequency, and start the edge channel. The edge terminal periodically obtains real-time values ​​from points within the agricultural park based on the point collection frequency.

[0059] The edge sends collected point data to the main platform via a persistent connection established with Netty. This data is then stored and sent to a message queue. The business platform monitors the message queue to obtain real-time data values. Users can view real-time data from the greenhouse site and use remote control functions to control greenhouse automation equipment such as automatic spraying, ventilation, and irrigation systems through relay control devices or analog output modules.

[0060] The system consists of wireless soil moisture monitoring stations, seedling monitoring cameras, visual automatic insect infestation detection lights, and disaster video monitoring cameras. Through the service platform, users can log in to their own dedicated web client anytime and anywhere via mobile devices and PCs to access real-time field data and manage the system, providing real-time monitoring of the environment, weather, pest and disease conditions, and crop growth at each monitoring point.

[0061] Combined with the system early warning model, real-time remote monitoring and diagnosis of crops can be achieved, and intelligent and automated solutions can be obtained to realize dynamic monitoring of crop growth and manual remote precise management, ensuring that crops grow under suitable environmental conditions, improving agricultural productivity and increasing farmers' income.

[0062] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for processing node serial data acquisition, characterized in that: The steps include: The data to be collected is statistically analyzed and data modeled, and the modeled data is stored as a first result in a MySQL database. The first result includes spatial information, device information, and information about the location of the device. Create a new edge channel, mount the modeled device point data to the corresponding channel, and set the point collection frequency; Start the edge channel. The edge obtains the real-time value of the point as the second result based on the point collection frequency, and sends it to the main platform by establishing a long connection through Netty. The second result includes the path of the point and the real-time value of the point. Get the second result and determine whether the path of the point in the second result exists in the database event monitoring. If so, store the path of the point in the second result and the real-time value of the point in the memory cache database Redis and the real-time historical database Influxdb to achieve multi-database storage; Get the second result, determine whether the path of the point in the second result exists in the business event monitoring, if so, get the message queue corresponding to each point, and send the path of the point and the real-time value of the point to the message queue; The third-party business platform monitors the message queue to obtain real-time data values; The edge channel refers to a transmission channel established between the main platform and the edge end, and a series of processes for transmission configuration. When the configuration is completed and the edge end is turned on, a transmission start signal is sent to the main platform.

2. The method for processing node serial data acquisition according to claim 1, wherein: The point acquisition frequency is similar to a timer. According to the time frequency set by the timer, the edge end regularly collects point real-time data.

3. A device based on node serial data collection and processing, characterized in that: include: A first obtaining unit is used to collect statistics of the data to be collected, perform data modeling, and store the modeled data as a first result in a MySQL database, where the first result includes spatial information, device information, and information about the location of the device; The second acquisition unit is used to create an edge channel, mount the modeled device point data under the corresponding channel, set the point acquisition frequency, and start the edge channel. The edge obtains the point real-time value as the second result according to the point acquisition frequency, and establishes a long connection through Netty to send it to the main platform. The second result includes the path of the point and the real-time value of the point; a database event monitoring unit, configured to obtain the second result obtained by the second obtaining unit, determine whether the path of the point in the second result exists in the database event monitoring, and if so, store the path of the point in the second result and the real-time value of the point in the in-memory database Redis and the real-time historical database Influxdb, thereby realizing multi-database storage; a business event monitoring unit, configured to obtain the second result obtained by the second obtaining unit, determine whether the path of the point in the second result exists in the business event monitoring, and if so, obtain the message queue corresponding to each point, and send the path of the point and the real-time value of the point to the message queue; A third-party monitoring unit is used to monitor the message queue and obtain real-time data values; The edge channel refers to a transmission channel established between the main platform and the edge end, and a series of processes for transmission configuration. When the configuration is completed and the edge end is turned on, a transmission start signal is sent to the main platform.

4. The device for processing node-based serial data acquisition according to claim 3, wherein: The point acquisition frequency is similar to a timer. According to the time frequency set by the timer, the edge end regularly collects point real-time data.

Images