Industrial Internet of Things high-concurrency access and data acquisition method and system

Abstract

The invention relates to an industrial internet of things high-concurrency access and data acquisition method, which comprises the following steps of: connecting a plurality of devices through a connection pool, allocating storage resources for access devices and storing device information by utilizing a memory pool, polling instruction queues of the devices according to a scheduling priority rule, and sending the instruction queues to a server; each polling result is put into the message queue of the working thread pool as a to-be-processed message, and each idle working thread in the working thread pool can directly take out the to-be-processed message from the message queue of the thread pool to communicate with corresponding equipment, so that data acquisition and analysis are realized, the response speed of the system is effectively improved, and the service life of the system is prolonged. The problem of system jamming caused by a large number of threads generated by access of multiple devices and a large message instruction queue jumping demand is solved, and the system is good in stability and high in reliability.

Description

Technical Field

[0001] The present invention relates to the field of data collection, and in particular to a method and system for high-concurrency access and data collection of an industrial Internet of Things. Background Art

[0002] The Industrial Internet of Things integrates various terminal devices such as sensors or controllers for data collection and control, as well as technologies such as mobile communications and intelligent analysis, into all aspects of the industrial production process, thereby greatly improving manufacturing efficiency, improving product quality, reducing product costs and resource consumption, and ultimately achieving a new stage of upgrading traditional industries to intelligence. It is widely used in industrial fields such as intelligent manufacturing, supply chain management, and energy management. With the upgrade of intelligence, a large number of terminal devices need to be connected to the Internet of Things system. The concurrent access of devices has huge resource overhead, and often leads to problems such as slow response or even crash, resulting in low system reliability.

[0003] At present, in order to solve the response problem caused by the access of a large number of devices, a separate thread will be generated when each device is connected to sort, collect, analyze and store data. However, when a large number of devices are connected concurrently, the platform instantly generates a large number of threads, and the Internet of Things system will also have the problem of slow response. The publication number is CN110868349B, which discloses a distributed high-concurrency data collection and processing method and system in the Internet of Things environment. The data of the terminal device is sent to the data acquisition module through the Internet of Things gateway. The data acquisition module generates processors of different protocols according to the communication protocol format of the data to access the system. When the device message enters the system, the system immediately generates a working thread to transmit and communicate with the device. This data processing method, when multiple devices enter the system, the system will instantly generate multiple corresponding working threads to process the message instructions of these devices, which seriously affects the system response speed and may even cause the system to crash, and the system reliability is low. Moreover, during the data processing of industrial equipment, there is often a need to jump the queue for data processing types. The system needs to sort and read and write instructions according to priority rules first, and then collect data according to the sorted order. When there are fewer messages, the system can sort and collect data well according to the rules. However, when there are more messages, this method of sorting first and then collecting data according to the sorting results will aggravate the problem of slow response of the IoT system. Summary of the invention

[0004] One purpose of the present invention is to provide an industrial Internet of Things high-concurrency access and data acquisition method in response to the deficiencies in the prior art. The method polls the command queue of each device according to the scheduling priority rule, puts the result of each polling into the working thread pool message queue as a message to be processed, and each idle working thread directly takes out the message to be processed from the working thread pool message queue to communicate with the corresponding device to realize data collection, effectively improve the system response speed, and parallelizes the message instruction priority processing with the data collection processing, solves the problem of system jamming caused by concurrent access of multiple devices and message instruction queue jumping requirements, and improves the stability and reliability of the system.

[0005] One object of the present invention is to adopt the following scheme to achieve: a high-concurrency access and data collection method for industrial Internet of Things, specifically comprising the following steps: S1: Monitor multiple communication protocol plug-in ports that are opened through the connection pool of the device access module. When the connection pool monitors a device connection request, the devices requesting connection are connected to the IoT system in sequence, and the device information requesting connection is passed to the memory pool of the device access module; S2: The memory pool allocates storage resources to each device according to the communication protocol type of the device, and the storage content of the storage resources includes the device name, slaveID, socket, and instruction queue; S3: The collection and analysis module polls the instruction queue of each device. Each time it polls, it takes out the highest priority instruction in the instruction queue of each device according to the scheduling priority rule, converts the type of the taken instruction into a message, and pushes it into the message queue of the working thread pool as a message to be processed; S4: The idle working threads in the thread pool of the collection and analysis module take out the pending messages from the message queue of the working thread pool in turn, read the socket information of the device in the pending messages, send instructions to the corresponding devices through the socket information, receive the response data returned by the devices, parse the response data, obtain the parsing results, and store the parsing results in the database; S5: The business application obtains the corresponding data of the device from the database according to the device information.

[0006] Preferably, the instruction queue includes a write instruction queue, a read device status queue, and a read history data queue, and the scheduling priority rule is set according to the importance of the business corresponding to the instruction queue.

[0007] Further preferably, the scheduling priority rule is specifically: if there is an instruction in the write instruction queue, then take out an instruction from the write instruction queue; otherwise, determine whether there is an instruction in the read history data queue, and if there is an instruction in the read history data queue, then take out an instruction from the read history data queue; otherwise, determine whether there is an instruction in the read device status queue, and if there is an instruction in the read device status queue, then take out an instruction from the read device status queue; otherwise, continue to wait for the next polling.

[0008] Preferably, the communication protocol of the communication protocol plug-in port is a long connection communication protocol.

[0009] Further preferably, the long connection communication protocol includes TCP, HTTP, MQTT, MODBUS RTU, and MODBUSTCP.

[0010] Preferably, the database is a time series database.

[0011] Another object of the present invention is to address the deficiencies in the prior art and provide an industrial Internet of Things high-concurrency access and data processing system, which utilizes a scheduling thread unit to sort messages and utilizes working threads to construct instructions, data collection, and analysis processing, thereby improving data processing efficiency and solving the problem of slow system response caused by a large number of threads generated instantly when a large number of devices are connected.

[0012] Another object of the present invention is to adopt the following scheme to achieve: a system of high-concurrency access and data acquisition method of industrial Internet of Things, including a device access module, an acquisition and analysis module, a work thread pool message queue, a database, a business application and multiple devices, the device access module includes a connection pool and a memory pool, the connection pool is used to monitor and connect each device to realize the communication between each work thread and the device, the memory pool is used to allocate storage resources for each connected device, and store each device information; the acquisition and analysis module includes a scheduling thread unit and a thread pool, the scheduling thread unit is used to take out each device instruction according to the scheduling priority rule, a plurality of work threads are set in the thread pool, the work thread is used to take out instructions from the work thread pool message queue, communicate with the corresponding device according to the instructions, and put the processing results into the database; the work thread pool message queue is used to store the taken out device instructions and perform type conversion on the instructions; the database is used to store the processing results of the work thread; the business application is used to query the information processed by the device and modify the device memory parameters through the device access module and the acquisition and analysis module.

[0013] Preferably, the business application is connected to a memory pool of a device access module for modifying device memory parameters.

[0014] The present invention identifies device connection requests by monitoring the communication protocol plug-in port through a connection pool, and accesses multiple devices. Different plug-ins can be selected according to actual needs, and the Internet of Things devices with different communication protocols are applicable, and the versatility is high. The memory pool is used to allocate storage resources for access devices and store device information. The device is polled by a scheduling thread unit, and the device message instructions are sorted and scheduled and put into a work thread pool message queue to wait for processing. Multiple work threads take out the to-be-processed messages according to their own states, collect data with corresponding devices, and process data. The message instruction priority processing is parallel to the data collection processing, so as to solve the problem that a large number of threads are generated at the moment of access of a large number of devices, resulting in slow system response. Message sorting and data collection and processing are processed by the scheduling thread unit and the thread pool respectively, and each work thread is only responsible for constructing instructions and parsing messages, without paying attention to message mutual exclusion and sorting rules, so as to have higher processing efficiency and effectively improve the system response speed. At the same time, the number of multiple work threads is based on the hardware configuration of the server. In this thread pool with a fixed number of pre-initialization, when a large number of devices are concurrently accessed, the work threads process messages according to their own states, and a large number of threads will not be suddenly generated, so as to avoid the drastic fluctuation of system resource consumption, and the stability of the system is high.

[0015] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the present invention; Figure 2 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0017] See also Figure 1 to Figure 2 , a high-concurrency access and data collection method for industrial Internet of Things, specifically comprising the following steps: S1: Through the connection pool of the device access module, multiple communication protocol plug-in ports that are opened are monitored. When the connection pool monitors the device connection request, the devices requesting connection are connected to the IoT system in turn, and it is determined whether they are new devices. The information of the device requesting connection and the judgment result are passed to the memory pool of the device access module.

[0018] The communication protocol of the communication protocol plug-in is a long connection communication protocol, which includes TCP, HTTP, MQTT, MODBUS RTU, and MODBUS TCP. It is connected to each device by means of a communication protocol plug-in. By selecting the corresponding communication protocol plug-in according to the actual needs of the Internet of Things system, the connection method with the device can be changed without modifying the internal program, and the versatility is high; the connection pool is implemented using the corresponding framework according to the communication protocol type. If it is TCP, HTTP, or MQTT, the Netty framework is used; if it is MODBUS RTU or MODBUS TCP, the modbus4j framework is used; the memory pool type uses ConcurrentLinkedQueue.

[0019] S2: The memory pool allocates storage resources to each device according to the information transmitted by the connection pool. If the device is a newly connected device, storage resources are directly allocated to the device according to the communication information of the device. If the device is not a newly connected device, the device-related data in the memory pool is first deleted and then new storage resources are allocated to the device according to the communication information of the device. The storage content stored in the storage resources includes the device name, slaveID, socket, and instruction queue; the instruction queue includes a write instruction queue, a read device status queue, and a read history data queue.

[0020] S3: The collection and analysis module polls the instruction queue of each device, and each time polling, it takes out the highest priority instruction in the instruction queue of each device according to the scheduling priority rule, converts the type of the taken instruction into a message, and pushes it into the working thread pool message queue as a message to be processed; the scheduling priority rule is set according to the importance of the business corresponding to the instruction queue, and the scheduling priority rule is specifically that if there is an instruction in the write instruction queue, an instruction is taken out from the write instruction queue, otherwise it is determined whether there is an instruction in the read history data queue, if there is an instruction in the read history data queue, an instruction is taken out from the read history data queue, otherwise it is determined whether there is an instruction in the read device status queue, if there is an instruction in the read device status queue, an instruction is taken out from the read device status queue, otherwise continue to wait for the next polling.

[0021] S4: The idle working threads in the thread pool of the acquisition and analysis module take out the pending messages from the working thread pool message queue in turn, and read the socket information of the device in the pending messages, send instructions to the corresponding device through the socket information, and receive the response data returned by the device at the same time, parse the response data, obtain the parsing result, and store the parsing result in the database; the working thread pool message queue adopts RabbitMq or rocketMq; the database is a time series database with fast data query speed. In this embodiment, the database adopts TD or ES.

[0022] S5: The business application obtains the corresponding data of the device from the database according to the device information.

[0023] A system for a high-concurrency access and data acquisition method for an industrial Internet of Things comprises a device access module, an acquisition and analysis module, a work thread pool message queue, a database, a business application and a plurality of devices, wherein the device access module comprises a connection pool and a memory pool, wherein the connection pool is used to monitor and connect each device to realize communication between each work thread and the device, and the memory pool is used to allocate storage resources for each connected device and store information of each device; the acquisition and analysis module comprises a scheduling thread unit and a thread pool, wherein the scheduling thread unit is used to take out instructions of each device according to a scheduling priority rule, wherein a plurality of work threads are arranged in the thread pool, wherein the work thread is used to take out instructions from a work thread pool message queue, communicate with a corresponding device according to the instructions, and put the processing results into a database; the work thread pool message queue is used to store the taken out instructions of each device and perform type conversion on the instructions; the database is used to store the processing results of the work thread; the business application is used to query the information processed by the device and modify the memory parameters of the device through the device access module and the acquisition and analysis module; the device is a sensor module or an intelligent instrument or an intelligent home device; the business application is a WEB application or a small program or a mobile phone APP.

[0024] Preferably, the business application is connected to the memory pool of the device access module and communicates through an interface to modify the device memory parameters; the business application writes instructions to the instruction queue of the device in the memory pool through the interface to update the corresponding device instructions, and the updated instructions pass through the scheduling thread unit and the working thread pool message queue in sequence to reach the working thread, and the working thread communicates with the corresponding device through the connection pool to realize the modification of the device's memory parameters.

[0025] Compared with the prior art, the present invention determines the priority of instructions through a scheduling thread unit, collects and analyzes data from each device through multiple working threads, has a fast response speed, and has high system security and stability. Example 1

[0026] The system of high-concurrency access and data acquisition methods of the Industrial Internet of Things is applied to the intelligent operation and maintenance system of instruments and meters, which is used to query instrument measurement data and operation status data and modify instrument parameter data, and perform instrument status monitoring and remote maintenance. The following is an introduction to the use of monitoring and maintenance of flow and pressure sensing equipment in industrial pipelines as an example. The intelligent operation and maintenance system of industrial pipelines is built, and various scheduling priority rules are set. Pressure transmitters and electromagnetic flow meters on multiple pipelines are connected to the system; when in use, MODBUS is opened on the intelligent operation and maintenance system of industrial pipelines. RTU protocol plug-in (listening port 10000 is used to accept the connection of pressure transmitter) and TCP protocol plug-in (listening port 10001 is used to accept the connection of electromagnetic flowmeter); when the connection pool in the system listens to the request connection of pressure transmitter or electromagnetic flowmeter, the corresponding device is connected to the industrial pipeline intelligent operation and maintenance system through the communication protocol plug-in, and the memory pool allocates storage resources for each connected device, the name, slaveID, socket and instruction queue of each pressure transmitter and electromagnetic flowmeter; when the user needs to query the operating status of the specified device in the specified pipeline, the query instruction is issued through the WEB application, applet or mobile phone APP of the industrial pipeline intelligent operation and maintenance system, and the instruction is pushed into the instruction queue of the specified pressure transmitter and electromagnetic flowmeter. The scheduling thread unit judges the priority of the instructions in the instruction queue of each device according to the scheduling priority rules, and then takes out the instruction with the highest priority, converts it and puts it into the message queue of the working thread pool. The idle working thread takes out the message and communicates with the pressure transmitter or electromagnetic flowmeter specified in the message to realize device data collection and analysis, and the analysis data is stored in the database of the industrial pipeline intelligent operation and maintenance system; the WEB application, applet or mobile phone APP queries the status data of the specified device in the database; if the parameters of the specified pressure transmitter or electromagnetic flowmeter in the specified pipeline are to be modified or adjusted, the modification instruction is issued through the WEB application, applet or mobile phone APP. After the above processing, the parameter modification results of the pressure transmitter or electromagnetic flowmeter are obtained from the database through the interface.

[0027] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification made to the present invention by those skilled in the art without departing from the spirit of the present invention shall fall within the protection scope of the present invention.

Claims

1. A high-concurrency access and data collection method for industrial Internet of Things, characterized in that: The specific steps include: S1: Monitor multiple communication protocol plug-in ports that are opened through the connection pool of the device access module. When the connection pool monitors a device connection request, the devices requesting connection are connected to the IoT system in sequence, and the device information requesting connection is passed to the memory pool of the device access module; S2: The memory pool allocates storage resources to each device according to the communication protocol type of the device, and the storage content of the storage resources includes the device name, slaveID, socket, and instruction queue; S3: The collection and analysis module polls the instruction queue of each device. Each time it polls, it takes out the highest priority instruction in the instruction queue of each device according to the scheduling priority rule, converts the type of the taken instruction into a message, and pushes it into the message queue of the working thread pool as a message to be processed; S4: The idle working thread in the thread pool of the collection and analysis module takes out the pending messages from the message queue of the working thread pool in turn, reads the socket information of the device in the pending messages, sends instructions to the corresponding device through the socket information, receives the response data returned by the device, parses the response data, obtains the parsing result, and stores the parsing result in the database; S5: The business application obtains the corresponding data of the device from the database according to the device information.

2. The high-concurrency access and data collection method for industrial Internet of Things according to claim 1 is characterized in that: The instruction queue includes a write instruction queue, a read device status queue, and a read history data queue. The scheduling priority rule is set according to the importance of the business corresponding to the instruction queue.

3. The high-concurrency access and data collection method for industrial Internet of Things according to claim 2 is characterized in that: The scheduling priority rule is specifically as follows: if there are instructions in the write instruction queue, then take out an instruction from the write instruction queue; otherwise, determine whether there are instructions in the read history data queue, and if there are instructions in the read history data queue, then take out an instruction from the read history data queue; otherwise, determine whether there are instructions in the read device status queue, and if there are instructions in the read device status queue, then take out an instruction from the read device status queue; otherwise, continue to wait for the next polling.

4. The high-concurrency access and data collection method for industrial Internet of Things according to claim 1 is characterized in that: The communication protocol of the communication protocol plug-in port is a long connection communication protocol.

5. The high-concurrency access and data collection method for industrial Internet of Things according to claim 4 is characterized in that: The long connection communication protocols include TCP, HTTP, MQTT, MODBUS RTU, and MODBUS TCP.

6. The high-concurrency access and data collection method for industrial Internet of Things according to claim 1 is characterized in that: The database is a time series database.

7. A system for the high-concurrency access and data collection method for the industrial Internet of Things according to claim 1, characterized in that: It includes a device access module, a collection and analysis module, a work thread pool message queue, a database, a business application and multiple devices. The device access module includes a connection pool and a memory pool. The connection pool is used to monitor and connect each device to realize the communication between each work thread and the device. The memory pool is used to allocate storage resources for each connected device and store the information of each device. The collection and analysis module includes a scheduling thread unit and a thread pool. The scheduling thread unit is used to take out the instructions of each device according to the scheduling priority rule. Multiple work threads are set in the thread pool. The work thread is used to take out instructions from the work thread pool message queue, communicate with the corresponding device according to the instructions, and put the processing results into the database; the work thread pool message queue is used to store the taken out device instructions and convert the instructions into different types; the database is used to store the processing results of the work thread; the business application is used to query the information processed by the device and modify the device memory parameters through the device access module and the collection and analysis module.

8. The system according to claim 7, characterized in that The business application is connected to the memory pool of the device access module for modifying the device memory parameters.

Images