Data acquisition method and device based on multi-order queue and electronic equipment

Abstract

The invention provides a data acquisition method and device based on a multi-order queue and electronic equipment, and the method comprises the steps: obtaining production data generated by industrial control equipment; according to the acquisition frequency, the production data is distributed to each level queue of the multi-level queue, the queue number of each level queue is adjusted according to the real-time data volume of each queue in each level queue, and each level queue is used for caching the production data of different acquisition frequencies; according to a data packaging strategy, the production data in the multi-order queue is packaged, and the data packaging strategy comprises a timing packaging strategy and a quantitative packaging strategy; and uploading the packaged production data to an industrial Internet of Things platform. Production data acquisition is performed through the multi-order queue, so that the processing pressure of a single queue can be effectively relieved, the data transmission frequency is reduced, and the resource consumption is reduced, thereby improving the efficiency and reliability of data acquisition.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data collection method, device and electronic equipment based on a multi-stage queue. Background Art

[0002] In the current industrial context, high-speed data acquisition technology is one of the key technologies for intelligent manufacturing. With the large-scale application of industrial equipment and sensors, manufacturing companies need to collect massive amounts of real-time data at high frequencies, covering a variety of parameters such as temperature, pressure, vibration, speed, and flow. However, the sharp increase in data volume makes it difficult for traditional data acquisition methods to cope with such large-scale high-speed data flows, and companies face huge challenges in data acquisition, transmission, and processing.

[0003] Among the related technologies, data acquisition technology still has many shortcomings in dealing with complex production environments and high-speed data streams. First, due to the complex production site environment, various types of equipment and sensors are widely used in different industrial scenarios. In order to fully cover low-speed and high-speed sampling scenarios, the data acquisition system needs to flexibly support the communication protocols of different devices to ensure that various devices can be flexibly accessed. The system also needs to have high availability and low latency characteristics to respond to various changes in the production site in a timely manner. Secondly, in high-speed acquisition scenarios, the acquisition frequency usually reaches milliseconds or even lower, which means that the amount of data generated per second has increased by hundreds or even thousands of times compared to traditional acquisition scenarios. In this case, extremely high requirements are placed on data processing capabilities and storage resources. Traditional data processing methods are difficult to meet the requirements of reliability and efficiency.

[0004] Therefore, it is necessary to propose a data acquisition solution that can improve the efficiency and reliability of data acquisition and can be applicable to various communication protocols, so as to fully cover low-speed and high-speed sampling scenarios. Summary of the invention

[0005] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0006] In view of the above-mentioned shortcomings of the prior art, the present invention discloses a data collection method, device and electronic device based on a multi-stage queue, which are used to solve the technical problems of low efficiency and insufficient reliability of data collection in the prior art.

[0007] In the first aspect, the present application provides a data collection method based on a multi-stage queue, the method comprising: acquiring production data generated by industrial control equipment; allocating the production data to queues of each level of the multi-stage queue according to the collection frequency, and adjusting the number of queues of each level of the queue according to the amount of real-time data in each queue in each level of the queue, wherein each level of the queue is used to cache the production data of different collection frequencies; packaging the production data in the multi-stage queue according to a data packaging strategy, the data packaging strategy including a timed packaging strategy and a quantitative packaging strategy; uploading the packaged production data to an industrial Internet of Things platform.

[0008] In one embodiment of the present invention, the production data is allocated to each level queue of a multi-stage queue according to the acquisition frequency, including: classifying the production data according to a preset frequency interval to obtain at least one frequency interval data; and allocating each frequency interval data to a corresponding level list according to a mapping relationship between a preset frequency interval and a queue level.

[0009] In one embodiment of the present invention, the queue quantity of each level queue is adjusted according to the real-time data volume of each queue in each level queue, including: obtaining the real-time data volume of any queue in each level queue; comparing the real-time data volume with a preset first quantity threshold and a second quantity threshold, respectively, the first quantity threshold is smaller than the second quantity threshold; if the real-time data volume is smaller than the first quantity threshold, recycling the arbitrary queue and setting it as an idle queue; if the real-time data volume is larger than the second quantity threshold, setting the idle queue in the target level queue corresponding to the arbitrary queue.

[0010] In one embodiment of the present invention, allocating the production data to queues of each level of a multi-level queue based on the acquisition frequency also includes: obtaining the total amount of real-time data in a queue of any level and the real-time data amount of each queue in the queue of any level; calculating the real-time data amount and the total amount of real-time data to obtain a data volume ratio; determining the allocation weight of each queue in the queue of any level based on the data volume ratio, the allocation weight being inversely proportional to the data volume ratio; and allocating the target frequency interval data corresponding to the queue of any level to the queue with the largest allocation weight in the queue of any level.

[0011] In one embodiment of the present invention, the production data in the multi-stage queue is packaged according to the data packaging strategy, including: responding to a queue detection instruction, reading the production data cached in each queue in the multi-stage queue according to a preset queue flush time, the queue detection instruction carrying the thread waiting time of the packaging thread; and packaging the production data read from each queue in the multi-stage queue respectively.

[0012] In one embodiment of the present invention, the packaging of the production data in the multi-stage queue according to the data packaging strategy also includes: obtaining the real-time data volume of each queue in the multi-stage queue in response to a queue detection instruction, the queue detection instruction carrying the thread waiting time of the packaging thread; judging whether the real-time data volume reaches a preset third quantity threshold; determining the queue whose real-time data volume reaches the third quantity threshold as the target queue; and packaging the production data in the target queue.

[0013] In one embodiment of the present invention, before obtaining the production data generated by the industrial control device, it also includes: configuring an industrial communication protocol, the industrial communication protocol includes communication protocols used by various industrial control devices, and the device identifiers of the various industrial control devices are bound to the communication protocols used in the industrial communication protocol; according to the target device identifier of the industrial control device, matching the target communication protocol corresponding to the industrial control device from the industrial communication protocol; and establishing a communication connection with the industrial control device according to the target communication protocol.

[0014] In one embodiment of the present invention, before uploading the packaged production data to the industrial Internet of Things platform, it also includes: obtaining the data length of the packaged production data; comparing the data length with a preset length threshold, the length threshold representing the length limit of the data received by the industrial Internet of Things platform; if the data length is greater than the length threshold, dividing the packaged production data according to the length threshold.

[0015] In the second aspect, the present application provides a data acquisition device based on a multi-stage queue, the device comprising: a data acquisition module, used to acquire production data generated by industrial control equipment; a queue resource division module, used to distribute the production data to queues of each level of the multi-stage queue according to the acquisition frequency, and adjust the number of queues of each level queue according to the real-time data volume of each queue in each level queue, wherein each level queue is used to cache the production data of different acquisition frequencies; a data packaging module, used to package the production data in the multi-stage queue according to a data packaging strategy, the data packaging strategy including a timed packaging strategy and a quantitative packaging strategy; a data upload module, used to upload the packaged production data to the industrial Internet of Things platform.

[0016] In a third aspect, an electronic device is provided, characterized in that the electronic device comprises: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the multi-stage queue-based data collection method as described in the first aspect.

[0017] As described above, the data collection method, device and electronic device based on multi-stage queues provided by the embodiments of the present invention have the following beneficial effects:

[0018] First, the production data generated by the industrial control equipment is obtained, and then the production data is distributed to each level of the multi-level queue according to the acquisition frequency, and the number of queues of each level is adjusted according to the real-time data volume of each queue in each level of the queue. Each level of the queue is used to cache the production data with different acquisition frequencies. Then, according to the data packaging strategy, the production data in the multi-level queue is packaged. The data packaging strategy includes a timing packaging strategy and a quantitative packaging strategy. Finally, the packaged production data is uploaded to the industrial Internet of Things platform;

[0019] (1) The collection frequency has been significantly improved. By introducing a multi-stage queue architecture and intelligent load balancing technology, the real-time and frequency of data collection have been greatly improved, meeting the needs for efficient capture of time-sensitive data.

[0020] (2) Resource utilization and data collection stability are enhanced. The multi-stage queue and dynamic resource allocation strategy are adopted to effectively improve the utilization of system resources and enhance the stability of the data collection process. Even in complex and changing industrial environments, it can maintain efficient and stable operation.

[0021] (3) Data transmission efficiency and stability are improved simultaneously. The batch packaging strategy significantly reduces the number of data transmissions and reduces the network load, thereby improving the stability and efficiency of data transmission and ensuring the integrity and accuracy of data during transmission.

[0022] (4) Cost-effectiveness is significantly optimized. Compared with the traditional high-frequency data collection solution that uses high-cost equipment, it has achieved cost reduction and efficiency improvement. It not only reduces the hardware procurement cost, but also reduces the consumption of network bandwidth and storage resources by improving data collection efficiency and reliability, thus bringing significant economic benefits to the enterprise.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0025] Figure 1 It is a schematic diagram of an implementation environment of a data acquisition device based on a multi-stage queue shown in an exemplary embodiment of the present application;

[0026] Figure 2 is a flow chart of a data collection method based on a multi-stage queue shown in an exemplary embodiment of the present application;

[0027] Figure 3 is a flowchart of packaging production data shown in an exemplary embodiment of the present application;

[0028] Figure 4 It is an architecture diagram of data collection and transmission of an industrial Internet of Things platform shown in an exemplary embodiment of the present application;

[0029] Figure 5 is a block diagram of a data acquisition device based on a multi-stage queue shown in an exemplary embodiment of the present application;

[0030] Figure 6 It is a structural schematic diagram of an electronic device shown as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0032] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic manner, and thus the illustrations only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0033] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0034] In the current industrial context, the rapid increase in the amount of data has brought huge challenges to enterprises in data collection, transmission and processing. Although high-speed data acquisition technology has been widely used, the inventors of this application have found that this technology still has many shortcomings in dealing with complex production environments and high-speed data flows. First, due to the complex production site environment, various types of equipment and sensors are widely used in different industrial scenarios. In order to fully cover low-speed and high-speed sampling scenarios, the data acquisition system needs to flexibly support the communication protocols of different devices to ensure that various devices can be flexibly accessed, and the system also needs to have high availability and low latency characteristics to respond to various changes in the production site in a timely manner. Secondly, in high-speed acquisition scenarios, the acquisition frequency usually reaches milliseconds or even lower, which means that the amount of data generated per second has increased by hundreds or even thousands of times compared to traditional acquisition scenarios. In this case, extremely high requirements are placed on data processing capabilities and storage resources, and traditional data processing methods are difficult to meet the requirements of reliability and efficiency.

[0035] Therefore, see Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a data acquisition device based on a multi-stage queue, shown in an exemplary embodiment of the present application. Figure 1As shown, the implementation environment may include a data acquisition device 110 based on a multi-stage queue and a computer device 120. The data acquisition device 110 based on a multi-stage queue may be arranged in the computer device 120 to realize data acquisition based on the multi-stage queue. The computer device 120 may be at least one of a desktop graphics processing unit (GPU) computer, a GPU computing cluster, a neural network computer, and the like. By using the multi-stage queue-based data acquisition device 110, the multi-stage queue architecture, intelligent load balancing technology and batch packaging strategy are comprehensively utilized to achieve (1) a significant improvement in the acquisition frequency. By introducing the multi-stage queue architecture and intelligent load balancing technology, the real-time and frequency of data acquisition are greatly improved, meeting the requirements for efficient capture of time-sensitive data; (2) resource utilization and acquisition stability are both enhanced. The multi-stage queue and dynamic resource allocation strategy are adopted to effectively improve the utilization of system resources and enhance the stability of the data acquisition process, so that efficient and stable operation can be maintained even in a complex and changeable industrial environment; (3) data transmission efficiency and stability are improved simultaneously. The batch packaging strategy significantly reduces the number of data transmissions and reduces the network load, thereby improving the stability and efficiency of data transmission and ensuring the integrity and accuracy of data during transmission; (4) cost-effectiveness is significantly optimized. Compared with the traditional high-frequency data acquisition solution that uses high-cost equipment, it achieves cost reduction and efficiency improvement. It not only reduces the hardware procurement cost, but also reduces the consumption of network bandwidth and storage resources by improving data acquisition efficiency and reliability, bringing significant economic benefits to the enterprise.

[0036] See also Figure 2 , Figure 2 is a flowchart of a data collection method based on a multi-stage queue shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown is specifically implemented by the multi-stage queue-based data acquisition device in the implementation environment. It should be understood that the method can also be applied to other exemplary implementation environments and be specifically implemented by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0037] like Figure 2 As shown, in an exemplary embodiment, the data collection method based on the multi-stage queue includes at least steps S210 to S240, which are described in detail as follows:

[0038] Step S210, obtaining production data generated by industrial control equipment.

[0039] The industrial control device may be an industrial PLC (Programmable Logic Controller) device.

[0040] In one embodiment, before obtaining the production data generated by the industrial control device, it also includes: configuring the industrial communication protocol, the industrial communication protocol includes the communication protocol used by various industrial control devices, and the device identification of various industrial control devices is bound to the communication protocol used in the industrial communication protocol; according to the target device identification of the industrial control device, matching the target communication protocol corresponding to the industrial control device from the industrial communication protocol; establishing a communication connection with the industrial control device according to the target communication protocol.

[0041] In this embodiment, by configuring the communication protocols used by various industrial control devices, it is possible to communicate with various industrial control devices to collect production data, thereby being suitable for industrial control devices with various communication protocols. In this way, for industrial control devices equipped with different communication protocols, the protocol conversion call work can be accurately completed, and communication connections can be established with them, so as to obtain the production data generated by the industrial control devices in real time and efficiently.

[0042] In this embodiment, the configured industrial communication protocols include mainstream industrial communication protocols such as PROFINET (Process Field Net, industrial Ethernet communication protocol), ETHERCAT (Ethernet for Control Automation Technology, Ethernet control automation technology), and TC-NET (Total Control Network, bus control network).

[0043] It should be noted that industrial control equipment can be integrated control equipment, that is, it can realize the control and management of complex systems through a single platform, and can generate production data in the production process of various production equipment. Among them, production data includes but is not limited to material data, process data, energy consumption data, and environmental data.

[0044] It should also be noted that, in this embodiment, the production data is high-frequency data.

[0045] In a possible embodiment, in order to collect production data of industrial control equipment, edge controller program is used as a carrier to realize data collection. The edge controller has the capabilities of function library, data fusion, multi-point operation, policy execution, edge cache, transparent transmission, etc., and accurately completes the protocol conversion call work by adapting to mainstream industrial communication protocols such as PROFINET, ETHERCAT, TC-NET, etc., establishes communication connection with industrial control equipment, and obtains the production data generated by industrial control equipment in real time and efficiently.

[0046] As a possible embodiment, the edge controller can directly obtain production data from the industrial control equipment, or establish a communication connection with each production equipment through a configured industrial communication protocol to directly obtain production data from the production equipment.

[0047] In one embodiment, before obtaining the production data generated by the industrial control device, parameter initialization settings are also included, and the parameters include but are not limited to the number of queues (T) of the multi-stage queue, the amount of dequeued data (Q), the queue refresh interval (I), the thread waiting time (W), and the maximum data length of a single message (M max ). Among them, the number of queues (T) of the multi-stage queue is dynamically adjusted according to the amount of data, which can avoid resource waste; the amount of dequeued data (Q) is used to trigger data packaging, which can ensure the timeliness of data processing; the queue refresh interval (I) is the time interval for controlling data dequeuing, which can prevent data processing from being too concentrated; the thread waiting time (W) is the time the thread waits for new data, which can optimize thread utilization; the maximum data length of a single message (M max ) is used to limit the data length of each message to prevent transmission failure due to excessive size of a single data item.

[0048] Step S220, distributes the production data to each level queue of the multi-stage queue according to the collection frequency, and adjusts the queue quantity of each level queue according to the real-time data volume of each queue in each level queue, wherein each level queue is used to cache production data with different collection frequencies.

[0049] In this embodiment, production data is allocated to corresponding level queues according to the frequency of collection, multi-level queue resources are divided, and load balancing is performed on queues of each level in real time. In this way, queue resource utilization can be optimized and the processing pressure of a single queue for high-speed data collection can be alleviated. Multi-level queues are processed in parallel to reduce data transmission times, thereby improving data collection efficiency and stability.

[0050] Specifically, the production data is allocated to each level queue of the multi-stage queue according to the collection frequency, including: classifying the production data according to the preset frequency interval to obtain at least one frequency interval data; according to the mapping relationship between the preset frequency interval and the queue level, each frequency interval data is allocated to the corresponding level list.

[0051] In this embodiment, the production data is divided into at least one frequency interval data according to the preset frequency interval of data collection and the frequency of the production data to be collected, and each frequency interval corresponds to a level queue. Therefore, each frequency interval data can be assigned to the corresponding level list according to the mapping relationship between the frequency interval and the queue level. In this way, the production data is cached by queues of different levels to realize parallel processing of production data, which not only reduces the number of data transmissions and reduces resource consumption, thereby effectively alleviating the processing pressure of a single queue, but also can meet the requirements of different collection frequencies, thereby greatly improving the real-time performance of data collection and meeting the requirements for efficient capture of time-sensitive data.

[0052] For example, the preset frequency intervals include between 1 millisecond and 10 milliseconds, between 10 milliseconds and 100 milliseconds, and between 100 milliseconds and 1000 milliseconds. The production data with a collection frequency between 1 millisecond and 10 milliseconds is defined as ultra-high-speed data, corresponding to the ultra-high-speed queue, the production data with a collection frequency between 10 milliseconds and 100 milliseconds is defined as medium-high-speed data, corresponding to the medium-high-speed queue, and the production data with a collection frequency between 100 milliseconds and 1000 milliseconds is defined as low-high-speed data, corresponding to the low-high-speed queue. Then, according to the collection frequency requirements, the production data is divided into data at three collection frequencies, that is, three frequency interval data. If the collection frequency of a certain production data is 150 milliseconds, it is between 100 milliseconds and 1000 milliseconds and is allocated to the low-high-speed queue for cache processing.

[0053] In another possible embodiment, initially, based on the number of queues (T) of the multi-stage queues, the number of queues of each level is in proportional relationship, wherein the higher the acquisition frequency, the greater the corresponding ratio of the level queue, thereby further meeting the demand for efficient capture of time-sensitive data.

[0054] For example, queue resources are automatically divided and initialized according to the collection frequency of production data (ultra-high speed 1-10ms, medium-high speed 10-100ms, low-high speed 100-1000ms), and resources are pre-allocated in a ratio of 3:2:1 to ensure that data in each frequency band is processed reasonably.

[0055] It should be understood that in any queue of each level of queue, the acquisition frequency corresponding to the cached production data is in the same frequency range, and these data can be production data of the same type from different devices, different types from different devices, or different types from the same device.

[0056] It should also be noted that the production data carries source information, type information and time information, where the source information indicates which device the production data comes from, the type information indicates the data type of the production data, and the time information indicates the time when the production data is generated. After receiving the data packet, the industrial Internet of Things platform can integrate a large number of production data based on the source information, type information and time information carried in the production data, so as to efficiently and securely store the received production data in the time series database.

[0057] Specifically, the number of queues in each level of queues is adjusted according to the real-time data volume of each queue in each level of queues, including: obtaining the real-time data volume of any queue in each level of queues; comparing the real-time data volume with a preset first quantity threshold and a second quantity threshold, respectively, and the first quantity threshold is less than the second quantity threshold; if the real-time data volume is less than the first quantity threshold, any queue is recycled and set as an idle queue; if the real-time data volume is greater than the second quantity threshold, the idle queue is set in the target level queue corresponding to any queue.

[0058] In this embodiment, by setting a threshold value for the number of data recovered by the queue and a threshold value for the number of data supplemented by the queue, and combining the queue recovery and queue supplement methods, the use of multi-stage queue resources is regularly monitored during the production data collection process, and the number of queues at each level is adjusted. In this way, the use of multi-level queues and dynamic resource allocation strategies effectively improves the utilization of system resources, while enhancing the stability of the data collection process, and can maintain efficient and stable operation even in complex and changeable industrial environments.

[0059] For example, during the collection process, the resource usage of multi-stage queues is monitored regularly. If the queue usage is found to be less than 25% during the monitoring process, its resources will be recycled. If the resource usage is higher than 75%, its resources will be supplemented.

[0060] It should be noted that the number of queues at each level will be dynamically adjusted according to the amount of real-time data in its queue, which involves queue recovery and queue replenishment. When the parameters are initialized, the number of queues (T) set for the multi-stage queues changes based on the changes in the number of queues at each level to avoid waste of resources.

[0061] In one embodiment, the production data is distributed to queues of each level of the multi-level queue according to the collection frequency, and also includes: obtaining the total amount of real-time data in any level queue and the real-time data amount of each queue in any level queue; calculating the real-time data amount and the total amount of real-time data to obtain a data amount ratio; determining the allocation weight of each queue in any level queue according to the data amount ratio, and the allocation weight is inversely proportional to the data amount ratio; and distributing the target frequency interval data corresponding to the queue of any level to the queue with the largest allocation weight in any level queue.

[0062] In this embodiment, a weighted polling algorithm is used to achieve load balancing. For each level of queue, the allocation strategy for subsequent production data is determined according to the real-time data volume of each queue. That is, the allocation weight obtained according to the real-time data volume is used to determine to which queue the subsequent production data will be allocated. In this way, production data is allocated according to the allocation weight, and queue resources in queues of the same level are load balanced to avoid jamming and delays caused by excessive queue load, alleviate the pressure on the system caused by high-speed data acquisition, and thus improve the stability and efficiency of data transmission.

[0063] For example, there are three queues a, b, and c in the low-speed queue, and the allocation weights are 0.2, 0.3, and 0.5 respectively. The production data with low-speed collection frequency is allocated to queue c. The data volume ratio changes will be continuously monitored later, thereby causing the allocation weight to change. If the allocation weight of the subsequent queue b is the largest, the subsequent production data with low-speed collection frequency will be allocated to queue b to achieve load balancing between queues.

[0064] As a possible implementation example, for each level of queue, a weighted round-robin algorithm is used to dynamically adjust the weight of each queue to evenly distribute high-speed data to multiple queue caches to improve system efficiency and stability. Assume that a certain level of queue has n queues, and the weight of each queue i is w i , the total weight is When new data arrives, the probability of selecting queue i is To achieve load balancing, new data is assigned to the queue with the largest weight, and the weight of each queue is adjusted regularly to optimize queue resource utilization.

[0065] Step S230, packaging the production data in the multi-stage queues according to the data packaging strategy, where the data packaging strategy includes a timing packaging strategy and a quantitative packaging strategy.

[0066] Among them, the timing packaging strategy refers to packaging data when a specific time is reached; the quantitative packaging strategy refers to packaging data when the amount of data in the queue reaches a specific amount.

[0067] In this embodiment, production data is packaged based on multi-stage queues to achieve batch packaging of production data, significantly reducing the number of data transmissions, reducing network load, and alleviating the processing pressure of uplink transmission, thereby improving the stability and efficiency of data transmission and ensuring the integrity and accuracy of data during transmission.

[0068] In one embodiment, production data in a multi-stage queue is packaged according to a data packaging strategy, including: responding to a queue detection instruction, reading the production data cached in each queue in the multi-stage queue according to a preset queue flush time, the queue detection instruction carrying a thread waiting time of the packaging thread; and packaging the production data read from each queue in the multi-stage queue respectively.

[0069] The queue flushing time is the queue refresh interval (I) mentioned above, and the thread waiting time of the packaging thread is the thread waiting time (W) mentioned above.

[0070] In this embodiment, a W time detection queue is specified, and a data flushing operation is performed at a preset time interval I. If t≥I, batch packaging processing is triggered.

[0071] In this embodiment, batch packaging of production data is achieved through a timed packaging strategy, which significantly reduces the number of data transmissions, reduces the network load, and alleviates the processing pressure of uplink transmission, thereby improving the stability and efficiency of data transmission and ensuring the integrity and accuracy of data during transmission.

[0072] In a possible embodiment, queues of different levels correspond to different queue flushing times, wherein the higher the level, the shorter the time interval of the queue flushing time, thereby further meeting the demand for efficient capture of time-sensitive data.

[0073] In one embodiment, according to the data packaging strategy, the production data in the multi-stage queues are packaged, and the method also includes: in response to a queue detection instruction, obtaining the real-time data volume of each queue in the multi-stage queue, the queue detection instruction carrying the thread waiting time of the packaging thread; judging whether the real-time data volume reaches a preset third quantity threshold; determining the queue whose real-time data volume reaches the third quantity threshold as the target queue; and packaging the production data in the target queue.

[0074] The third quantity threshold is the above-mentioned dequeued data volume (Q), and the thread waiting time of the packing thread is the above-mentioned thread waiting time (W).

[0075] In this embodiment, a queue is checked at a specified time W, and when the amount of data D in the queue reaches the number of queue dequeues Q, that is, if D≥Q, batch packaging processing is triggered.

[0076] In this embodiment, batch packaging of production data is achieved through a quantitative packaging strategy, which significantly reduces the number of data transmissions, reduces the network load, and alleviates the processing pressure of uplink transmission, thereby improving the stability and efficiency of data transmission and ensuring the integrity and accuracy of data during transmission.

[0077] In one possible embodiment, a packaging strategy combining queue flushing time and threshold packaging is used to package batch data. If the time reaches the queue refresh interval (I), data packaging processing is triggered even if the queue data volume D does not reach the dequeue quantity Q. If the queue data volume D reaches the dequeue quantity Q, data packaging processing is triggered even if the time does not reach the queue refresh interval (I).

[0078] In this way, the timing packaging strategy and the quantitative packaging strategy are used together to further improve the efficiency of data processing and resource utilization.

[0079] See also Figure 3 , Figure 3 FIG. 1 is a flowchart of packaging production data shown in an exemplary embodiment of the present application. Figure 3 After the production data is assigned to the multi-stage queue, it will simultaneously determine whether the number of data in the queue meets the number of queues to be dequeued and whether the first element of the queue (time element) has reached the queue flushing time. If neither is met or reached, the queue status will be regularly checked according to the thread waiting time. If the number of data in the queue meets the number of queues to be dequeued, data packaging will be triggered to batch pack and dequeue the production data. If the first element of the queue has reached the queue flushing time, data packaging will be triggered to batch pack and dequeue the production data.

[0080] Step S240, uploading the packaged production data to the industrial Internet of Things platform.

[0081] It should be understood that the data in each data packet may come from the same type of production data from different devices, different types of production data from different devices, or different types of production data from the same device. After receiving the data packet, the industrial Internet of Things platform can integrate a large number of production data based on the source information, type information and time information carried in the production data, thereby efficiently and securely storing the received production data in the time series database.

[0082] In one embodiment, before uploading the packaged production data to the industrial Internet of Things platform, it also includes: obtaining the data length of the packaged production data; comparing the data length with a preset length threshold, the length threshold represents the length limit of the data received by the industrial Internet of Things platform; if the data length is greater than the length threshold, the packaged production data is segmented according to the length threshold.

[0083] In this embodiment, before transmitting data to the time series database of the industrial Internet of Things platform, length detection is required. If the length of the transmitted data exceeds the preset length threshold, data segmentation is required to limit the data length of each piece of information to prevent transmission failure caused by a single piece of data being too large, thereby ensuring the safe transmission and effective storage of data.

[0084] See also Figure 4 , Figure 4 FIG. 1 is an architecture diagram of data collection and transmission of an industrial Internet of Things platform shown in an exemplary embodiment of the present application. Figure 4 As shown in the figure, the industrial PLC device layer includes multiple industrial PLCs, marked as PLC1 to PLCN. These PLCs are responsible for the control and data collection of on-site production equipment. In the edge acquisition controller layer, the edge acquisition controller receives data from the PLC device and supports a variety of mainstream industrial communication protocols, such as PROFINET, TC-NET, ETHERCAT and other mainstream industrial acquisition protocols. This layer, as a data middleware, converts the data of the underlying PLC device into a unified format for subsequent processing, and performs protocol conversion to integrate different types of industrial equipment so that the upper-layer services can process these data and solve the problem of communication compatibility between heterogeneous devices. This layer also includes function libraries, data aggregation, multi-point computing, strategy execution, edge caching and genetics and other functional modules, which are responsible for different processing tasks. In the multi-stage queue layer, the production data is processed in stages and in parallel, and queue load balancing is achieved according to the dynamic resource allocation strategy. In the batch packaging layer, the data after group parallel processing is sorted and compressed according to the batch packaging strategy, and finally uploaded to the industrial Internet of Things platform.

[0085] In this way, data is collected from the underlying industrial equipment, integrated through industrial communication protocols, and introduced into the multi-stage queue architecture and batch packaging strategy, ultimately realizing data collection on the industrial Internet of Things platform, and jointly achieving efficient and reliable data transmission and collection.

[0086] The above-mentioned data collection method based on multi-stage queues first obtains the production data generated by the industrial control equipment, and then distributes the production data to each level of the multi-stage queue according to the collection frequency, and adjusts the number of queues of each level according to the real-time data volume of each queue in each level of the queue. Each level of queue is used to cache production data with different collection frequencies, and then the production data in the multi-stage queue is packaged according to the data packaging strategy. The data packaging strategy includes a timing packaging strategy and a quantitative packaging strategy. Finally, the packaged production data is uploaded to the industrial Internet of Things platform; (1) The collection frequency is significantly improved. By introducing the multi-stage queue architecture and intelligent load balancing technology, the real-time and frequency of data collection are greatly improved, meeting the needs of efficient capture of time-sensitive data; (2) Resource utilization is improved. The system has enhanced both utilization rate and data collection stability. It adopts multi-stage queues and dynamic resource allocation strategies to effectively improve the utilization rate of system resources and enhance the stability of the data collection process. It can maintain efficient and stable operation even in complex and changeable industrial environments. (3) Data transmission efficiency and stability are improved simultaneously. The batch packaging strategy significantly reduces the number of data transmissions and reduces the network load, thereby improving the stability and efficiency of data transmission and ensuring the integrity and accuracy of data during transmission. (4) Cost-effectiveness is significantly optimized. Compared with the traditional high-frequency data collection solution that uses high-cost equipment, it has achieved cost reduction and efficiency improvement. It not only reduces the hardware procurement cost, but also reduces the consumption of network bandwidth and storage resources by improving data collection efficiency and reliability, bringing significant economic benefits to the enterprise.

[0087] See also Figure 5 , Figure 5 is a block diagram of a data acquisition device based on a multi-stage queue, shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The implementation environment shown is a schematic diagram. It should be understood that the device may also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the device is applicable.

[0088] like Figure 5 As shown, in an exemplary embodiment, a data acquisition device 500 based on a multi-stage queue includes at least a data acquisition module 510, a queue resource division module 520, a data packaging module 530 and a data upload module 540, which are described in detail as follows:

[0089] The data acquisition module 510 is used to acquire the production data generated by the industrial control equipment;

[0090] The queue resource division module 520 is used to distribute the production data to each level of the multi-level queue according to the collection frequency, and adjust the number of queues of each level according to the real-time data volume of each queue in each level of the queue, wherein each level of the queue is used to cache the production data of different collection frequencies;

[0091] The data packaging module 530 is used to package the production data in the multi-stage queue according to the data packaging strategy, and the data packaging strategy includes a timing packaging strategy and a quantitative packaging strategy;

[0092] The data uploading module 540 is used to upload the packaged production data to the industrial Internet of Things platform.

[0093] It should be noted that the data acquisition device based on multi-stage queues provided in the above embodiment and the data acquisition method based on multi-stage queues provided in the above embodiment belong to the same concept, and the contents of the operations performed by each module have been described in detail in the method embodiment and will not be repeated here.

[0094] See also Figure 6 , Figure 6 It is a structural schematic diagram of an electronic device shown as an exemplary embodiment of the present application. Figure 6 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0095] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603, such as executing the method in the above embodiment. In the RAM 603, various programs and data required for system operation are also stored. The CPU 601, the ROM 602 and the RAM 603 are connected to each other through the bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.

[0096] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0097] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 609, and / or installed from a removable medium 611. When the computer program is executed by a central processing unit (CPU) 601, various functions defined in the system of the present application are executed.

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0100] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A data collection method based on a multi-stage queue, characterized in that: The method comprises: Obtain production data generated by industrial control equipment; Allocating the production data to each level queue of the multi-level queue according to the acquisition frequency, and adjusting the number of queues of each level queue according to the real-time data volume of each queue in each level queue, wherein each level queue is used to cache the production data of different acquisition frequencies; Packaging the production data in the multi-stage queue according to a data packaging strategy, wherein the data packaging strategy includes a timing packaging strategy and a quantitative packaging strategy; The packaged production data is uploaded to the industrial Internet of Things platform.

2. The data collection method based on multi-stage queues according to claim 1, characterized in that: The distributing the production data to each level queue of the multi-level queue according to the collection frequency includes: Classifying the production data according to a preset frequency interval to obtain at least one frequency interval data; According to the mapping relationship between the preset frequency intervals and the queue levels, each frequency interval data is allocated to a corresponding level list.

3. The data collection method based on multi-stage queues according to claim 2, characterized in that: The step of adjusting the number of queues of each level according to the amount of real-time data of each queue in each level queue includes: Obtaining the real-time data volume of any queue in each of the level queues; Compare the real-time data volume with a preset first quantity threshold and a second quantity threshold respectively, the first quantity threshold being smaller than the second quantity threshold; If the real-time data volume is less than the first quantity threshold, the arbitrary queue is recycled and set as an idle queue; If the real-time data volume is greater than the second quantity threshold, the idle queue is set in the target level queue corresponding to the arbitrary queue.

4. The data collection method based on multi-stage queues according to claim 2, characterized in that: The step of allocating the production data to each level of the multi-level queue according to the acquisition frequency further includes: Obtaining the total amount of real-time data in any level of queue and the real-time data amount of each queue in the any level of queue; Calculating the real-time data volume and the total real-time data volume to obtain a data volume ratio; Determine, according to the data volume ratio, an allocation weight of each queue in the arbitrary level queue, wherein the allocation weight is inversely proportional to the data volume ratio; The target frequency interval data corresponding to the queues of any level are allocated to the queue with the maximum allocation weight among the queues of any level.

5. The data collection method based on multi-stage queues according to claim 1, characterized in that: Packaging the production data in the multi-stage queues according to the data packaging strategy includes: In response to a queue detection instruction, according to a preset queue flushing time, the production data cached in each queue in the multi-stage queue is read, wherein the queue detection instruction carries a thread waiting time of a packing thread; The production data read by each queue in the multi-stage queues is packaged respectively.

6. The data collection method based on multi-stage queues according to claim 1, characterized in that: The step of packaging the production data in the multi-stage queue according to the data packaging strategy further includes: In response to a queue detection instruction, obtaining the real-time data volume of each queue in the multi-stage queue, wherein the queue detection instruction carries a thread waiting time of a packing thread; Determining whether the real-time data volume reaches a preset third volume threshold; Determine the queue whose real-time data volume reaches the third quantity threshold as the target queue; The production data in the target queue is packaged.

7. The data collection method based on multi-stage queues according to any one of claims 1 to 6, characterized in that: Before obtaining the production data generated by the industrial control equipment, the method further includes: Configuring an industrial communication protocol, wherein the industrial communication protocol includes communication protocols used by various industrial control devices, and the device identifiers of the various industrial control devices are bound to the communication protocols used by the industrial communication protocol; According to the target device identifier of the industrial control device, matching a target communication protocol corresponding to the industrial control device from the industrial communication protocol; A communication connection is established with the industrial control device according to the target communication protocol.

8. The data collection method based on multi-stage queues according to claim 7, characterized in that: Before uploading the packaged production data to the industrial Internet of Things platform, the method further includes: Obtaining the data length of the packaged production data; Compare the data length with a preset length threshold, where the length threshold represents a length limit of data received by the industrial Internet of Things platform; If the data length is greater than the length threshold, the packaged production data is segmented according to the length threshold.

9. A data collection device based on a multi-stage queue, characterized in that: The device comprises: A data acquisition module, used to acquire production data generated by industrial control equipment; A queue resource division module, used to distribute the production data to each level queue of the multi-level queue according to the collection frequency, and adjust the number of queues of each level queue according to the real-time data volume of each queue in each level queue, wherein each level queue is used to cache the production data of different collection frequencies; A data packaging module, used for packaging the production data in the multi-stage queue according to a data packaging strategy, wherein the data packaging strategy includes a timing packaging strategy and a quantitative packaging strategy; The data uploading module is used to upload the packaged production data to the industrial Internet of Things platform.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the multi-stage queue-based data acquisition method as described in any one of claims 1 to 8.

Images