Large-throughput low-delay access and scheduling method for industrial internet platform equipment data

Abstract

The invention relates to a large-throughput low-delay access and scheduling method for industrial internet platform equipment data, which comprises the following steps of: separately processing real-time data and historical data reported by edge equipment, and respectively processing the real-time data and the historical data through a real-time data processor and a historical data processor according to the data volume characteristics of the two types of data; and dividing different computing resources to the real-time data processor and the historical data processor. For a real-time data processor, a multi-stage feedback queue task scheduling mechanism is adopted, so that real-time data with smaller data packets can be processed preferentially, real-time data with larger data packets can be processed in time, and data loss caused by overlong waiting time of the smaller data packets is avoided. For a historical data processor, an M / M / C / infinity / infinity / FCFS queuing model in a queuing theory (stochastic service system theory) is adopted to dynamically adjust the size of computing resources, and it is ensured that when queuing is too long, historical data can be processed in time.

Description

Technical Field

[0001] This invention relates to the field of industrial internet platform technology, specifically a method for high-throughput, low-latency access and scheduling of data from industrial internet platform devices. Background Technology

[0002] Industrial Internet platforms are typically structured into three layers: the edge layer, the platform layer, and the application layer. The edge layer builds the data foundation of the industrial Internet platform through large-scale, in-depth data acquisition, as well as protocol conversion and edge processing of heterogeneous data. As a bridge between real devices and the platform, the edge layer integrates functions such as device access, device management, data communication, data storage, and data forwarding. It usually needs to connect to a massive number of devices, receive data uploaded from these devices, and parse, transform, and store the data.

[0003] Data uploaded by edge acquisition devices is typically divided into real-time data and historical data, with real-time data comprising the majority and historical data a smaller portion. Real-time data is collected and generated by the edge acquisition devices in real time and sent to the platform in real time. Real-time data is characterized by a majority of smaller data packets and a smaller portion of larger data packets. Historical data is generated when the edge acquisition devices experience network anomalies, platform upgrades, or other situations that accumulate a large amount of data. After the network is restored, the historical data is uploaded to the platform. Historical data is characterized by a smaller portion of smaller data packets and a majority of larger data packets.

[0004] Currently, platform-side data packet processing typically mixes real-time and historical data, employing a First-Come, First-Served (FCFS) principle. When a program is processing a large historical data packet, the long processing time causes some important smaller data packets that need to be processed quickly to wait too long, resulting in some data corruption and network congestion. To meet the low-latency, high-throughput network performance requirements of the Industrial Internet, reasonable scheduling is needed at the platform level to ensure that each data packet is processed quickly and in an orderly manner. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for high-throughput, low-latency data access and scheduling for industrial internet platform devices. This method separates real-time and historical data processing and allocates different computing resources to their respective processors. For real-time data, a multi-level feedback queue task scheduling mechanism is employed, prioritizing smaller real-time data packets while ensuring timely processing of larger ones. For historical data, the M / M / C / ∞ / ∞ / FCFS queuing model from queuing theory (random service system theory) is used to dynamically adjust the computing resources of the historical data processor. This improves the throughput and low-latency network performance of data access for industrial internet platform devices, preventing data loss due to network congestion.

[0006] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0007] A method for high-throughput, low-latency access and scheduling of equipment data on an industrial internet platform includes the following steps:

[0008] 1) The industrial internet platform assigns a unique device ID and device key to each device, and defines tags for the collected device data;

[0009] 2) Initialize the data processing and computing resources of the industrial internet platform;

[0010] 3) Connect the edge acquisition device to the industrial internet platform using the device ID and device key, and obtain the collected real-time data and historical data. Based on the data identifier, send them to the real-time data processor and the historical data processor respectively.

[0011] 4) The real-time data processor uses a multi-level feedback queue task scheduling mechanism to process data and stores the processed data in a time-series database;

[0012] 5) The historical data processor dynamically adjusts data processing computing resources according to the M / M / C / ∞ / ∞ / FCFS queuing model.

[0013] Step 2) specifically refers to:

[0014] The total number of threads used for data processing is divided into threads for processing real-time data and threads for processing historical data, based on an initial ratio of 4:1.

[0015] Step 4) specifically involves:

[0016] 4.1) The real-time data processor adopts a multi-level feedback queue scheduling strategy, setting up three queues with different priorities, namely Q1, Q2 and Q3. Among them, the first-level queue Q1 and the second-level queue Q2 both adopt the round-robin scheduling strategy, with time slices set to T1 and T2 respectively. The third-level queue Q3 adopts the first-come, first-served scheduling strategy.

[0017] 4.2) Arrange the real-time data to be processed in ascending order of data packet size, input the data packets into queue Q1, and process the data according to time slice T1. Within time T1, the data packets that have been processed are removed from the queue, and the unprocessed data packets are marked as partially processed and transferred to queue Q2.

[0018] 4.3) In queue Q2, the data processing process processes data according to time slice T2. During time T2, the data packets that have been processed are removed from the queue, and the data packets that have not been processed are marked as partially completed and transferred to queue Q3.

[0019] 4.4) In the Q3 queue, each data packet is processed sequentially according to the first-come, first-served principle until one data packet is completed before proceeding to the next.

[0020] 4.5) Update time slices T1 and T2 at regular intervals.

[0021] The time slices T1 and T2 are set as follows:

[0022] The processing time for retrieving historical data from the past month is sorted in ascending order and denoted as t1, t2, ... t3. n ,but

[0023]

[0024] T2=t [( [n+1)*0.8]

[0025] Where n is the total number of data points.

[0026] Step 5) specifically involves:

[0027] 5.1) Read historical data for a set time period to obtain the average service rate μ of a single thread;

[0028] 5.2) Read the arrival status of historical data packets within the current time period and calculate the average arrival rate λ;

[0029] 5.3) Based on the M / M / C / ∞ / ∞ / FCFS queuing model in queuing theory, obtain the value of the number of threads c;

[0030] 5.4) Adjust the number of threads in the real-time data processor and the historical data processor. Set the number of threads in the historical data processor to c, and allocate the remaining threads to the real-time data processor.

[0031] 5.5) Update the value of c at regular intervals.

[0032] The M / M / C / ∞ / ∞ / FCFS queuing model is as follows:

[0033] Assuming the historical average arrival rate of data packets is λ and the average service rate of a single thread is μ, then the metrics of the M / M / C / ∞ / ∞ / FCFS queuing model are:

[0034]

[0035] Where p0 represents the probability that there are no historical data packets in the current system, p n L represents the probability that there are n historical data packets in the system. q W represents the queue length waiting to be processed in the system.q ρ represents the average waiting time for data packets to be processed, ρ represents the average utilization of threads, and i represents the i-th thread.

[0036] The dynamic adjustment of the value of c specifically refers to:

[0037] System settings W q If ≤1, take the smallest value of c that satisfies this condition.

[0038] A high-throughput, low-latency data access and scheduling system for industrial internet platform devices includes a memory and a processor; the memory is used to store computer programs; the processor is used to implement the high-throughput, low-latency data access and scheduling method for industrial internet platform devices when executing the computer programs.

[0039] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements a method for high-throughput, low-latency access and scheduling of data for industrial internet platform devices.

[0040] The present invention has the following beneficial effects and advantages:

[0041] 1. Process real-time data and historical data separately, and allocate different computing resources to real-time data processors and historical data processors according to the characteristics of the data volume of the two types of data.

[0042] 2. For real-time data, a multi-level feedback queue task scheduling mechanism is adopted to ensure that real-time data with smaller data packets can be processed first, while also ensuring that real-time data with larger data packets can be processed in a timely manner, thus avoiding data loss due to excessive waiting time for smaller data packets.

[0043] 3. For historical data, the M / M / C / ∞ / ∞ / FCFS queuing model from queuing theory (theory of random service systems) is used to dynamically adjust the computing resources of the historical data processor to ensure that the data can be processed in a timely manner when the queue is too long. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention;

[0045] Figure 2 It is a flowchart of real-time data processing;

[0046] Figure 3 This is a flowchart illustrating the historical data processing workflow. Detailed Implementation

[0047] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The present invention relates to a method for high-throughput, low-latency access and scheduling of device data on an industrial internet platform. Currently, industrial internet platforms or IoT platforms generally use a first-come, first-served (FCFS) principle to process device data. When a program is processing a large data packet, smaller data packets may experience excessively long waiting times, leading to network congestion and partial data loss. To meet the low-latency, high-throughput network performance requirements of the industrial internet, reasonable scheduling is needed at the platform level to ensure that each data packet is processed quickly and orderly.

[0048] Figure 1 The method flowchart of the present invention includes the following steps:

[0049] Step 1: Register the device. The industrial internet platform will assign a device ID and a device key to the device.

[0050] Step 2: Define data labels. Label attributes include label code, label name, data type, data unit, and upper and lower limits of data.

[0051] Step 3: Allocate computing resources for data processing on the industrial internet platform. The initial ratio of threads for processing real-time data to those for processing historical data is 4:1, and this can be dynamically adjusted according to the task requirements.

[0052] Step 4: The edge acquisition device connects to the industrial internet platform via the MQTT protocol, where the clientID uses the device ID assigned by the industrial internet platform, the password uses the device key assigned by the industrial internet platform, and the acquired real-time data and historical data are uploaded to the industrial internet platform.

[0053] Step 5: After receiving the data uploaded by the edge acquisition device, the industrial internet platform distributes the data packets marked with R to the real-time data processor and the data packets marked with H to the historical data processor.

[0054] Step 6: The real-time data processor uses a multi-level feedback queue task scheduling mechanism for data processing. Three queues with different priorities are set up. The first and second level queues use a round-robin scheduling strategy, and the third level queue uses a first-come, first-served scheduling strategy.

[0055] Step 7: The historical data processor adopts a first-come, first-served scheduling mechanism and dynamically adjusts the number of threads of the historical data processor according to the M / M / C / ∞ / ∞ / FCFS queuing model in queuing theory (random service system theory).

[0056] Registering devices in the industrial internet platform includes two methods: one is to manually operate in the industrial internet platform interface, fill in the device name, product ID, node identification code, and create the device; the other is for edge devices to create the device by calling the device registration API interface, carrying information such as registration code, node identification code, and device name.

[0057] The allocation of data processing computing resources in the industrial internet platform involves dividing the total number of threads used for data processing in the system. The initial ratio of threads for processing real-time data to threads for processing historical data is 4:1, which can be dynamically adjusted according to the task requirements.

[0058] The steps for edge devices to access the industrial internet platform are as follows: the edge device connects to the industrial internet platform via the MQTT protocol, where the clientID uses the device ID assigned by the industrial internet platform and the password uses the device key assigned by the industrial internet platform. After the edge device initiates a connection request, the industrial internet platform will verify the device ID and device key. If the verification is successful, the device is allowed to access; if the verification fails, the device is rejected.

[0059] The real-time data processor employs a multi-level feedback queue task scheduling mechanism for data processing, as follows: Figure 2 As shown, the specific steps are as follows:

[0060] Step 1: Set up three queues with different priorities, namely Q1, Q2, and Q3.

[0061] Step 2: The first-level queue Q1 adopts the round-robin scheduling strategy (RR strategy) with a time slice of T1. The second-level queue Q2 also adopts the round-robin scheduling strategy (RR strategy) with a time slice of T2. The third-level queue Q3 adopts the first-come, first-served scheduling strategy (FCFS).

[0062] Step 3: The time slices T1 and T2 are set as follows: Read the processing time of historical data from the past month, sort them in ascending order of time, and denot them as t1, t2...t n ,but

[0063]

[0064] T2=t [(n+1)*0.8]

[0065] Step 4: Arrange the real-time data to be processed in ascending order of data packet size. Then, the data packets first enter the Q1 queue. The data processing process processes the data according to the pre-set time slice T1. Within the T1 time, the data packets that have been processed are removed from the queue, and the unprocessed data packets are marked as partially completed and transferred to the Q2 queue.

[0066] Step 5: In queue Q2, the data processing process processes data according to time slice T2. During time T2, the data packets that have been processed are removed from the queue, and the unprocessed data packets are marked as partially processed and transferred to queue Q3.

[0067] Step 6: In the Q3 queue, each data packet is processed sequentially according to the first-come, first-served principle until one data packet is completed before proceeding to the next.

[0068] Step 7: Time slices T1 and T2 are updated at regular intervals.

[0069] Figure 3 This is a schematic diagram of the historical data processing flow. The number of threads in the historical data processor is dynamically adjusted. The specific steps are as follows:

[0070] Step 1: Based on the characteristics that the arrival of historical data packets within a certain period follows a Poisson distribution and the data processing time follows a negative exponential distribution, the M / M / C / ∞ / ∞ / FCFS queuing model in queuing theory (theory of random service systems) is used to optimize and adjust the number of threads of the historical data processor.

[0071] Step 2: When the historical average arrival rate of data packets is λ and the average service rate of a single thread is μ, some metrics of the M / M / C / ∞ / ∞ / FCFS queuing model are as follows:

[0072]

[0073]

[0074] Where p0 represents the probability that there are no historical data packets in the current system, p n L represents the probability that there are n historical data packets in the system. q W represents the queue length waiting to be processed in the system. q ρ represents the average waiting time for historical data packets to be processed, ρ represents the average utilization of threads, and i represents the i-th thread.

[0075] Step 3: From the above formulas, it can be seen that when ρ > 1, the rate at which data packets enter the system is greater than the rate at which they leave the system, leading to an increasing number of queues in the system. It is necessary to increase the number of threads in the historical data processor in a timely manner; otherwise, data loss will occur over time. When ρ = 1, the rate at which data packets enter the system is equal to the rate at which they leave the system, and the queue length in the system remains unchanged, also requiring an increase in the number of threads. When ρ < 1, λ and μ in the system are fixed, and W... q As c increases, it decreases. To ensure that a large number of historical data packets can be processed in a timely manner, W is set in the system. q ≤1, thus determining the value of c that meets the requirements.

[0076] Step 4: Read historical data from the past three months and calculate the average service rate μ of a single thread in the system;

[0077] Step 5: Read the arrival status of historical data packets in the system over the past hour, calculate the average arrival rate λ, and dynamically adjust the value of c according to the strategy in Step 3.

[0078] Step 6: Adjust the number of threads for the real-time data processor and the historical data processor. The historical data processor has c threads, and the rest are allocated to the real-time data processor.

[0079] Step 7: Repeat step 5 every hour.

[0080] The specific implementation methods will be described in further detail below.

[0081] In this embodiment, for real-time data processing, the real-time data in the system for a certain period of time is shown in Table 1. The processing time of historical data from the past month is read, resulting in time slices T1 = 82ms and T2 = 131ms. All data packets first enter the first-level queue Q1. After each data packet is processed by time slice T1, data packets with sequence numbers 1, 2, 4, 5, 6, 9, 13, 14, 15, 16, 17, and 20 are processed and removed from the queue, while the other data packets enter the second-level queue Q2. After each data packet is processed by time slice T2, data packets with sequence numbers 7, 8, 18, and 19 are processed and removed from the queue, while the other data packets enter the third-level queue Q3. In queue Q3, data packets with sequence numbers 3, 10, 11, and 12 are processed sequentially. Through processing by the three queues, most of the smaller data packets can be processed in Q1 and Q2, and larger data packets can also be partially processed.

[0082] Table 1

[0083] Serial Number Device ID Timestamp Data packet size (kb) Estimated processing time (ms) 1 100035 1725936037002 1.22 12 2 100035 1725936037022 2.85 28 3 100037 1725936037023 27.65 270 4 100005 1725936037045 7.56 75 5 100043 1725936037047 5.42 54 6 100107 1725936037051 0.79 8 7 100043 1725936037089 14.25 140 8 100048 1725936037100 10.11 100 9 100024 1725936037127 5.36 53 10 100272 1725936037206 24.25 240 11 100279 1725936037245 395.58 395 12 100272 1725936037248 25.47 250 13 100475 1725936037248 5.98 59 14 100025 1725936037369 3.56 35 15 100475 1725936037895 4.25 42 16 100068 1725936038106 4.36 43 17 100103 1725936038115 1.58 15 18 100156 1725936038198 11.5 110 19 100252 1725936038365 14.65 140 20 100039 1725936038452 4.98 50

[0084] In this embodiment, for the processing of historical data, the queuing status of historical data packets in the system for a certain period is selected. Some parameters set in the system are read, where μ = 2.2 and the initial value of c is 8. The arrival status of historical data packets is observed for 8 consecutive hours. Specific data is shown in Table 2. In the table, the time periods numbered 1 and 8 have ρ < 1 and satisfy W... q ≤1, no thread count adjustment needed; for time period 2, ρ=1, thread count needs adjustment to ensure that the adjusted ρ′<1 and W q If ρ ≤ 1, the number of threads after adjustment is 9; for time periods numbered 3, 4, 5, 6, and 7, if ρ > 1, the number of threads needs to be adjusted to ensure that the condition ρ′ < 1 and W is satisfied after adjustment. q ≤1. As can be seen from the data in the table, different average arrival rates correspond to different numbers of threads.

[0085] Table 2

[0086]

Claims

1. A method for high-throughput, low-latency access and scheduling of equipment data in an industrial internet platform, characterized in that, Includes the following steps: 1) The industrial internet platform assigns a unique device ID and device key to each device, and defines tags for the collected device data; 2) Initialize the data processing and computing resources of the industrial internet platform; 3) Connect the edge acquisition device to the industrial internet platform using the device ID and device key, and obtain the collected real-time data and historical data. Based on the data identifier, send them to the real-time data processor and the historical data processor respectively. 4) The real-time data processor uses a multi-level feedback queue task scheduling mechanism to process data and stores the processed data in a time-series database; 5) The historical data processor dynamically adjusts data processing computing resources according to the M / M / C / ∞ / ∞ / FCFS queuing model.

2. The method for high-throughput, low-latency access and scheduling of industrial internet platform equipment data according to claim 1, characterized in that, Step 2) specifically refers to: The total number of threads used for data processing is divided into threads for processing real-time data and threads for processing historical data, based on an initial ratio of 4:

1.

3. The method for high-throughput, low-latency access and scheduling of industrial internet platform equipment data according to claim 1, characterized in that, Step 4) specifically involves: 4.1) The real-time data processor adopts a multi-level feedback queue scheduling strategy, setting up three queues with different priorities, namely Q1, Q2 and Q3. Among them, the first-level queue Q1 and the second-level queue Q2 both adopt the round-robin scheduling strategy, with time slices set to T1 and T2 respectively. The third-level queue Q3 adopts the first-come, first-served scheduling strategy. 4.2) Arrange the real-time data to be processed in ascending order of data packet size, input the data packets into queue Q1, and process the data according to time slice T1. Within time T1, the data packets that have been processed are removed from the queue, and the unprocessed data packets are marked as partially processed and transferred to queue Q2. 4.3) In queue Q2, the data processing process processes data according to time slice T2. During time T2, the data packets that have been processed are removed from the queue, and the data packets that have not been processed are marked as partially completed and transferred to queue Q3. 4.4) In the Q3 queue, each data packet is processed sequentially according to the first-come, first-served principle until one data packet is completed before proceeding to the next. 4.5) Update time slices T1 and T2 at regular intervals.

4. The method for high-throughput, low-latency access and scheduling of industrial internet platform equipment data according to claim 3, characterized in that, The time slices T1 and T2 are set as follows: The processing time for retrieving historical data from the past month is sorted in ascending order and denoted as t1, t2, ... t3. n ,but T2=t [(n+1)*0.8] Where n is the total number of data points.

5. The method for high-throughput, low-latency access and scheduling of industrial internet platform equipment data according to claim 1, characterized in that, Step 5) specifically involves: 5.1) Read historical data for a set time period to obtain the average service rate μ of a single thread; 5.2) Read the arrival status of historical data packets within the current time period and calculate the average arrival rate λ; 5.3) Based on the M / M / C / ∞ / ∞ / FCFS queuing model in queuing theory, obtain the value of the number of threads c; 5.4) Adjust the number of threads in the real-time data processor and the historical data processor. Set the number of threads in the historical data processor to c, and allocate the remaining threads to the real-time data processor. 5.5) Update the value of c at regular intervals.

6. The method for high-throughput, low-latency access and scheduling of industrial internet platform equipment data according to claim 5, characterized in that, The M / M / C / ∞ / ∞ / FCFS queuing model is as follows: Assuming the historical average arrival rate of data packets is λ and the average service rate of a single thread is μ, then the metrics of the M / M / C / ∞ / ∞ / FCFS queuing model are: Where p0 represents the probability that there are no historical data packets in the current system, p n L represents the probability that there are n historical data packets in the system. q W represents the queue length waiting to be processed in the system. q ρ represents the average waiting time for data packets to be processed, ρ represents the average utilization of threads, and i represents the i-th thread.

7. The method for high-throughput, low-latency access and scheduling of industrial internet platform equipment data according to claim 5, characterized in that, The dynamic adjustment of the value of c specifically refers to: System settings W q If ≤1, take the smallest value of c that satisfies this condition.

8. A system for high-throughput, low-latency access and scheduling of equipment data on an industrial internet platform, characterized in that: It includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement, when executing the computer program, a method for high-throughput, low-latency access and scheduling of industrial internet platform equipment data as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements a method for high-throughput, low-latency access and scheduling of industrial internet platform equipment data as described in any one of claims 1-7.

Images