A method for improving data transmission efficiency in an industrial control environment

Abstract

The application relates to the technical field of data transmission, in particular to a method for improving data transmission efficiency in an industrial control environment, which comprises the following steps: S1, a parameter setting interface receives parameter configuration and sets an allowed error range; S2, an input interface receives measurement point data and puts the data into a compression module buffer; S3, a first-order compression module obtains data from the compression module buffer, executes first-order compression, and puts the compressed data block into a second-order compression module buffer to execute second-order compression, and the compressed data is put into an aggregation module aggregation queue; S4, the aggregation module aggregates the compressed block data packets according to MTU, if the output condition is met, the aggregation packet is sent out through an output interface; otherwise, the compressed block data is continuously waited for; and S5, the output interface sends the given compressed block data of the aggregation module out. The application provides a lightweight data compression and aggregation packaging strategy, uses a 1-bit bitmap + 64-bit difference value double-mode encoding mode to combine second-order general compression and MTU packaging technology, and provides extremely high data compression ratio and aggregation transmission performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data transmission, and in particular to a method for improving data transmission efficiency in an industrial control environment. BACKGROUND

[0002] In the industrial 4.0 and edge computing scenarios, a large number of devices generate hundreds of millions of time series data (temperature, vibration, current, etc.) per second, and these data exhibit bimodal characteristics:

[0003] Long-term stability: the data fluctuation is less than 0.1% in 99% of the time, such as constant temperature production lines, pressure parameters, etc.

[0004] Occasional jump: 1% of the time has a sudden and severe fluctuation, such as device failure, high-temperature melting, switch tripping, etc.

[0005] The traditional way of using raw data directly after collecting such data will face the following problems:

[0006] 1. A large number of small data packets account for more than 60%, and the bandwidth utilization is insufficient.

[0007] 2. The general data compression method has insufficient compression rate, and the compressed data packets are smaller, resulting in lower bandwidth utilization.

[0008] 3. The traditional time series data compression algorithm performs well on stable data, but has insufficient adaptability to mixed jump data, and the general compression algorithm consumes more computing power, resulting in high overhead on edge devices.

[0009] Based on the above defects, there is an urgent need for a compression and transmission method that can adapt to the characteristics of industrial time series data, efficiently run on edge devices, and significantly improve network transmission efficiency. SUMMARY

[0010] To overcome the shortcomings of the prior art, the present application provides a method for improving data transmission efficiency in an industrial control environment, which is based on a binary stream compression + time series packaging scheme based on jump perception, which can effectively alleviate the above problems.

[0011] The present application is realized by the following technical solutions:

[0012] A method for improving data transmission efficiency in an industrial control environment is provided, which is realized by the cooperation of a parameter setting interface, an input interface, a compression module, an aggregation module, and an output interface, including the following steps:

[0013] S1, the parameter setting interface receives parameter configuration and sets the allowed error range.

[0014] S2, the input interface receives the measurement point data and puts it into the compression module buffer, and the compression module includes a first-order compression module and a second-order compression module.

[0015] S3. The first-order compression module obtains data from the compression module cache and executes the first-order compression process. The compressed data block enters the second-order compression module, executes the second-order compression process, and puts the second-order compressed data into the aggregation queue of the aggregation module.

[0016] S4. The aggregation module aggregates the compressed block data packet according to the MTU. If the output conditions are met, the aggregated packet is sent out through the output interface; otherwise, it continues to wait for the compressed block data.

[0017] S5. The output interface sends out the compressed block data given by the aggregation module.

[0018] Furthermore, the first-order compression process includes:

[0019] Step X1: Obtain data from the first-order compression queue, including device ID, acquisition timestamp, acquisition interval, data type, data tolerance, and real-time data of the measurement point;

[0020] Step X2: If the ID is entered for the first time, create a compressed structure and fill it with data including device ID, timestamp, time step, data type, and allowable error. Set the initial value to the current measurement point data and create the data sequence number index = 0 for the ID entered.

[0021] If the ID already exists, determine whether the difference between the current measurement point data and the initial value in the existing structure is within the allowable error range. If so, obtain the serial number index of the ID and set the bitmap Bitmap[index+1]=0 of the data structure; otherwise, set the bitmap Bitmap[index+1]=1 of the data structure and write the difference into the jump difference column of the structure. Finally, set index=index+1;

[0022] Step X3: Calculate the number of bytes occupied by the structure to which the ID belongs, where the first-order compressed packet data length S = index + 4B + 8B + 2B + 1B + 8B + the total number of bytes in the jump difference column;

[0023] If S≤1460 and if the next data is a jump data, the above calculation result S≥1460, then the bitmap in this structure is reduced to the index size and put into the second-order compression queue, and finally delete the structure to which this ID belongs from the compression cache;

[0024] Otherwise, continue to step X1.

[0025] Furthermore, the second-stage compression process includes:

[0026] Step Y1: Obtain first-order compressed data from the second-order compression queue;

[0027] Step Y2: compress the data again using the general compression algorithm SprintZ to obtain a second-order compressed data block;

[0028] Step Y3: The second-order compressed block data is placed into the aggregation module queue.

[0029] Furthermore, the aggregation process of the aggregation module is as follows:

[0030] Step Z1: Get the second-order compressed block from the aggregation queue;

[0031] Step Z2: If there is no unit being aggregated, create a new aggregation unit and a variable-length data pointer PTR, calculate the current compressed block size S, convert S into 2B byte data, append it to PTR, and then append the compressed block byte data to PTR; if there is a unit PTR being aggregated, calculate the current compressed block size S, where the current data length of PTR is S1. If S+S1+20+20>MTU, send this aggregated data to the output interface, create a new aggregation unit and a variable-length data pointer PTR, convert S into 2B byte data, append it to PTR, and then append the compressed block byte data to PTR;

[0032] Step Z3: Continue with step Z1.

[0033] Furthermore, the aggregation rule is based on the maximum length of MTU = 1500 bytes, which includes 20 bytes of IP header and 20 bytes of TCP header.

[0034] Beneficial effects of the present invention:

[0035] The present invention provides a lightweight data compression and aggregation packaging strategy in an industrial control environment. It uses a dual-mode encoding mode of "1-bit bitmap + 64-bit difference" combined with second-order universal compression and MTU intelligent packaging technology to provide extremely high data compression ratio and aggregation transmission performance.

[0036] The present invention adopts a minimalist compression mode of first-order dual-mode encoding combined with a general second-order compression algorithm and an MTU intelligent packaging engine, providing a compression ratio of approximately 99.5% in a stable state, while achieving a bandwidth utilization rate of over 90%. In extreme jump environments, the second-order compression process can be abandoned because the second-order compression has become inefficient. In this case, the use of an intelligent aggregation engine can still provide transmission efficiency superior to the traditional mode, which is of great value for data collection efficiency and network / device anomaly monitoring, and can also effectively reduce the high energy consumption problem caused by high-speed encoding and decoding. In addition, the first-order dual-mode encoding mode can also be applied to hardware algorithms such as FPGA due to its simple and efficient working mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1It is the overall flow chart of the present invention.

[0038] Figure 2 This is a schematic diagram of the first-order compression process in the present invention.

[0039] Figure 3 Flowchart of the second-order compression and aggregation data in the present invention.

[0040] Figure 4 It is the data structure of the first-order compression in the present invention.

[0041] Figure 5 The data structure of the aggregation module in the present invention that aggregates the second-order compressed data blocks. DETAILED DESCRIPTION

[0042] In order to clearly illustrate the technical features of this solution, this solution is described below through specific implementation methods.

[0043] like Figure 1-Figure 3 As shown, a method for improving data transmission efficiency in an industrial control environment is implemented by cooperating with a parameter setting interface, an input interface, a compression module, an aggregation module, and an output interface, including the following steps:

[0044] S1. The parameter setting interface receives parameter configuration and sets the allowable error range.

[0045] S2. The input interface receives the measurement point data and puts it into the compression module cache. The compression module includes a first-order compression module and a second-order compression module.

[0046] S3. The first-order compression module obtains data from the compression module cache and executes the first-order compression process. The compressed data block enters the second-order compression module, executes the second-order compression process, and puts the second-order compressed data into the aggregation queue of the aggregation module.

[0047] S4. The aggregation module aggregates the compressed block data packets according to the MTU. If the output conditions are met, the aggregated packet is sent out through the output interface; otherwise, it continues to wait for the compressed block data. The aggregation process of the aggregation module is as follows:

[0048] Step Z1: Get the second-order compressed block from the aggregation queue;

[0049] Step Z2: If there is no unit being aggregated, create a new aggregation unit and a variable-length data pointer PTR, calculate the current compressed block size S, convert S into 2B byte data, append it to PTR, and then append the compressed block byte data to PTR; if there is a unit PTR being aggregated, calculate the current compressed block size S, where the current data length of PTR is S1. If S+S1+20+20>MTU, send this aggregated data to the output interface, create a new aggregation unit and a variable-length data pointer PTR, convert S into 2B byte data, append it to PTR, and then append the compressed block byte data to PTR;

[0050] Step Z3, continue with Z1. The data structure of the aggregation module aggregating the second-order compressed data block is as follows: Figure 5 The aggregation rule is based on the maximum length of MTU = 1500 bytes, which includes the 20-byte IP header and the 20-byte TCP header.

[0051] S4. The output interface sends out the compressed block data given by the aggregation module.

[0052] Among them: The first-order compression process includes:

[0053] Step X1: Obtain data from the first-order compression queue, including device ID, acquisition timestamp, acquisition interval, data type, data tolerance, and real-time data of the measurement point;

[0054] Step X2: If the ID is entered for the first time, create a compressed structure and fill it with data including device ID, timestamp, time step, data type, and allowable error. Set the initial value to the current measurement point data and create the data sequence number index = 0 for the ID entered.

[0055] If the ID already exists, determine whether the difference between the current measurement point data and the initial value in the existing structure is within the allowable error range. If so, obtain the serial number index of the ID and set the bitmap Bitmap[index+1]=0 of the data structure; otherwise, set the bitmap Bitmap[index+1]=1 of the data structure and write the difference into the jump difference column of the structure. Finally, set index=index+1;

[0056] Step X3: Calculate the number of bytes occupied by the structure to which the ID belongs, where the first-order compressed packet data length S = index + 4B + 8B + 2B + 1B + 8B + the total number of bytes in the jump difference column;

[0057] If S≤1460 and if the next data is a jump data, the above calculation result S≥1460, then the bitmap in this structure is reduced to the index size and put into the second-order compression queue, and finally delete the structure to which this ID belongs from the compression cache;

[0058] Otherwise, continue to step X1.

[0059] The compressed data format obtained by the first-order compression module is as follows Figure 4 shown.

[0060] Among them: The second-stage compression process includes:

[0061] Step Y1: Obtain first-order compressed data from the second-order compression queue;

[0062] Step Y2: compress the data again using the general compression algorithm SprintZ to obtain a second-order compressed data block;

[0063] Step Y3: The second-order compressed block data is placed into the aggregation module queue.

[0064] The data transmission method of the present invention is applied to a wind power monitoring network and a constant temperature production line, respectively, because both monitoring networks have the following characteristics:

[0065] 99% of the time, data fluctuations are minimal, meaning they are in a stable state for a long time;

[0066] 60% of the data packets are smaller than 200B, which means that their MTU utilization is less than 35%.

[0067] Based on the above characteristics, in the conventional data collection process, each point needs to collect data thousands of times per second. The collection and transmission of a single point (voltage value) is used as an example to illustrate the beneficial effects of this solution:

[0068] When reporting data collected in a single transaction, it includes the point ID, timestamp, and current voltage value. The data ID occupies 4 bytes, the timestamp occupies 4 bytes, and the voltage value occupies 4 bytes. The combined payload length is 12 bytes. Taking MTU = 1500 bytes as the standard, the reported communication payload is a 20-byte TCP header, a 20-byte IP header, and an actual payload of 12 bytes, totaling 52 bytes, which is far less than the MTU.

[0069] When this solution is used, each collected data will be continuously sent to the first-order compression module, which will compress it according to the steady-state initial value and the steady-state error range; the data format contained in the first-order compressed data block is 4B+8B+2B+1B+8B+1437B bytes of data ( Figure 4), where 4B+8B+2B+1B+8B is the packet header information, and the changing part is the subsequent 1437B, where 1437B contains 11496 bits. Each bit can identify a voltage value after a fixed time step. When in steady state, the corresponding bit of the data is 0, that is, in the extreme case, all 11496 bits are equal to 0 and do not contain jump data; when a jump occurs, the corresponding bit is equal to 1. At the same time, the current voltage value is subtracted from the steady-state initial value to obtain the difference and write it into the jump difference column. The data index position and the number of jump interpolation bytes are calculated. When the index plus the number of bytes in the difference column equals 1437 bytes, a compression is completed.

[0070] When this solution is used, the second-order compression compresses the first-order compressed block again. When the first-order compressed block is full steady-state data, all 1437 bytes in the data block are 0. At this time, the first-order data block is compressed using the SprintZ general compression algorithm. The data block size is approximately 40 bytes, which is much lower than the second-order compression MTU. This compressed block is appended to the payload of the data packet to be sent after adding 2 bytes of data length to the header. At this time, the payload length is 42 bytes. It can be seen that in steady-state, at least 33 42-byte first-order compressed blocks are required to fill the MTU. At this time, the fully loaded data packet to be sent will be It contains 34*11496 collected voltage values, and the time taken is 34*11496 / 1000, which is about 391 seconds. When the first-order compression block contains transition data, due to the small number of transition values, the vast majority of the 1437 bytes in the data block are still all 0s. When using SprintZ compression, a high compression ratio can still be achieved, and the efficiency of subsequent aggregation and transmission is still greatly improved. When the vast majority of voltage values ​​are transition data, a single first-order compression package can still reach full load. Although the efficiency of second-order compression is reduced at this time, aggregation can still increase the effective point data of a single package and slightly improve transmission efficiency.

[0071] Based on the above description, it can be seen that if this solution is not adopted in steady state, 52B * 34 * 11496 = 20324928 bytes of data will need to be sent, which is about 19.4MB. When this solution is adopted, only one fully loaded second-order compressed block needs to be sent once to complete the transmission, with an extremely high compression rate and greatly improved efficiency. Even if individual jump data occurs in the middle, the efficiency can be greatly improved. At the same time, because data only needs to be sent once within 391 seconds, the bandwidth is also fully utilized. Although the transmission efficiency in non-steady state will be lower than that in steady state environment, it can still be maintained much higher than the transmission efficiency without using this solution.

[0072] Of course, the above description is not limited to the above examples. Technical features not described in the present invention can be achieved by or by adopting existing technologies, which will not be described here. The above embodiments and drawings are only used to illustrate the technical solutions of the present invention and are not limitations of the present invention. The present invention is described in detail with reference to the preferred implementation methods. Ordinary technicians in this field should understand that changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention do not depart from the purpose of the present invention and should also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for improving data transmission efficiency in an industrial control environment, implemented by the cooperation of a parameter setting interface, an input interface, a compression module, an aggregation module, and an output interface, characterized in that: The following steps are involved: S1. The parameter setting interface receives parameter configuration and sets the allowable error range; S2, the input interface receives the measurement point data and puts it into the compression module cache, the compression module includes a first-order compression module and a second-order compression module; S3: The first-order compression module obtains data from the compression module cache and performs the first-order compression process. The compressed data blocks enter the second-order compression module, perform the second-order compression process, and put the second-order compressed data into the aggregation queue of the aggregation module. S4. The aggregation module aggregates the compressed block data packet according to the MTU. If the output conditions are met, the aggregated packet is sent out through the output interface; otherwise, it continues to wait for the compressed block data; S5. The output interface sends the compressed block data given by the aggregation module; Among them: The first-order compression process includes: Step X1: Obtain data from the first-order compression queue, including device ID, acquisition timestamp, acquisition interval, data type, data tolerance, and real-time data of the measurement point; Step X2: If the ID is entered for the first time, create a compressed structure and fill it with data including device ID, timestamp, time step, data type, and allowable error. Set the initial value to the current measurement point data and create the data sequence number index = 0 for the ID entered. If the ID already exists, determine whether the difference between the current measurement point data and the initial value in the existing structure is within the allowable error range. If so, obtain the serial number index of the ID and set the bitmap Bitmap[index+1]=0 of the data structure; otherwise, set the bitmap Bitmap[index+1]=1 of the data structure and write the difference into the jump difference column of the structure. Finally, set index=index+1; Step X3: Calculate the number of bytes occupied by the structure to which the ID belongs, where the first-order compressed packet data length S = index + 4B + 8B + 2B + 1B + 8B + the total number of bytes in the jump difference column; If S≤1460 and if the next data is a jump data, the above calculation result S≥1460, then the bitmap in this structure is reduced to the index size and put into the second-order compression queue, and finally delete the structure to which this ID belongs from the compression cache; Otherwise, continue to step X1; The aggregation process of the aggregation module is as follows: Step Z1: Get the second-order compressed block from the aggregation queue; Step Z2: If there is no unit being aggregated, create a new aggregation unit and a variable-length data pointer PTR, calculate the current compressed block size S, convert S into 2B byte data, append it to PTR, and then append the compressed block byte data to PTR; if there is a unit PTR being aggregated, calculate the current compressed block size S, where the current data length of PTR is S1. If S+S1+20+20>MTU, send this aggregated data to the output interface, create a new aggregation unit and a variable-length data pointer PTR, convert S into 2B byte data, append it to PTR, and then append the compressed block byte data to PTR; Step Z3: Continue with step Z1.

2. The method for improving data transmission efficiency in an industrial control environment according to claim 1, characterized in that: The second-stage compression process includes: Step Y1: Obtain first-order compressed data from the second-order compression queue; Step Y2: compress the data again using the general compression algorithm SprintZ to obtain a second-order compressed data block; Step Y3: The second-order compressed block data is placed into the aggregation module queue.

3. The method for improving data transmission efficiency in an industrial control environment according to claim 1, characterized in that: The aggregation rule is based on the maximum length of MTU = 1500 bytes, which includes 20 bytes of IP header and 20 bytes of TCP header.

Images