Data compression and transmission optimization method and device for cross-level industrial data pipeline

Abstract

The application provides a data compression and transmission optimization method and device for a cross-level industrial data pipeline, wherein the device for the data compression and transmission optimization method of the cross-level industrial data pipeline comprises a field layer data processing unit, an edge layer cooperative optimization unit and a cloud end management and control unit, each unit is interconnected through an industrial data pipeline, and each unit is internally modularly designed; the device is a modular distributed architecture, which is suitable for the hardware deployment characteristics of the industrial field layer and the edge layer; the data compression and transmission optimization method of the cross-level industrial data pipeline is based on a cross-level architecture composed of industrial field layer nodes, edge layer processing nodes and cloud end management and control nodes, and is respectively optimized for two industrial data pipelines of the field-edge and the edge-cloud end, so as to realize the matching of data characteristics and compression algorithms, the matching of network states and transmission parameters and the cooperation of compression and transmission.

Description

Technical Field

[0001] This invention relates to the fields of industrial internet, data communication and data compression technology, and in particular to a data compression and transmission optimization method and apparatus for a cross-level industrial data pipeline. Background Technology

[0002] As we all know, the popularization of the Industrial Internet has promoted the cross-level flow of industrial data at the field, edge, and cloud levels, and the industrial data pipeline has become the core carrier for data interaction at all levels.

[0003] However, industrial data is characterized by strong heterogeneity, massive volume, complex types, and high real-time requirements for critical data. Existing cross-level industrial data pipeline data transmission solutions have many shortcomings: First, using a single compression algorithm to process all industrial data cannot adapt to the characteristics of heterogeneous data such as time-series sensor data, structured equipment parameters, and unstructured images or videos, resulting in low compression efficiency and redundant data still consuming a large amount of transmission bandwidth. Second, there is no priority scheduling mechanism for data transmission, and core production control data is transmitted together with ordinary equipment log data, making it impossible to guarantee the real-time performance and reliability of critical data. Third, the cross-level network environments vary greatly, with the field layer mostly using wired private networks and the edge-cloud mostly using wireless or public networks, but the transmission parameters are fixed and cannot be dynamically adjusted according to network bandwidth, packet loss rate, and other conditions, easily leading to packet loss and excessive latency. Fourth, data compression and transmission are disconnected from each other, and no collaborative optimization mechanism has been established, resulting in poor matching between compression rate and transmission rate and low overall data pipeline transmission efficiency. Fifth, existing devices lack modular design for cross-level architectures, have poor adaptability, and are difficult to deploy flexibly on hardware nodes at different industrial levels.

[0004] The aforementioned problems result in low bandwidth utilization, high data transmission latency, and insufficient reliability of critical data in cross-level industrial data pipelines, which restricts the implementation of industrial edge computing and cloud-based remote control. Summary of the Invention

[0005] One objective of this invention is to provide a data compression and transmission optimization method and apparatus for cross-level industrial data pipelines, in order to overcome the problems of low bandwidth utilization, large data transmission latency, and insufficient reliability of key data in existing cross-level industrial data pipeline data compression and transmission optimization methods and apparatuses, which restrict the implementation effect of industrial edge computing and cloud-based remote control.

[0006] Another objective of this invention is to provide a data compression and transmission optimization method and apparatus for cross-level industrial data pipelines. The method comprises a cross-level architecture based on industrial field layer nodes, edge layer processing nodes, and cloud management nodes. It optimizes two segments of the industrial data pipeline: field-edge and edge-cloud. Core steps include data feature perception and classification, type-based adaptive compression, data priority grading and encapsulation, cross-level network status monitoring, cross-level dynamic transmission scheduling, compression-transmission collaborative optimization, and data decompression and integrity verification. By adapting to heterogeneous industrial data through type-based compression, ensuring core data transmission needs through priority scheduling, adapting to cross-level network differences through dynamic transmission parameter adjustment, and achieving dynamic matching between the two stages through compression-transmission collaborative optimization, the overall transmission efficiency and reliability of the cross-level industrial data pipeline are improved.

[0007] Another objective of this invention is to provide a data compression and transmission optimization method and apparatus for cross-level industrial data pipelines. The data feature perception and classification steps are as follows: after the field layer or edge layer nodes collect industrial data, they perceive the data type, temporal features, data redundancy, and importance features through feature extraction, and classify the industrial data into three categories: temporal sensing type, structured parameter type, and unstructured media type. The data is then marked with a primary priority based on industrial business needs, providing a feature basis for subsequent compression and scheduling.

[0008] Another objective of this invention is to provide a data compression and transmission optimization method and apparatus for cross-level industrial data pipelines. The type-based adaptive compression step specifically involves: pre-setting dedicated compression algorithms adapted to three types of industrial data; calling the corresponding algorithms for type-based compression based on the data classification results; using differential compression algorithm for time-series sensor data, dictionary compression algorithm for structured parameter data, and lightweight media compression algorithm for unstructured media data; and adding a unique compression identifier and a data integrity check code to the data after compression. The compression identifier is used by the receiving end to match the decompression algorithm, and the check code is used for data integrity verification.

[0009] Another objective of this invention is to provide a data compression and transmission optimization method and apparatus for cross-level industrial data pipelines. The data priority classification and encapsulation steps specifically involve: dividing the compressed data into three priority levels—core production level, equipment monitoring level, and log statistics level—based on the business needs of the industrial site; segmenting and encapsulating the data according to priority; assigning exclusive priority identifiers and transmission thresholds to data packets of different priorities; and allocating the highest transmission resource occupancy threshold to the core production level data packets to ensure their transmission priority.

[0010] Another objective of this invention is to provide a data compression and transmission optimization method and apparatus for cross-level industrial data pipelines. The cross-level network status monitoring step specifically involves: edge layer processing nodes monitoring network status parameters of the field-edge and edge-cloud data pipelines in real time, including network bandwidth, packet loss rate, and transmission delay. After normalizing each parameter, real-time network status feature values ​​are generated to provide a quantitative basis for adjusting transmission parameters. The frequency of network status monitoring is dynamically adjusted according to the transmission rate of industrial data; the higher the data transmission rate, the higher the monitoring frequency.

[0011] Another objective of this invention is to provide a data compression and transmission optimization method and apparatus for cross-level industrial data pipelines. The cross-level dynamic transmission scheduling step specifically involves: dynamically adjusting transmission parameters, including data packet size, transmission rate, and retransmission mechanism, for the field-edge and edge-cloud data pipelines respectively, based on network status characteristics. Simultaneously, transmission scheduling is performed according to data priority, with high-priority data packets preempting transmission resources and low-priority data packets being transmitted during off-peak hours. When network resources are scarce, low-priority data packets can be temporarily stored and transmitted after the network status improves.

[0012] Another objective of this invention is to provide a data compression and transmission optimization method and apparatus for cross-level industrial data pipelines. The compression-transmission collaborative optimization step specifically involves: establishing a collaborative mapping relationship between compression ratio and transmission parameters; dynamically increasing the compression ratio of the corresponding data and reducing bandwidth usage when network bandwidth is reduced, packet loss rate or transmission delay exceeds a preset threshold; and appropriately reducing the data compression ratio and reducing the decompression computation overhead at the receiving end when the network status recovers to a preset good range. The collaborative optimization strategy is executed by the edge layer processing node and globally controlled by the cloud management node.

[0013] Another objective of this invention is to provide a data compression and transmission optimization method and apparatus for cross-level industrial data pipelines. The data decompression and integrity verification steps specifically involve: the receiving node calling the corresponding decompression algorithm based on the compression identifier in the data packet to decompress the data; verifying the data integrity using a checksum; if the data integrity verification fails, triggering a tiered retransmission mechanism based on data priority; core production-level data packets immediately triggering single-packet retransmission; and equipment monitoring-level and log statistics-level data packets triggering batch retransmission and timed retransmission, respectively, to avoid low-priority data retransmission occupying core network resources. This cross-level industrial data pipeline data compression and transmission optimization method is based on a cross-level architecture consisting of industrial field layer nodes, edge layer processing nodes, and cloud management nodes. It optimizes the field-edge and edge-cloud segments of the industrial data pipeline separately, achieving matching of data characteristics and compression algorithms, matching of network status and transmission parameters, and coordinated compression and transmission.

[0014] Another objective of this invention is to provide a method and apparatus for optimizing data compression and transmission in a cross-level industrial data pipeline. The apparatus for optimizing data compression and transmission in a cross-level industrial data pipeline executes the method and includes a field-level data processing unit, an edge-level collaborative optimization unit, and a cloud-based management unit. Each unit is interconnected through the industrial data pipeline, and each unit is adapted to lightweight or high-performance configurations based on the hardware resource characteristics of the industrial level. It can be flexibly deployed on hardware nodes at the corresponding level. The units collaboratively realize the acquisition, classification, compression, scheduling, transmission, decompression, and verification of industrial data, completing the optimization of compression and transmission in the cross-level data pipeline. The device for optimizing data compression and transmission in a cross-level industrial data pipeline is a modular distributed architecture, adapted to the hardware deployment characteristics of the industrial field layer and edge layer.

[0015] Another objective of this invention is to provide a data compression and transmission optimization method and apparatus for a cross-level industrial data pipeline. The field-level data processing unit includes a feature-aware classification module, an adaptive compression module, and a primary priority marking module, adapting to the limited hardware resources of the field layer to achieve local data acquisition, classification, adaptive compression, and primary priority marking of industrial data. The edge-level collaborative optimization unit includes a priority-level encapsulation module, a network status monitoring module, a dynamic transmission scheduling module, a compression-transmission collaboration module, and a decompression verification module, achieving fine-grained data classification, network monitoring, dynamic scheduling, collaborative optimization, and decompression verification. The cloud-level management unit includes a global collaborative scheduling module, a decompression verification module, and a data feedback module, achieving closed-loop feedback adjustment of edge-cloud data decompression verification, global scheduling, and optimization strategies. The core key and optimization strategy parameters are encrypted for storage, ensuring the security of the device operation.

[0016] According to one aspect of the present invention, the present invention provides a data compression and transmission optimization method for a cross-level industrial data pipeline. The method includes: optimization of two segments of the industrial data pipeline—field-to-edge and edge-to-cloud—based on a cross-level architecture consisting of industrial field layer nodes, edge layer processing nodes, and cloud management nodes. The core steps include data feature perception and classification, type-based adaptive compression, data priority grading and encapsulation, cross-level network status monitoring, cross-level dynamic transmission scheduling, compression-transmission collaborative optimization, and data decompression and integrity verification. By adapting to heterogeneous industrial data through type-based compression, ensuring core data transmission needs through priority scheduling, adapting to cross-level network differences through dynamic transmission parameter adjustment, and achieving dynamic matching between the two stages through compression-transmission collaborative optimization, the overall transmission efficiency and reliability of the cross-level industrial data pipeline are improved.

[0017] In one embodiment, the data feature perception and classification step specifically involves: after the field layer or edge layer node collects industrial data, it perceives the data type, temporal features, data redundancy, and importance features through feature extraction, and classifies the industrial data into three categories: temporal sensing type, structured parameter type, and unstructured media type. It also performs primary priority marking on the data in conjunction with industrial business needs, providing feature basis for subsequent compression and scheduling.

[0018] In one embodiment, the type-based adaptive compression step specifically involves: pre-setting dedicated compression algorithms adapted to three types of industrial data; calling the corresponding algorithms for type-based compression based on the data classification results; using differential compression algorithm for time-series sensor data, dictionary compression algorithm for structured parameter data, and lightweight media compression algorithm for unstructured media data; and adding a unique compression identifier and data integrity check code to the data after compression. The compression identifier is used by the receiving end to match the decompression algorithm, and the check code is used for data integrity verification.

[0019] In one embodiment, the data priority classification and encapsulation steps specifically involve: dividing the compressed data into three priority levels—core production level, equipment monitoring level, and log statistics level—based on the business needs of the industrial site; segmenting and encapsulating the data according to priority; assigning exclusive priority identifiers and transmission thresholds to data packets of different priorities; and allocating the highest transmission resource occupancy threshold to the core production level data packets to ensure their transmission priority.

[0020] In one embodiment, the cross-level network status monitoring step specifically involves: the edge layer processing node monitoring the network status parameters of the two data pipelines, namely, the field-edge and edge-cloud, in real time, including network bandwidth, packet loss rate, and transmission delay. After normalizing each parameter, real-time network status feature values ​​are generated to provide a quantitative basis for adjusting transmission parameters. The frequency of network status monitoring is dynamically adjusted according to the transmission rate of industrial data. The higher the data transmission rate, the higher the monitoring frequency.

[0021] In one embodiment, the cross-level dynamic transmission scheduling step specifically involves: dynamically adjusting transmission parameters, including data packet size, transmission rate, and retransmission mechanism, according to the network status characteristics of the two data pipelines: the field-edge and the edge-cloud. Simultaneously, transmission scheduling is performed according to data priority, with high-priority data packets preempting transmission resources and low-priority data packets being transmitted during off-peak hours. When network resources are scarce, low-priority data packets can be temporarily stored and transmitted after the network status improves.

[0022] In one embodiment, the compression-transmission collaborative optimization step specifically involves: establishing a collaborative mapping relationship between compression ratio and transmission parameters; dynamically increasing the compression ratio of the corresponding data and reducing bandwidth usage when network bandwidth is reduced, packet loss rate or transmission delay exceeds a preset threshold; and appropriately reducing the data compression ratio and reducing the decompression computation overhead at the receiving end when the network status recovers to a preset good range. The collaborative optimization strategy is executed by the edge layer processing node and globally controlled by the cloud management node.

[0023] In one embodiment, the data decompression and integrity verification steps are as follows: the receiving node calls the corresponding decompression algorithm to decompress the data according to the compression identifier in the data packet, verifies the integrity of the data through the check code, and if the data integrity verification fails, a hierarchical retransmission mechanism is triggered according to the data priority. Core production-level data packets are immediately retransmitted as single packets, while device monitoring-level and log statistics-level data packets are retransmitted in batches and at regular intervals, respectively, to avoid low-priority data retransmission occupying core network resources.

[0024] According to another aspect of the present invention, the present invention provides an apparatus for a cross-level industrial data pipeline data compression and transmission optimization method. The apparatus executes the cross-level industrial data pipeline data compression and transmission optimization method. The apparatus is a modular distributed architecture, including a field-level data processing unit, an edge-level collaborative optimization unit, and a cloud-based management and control unit. Each unit is interconnected through the industrial data pipeline, and each unit is adapted to be lightweight or high-performance according to the hardware resource characteristics of the industrial level. It can be flexibly deployed on the hardware nodes of the corresponding level. The units work together to realize the collection, classification, compression, scheduling, transmission, decompression, and verification of industrial data, thereby completing the compression and transmission optimization of the cross-level data pipeline.

[0025] In one embodiment, the field-layer data processing unit includes a feature-aware classification module, an adaptive compression module, and a primary priority marking module, adapting to the limited hardware resources of the field layer to achieve local acquisition, classification, adaptive compression, and primary priority marking of industrial data. The edge-layer collaborative optimization unit includes a priority-level encapsulation module, a network status monitoring module, a dynamic transmission scheduling module, a compression-transmission collaboration module, and a decompression verification module, achieving fine-grained data classification, network monitoring, dynamic scheduling, collaborative optimization, and decompression verification. The cloud-based management and control unit includes a global collaborative scheduling module, a decompression verification module, and a data feedback module, achieving closed-loop feedback adjustment of edge-cloud data decompression verification, global scheduling, and optimization strategies. The core key and optimization strategy parameters are encrypted and stored to ensure the security of the device operation. Attached Figure Description

[0026] Figure 1This is a schematic diagram of an apparatus for a cross-level industrial data compression and transmission optimization method according to an embodiment of the present invention.

[0027] Figure 2 This is an overall architecture diagram of the apparatus for the cross-level industrial data compression and transmission optimization method according to the above embodiments of the present invention.

[0028] Figure 3 This is a flowchart of the core method for optimizing cross-level industrial data compression and transmission according to the present invention.

[0029] Figure 4 This is a matching diagram of the industrial data classification and adaptive compression algorithm of the present invention.

[0030] Figure 5 This is a diagram illustrating the data priority grading and differentiated encapsulation strategy of the present invention.

[0031] Figure 6 This is the logic diagram for the compression-transmission collaborative optimization of the present invention.

[0032] Figure 7 This is a diagram of the data decompression verification and hierarchical retransmission mechanism of the present invention. Detailed Implementation

[0033] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0034] Those skilled in the art should understand that, in the disclosure of this invention, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting this invention.

[0035] It is understood that the term "a" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0036] See Figures 1 to 7As shown, a data compression and transmission optimization method and apparatus for a cross-level industrial data pipeline according to the present invention are illustrated. The method is based on a cross-level architecture consisting of industrial field layer nodes, edge layer processing nodes, and cloud control nodes. It optimizes two segments of the industrial data pipeline 4: field-edge and edge-cloud. Core steps include data feature perception and classification, type-based adaptive compression, data priority grading and encapsulation, cross-level network status monitoring, cross-level dynamic transmission scheduling, compression-transmission collaborative optimization, and data decompression and integrity verification. The device for the transmission optimization method is a modular distributed architecture. It executes the data compression and transmission optimization method for the cross-level industrial data pipeline. The device for the data compression and transmission optimization method for the cross-level industrial data pipeline includes a field-level data processing unit 1, an edge-level collaborative optimization unit 2, and a cloud-based management and control unit 3. Each unit is interconnected through an industrial data pipeline 4. Each unit is adapted to be lightweight or high-performance according to the hardware resource characteristics of the industrial level. It can be flexibly deployed on the hardware nodes of the corresponding level. The units work together to realize the collection, classification, compression, scheduling, transmission, decompression, and verification of industrial data, and complete the compression and transmission optimization of the cross-level data pipeline.

[0037] The cross-layer architecture described herein serves to establish a complete flow channel for industrial data from field collection to cloud-based management and control. Through hierarchical division of labor, it achieves efficient collaborative data processing. The field layer focuses on data collection and initial processing, the edge layer is responsible for core collaborative optimization, and the cloud layer coordinates overall control, avoiding excessive load on any single layer. The independent optimization design of the two data pipelines is because the field-edge connection is often a wired private network with a stable but limited bandwidth, while the edge-cloud connection is often wireless or a public network with significant network fluctuations. Separate optimization allows for precise adaptation to different network characteristics. The core function of this method is to solve problems such as poor heterogeneous data compression adaptation, chaotic transmission priorities, mismatched network states, and disconnect between compression and transmission through closed-loop optimization throughout the entire process. The modular distributed architecture of the device is designed to adapt to the differentiated needs of the industrial field layer (limited hardware resources), the edge layer (needing to balance performance and flexibility), and the cloud layer (requiring high-performance processing), enabling on-demand deployment and flexible expansion, and ensuring the feasibility of the method in different industrial scenarios.

[0038] First, refer to Figures 2 to 7As shown, in this embodiment, the data feature perception and classification steps are implemented collaboratively by the feature perception and classification module 5 and the primary priority marking module 7 of the field layer data processing unit 1. After the field layer nodes (such as sensors, PLCs, etc.) collect industrial data, they transmit the data to the field layer data processing unit 1. The feature perception and classification module 5 starts the feature extraction process and, through a preset feature recognition algorithm, perceives the data type (determines whether it is time-series continuous data, structured key-value data, or media data such as images or videos), time-series features (data acquisition interval, change frequency), data redundancy (percentage of repeated data, information entropy), and importance features (whether it is associated with core businesses such as production control and safety monitoring). Based on the perception results, the industrial data is divided into three categories: time-series sensing data (such as continuous data collected by temperature and pressure sensors), structured parameter data (such as equipment operating parameters and production process setpoints), and unstructured media data (such as equipment inspection images and operating condition monitoring videos).

[0039] The primary priority marking module 7 combines the basic business rules of the industrial site to initially mark three types of data: data directly related to production start-up and shutdown and process control is marked as high initial priority, data related to equipment operation status monitoring is marked as medium initial priority, and data related to log recording and statistical analysis is marked as low initial priority. The core function of the feature perception classification module 5 is to provide accurate data profiles for subsequent compression stages, ensuring that the compression algorithm and data characteristics are accurately matched, and avoiding the problem of low compression efficiency caused by a single algorithm processing all data. The primary priority marking module 7 is to establish preliminary distinction standards for data importance, provide a basis for fine classification of the edge layer, and reduce the data processing pressure of the edge layer.

[0040] Both modules of the field layer data processing unit 1 adopt a lightweight design to adapt to the limited hardware resources of the field layer (such as small PLC memory and weak computing power). The feature extraction algorithm simplifies complex matrix operations, and the primary labeling rules are stored in the form of configuration files, which can be quickly modified according to different industrial scenarios without refactoring the module code, thus improving the scenario adaptability of the device.

[0041] Then, refer to Figures 2 to 7As shown, in this embodiment, the type-based adaptive compression step is implemented by the adaptive compression module 6 of the field layer data processing unit 1. The adaptive compression module 6 has three sets of dedicated compression algorithms pre-set, which are adapted to the characteristics of three types of industrial data: For time-series sensor data, which has strong time sequence and small differences between adjacent data, a differential compression algorithm is adopted. By calculating the difference between adjacent data and storing the difference information, redundant data is greatly reduced. For structured parameter data, which has a large number of repeated fields (such as equipment number and parameter name), a dictionary compression algorithm is adopted. A dictionary mapping table is established for repeated fields, and only the dictionary index is sent during transmission to reduce the amount of data. For unstructured media data, which has high real-time requirements and cannot lose too much information, a lightweight media compression algorithm is adopted. While ensuring the clarity of key information in images or videos, the resolution or compression bit rate is reduced to balance compression efficiency and information integrity.

[0042] After receiving the classification results from the feature perception and classification module 5, the adaptive compression module 6 automatically calls the corresponding algorithm for compression processing: time-series sensor data enters the differential compression channel, structured parameter data enters the dictionary compression channel, and unstructured media data enters the lightweight media compression channel. During the compression process, the module monitors the compression efficiency in real time. When the compression rate of a certain type of data is lower than the preset threshold, the algorithm parameters (such as the step size of differential compression and the dictionary update frequency of dictionary compression) are automatically adjusted to ensure that the compression effect meets the standard.

[0043] After compression, the adaptive compression module 6 adds a unique compression identifier and a data integrity check code to each segment of compressed data: The compression identifier consists of a data type code + algorithm number (such as the identifier corresponding to the time sequence sensor class and the differential compression algorithm), which is used by the receiver to quickly match the decompression algorithm. The check code is generated using the CRC32 algorithm and is calculated based on the compressed data content. It is used to verify whether the data has been tampered with, lost or damaged during transmission.

[0044] The core design principles of the adaptive compression module 6 are adaptability and lightweight design. All three algorithms have been trimmed and optimized to avoid complex calculations consuming too much hardware resources at the field layer. At the same time, the addition of identifiers and verification codes provides a basic guarantee for subsequent transmission and decompression, ensuring that the compressed data can be accurately restored.

[0045] Secondly, see Figures 2 to 7As shown, in this embodiment, the data priority classification and encapsulation steps are implemented by the priority classification and encapsulation module 8 of the edge layer collaborative optimization unit 2. After the edge layer processing node receives the compressed data transmitted from the field layer, it forwards the data to the priority classification and encapsulation module 8. The module first reads the preliminary priority mark of the data, and then, in combination with the refined business needs of the industrial field (such as clarifying the core production level corresponding to production process control and safety interlock related data, the equipment monitoring level corresponding to equipment temperature, vibration and other operating status data, and the log statistics level corresponding to operation logs, data statistics reports and other data through a preset business rule base), the data is divided into three final priorities: core production level, equipment monitoring level and log statistics level.

[0046] For data of three priorities, the module adopts a differentiated fragmentation and encapsulation strategy: core production-level data is encapsulated in small fragments (each fragment is no more than 1KB) to ensure that even if packet loss occurs during transmission, the amount of retransmitted data is small and the time consumption is short; device monitoring-level data is encapsulated in medium fragments (each fragment is no more than 5KB) to balance transmission efficiency and retransmission overhead; and log statistics-level data is encapsulated in large fragments (each fragment is no more than 20KB) to improve the transmission efficiency of batch data.

[0047] Meanwhile, the module adds a unique identifier to each fragmented data packet: priority identifier, fragment sequence number, and transmission resource occupancy threshold. The role of the priority hierarchical encapsulation module 8 is to provide a clear scheduling basis for subsequent transmission scheduling through fine-grained hierarchical and differentiated encapsulation, ensuring that core data is given priority in resource competition. At the same time, through fragmentation processing, the cost of retransmission after data transmission failure is reduced, and transmission reliability is improved.

[0048] The edge layer collaborative optimization unit 2 is designed with medium performance, supports multi-threaded parallel processing of data with different priorities, and can dynamically adjust the number of threads according to the hardware configuration of the edge layer node (such as the number of CPU cores) to avoid resource waste or processing bottlenecks.

[0049] Again, see Figures 2 to 7 As shown, in this embodiment, the cross-level network status monitoring step is implemented by the network status monitoring module 9 of the edge layer collaborative optimization unit 2. This module simultaneously monitors the network status of two industrial data pipelines 4: the field-edge and the edge-cloud. For each pipeline, three core parameters are collected in real time: network bandwidth, packet loss rate, and transmission delay.

[0050] To ensure the accuracy of parameter acquisition, the module adopts a periodic acquisition + trigger-based supplementary acquisition method: the default acquisition period is 1 second, and when the rate of change of a certain parameter is detected to exceed 10%, the acquisition period is automatically shortened to 0.2 seconds until the parameter tends to stabilize. When the data transmission rate exceeds 100Mbps, the acquisition period is fixed at 0.5 seconds to ensure monitoring accuracy under high transmission load.

[0051] The collected raw parameters need to be normalized and converted into network state feature values ​​between 0 and 1. The processing formula is: Feature value = (Original parameter - Minimum value) / (Maximum value - Minimum value), where the maximum and minimum values ​​are preset according to common industrial network scenarios. After normalization, the module will sum the feature values ​​of the three parameters by weight to generate the real-time network state feature value of each pipeline segment.

[0052] The network status monitoring module 9 is designed to provide a quantitative network profile for transmission scheduling and collaborative optimization, preventing transmission strategies from becoming disconnected from network status. At the same time, by dynamically adjusting the acquisition cycle, it reduces the module's own resource consumption while ensuring monitoring accuracy, thus balancing monitoring effectiveness and system performance.

[0053] In addition, see Figures 2 to 7 As shown, in this embodiment, the cross-level dynamic transmission scheduling step is implemented by the dynamic transmission scheduling module 10 of the edge layer collaborative optimization unit 2. After receiving the network status feature value from the network status monitoring module 9 and the data packet from the priority hierarchical encapsulation module 8, this module initiates a two-dimensional scheduling strategy: The first dimension is pipeline adaptation and scheduling: For field-edge pipelines (where the network status is relatively stable), dynamically adjust the data packet size (appropriately increase the fragment size when the network bandwidth is sufficient, and decrease the fragment size when the bandwidth is insufficient) and transmission rate (increase the transmission rate when the latency is low, and decrease the transmission rate when the latency is high). For edge-cloud pipelines (where the network status fluctuates greatly), focus on adjusting the retransmission mechanism (use one retransmission when the packet loss rate is less than 1%, use three retransmissions when the packet loss rate is 1%-5%, and use exponential backoff retransmission when the packet loss rate is higher than 5%).

[0054] The second dimension is priority scheduling: Based on the priority identifier of the data packet and the transmission resource occupancy threshold, transmission resources are allocated: core production-level data packets occupy pipeline resources first. When resources are scarce, they can preempt the transmission resources of device monitoring-level and log statistics-level data packets (suspend the transmission of low-priority data packets). Device monitoring-level data packets are transmitted during the transmission gap of core production-level data packets or when resources are sufficient. They can preempt log statistics-level data packet resources. Log statistics-level data packets are transmitted after the transmission of the other two types of data packets is completed or when resources are idle. They cannot preempt other priority resources.

[0055] When the network status characteristic value is below 0.3 (extremely poor network), the dynamic transmission scheduling module 10 activates the core resource protection mechanism: suspending the transmission of log statistics-level data packets and temporarily storing them in the local cache of the edge layer node (the cache capacity is configurable, with a default maximum of 10GB). After the network status characteristic value recovers to above 0.6 (good network), the temporarily stored data is transmitted in batches. Only key fields of device monitoring-level data packets are retained for transmission, while non-key fields are temporarily stored. Core production-level data packets are transmitted normally, ensuring that production control is not affected. The role of the dynamic transmission scheduling module 10 is to achieve dual protection of network status adaptation and data priority adaptation through dual-dimensional scheduling, which avoids transmission failures caused by network fluctuations and ensures the real-time and reliability of core data.

[0056] In addition, see Figures 2 to 7 As shown, in this embodiment, the compression-transmission collaborative optimization step is implemented by the compression-transmission collaborative module of the edge layer collaborative optimization unit 2. The core of this module is to establish a collaborative mapping relationship between compression ratio and transmission parameters, linking the compression link and the transmission link together, so as to avoid the overall inefficiency caused by the optimization of a single link.

[0057] The collaborative mapping relationship is stored in the form of a configuration table, which specifies the compression ratio adjustment range corresponding to different network state characteristic values: when the network state characteristic bandwidth is low, packet loss rate or latency is high, the corresponding compression ratio is increased to further compress the data volume and reduce bandwidth consumption; when the network state is medium, the compression ratio remains at the default value; when the network state is good, the compression ratio is reduced to reduce the decompression calculation overhead of the receiving end and improve data restoration efficiency.

[0058] The compression-transmission coordination module reads the feature values ​​of the network status monitoring module 9 and the current compression ratio of the adaptive compression module 6 in real time, and dynamically adjusts the compression parameters according to the mapping relationship: when the edge-cloud pipeline network bandwidth is detected to drop below 50Mbps, the packet loss rate exceeds 5%, or the transmission delay exceeds 50ms, it immediately sends an instruction to the field layer adaptive compression module 6 to increase the compression ratio of the corresponding data; when the network status recovers to a bandwidth of over 100Mbps, a packet loss rate of less than 1%, and a delay of less than 20ms, it sends an instruction to reduce the compression ratio.

[0059] The main body for implementing the collaborative optimization strategy is the edge layer processing node. Because the edge layer is close to the field, it can quickly respond to network changes and reduce the latency of cloud management. The cloud management node performs global control of the collaborative strategy through the global collaborative scheduling module 11. When multiple edge nodes transmit data to the cloud at the same time, causing network congestion, the cloud can send a unified compression rate adjustment command to each edge node to avoid the intensification of resource competition caused by each edge node working on its own. The role of the compression-transmission collaborative module is to realize the dynamic matching of compression and transmission, so that the system as a whole can reach the optimal state of highest transmission efficiency and lowest resource consumption.

[0060] Finally, see Figures 2 to 7 As shown, in this embodiment, the data decompression and integrity verification steps are implemented by the decompression verification module 12 of the edge layer collaborative optimization unit 2 or the cloud management and control unit 3, and are specifically determined by the receiving end of the data transmission: the receiving end (edge ​​layer node) of the field-edge pipeline calls its own decompression verification module 12, and the receiving end (cloud node) of the edge-cloud pipeline calls the decompression verification module 12 of the cloud management and control unit 3.

[0061] The decompression verification module 12 operates in three steps: First, it reads the compression identifier in the data packet, matches the corresponding decompression algorithm according to the identifier, starts the decompression process, and restores the compressed data to the original data format. Second, it reads the integrity check code in the data packet, recalculates the CRC32 check code on the decompressed original data, and compares it with the check code in the data packet to verify data integrity. Third, it executes the corresponding processing logic according to the verification result and data priority: when the verification matches, the data is forwarded to the subsequent processing module (edge ​​layer data is forwarded to the cloud, and cloud data is forwarded to the business system); when the verification does not match, a hierarchical retransmission mechanism is triggered.

[0062] The hierarchical retransmission mechanism is designed with differentiated priorities: Core production-level data packets immediately send a single-packet retransmission request to the sender. Upon receiving the request, the sender retransmits the fragment within 100ms to ensure that core data is not delayed. After three fragments of device monitoring-level data packets fail verification, a batch retransmission request is sent. The sender retransmits the fragments in batches according to their sequence numbers to avoid frequent single-packet retransmissions consuming resources. Log statistics-level data packets summarize the failed fragments every 5 minutes and send a timed retransmission request to complete the retransmission during network idle periods without affecting other data transmission.

[0063] The decompression and verification module 12 is used to ensure the accuracy and integrity of transmitted data, prevent erroneous data from entering subsequent business processes, and reduce the network resource consumption of low-priority data retransmission through a hierarchical retransmission mechanism while ensuring the reliability of core data, thus balancing reliability and efficiency.

[0064] The three units of the device are interconnected through industrial data pipeline 4. The data flow path is as follows: field-level data processing unit 1 - field-edge industrial data pipeline 4 - edge-layer collaborative optimization unit 2 - edge-cloud industrial data pipeline 4 - cloud management and control unit 3. The internal modules of each unit are connected through an internal bus, supporting fast data transmission and command interaction. Core keys (such as compression algorithm parameter encryption keys and data transmission encryption keys) and optimization strategy parameters (such as priority thresholds and network status thresholds) are stored using AES-256 encryption and can only be read by authorized modules, ensuring the security of device operation and preventing transmission risks caused by parameter tampering or key leakage.

[0065] Working principle: The data compression and transmission optimization method and device for cross-level industrial data pipelines, when in use, first completes the device deployment and initialization. The specific operation is as follows: according to the hierarchical hardware configuration of the industrial scenario, the field layer data processing unit 1 is deployed on field devices such as PLCs and sensor gateways, the edge layer collaborative optimization unit 2 is deployed on edge computing gateways or industrial servers, and the cloud management and control unit 3 is deployed in a cloud server cluster. Industrial data pipelines 4 between the units are established through networks such as industrial Ethernet and 5G. Pipeline communication parameters (such as port number and transmission protocol) are configured, and the parameters of each module are initialized, including data classification standards, default parameters of compression algorithms, priority classification rules, network status thresholds, collaborative mapping relationship tables, etc. The core key and optimization strategy parameters are encrypted and stored in the secure storage module of each unit.

[0066] Then, data acquisition, classification, and preliminary compression are performed. Specifically, the field layer nodes acquire industrial data (such as real-time sensor data, equipment operating parameters, and monitoring images), transmit it to the field layer data processing unit 1, the feature perception and classification module 5 extracts data features and divides them into three categories, the primary priority marking module 7 performs preliminary priority marking, the adaptive compression module 6 calls the corresponding dedicated algorithm to compress the data, adds compression identifiers and integrity verification codes, and transmits the compressed data to the edge layer collaborative optimization unit 2 through the field-edge pipeline.

[0067] Secondly, data is finely classified, network is monitored, and transmission is scheduled. The specific operations are as follows: After receiving the data, the priority classification and encapsulation module 8 of the edge layer divides it into three final priorities according to the business rules, performs differentiated fragmentation and encapsulation, and adds priority identifiers and transmission thresholds. The network status monitoring module 9 collects the network parameters of the two pipelines in real time, generates network status feature values ​​after normalization, and the dynamic transmission scheduling module 10 adjusts the transmission parameters (data packet size, rate, retransmission mechanism) according to the network feature values ​​and data priorities, allocates transmission resources according to priority, and temporarily stores low-priority data when the network is busy.

[0068] Furthermore, compression-transmission collaborative optimization is performed. Specifically, the compression-transmission collaborative module dynamically adjusts the compression rate of the field layer data based on network status characteristics and collaborative mapping relationships. When the network deteriorates, the compression rate is increased to reduce bandwidth usage, and when the network is good, the compression rate is decreased to reduce decompression overhead. The global collaborative scheduling module 11 of the cloud management unit 3 monitors the transmission status of each edge node in real time. When global network congestion occurs, a unified compression rate adjustment command is issued to achieve global collaborative optimization.

[0069] Finally, data decompression, verification, and retransmission are completed. The specific operations are as follows: the decompression and verification module 12 of the receiving end (edge ​​layer or cloud) calls the corresponding algorithm to decompress the data according to the compression identifier, and verifies the integrity through the verification code. If the verification is consistent, the data is forwarded to the subsequent business module. If the verification is inconsistent, hierarchical retransmission is triggered according to priority. The data feedback module 13 of the cloud management unit 3 records the optimization effect of each link (such as compression rate, transmission delay, retransmission rate), and periodically issues strategy adjustment instructions to the edge layer and field layer to form a closed-loop optimization, continuously improving the transmission efficiency and reliability of the cross-level industrial data pipeline 4.

[0070] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0071] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments, and any variations or modifications may be made to the implementation of the present invention without departing from the stated principles.

Claims

1. A method for data compression and transmission optimization in a cross-level industrial data pipeline, characterized in that, include: Based on a cross-layer architecture consisting of industrial field layer nodes, edge layer processing nodes, and cloud management nodes, the system optimizes the industrial data pipelines at both the field-edge and edge-cloud levels. Key steps include data feature perception and classification, type-based adaptive compression, data priority grading and encapsulation, cross-layer network status monitoring, cross-layer dynamic transmission scheduling, compression-transmission collaborative optimization, and data decompression and integrity verification. By adapting to heterogeneous industrial data through type-based compression, ensuring core data transmission needs through priority scheduling, adapting to cross-layer network differences through dynamic transmission parameter adjustment, and achieving dynamic matching between the two stages through compression-transmission collaborative optimization, the system comprehensively improves the transmission efficiency and reliability of the cross-layer industrial data pipeline.

2. The data compression and transmission optimization method for cross-level industrial data pipelines according to claim 1, wherein the data feature perception and classification step specifically comprises: after the field layer or edge layer node collects industrial data, it perceives the type, temporal features, data redundancy and importance features of the data through feature extraction, and classifies the industrial data into three categories: temporal sensing type, structured parameter type and unstructured media type, and performs primary priority marking on the data in combination with industrial business needs, providing feature basis for subsequent compression and scheduling.

3. The data compression and transmission optimization method for cross-level industrial data pipelines according to claim 2, wherein the type-based adaptive compression step specifically comprises: pre-setting dedicated compression algorithms adapted to three types of industrial data; calling the corresponding algorithm for type-based compression according to the data classification results; using differential compression algorithm for time-series sensor data, dictionary compression algorithm for structured parameter data, and lightweight media compression algorithm for unstructured media data; adding a unique compression identifier and a data integrity check code to the data after compression; the compression identifier is used by the receiving end to match the decompression algorithm, and the check code is used for data integrity verification.

4. The data compression and transmission optimization method for cross-level industrial data pipelines according to claim 3, wherein the data priority classification and encapsulation steps are specifically as follows: combining the business needs of the industrial site, the compressed data is divided into three priorities: core production level, equipment monitoring level, and log statistics level. The data is fragmented and encapsulated according to the priority, and exclusive priority identifiers and transmission thresholds are assigned to data packets of different priorities. Among them, the core production level data packets are assigned the highest transmission resource occupancy threshold to ensure their transmission priority.

5. The data compression and transmission optimization method for cross-level industrial data pipelines according to claim 4, wherein the cross-level network status monitoring step specifically comprises: the edge layer processing node monitoring the network status parameters of the field-edge and edge-cloud data pipelines in real time, including network bandwidth, packet loss rate, and transmission delay; generating real-time network status feature values ​​after normalizing each parameter, providing a quantitative basis for adjusting transmission parameters; and dynamically adjusting the frequency of network status monitoring according to the transmission rate of industrial data, wherein the higher the data transmission rate, the higher the monitoring frequency.

6. The data compression and transmission optimization method for cross-level industrial data pipelines according to claim 5, wherein the cross-level dynamic transmission scheduling step specifically comprises: dynamically adjusting transmission parameters, including data packet size, transmission rate, and retransmission mechanism, according to network status characteristics for the two data pipeline segments of field-edge and edge-cloud, and simultaneously scheduling transmission according to data priority, with high-priority data packets preempting transmission resources and low-priority data packets being transmitted off-peak; when network resources are scarce, low-priority data packets can be temporarily stored and transmitted after the network status improves.

7. The data compression and transmission optimization method for cross-level industrial data pipelines according to claim 6, wherein the compression-transmission collaborative optimization step specifically comprises: establishing a collaborative mapping relationship between compression ratio and transmission parameters; dynamically increasing the compression ratio of the corresponding data and reducing bandwidth usage when network bandwidth is reduced, packet loss rate or transmission delay exceeds a preset threshold; and appropriately reducing the data compression ratio and reducing the decompression computation overhead at the receiving end when the network status recovers to a preset good range. The collaborative optimization strategy is executed by the edge layer processing node and globally controlled by the cloud management node.

8. The data compression and transmission optimization method for cross-level industrial data pipelines according to claim 7, wherein the data decompression and integrity verification steps are specifically as follows: the receiving node calls the corresponding decompression algorithm to decompress the data according to the compression identifier in the data packet, verifies the integrity of the data through the check code, and if the data integrity verification fails, a hierarchical retransmission mechanism is triggered according to the data priority. Core production-level data packets are immediately retransmitted as single packets, while equipment monitoring-level and log statistics-level data packets are retransmitted in batches and at regular intervals, respectively, to avoid low-priority data retransmission occupying core network resources.

9. An apparatus for data compression and transmission optimization of a cross-level industrial data pipeline, characterized in that... The data compression and transmission optimization method for cross-level industrial data pipelines according to any one of claims 1 to 8 is wherein the apparatus for the data compression and transmission optimization method for cross-level industrial data pipelines is a modular distributed architecture, including a field-level data processing unit, an edge-level collaborative optimization unit, and a cloud-based management and control unit. Each unit is interconnected through the industrial data pipeline, and each unit is adapted to be lightweight or high-performance according to the hardware resource characteristics of the industrial level, and can be flexibly deployed on the hardware nodes of the corresponding level. Each unit works together to realize the collection, classification, compression, scheduling, transmission, decompression and verification of industrial data, thereby completing the compression and transmission optimization of cross-level data pipelines.

10. The apparatus for the data compression and transmission optimization method of the cross-level industrial data pipeline according to claim 9, characterized in that, The field-layer data processing unit includes a feature perception and classification module, an adaptive compression module, and a primary priority marking module. Adapting to the limited hardware resources of the field layer, it enables local acquisition, classification, adaptive compression, and primary priority marking of industrial data. The edge-layer collaborative optimization unit includes a priority hierarchical encapsulation module, a network status monitoring module, a dynamic transmission scheduling module, a compression-transmission collaboration module, and a decompression verification module. This enables fine-grained data hierarchical classification, network monitoring, dynamic scheduling, collaborative optimization, and decompression verification. The cloud-based management and control unit includes a global collaborative scheduling module, a decompression verification module, and a data feedback module. This enables closed-loop feedback adjustment of edge-cloud data decompression verification, global scheduling, and optimization strategies. Core keys and optimization strategy parameters are encrypted for storage, ensuring the security of the device operation.

Images