Transmission efficiency improvement system based on big data

Abstract

The invention relates to the technical field of data transmission, in particular to a transmission efficiency improvement system based on big data, which comprises a parameter acquisition module for periodically acquiring the transmission rate of each network, bandwidth occupation information in different time periods and transmission delay information, the efficiency evaluation module evaluates the efficiency level of each network bandwidth according to the transmission rate, the bandwidth occupation information and the transmission delay information; the classification module monitors time periods in a period for classification and classifies each network; the transmission optimization module determines network adjustment modes in different time periods based on a time period classification result and a network classification result, the dynamic adjustment module dynamically adjusts the data sending rate through a back pressure mechanism in the transmission process, and transmission fault tolerance and rapid recovery are achieved based on the asynchronous check point technology. According to the method, the network communication delay is reduced, the transmission efficiency is improved, and overload of a receiver is avoided; business pause is reduced, and the fault recovery speed is improved.

Description

Technical Field

[0001] The present invention relates to the field of data transmission technology, and in particular to a transmission efficiency improvement system based on big data. Background Art

[0002] Traditional layered protocols face systemic bottlenecks in wireless environments, large data volumes, and low-latency scenarios. Global optimization is required through cross-layer collaboration (such as QUIC / RDMA), data processing innovations (Arrow / Avro), and refined resource management (back pressure / memory pools). The current trend has shifted from single-layer protocol improvements to full-stack refactoring and dynamic parameter adjustment to further enhance adaptability. At the protocol level, while QUIC reduces connection overhead, it lacks compatibility in complex network environments and has high adaptation costs for some legacy devices. SDN dynamic routing relies on accurate real-time data, making decision delays prone to sudden traffic surges, which in turn increases short-term energy consumption. Regarding intelligent scheduling, AI load prediction models require extensive historical data training, which increases error rates in data-sparse edge scenarios and can lead to resource misallocation. While the distributed architecture of edge computing reduces transmission volume, it increases the energy consumption and complexity of inter-node coordination. Furthermore, green energy scheduling is constrained by geography and weather conditions. In areas with unstable power grids, power fluctuations can reduce transmission efficiency. Summary of the Invention

[0003] To this end, the present invention provides a transmission efficiency improvement system based on big data to overcome the problems of low transmission efficiency and long response time faced by traditional layered protocols in the prior art in wireless environments, large data volumes, and low-latency scenarios due to systemic bottlenecks.

[0004] To achieve the above objectives, the preferred technical solution for the big data-based transmission efficiency improvement system includes: The parameter collection module is used to periodically collect the transmission rate of each network, bandwidth occupancy information in different time periods, and transmission delay information. The transmission rate includes the real-time transmission rate and average transmission rate. The bandwidth occupancy information includes the occupancy ratio of each network in different time periods. The transmission delay information includes the round-trip delay and processing delay during the data transmission process. An efficiency evaluation module, connected to the parameter acquisition module, for evaluating the efficiency level of each network bandwidth based on the transmission rate, bandwidth occupancy information and transmission delay information; a classification module connected to the performance evaluation module and the parameter collection module, for classifying time periods within a monitoring period according to bandwidth occupancy information, and classifying each network according to network stability and delay fluctuation; A transmission optimization module, connected to the classification module and the performance evaluation module, is used to determine network adjustment methods within different time periods based on the time period classification results and the network classification results, combined with the QUIC / RDMA technology of the protocol layer, the Avro serialization and Snappy compression algorithm of the data layer, the back pressure mechanism of the flow control layer, and the asynchronous checkpoint technology of the fault tolerance layer. The network adjustment method includes a network replacement strategy and a transmission parameter adjustment plan; A dynamic adjustment module is connected to the transmission optimization module to dynamically adjust the data transmission rate through the back pressure mechanism during the transmission process, realize transmission fault tolerance and rapid recovery based on asynchronous checkpoint technology, and optimize cross-system data interaction using a vectorized format.

[0005] As a preferred technical solution for the transmission efficiency improvement system based on big data, the period of the periodic collection is a fixed duration, and the occupancy ratio in the bandwidth occupancy information includes a peak occupancy ratio and a continuous occupancy ratio.

[0006] As a preferred technical solution for the transmission efficiency improvement system based on big data, the evaluation of the efficiency level of each network bandwidth includes: If the transmission rate is at a high level, the bandwidth occupancy ratio is at a low level, and the transmission delay is at a low level, the performance level is high efficiency; If the transmission rate is at a medium level, the bandwidth occupancy ratio is at a medium level, and the transmission delay is at a medium level, the performance level is medium; If the transmission rate is at a low level, the bandwidth occupancy ratio is at a high level, or the transmission delay is at a high level, the performance level is low.

[0007] As a preferred technical solution for the transmission efficiency improvement system based on big data, the classification of time periods within the monitoring cycle includes: The high-frequency usage period is the period when the bandwidth peak occupancy ratio remains high for several consecutive collection cycles; The low-frequency usage period is a period when the bandwidth peak occupancy ratio remains low for several consecutive collection cycles.

[0008] As the preferred technical solution for the transmission efficiency improvement system based on big data, the classification of each network includes: A high-quality network is one where the transmission rate fluctuation is within the preset fluctuation range and the delay fluctuation is within the allowed fluctuation range. A low-quality network is a network whose transmission rate fluctuation exceeds the preset fluctuation range or whose delay fluctuation exceeds the allowed fluctuation range.

[0009] As a preferred technical solution for improving the transmission efficiency of big data, the protocol layer uses QUIC / RDMA technology, including: During the high-frequency usage period, selecting the high-quality network and enabling RDMA technology; During the low-frequency usage period or when the network is of low quality, QUIC technology is used to improve transmission stability.

[0010] As a preferred technical solution for improving the transmission efficiency of big data, in step S3, the data layer uses Avro serialization and Snappy compression algorithms, including: Use Avro serialization format to uniformly encode structured data; For serialized data, select the Snappy compression strength based on the data size. High-intensity compression is used for large-capacity data, and basic compression is used for small-capacity data.

[0011] As a preferred technical solution for improving the transmission efficiency of big data, the back pressure mechanism adopted by the flow control layer includes: monitoring the occupancy of the receiving end buffer in real time, and when the occupancy is at a high level, triggering the back pressure signal to cause the sending end to reduce the transmission rate; When the buffer occupancy rate of the receiving end is at a low level, the back pressure signal is released to enable the sending end to restore the initial transmission rate.

[0012] As a preferred technical solution for improving the transmission efficiency of big data, the fault-tolerant layer adopts asynchronous checkpoint technology, including: generating an asynchronous checkpoint at a first interval during the high-frequency usage time period; During the low-frequency usage time period, an asynchronous checkpoint is generated at a second interval, and the checkpoint data is stored in a vectorized format.

[0013] As a preferred technical solution for the big data-based transmission efficiency improvement system, the network replacement strategy includes: When the current network is of low quality and in a high-frequency usage period, it will automatically switch to a backup high-quality network; After switching, core data processed in vectorized format is transmitted first.

[0014] The beneficial effects of the present invention are: This system maps network time periods, link quality, data characteristics, and protocol stack capabilities in real time. It first uses periodic big data for judgment, then dynamically selects Avro / Snappy compression based on capacity at the data layer. The flow control layer uses back pressure to match send and receive buffers in real time, and the fault tolerance layer uses asynchronous vectorized checkpoints for sub-second recovery. In the event of network degradation, it automatically switches to a backup link and prioritizes the transmission of core data. This minimizes link latency and bandwidth waste in wireless, high-concurrency, and low-latency scenarios, while avoiding receiver overload and failures. This ensures high throughput, low energy consumption, high reliability, and zero maintenance for big data transmission.

[0015] In particular, by dynamically adapting protocol layer technologies and optimizing data encoding and compression strategies to introduce flow control and fault tolerance mechanisms, the system simultaneously addresses issues such as low transmission rates, unreasonable bandwidth utilization, and large latency fluctuations. This combination of technologies enables efficient transmission regardless of high-load periods or poor network quality. This ensures high rates on high-quality networks while enhancing transmission stability on low-quality networks or during complex periods, comprehensively improving the overall performance of big data transmission.

[0016] In particular, the system achieves precise allocation of bandwidth, computing, and storage resources by categorizing time periods and network quality, dynamically adjusting compression strength based on data capacity, generating checkpoints on demand, and utilizing backpressure mechanisms to regulate traffic. High-frequency transmission is focused on efficient transmission to reduce resource waste, while low-frequency transmission optimizes resource utilization to reduce energy consumption. Furthermore, vectorized formats are used to improve data exchange efficiency, reducing resource consumption during transmission from multiple dimensions while balancing performance and cost control.

[0017] In particular, through periodic parameter collection, automatic classification, and dynamic policy adjustment, the system can adapt to diverse network environments and load fluctuations without manual intervention. Features such as real-time adjustment of backpressure mechanisms and automatic adaptation of checkpoint intervals further enhance the system's responsiveness to dynamic scenarios, reduce manual operation and maintenance costs, and ensure that the transmission process is more aligned with fluctuations in actual business needs.

[0018] In particular, the system significantly enhances its risk resilience by enabling rapid fault recovery through asynchronous checkpointing technology, combined with a network replacement strategy that automatically switches to a backup network in the event of an anomaly and prioritizes core data transmission. Whether it's a sudden network failure or data transmission interruption, the fault-tolerance mechanism minimizes data loss, and the prioritization of critical data ensures that core business processes are not affected, providing comprehensive protection for the continuity and security of big data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic diagram of the structure of a system for improving transmission efficiency based on big data according to an embodiment of the present invention; Figure 2 A logical diagram for classifying networks according to an embodiment of the present invention; Figure 3 This is a logical diagram of the asynchronous checkpoint technology used in the fault-tolerant layer of an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention.

[0021] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0022] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside", and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0023] Furthermore, it should be noted that, in the description of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0024] See also Figure 1 , which is a schematic diagram of the structure of a transmission efficiency improvement system based on big data according to an embodiment of the present invention; the present invention provides a transmission efficiency improvement system based on big data, comprising: The parameter collection module is used to periodically collect the transmission rate of each network, bandwidth occupancy information in different time periods, and transmission delay information. The transmission rate includes the real-time transmission rate and average transmission rate. The bandwidth occupancy information includes the occupancy ratio of each network in different time periods. The transmission delay information includes the round-trip delay and processing delay during the data transmission process. An efficiency evaluation module, connected to the parameter acquisition module, for evaluating the efficiency level of each network bandwidth based on the transmission rate, bandwidth occupancy information and transmission delay information; a classification module connected to the performance evaluation module and the parameter collection module, for classifying time periods within a monitoring period according to bandwidth occupancy information, and classifying each network according to network stability and delay fluctuation; A transmission optimization module, connected to the classification module and the performance evaluation module, is used to determine network adjustment methods within different time periods based on the time period classification results and the network classification results, combined with the QUIC / RDMA technology of the protocol layer, the Avro serialization and Snappy compression algorithm of the data layer, the back pressure mechanism of the flow control layer, and the asynchronous checkpoint technology of the fault tolerance layer. The network adjustment method includes a network replacement strategy and a transmission parameter adjustment plan; A dynamic adjustment module is connected to the transmission optimization module to dynamically adjust the data transmission rate through the back pressure mechanism during the transmission process, realize transmission fault tolerance and rapid recovery based on asynchronous checkpoint technology, and optimize cross-system data interaction using a vectorized format.

[0025] This system maps network time periods, link quality, data characteristics, and protocol stack capabilities in real time. It first uses periodic big data for judgment, then dynamically selects Avro / Snappy compression based on capacity at the data layer. The flow control layer uses back pressure to match send and receive buffers in real time, and the fault tolerance layer uses asynchronous vectorized checkpoints for sub-second recovery. In the event of network degradation, it automatically switches to a backup link and prioritizes the transmission of core data. This minimizes link latency and bandwidth waste in wireless, high-concurrency, and low-latency scenarios, while avoiding receiver overload and failures. This ensures high throughput, low energy consumption, high reliability, and zero maintenance for big data transmission.

[0026] Specifically, the period of the periodic collection is a fixed duration, and the occupancy ratio in the bandwidth occupancy information includes a peak occupancy ratio and a continuous occupancy ratio.

[0027] During implementation, a fixed collection period (e.g., 5 minutes / time) is set to ensure uniform data collection time granularity. Within each period, the peak and sustained network bandwidth utilization ratios, as well as the ratio of the average utilization value to the total bandwidth within the period, are recorded. These two types of ratio data are then associated with the corresponding time period to provide basic data support for subsequent performance evaluation and time period classification. The collection cycle is preferably set to 5 minutes. Short-term fluctuations in network status (bandwidth usage, rate) have a certain degree of continuity. 5 minutes can capture network changes in a timely manner (avoiding lags caused by too long a cycle) without increasing system collection and computing overhead due to too frequent collection (such as within 1 minute), thus balancing real-time performance and resource consumption.

[0028] Specifically, the evaluation of the performance level of each network bandwidth includes: If the transmission rate is at a high level, the bandwidth occupancy ratio is at a low level, and the transmission delay is at a low level, the performance level is high efficiency; If the transmission rate is at a medium level, the bandwidth occupancy ratio is at a medium level, and the transmission delay is at a medium level, the performance level is medium; If the transmission rate is at a low level, the bandwidth occupancy ratio is at a high level, or the transmission delay is at a high level, the performance level is low.

[0029] Three thresholds (high, medium, and low) are preset for transmission rate, bandwidth occupancy ratio, and transmission delay. The collected network parameters are compared. If the transmission rate is higher than the high threshold, the bandwidth occupancy ratio is lower than the low threshold, and the transmission delay is lower than the low threshold, it is judged as efficient; if all three are within the medium threshold range, it is judged as medium; if the transmission rate is lower than the low threshold, the bandwidth occupancy ratio is higher than the high threshold, or the transmission delay is higher than the high threshold, it is judged as inefficient, thereby completing the quantitative evaluation of the bandwidth efficiency level of each network.

[0030] In this implementation, the transmission rate is: high ≥ 10Gbps, medium 1-10Gbps, low < 1Gbps; Bandwidth usage: high level ≥ 80%, medium level 30%-80%, low level < 30%; Transmission delay: high ≥ 100ms, medium 10-100ms, low < 10ms.

[0031] It is understandable that the network speed of modern data centers is mostly 10Gbps and above. Bandwidth occupancy of more than 80% is considered congestion, latency within 10ms is low latency (such as local area network), and latency above 100ms is high latency (such as cross-regional transmission). The three-level division can clearly distinguish the performance levels.

[0032] Specifically, classifying the time periods within the monitoring period includes: The high-frequency usage period is the period when the bandwidth peak occupancy ratio remains high for several consecutive collection cycles; The low-frequency usage period is a period when the bandwidth peak occupancy ratio remains low for several consecutive collection cycles.

[0033] During implementation, high and low level thresholds for bandwidth peak occupancy ratio and the number of continuous determination cycles are preset; for each sub-period within the monitoring period, the performance of its bandwidth peak occupancy ratio within the continuous collection period is counted. If it continuously reaches or exceeds the high level threshold, it is classified as a high-frequency usage time period; if it continuously remains at or below the low level threshold, it is classified as a low-frequency usage time period, thus realizing time period division based on actual load.

[0034] In this embodiment, consecutive cycles are preferred. It is understandable that high / low bandwidth occupancy in 1-2 cycles may be accidental fluctuations (such as burst data transmission), and 3 consecutive cycles (such as 5 minutes × 3 = 15 minutes) can reflect a stable trend and avoid misjudging high or low frequency.

[0035] See also Figure 2 As shown, it is a logical diagram of network classification according to an embodiment of the present invention. Classification of each network includes: A high-quality network is one where the transmission rate fluctuation is within the preset fluctuation range and the delay fluctuation is within the allowed fluctuation range. A low-quality network is a network whose transmission rate fluctuation exceeds the preset fluctuation range or whose delay fluctuation exceeds the allowed fluctuation range.

[0036] During implementation, the allowable range of transmission rate fluctuation and the allowable range of delay fluctuation are preset; the transmission rate and delay changes of each network are monitored in real time, and the fluctuation amplitude, such as the ratio of the standard deviation to the mean, is calculated; if the fluctuation amplitudes of both are within the allowable range, it is judged as a high-quality network; if any fluctuation amplitude exceeds the corresponding range, it is judged as a low-quality network, completing the stability-based network classification.

[0037] The fluctuation range, that is, the coefficient of variation ≤ 20%. Exceeding this value indicates that the rate or latency fluctuates drastically, such as sudden packet loss or bandwidth drop, which does not meet the stability requirements of a high-quality network. This threshold can effectively identify network stability.

[0038] In this invention, the system achieves precise allocation of bandwidth, computing, and storage resources by classifying time periods and network quality, dynamically adjusting compression strength based on data capacity, generating checkpoints on demand, and utilizing backpressure mechanisms to regulate traffic. High-frequency periods focus on efficient transmission to reduce resource waste, while low-frequency periods optimize resource utilization to reduce energy consumption. Furthermore, vectorized formats improve data exchange efficiency, reducing resource consumption during transmission from multiple dimensions while balancing performance and cost control.

[0039] Specifically, the protocol layer adopts QUIC / RDMA technology including: During the high-frequency usage period, selecting the high-quality network and enabling RDMA technology; During the low-frequency usage period or when the network is of low quality, QUIC technology is used to improve transmission stability.

[0040] During implementation, during high-frequency usage periods and when the current network is of high quality, RDMA technology is enabled to increase the transmission rate by bypassing the operating system kernel and directly accessing the memory; during low-frequency usage periods or when the network is of low quality, it switches to the QUIC protocol, utilizing its multiplexing, connection migration, and strong error correction capabilities to ensure transmission stability and achieve dynamic adaptation of protocol layer technology.

[0041] For example, see Table 1 for the performance optimization effects brought about by the system's dynamic adaptation strategy in different scenarios: Table 1 Comparison of optimization effects of dynamic adaptation strategies

[0042] This implementation system can dynamically combine technology stacks based on real-time scenarios, and can provide performance significantly better than traditional fixed strategies under various conditions.

[0043] In this invention, by dynamically adapting protocol layer technologies (QUIC / RDMA), optimizing data encoding and compression strategies (Avro / Snappy), and introducing flow control and fault tolerance mechanisms, the system simultaneously addresses issues such as low transmission rates, unreasonable bandwidth utilization, and large latency fluctuations. This combination of technologies enables efficient transmission in both high-load periods and poor network conditions. This ensures high transmission rates on high-quality networks while enhancing transmission stability on low-quality networks or during complex periods, comprehensively improving the overall performance of big data transmission.

[0044] Specifically, the data layer uses Avro serialization and Snappy compression algorithms including: Use Avro serialization format to uniformly encode structured data; For serialized data, select the Snappy compression strength based on the data size. High-intensity compression is used for large-capacity data, and basic compression is used for small-capacity data.

[0045] During implementation, all structured data is serialized in Avro format. By defining a unified schema to describe the data structure, cross-system parsing compatibility is ensured. For serialized data, the Snappy compression strength is selected based on the preset capacity threshold. High-intensity compression, such as the highest compression level, is used for large-capacity data exceeding the threshold, while basic compression is used for small-capacity data that does not exceed the threshold, balancing compression efficiency and computational overhead.

[0046] The preferred capacity threshold for this implementation is 100MB. It is understandable that the compression benefits of data above 100MB are significant, and strong compression can reduce the volume by more than 60%. However, the computational overhead (such as CPU usage) of strong compression of data below 100MB may exceed the benefits, and basic compression is more efficient.

[0047] Specifically, the back pressure mechanism adopted by the fluidic layer includes: monitoring the occupancy of the receiving end buffer in real time, and when the occupancy is at a high level, triggering the back pressure signal to cause the sending end to reduce the transmission rate; When the buffer occupancy rate of the receiving end is at a low level, the back pressure signal is released to enable the sending end to restore the initial transmission rate.

[0048] The occupancy rate of the receiving end's buffer (the ratio of used space to total capacity) is monitored in real time, and high and low occupancy thresholds are preset. When the occupancy rate reaches or exceeds the high occupancy threshold, the receiving end sends a backpressure signal to the sending end, and the sending end reduces the transmission rate after receiving the signal. When the occupancy rate drops below the low occupancy threshold, the receiving end sends a release signal, and the sending end resumes the initial rate, thus achieving dynamic matching of sending and receiving capabilities.

[0049] In this implementation, the high occupancy threshold is 80% and the low occupancy threshold is 30%. It is understandable that when the buffer occupancy rate reaches 80%, it is close to the risk of overflow, which may cause data loss and trigger back pressure to reduce the speed; when it drops to 30%, there is sufficient remaining space to safely restore the initial rate, which complies with the management logic of preventing buffer overflow while ensuring efficiency.

[0050] For example, see Table 2, which simulates the receiver buffer's performance under different loads, comparing the cases with and without back pressure enabled: Table 2 Comparison of back pressure effects under different loads

[0051] In this implementation, the back pressure mechanism effectively avoids data loss caused by overload at the receiving end and ensures the reliability of data transmission.

[0052] In this invention, the system adapts to diverse network environments and load variations without manual intervention, leveraging periodic parameter collection, automatic classification, and dynamic policy adjustments, such as automatic network switching and on-demand protocol selection. Features such as real-time adjustment of backpressure mechanisms and automatic adaptation of checkpoint intervals further enhance the system's responsiveness to dynamic scenarios, reduce manual operation and maintenance costs, and ensure that the transmission process is more aligned with fluctuations in actual business needs.

[0053] See also Figure 3 As shown, it is a logic diagram of the asynchronous checkpoint technology used in the fault-tolerant layer of an embodiment of the present invention. The asynchronous checkpoint technology used in the fault-tolerant layer includes: generating an asynchronous checkpoint at a first interval during the high-frequency usage time period; During the low-frequency usage time period, an asynchronous checkpoint is generated at a second interval, and the checkpoint data is stored in a vectorized format.

[0054] Preset checkpoint generation intervals for high-frequency usage periods (first interval) and intervals for low-frequency usage periods (second interval, which is longer than the first interval). Asynchronous checkpoints are automatically generated at the first interval during high-frequency periods and at the second interval during low-frequency periods. Checkpoint data is uniformly stored in a vectorized format (such as Arrow). Columnar storage and pre-compression optimization reduce storage space and accelerate read and write operations, balancing fault tolerance and resource efficiency.

[0055] In implementation, the first interval is preferably 5 minutes; the second interval is preferably 30 minutes. During high-frequency periods, data is dense and the impact of failures is large, so a checkpoint every 5 minutes can enable rapid recovery. During low-frequency periods, data volume is small and risk is low, so a checkpoint every 30 minutes can reduce backup overhead. Checkpoint generation requires CPU / bandwidth to balance fault tolerance and resource consumption.

[0056] Specifically, the network replacement strategy includes: When the current network is of low quality and in a high-frequency usage period, it will automatically switch to a backup high-quality network; After switching, core data processed in vectorized format is transmitted first.

[0057] The system monitors the quality type and time period of the current network in real time. When it is determined to be a low-quality network and is in a high-frequency usage period, the system automatically triggers the network switching mechanism and connects to the preset backup high-quality network. After the switch is completed, the content marked as core data (such as key business indicators) is transmitted first, and the core data must be pre-processed in a vectorized format to ensure priority delivery by reducing parsing delays and guaranteeing the continuity of critical data transmission.

[0058] This invention uses asynchronous checkpointing technology to achieve rapid fault recovery. Combined with a network replacement strategy, it automatically switches to a backup network in the event of an anomaly, prioritizing core data transmission. This significantly improves the system's risk tolerance. Whether it's a sudden network failure or data transmission interruption, the fault-tolerance mechanism reduces data loss. A critical data prioritization strategy ensures that core business processes are not affected, providing comprehensive protection for the continuity and security of big data transmission.

[0059] Example 1 To evaluate the performance of this system, a 24-hour simulation test was conducted in a large internet company's data center. The test data volume was approximately 500 TB, and the network environment included a high-quality local area network (LAN) and a cross-regional public network (WAN). The comparison results are shown in Figure 3: Table 3 Comparison of lifting effects of the examples

[0060] The system of this embodiment has significant improvements in transmission rate, latency, stability, and resource utilization, with only negligible additional computing overhead.

[0061] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based device that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0062] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

[0063] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A transmission efficiency improvement system based on big data, characterized in that: include: The parameter collection module is used to periodically collect the transmission rate of each network, bandwidth occupancy information in different time periods, and transmission delay information. The transmission rate includes the real-time transmission rate and average transmission rate. The bandwidth occupancy information includes the occupancy ratio of each network in different time periods. The transmission delay information includes the round-trip delay and processing delay during the data transmission process. An efficiency evaluation module, connected to the parameter acquisition module, for evaluating the efficiency level of each network bandwidth based on the transmission rate, bandwidth occupancy information and transmission delay information; a classification module connected to the performance evaluation module and the parameter collection module, for classifying time periods within a monitoring period according to bandwidth occupancy information, and classifying each network according to network stability and delay fluctuation; A transmission optimization module, connected to the classification module and the performance evaluation module, is used to determine network adjustment methods within different time periods based on the time period classification results and the network classification results, combined with the QUIC / RDMA technology of the protocol layer, the Avro serialization and Snappy compression algorithm of the data layer, the back pressure mechanism of the flow control layer, and the asynchronous checkpoint technology of the fault tolerance layer. The network adjustment method includes a network replacement strategy and a transmission parameter adjustment plan; A dynamic adjustment module is connected to the transmission optimization module to dynamically adjust the data transmission rate through the back pressure mechanism during the transmission process, realize transmission fault tolerance and rapid recovery based on asynchronous checkpoint technology, and optimize cross-system data interaction using a vectorized format.

2. The transmission efficiency improvement system based on big data according to claim 1, characterized in that: The period of the periodic collection is a fixed duration, and the occupancy ratio in the bandwidth occupancy information includes a peak occupancy ratio and a continuous occupancy ratio.

3. The transmission efficiency improvement system based on big data according to claim 2, characterized in that: The evaluation of the performance level of each network bandwidth includes: If the transmission rate is at a high level, the bandwidth occupancy ratio is at a low level, and the transmission delay is at a low level, the performance level is high efficiency; If the transmission rate is at a medium level, the bandwidth occupancy ratio is at a medium level, and the transmission delay is at a medium level, the performance level is medium; If the transmission rate is at a low level, the bandwidth occupancy ratio is at a high level, or the transmission delay is at a high level, the performance level is low.

4. The transmission efficiency improvement system based on big data according to claim 3, characterized in that: Classification of time periods within the monitoring period includes: The high-frequency usage period is the period when the bandwidth peak occupancy ratio remains high for several consecutive collection cycles; The low-frequency usage period is a period when the bandwidth peak occupancy ratio remains low for several consecutive collection cycles.

5. The transmission efficiency improvement system based on big data according to claim 4, characterized in that: The classification of each network includes: A high-quality network is one where the transmission rate fluctuation is within the preset fluctuation range and the delay fluctuation is within the allowed fluctuation range. A low-quality network is a network whose transmission rate fluctuation exceeds the preset fluctuation range or whose delay fluctuation exceeds the allowed fluctuation range.

6. The transmission efficiency improvement system based on big data according to claim 5, characterized in that: The protocol layer adopts QUIC / RDMA technology including: During the high-frequency usage period, selecting the high-quality network and enabling RDMA technology; During the low-frequency usage period or when the network is of low quality, QUIC technology is used to improve transmission stability.

7. The transmission efficiency improvement system based on big data according to claim 6, characterized in that: The data layer uses Avro serialization and Snappy compression algorithms including: Use Avro serialization format to uniformly encode structured data; For serialized data, select the Snappy compression strength based on the data size. High-intensity compression is used for large-capacity data, and basic compression is used for small-capacity data.

8. The transmission efficiency improvement system based on big data according to claim 7, characterized in that: The back pressure mechanism adopted by the fluidic control layer includes: Monitor the occupancy of the receiving end's buffer in real time. When the occupancy rate is high, trigger a back pressure signal to make the sending end reduce the transmission rate. When the buffer occupancy rate of the receiving end is at a low level, the back pressure signal is released to enable the sending end to restore the initial transmission rate.

9. The transmission efficiency improvement system based on big data according to claim 8, characterized in that: The fault-tolerant layer adopts asynchronous checkpoint technology including: generating an asynchronous checkpoint at a first interval during the high-frequency usage time period; During the low-frequency usage time period, an asynchronous checkpoint is generated at a second interval, and the checkpoint data is stored in a vectorized format.

10. The transmission efficiency improvement system based on big data according to claim 9, characterized in that: The network replacement strategy includes: When the current network is of low quality and in a high-frequency usage period, it will automatically switch to a backup high-quality network; After switching, core data processed in vectorized format is transmitted first.

Images