Dynamic data compression and transmission system and method based on big data

Abstract

The invention relates to the technical field of data compression and transmission, in particular to a dynamic data compression and transmission system and method based on big data. The method comprises the following steps: a data acquisition and processing unit acquires original data from various data sources and processes the original data; the data compression execution unit compresses original data through an LZ4 compression algorithm, and introduces a layered matching DEW-LZ4 technology based on dynamic entropy weight into the LZ4 compression algorithm to optimize the compression process; the data transmission unit performs network transmission on the compressed data through a transmission protocol HTTPS, and dynamically adjusts transmission parameters by adopting a machine learning Q-Learning model according to a network state; and a receiving end decompression unit obtains the transmitted compressed data from the network, decompresses the compressed data by using an LZ4 decompression algorithm, and recovers the compressed data to an original data format. According to the design of the invention, by introducing a layered matching DEW-LZ4 technology based on dynamic entropy weight, the system can intelligently select an optimal compression strategy according to different characteristics (Shannon entropy values) of data.

Description

Technical Field

[0001] The present invention relates to the technical field of data compression and transmission, and in particular to a dynamic data compression and transmission system and method based on big data. Background Art

[0002] With the development of technologies such as the Internet of Things, social media, and mobile devices, the speed and scale of data generation have increased dramatically. How to efficiently store, process, and transmit these massive amounts of data has become an important challenge. The data is not only large in volume, but also comes from a wide range of sources and in various formats, including sensor data, log files, database records, streaming data, and other types. Traditional data processing methods are difficult to effectively cope with this multi-source heterogeneous data environment. In the context of big data, transmitting a large amount of data from one place to another requires a lot of bandwidth resources and may cause increased latency and increased packet loss rates. Especially in the case of unstable network conditions or limited bandwidth, how to ensure that data can be transmitted quickly and reliably is an urgent problem to be solved. In order to reduce the amount of data transmitted, the data is usually compressed. However, traditional compression algorithms may perform poorly on certain types of high entropy data (such as highly random log data), resulting in low compression efficiency or a long compression / decompression process. Therefore, a dynamic data compression transmission system and method based on big data are provided. Summary of the invention

[0003] The purpose of the present invention is to provide a dynamic data compression and transmission system and method based on big data, so as to solve the problems raised in the above background technology, namely, the explosive growth of data volume, the difficulty in integrating heterogeneous data sources, the high efficiency and cost of network transmission, and the difficulty in achieving a balance between data compression and decompression speed and effect.

[0004] To achieve the above objectives, on the one hand, the present invention aims to provide a dynamic data compression transmission system based on big data, comprising: A data acquisition and processing unit, which acquires raw data from various data sources and processes them; A data compression execution unit, wherein the data compression execution unit compresses the original data by using an LZ4 compression algorithm, and introduces a layered matching DEW-LZ4 technology based on dynamic entropy weight into the LZ4 compression algorithm to optimize the compression process; A data transmission unit, wherein the data transmission unit transmits the compressed data over the network through the transmission protocol HTTPS, and dynamically adjusts the transmission parameters according to the network status by using a machine learning Q-Learning model; The receiving end decompression unit obtains the transmitted compressed data from the network, decompresses the data using the LZ4 decompression algorithm, and restores the data to the original data format.

[0005] As a further improvement of the technical solution, the data acquisition and processing unit includes a data acquisition module and a data processing module; The data acquisition module supports real-time acquisition of multi-source heterogeneous data and performs preliminary classification and priority marking on the original multi-source heterogeneous data; The data processing module analyzes data features based on the preliminary classified and labeled data, generates metadata tags, and divides the data into blocks of fixed size m.

[0006] As a further improvement of the technical solution, the data compression execution unit compresses the original data through the LZ4 compression algorithm, and introduces the hierarchical matching DEW-LZ4 technology based on dynamic entropy weight into the LZ4 compression algorithm to optimize the compression process, including the following steps: S1.1. Use the DEW-LZ4 technology based on dynamic entropy weight to find repeated data sequences in the data block; S1.2. For the matching part, LZ4 uses a pointer to indicate the position and length information of the matching sequence relative to the current position; S1.3, if the length of the matching sequence found is greater than the threshold n, the original data is replaced by the dictionary pointer; for the part where no match is found, the original data is copied; S1.4. Concatenate the compressed data blocks and matching information through LZ4 to form compressed output data.

[0007] As a further improvement of the technical solution, in S1.1, the DEW-LZ4 technology based on dynamic entropy weight is used to find the data sequence that appears repeatedly in the data block, including the following steps: S1.11. Calculate the Shannon entropy of the current data block , and based on metadata tags, establish an entropy value-matching weight model; S1.12. Calculate the entropy weight of the data block based on the entropy value-matching weight model ; S1.13, if the entropy weight If the entropy weight is greater than the preset threshold A, a convolutional neural network is used to predict matching hotspots. If the entropy weight is greater than the preset threshold B and less than or equal to the preset threshold A, a three-level hash index is constructed for matching; If it is less than or equal to the preset threshold B, the traditional LZ4 algorithm is continued to be used for matching; S1.14. In all cases, when the Shannon entropy value change rate exceeds the set threshold When the double buffer dictionary structure is updated, the update mechanism is triggered.

[0008] As a further improvement of the technical solution, in S1.11, the entropy value-matching weight model is: ; in, represents the regulating factor; Represents the benchmark Shannon entropy value.

[0009] As a further improvement of the technical solution, the data transmission unit transmits the compressed data over the network through the transmission protocol HTTPS, and dynamically adjusts the transmission parameters according to the network status, including the following steps: S2.1, receiving a compressed data block from a data compression execution unit; S2.2. Establish a secure HTTPS connection based on the target server information; S2.3, monitor the current network status in real time, and execute adaptive strategies to adjust the data segment size and compression mode according to the real-time network status; S2.4, using a sliding window mechanism to control the sending rate, through ACK confirmation and timeout retransmission, when the network fluctuates, using selective confirmation to retransmit only the lost data segments; S2.5. Use the built-in security features of the TLS protocol to encrypt and verify the integrity of the transmitted data; S2.6. Train the machine learning Q-Learning model based on historical data to predict the trend of network status changes and adjust the transmission parameters in advance.

[0010] As a further improvement of the technical solution, the S2.6, training a machine learning Q-Learning model based on historical data, predicting the trend of network status changes, and adjusting transmission parameters in advance, includes the following steps: S2.61. Collect historical data of network status and record associated transmission parameters; S2.62, discretize the network state into a multi-dimensional vector; S2.63, define the state of the network and define adjustable transmission parameters as actions; S2.64. Design a reward function to evaluate the quality of each action; S2.65, create a state-action matrix, assign initial values ​​to unknown state-action pairs, and initialize Q values; S2.66, use the ε-greedy strategy to balance the relationship between exploring new strategies and using existing knowledge, and update the Q value according to the Q value update formula. In the Q value update formula, an algorithm for dynamically adjusting the learning rate based on entropy weight is introduced to optimize the learning rate. Entropy weight of the adaptive system In order to balance compression efficiency and computing resources and network environment changes, we added additional reward items to the Q value update formula for further optimization. S2.67. Dynamically adjust transmission parameters based on the action recommendations output by the machine learning Q-Learning model.

[0011] As a further improvement of the technical solution, in S2.66, the Q value update formula is: ; in, Indicates the current state; Indicates the latter state; Indicates the action taken; Indicates the immediate reward after taking an action; represents the learning rate; represents the discount factor; Indicates status Take action The expected cumulative rewards that can be obtained; represents the time step; The algorithm for dynamically adjusting the learning rate based on entropy weight is introduced into the Q value update formula for optimization, so that the learning rate Entropy weight of the adaptive system Changes: Among them, the algorithm for dynamically adjusting the learning rate based on entropy weight is: ; in, Represents the learning rate dynamically adjusted based on the entropy weight; represents the initial learning rate; represents the average value of entropy weight; represents the standard deviation of the change in entropy weight; The algorithm for dynamically adjusting the learning rate based on entropy weight is introduced into the Q value update formula as follows: ; In order to balance compression efficiency with computing resources and network environment changes, we add additional bonus items to the Q value update formula for further optimization: Among them, additional rewards include compression efficiency rewards and entropy stability bonus : ; in, represents the adjustment parameter; Considering that high compression rate will cause high CPU usage, reward compression efficiency A dynamic synergy factor is introduced to quantify the conflict between compression rate and CPU usage, and the compression efficiency reward is adjusted through the second-order interaction term: ; in, The weight coefficient representing the compression ratio; Indicates the weight coefficient of CPU usage; The weight coefficient representing the dictionary hit rate; represents the synergy strength coefficient; Indicates the standard deviation of bandwidth fluctuation.

[0012] As a further improvement of the technical solution, the receiving end decompression unit includes a decompression and verification module and a data reorganization module; The decompression and verification module calls the LZ4 decompression algorithm to decompress and verify the data integrity according to the metadata; The data reorganization module merges the block data and restores the original structure.

[0013] On the other hand, the present invention provides a dynamic data compression transmission method based on big data, which is used in any one of the above-mentioned dynamic data compression transmission systems based on big data, and comprises the following steps: S3.1. Obtain and process raw data from various data sources; S3.2, compressing the original data by LZ4 compression algorithm, and introducing the layered matching DEW-LZ4 technology based on dynamic entropy weight into the LZ4 compression algorithm to optimize the compression process; S3.3, the compressed data is transmitted over the network through the transmission protocol HTTPS, and the transmission parameters are dynamically adjusted according to the network status using the machine learning Q-Learning model; S3.4, obtain the transmitted compressed data from the network, and decompress it using the LZ4 decompression algorithm to restore it to the original data format.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. In the dynamic data compression and transmission system and method based on big data, by introducing the hierarchical matching DEW-LZ4 technology based on dynamic entropy weights, the system can intelligently select the optimal compression strategy according to the different characteristics of the data (such as Shannon entropy value). For high-entropy data, a convolutional neural network is used to predict hot spots to accelerate compression; for medium-entropy data, a three-level hash index is used to reduce invalid searches and improve the compression rate; for low-entropy data, the traditional LZ4 algorithm is continued to be used for efficient compression. This method not only improves the overall compression efficiency, but also enhances the adaptability to data of different types and complexity, and is particularly suitable for processing large-scale, heterogeneous data sets. In addition, the application of a double-cache dictionary structure further optimizes memory usage, reduces CPU occupancy peaks, and improves system stability and performance.

[0015] 2. In the dynamic data compression transmission system and method based on big data, the HTTPS protocol is used to ensure security during the data transmission process, and the machine learning Q-Learning model is used to dynamically adjust the transmission parameters, such as data segment size, compression mode, etc., according to the real-time network status, so as to maintain efficient data transmission under different network conditions. For example, the data segment is increased to improve transmission efficiency when the bandwidth is high, and the segment size is reduced in a low bandwidth or high latency environment to ensure the stability of transmission. At the same time, the sending rate is controlled by the sliding window mechanism, combined with the ACK confirmation and timeout retransmission mechanism to ensure the reliability of data transmission. This adaptive strategy greatly reduces the delay and packet loss rate, providing users with a more stable service quality, especially suitable for scenarios with large fluctuations in the network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the overall flow chart of the present invention; The meaning of each number in the figure is: 1. Data acquisition and processing unit; 11. Data acquisition module; 12. Data processing module; 2. Data compression execution unit; 3. Data transmission unit; 4. Receiving end decompression unit; 41. Decompression and verification module; 42. Data reorganization module. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example 1: Please refer to Figure 1 As shown, a dynamic data compression transmission system based on big data is provided, including.

[0019] The data acquisition and processing unit 1 obtains raw data from various data sources (data sources include sensors, devices, user terminals, etc.) and processes them; In this embodiment, the data acquisition and processing unit 1 includes a data acquisition module 11 and a data processing module 12; The data acquisition module 11 supports real-time acquisition of multi-source heterogeneous data (sensors, logs, databases, streaming data), and performs preliminary classification and priority marking (including data volume, timeliness, and sensitivity) on the original multi-source heterogeneous data. The data processing module 12 analyzes data features (including entropy, repetition rate, and time series correlation) based on the preliminary classified and labeled data, generates metadata tags, and divides the data into blocks of a fixed size m for subsequent compression.

[0020] The data compression execution unit 2 compresses the original data by using the LZ4 compression algorithm, and introduces the layered matching DEW-LZ4 technology based on dynamic entropy weight into the LZ4 compression algorithm to optimize the compression process; In this embodiment, traditional LZ4 relies on fixed hash tables and sliding windows to detect repeated sequences, while DEW-LZ4 dynamically adjusts the matching strategy through entropy weights, solving the problem of low matching efficiency of traditional methods in heterogeneous data (such as highly random logs); for high-entropy data (such as sensor streams), CNN predicts hot spots to increase compression speed; for medium-entropy data (such as database records), the three-level hash index reduces invalid searches and improves compression rates; the dynamic entropy weight model can identify different features of text, binary, and time series data, and automatically switch compression strategies. For example, for highly random log data (high entropy), the optimized compression rate is higher than that of traditional LZ4; the double-buffered dictionary reduces memory usage and avoids the overhead of frequent hash table reconstruction; the dictionary update is triggered by the entropy change rate to reduce the peak CPU usage; based on the extremely fast compression of traditional LZ4, DEW-LZ4 breaks through the dependence of fixed algorithms on data features through information entropy theory + machine learning, matches appropriate algorithms for data of different complexities, and calculates the intelligent trade-off between resource consumption and compression efficiency; The data compression execution unit 2 compresses the original data by using the LZ4 compression algorithm, and introduces the layered matching DEW-LZ4 technology based on dynamic entropy weight into the LZ4 compression algorithm to optimize the compression process, including the following steps: S1.1. Use the DEW-LZ4 technology based on dynamic entropy weights to find repeated data sequences in data blocks. Repeated sequences are recorded as "dictionary" entries. DEW-LZ4 is a technology designed to optimize the performance of the LZ4 compression algorithm. It improves data compression efficiency by introducing a dynamic entropy perception mechanism and a hierarchical matching architecture, especially when processing large-scale, heterogeneous data sets. The method of using the hierarchical matching DEW-LZ4 technology based on dynamic entropy weight to find repeated data sequences in the data block includes the following steps: S1.11. Calculate the Shannon entropy of the current data block (Shannon entropy The entropy value-matching weight model is a mechanism that calculates the entropy weight of a data block based on its Shannon entropy, thereby guiding the selection of matching strategies in the data compression process. The entropy-matching weight model is: ; in, represents the adjustment factor (recommended value 0.5); represents the benchmark Shannon entropy value (obtained through historical data analysis); S1.12. Calculate the entropy weight of the data block based on the entropy value-matching weight model ; S1.13, if the entropy weight is greater than a preset threshold A (the preset threshold A in this embodiment is 0.7), a convolutional neural network is used to predict matching hotspots; if the entropy weight If the entropy weight is greater than the preset threshold B (the preset threshold B in this embodiment is 0.3) and less than or equal to the preset threshold A, a three-level hash index is constructed for matching; If it is less than or equal to the preset threshold value B, the traditional LZ4 algorithm is used for matching (the traditional LZ4 algorithm uses sliding window technology to find and replace repeated sequences in the data with short references to achieve compression purposes); In this embodiment, the steps of using a convolutional neural network to predict matching hot spots are as follows: the input of the CNN is the entropy weight feature of the current data block (such as Shannon entropy E, repetition rate in metadata tags, time series correlation, etc.), and the local pattern features are extracted through the convolution layer. The model architecture includes multiple convolution layers + pooling layers, and the output is a probability distribution map of potential matching hot spots in the data block. High entropy areas (such as log data with strong randomness) may be identified as periodic or local repetitive patterns through CNN; when the entropy weight is greater than the preset threshold A (0.7), the data block complexity is high and the traditional hash matching efficiency is low. CNN predicts hot spots (such as high-frequency repeated byte sequences) to guide the hash table to match these areas first, thereby reducing invalid searches; The steps of building a three-level hash index for matching are as follows: design a three-level hash structure: first-level hash: use the hash value of the 4-byte sequence in the data block as the key and map it to the initial bucket (similar to the hash table of LZ4); second-level hash: build a chain hash for the conflicting key values ​​and disperse the conflict through the secondary hash function; third-level hash: based on the first two levels, build a prefix tree (Trie) index for the long matching sequence (>8 bytes) to accelerate the matching of long sequences; when the entropy weight is between the preset thresholds B (0.3) and A (0.7), the data block repetition rate is medium, and the three-level hash index narrows the search range layer by layer: the first-level hash quickly locates the candidate position; the second-level hash resolves the conflict; the third-level hash confirms the longest repetitive sequence through prefix matching, significantly improving the matching accuracy; compared with the single-layer hash of traditional LZ4, the three-level index reduces the hash conflict rate by about 60%, which is especially suitable for heterogeneous data (such as mixed text and binary streams); S1.14. In all cases, when the Shannon entropy value change rate exceeds the set threshold ( ) or the matching hit rate decreases continuously, the update mechanism of the double-buffer dictionary structure is triggered. The specific composition of the double-buffer dictionary structure includes the coordination mechanism of the main dictionary and the backup dictionary. The trigger condition of the update mechanism is: the entropy value change rate exceeds the standard ( ) (indicating that the data pattern has mutated and the dictionary needs to be updated), the update process is: gradually migrate the low-frequency entries in the main dictionary to the backup dictionary, and load the newly generated metadata tags (hotspot areas output by the dynamic entropy weight model). After the migration is completed, the main and backup dictionaries are instantly switched through the lock mechanism or CAS operation to ensure that the compression process is not blocked; S1.2. For the matching part, LZ4 uses a pointer to indicate the position and length information of the matching sequence relative to the current position; S1.3. If the length of the matching sequence found is greater than the threshold n, the original data is replaced by a dictionary pointer (indicating the position and length of the match). The length of these pointers is usually fixed, such as 16 bits for the position and 8 bits for the length of the match. For the part where no match is found, the original data is copied. S1.4. The compressed data blocks and matching information (position and length) are concatenated through LZ4 to form compressed output data. For parts with a high repetition rate, LZ4 will build a compression dictionary and store commonly used data patterns in the dictionary instead of repeatedly storing the same bytes.

[0021] The data transmission unit 3 transmits the compressed data over the network through the transmission protocol HTTPS, and dynamically adjusts the transmission parameters according to the network status using the machine learning Q-Learning model; In this embodiment, the HTTPS protocol provides end-to-end encryption to ensure the security and integrity of data during transmission. This is particularly important for the transmission of sensitive information, and can effectively prevent data from being eavesdropped or tampered with; by compressing the data before transmitting it, the amount of data that actually needs to be transmitted on the network is reduced, which not only speeds up the transmission speed, but also reduces the cost of bandwidth usage; the system can monitor the current network status (such as bandwidth, latency, packet loss rate, etc.) in real time, and dynamically adjust the transmission parameters (such as data segment size and compression mode) based on this information. This adaptive strategy ensures effective data transmission even under poor network conditions, while optimizing resource consumption; The data transmission unit 3 transmits the compressed data over the network through the transmission protocol HTTPS, and dynamically adjusts the transmission parameters according to the network status, including the following steps: S2.1, receiving the compressed data block from the data compression execution unit 2; S2.2. Establish a secure HTTPS connection based on the target server’s information (such as URL, port, etc.), which includes a handshake process to ensure that the communication between the client and the server is encrypted; S2.3. Real-time monitoring of current network conditions, including but not limited to key indicators such as bandwidth, latency, and packet loss rate, and implementation of adaptive strategies to adjust data fragmentation size and compression mode according to real-time network conditions, thereby ensuring efficient and reliable data transmission in environments with limited or fluctuating bandwidth, while reducing latency and resource consumption; The adaptive strategy is: increase the data fragmentation when the bandwidth is high (1MB→2MB in this embodiment), and reduce the fragmentation (such as 512KB) when the bandwidth is low or the latency is high; dynamically adjust the number of parallel connections according to the degree of network congestion (HTTP / 1.1 defaults to 6 connections, HTTP / 2 enables multiplexing); under extreme network conditions (bandwidth <1Mbps in this embodiment), downgrade to fast compression mode (LZ4-HC→LZ4-fast); S2.4, use the sliding window mechanism to control the sending rate, and use ACK confirmation and timeout retransmission (RTO dynamic calculation) to ensure the reliability of data transmission (using the sliding window mechanism to control the sending rate, using ACK confirmation and timeout retransmission is a network data transmission control method, which ensures reliable data transmission and optimizes transmission efficiency by adjusting the sending window size, using confirmation reply (ACK) and setting timeout retransmission). When the network fluctuates, use selective acknowledgment (SACK) to retransmit only the lost data segment instead of the entire data packet. SACK is a technology that optimizes the data retransmission process and improves network transmission efficiency; S2.5. Use the built-in security features of the TLS protocol to encrypt (AES-256) and perform integrity verification (HMAC) on the transmitted data. This part of the operation is automatically handled by the TLS layer and no additional configuration is required. S2.6. Train the machine learning Q-Learning model based on historical data to predict the trend of network status changes and adjust transmission parameters in advance; Among them, the machine learning Q-Learning model is a reinforcement learning algorithm that makes decisions by learning action-value functions. It enables the agent to choose the best action in a given state to maximize long-term rewards. By learning and predicting the changing trend of network status, the system can dynamically adjust transmission parameters (such as shard size, compression mode, etc.) to adapt to different network conditions, thereby maintaining efficient data transmission in various network environments. The model can respond to impending changes in network status in advance, such as adjusting transmission strategies before network congestion, which helps reduce latency and packet loss and provide more stable service quality. It intelligently adjusts the configuration of data transmission according to real-time network conditions to avoid unnecessary waste of resources. For example, increasing the data shard size when the bandwidth is high and reducing the shard size when the bandwidth is low or the latency is high makes resource usage more reasonable. Training a machine learning Q-Learning model based on historical data, predicting the trend of network status changes, and adjusting transmission parameters in advance includes the following steps: S2.61. Collect historical data on network status: including bandwidth, latency, packet loss rate, transmission success rate and other indicators, and record related transmission parameters: including compression rate, data block size, transmission protocol type (HTTPS / MQTT), etc. (refer to the transmission unit design in the historical dialogue); S2.62, discretize the network state into a multi-dimensional vector; S2.63, define the state of the network, including current bandwidth, latency, etc., and define adjustable transmission parameters as actions, such as increasing / decreasing the fragment size, etc.; S2.64. Design a reward function to evaluate the quality of each action. For example, improving transmission efficiency can get positive rewards, while negative rewards can be obtained otherwise. S2.65. Create a state-action matrix, assign initial values ​​to unknown state-action pairs (such as all zeros or random small values), and initialize the Q value (the Q value represents the expected long-term return (or cumulative reward) of taking an action in a given state); S2.66. Use the ε-greedy strategy to balance the relationship between exploring new strategies and using existing knowledge (the ε-greedy strategy is a strategy used in reinforcement learning to balance exploration and utilization. It randomly selects actions with a probability of ε for exploration and selects the best currently known action with a probability of 1-ε for utilization). Update the Q value according to the Q value update formula. In the Q value update formula, an algorithm for dynamically adjusting the learning rate based on entropy weight is introduced for optimization, so that the learning rate Entropy weight of the adaptive system In order to balance compression efficiency and computing resources and network environment changes, we added additional reward items to the Q value update formula for further optimization. Furthermore, the Q value update formula is: ; in, Indicates the current state; Indicates the latter state; Indicates the action taken; Indicates the immediate reward after taking an action; represents the learning rate; represents the discount factor; Indicates status Take action The expected cumulative rewards that can be obtained; represents the time step; Entropy Weight Reflects the degree of uncertainty of the system: When the system is in a high entropy state (such as multi-objective conflict, drastic environmental dynamic changes), Increases, indicating that a more cautious update strategy is needed; on the contrary, in a low entropy state (clear goals and stable environment), Reduced, allowing more aggressive parameter updates; in a complex network environment, a fixed learning rate may not be able to adapt to different state changes. In a high-entropy environment, the network state changes greatly, and the learning process requires a higher learning rate to accelerate learning and adapt to these changes. In a low-entropy environment, the system is relatively stable, and too high a learning rate may cause overfitting or unnecessary fluctuations, reducing system performance; the purpose of introducing dynamic adjustment of the learning rate based on entropy weight is to enable Q-Learning to automatically adjust the learning speed according to the complexity of the current network state, thereby improving the adaptability and effectiveness of the algorithm. Especially in network transmission optimization, the fluctuation and uncertainty of the network state are large, and the dynamic learning rate can effectively improve the transmission performance and stability of the network; The algorithm for dynamically adjusting the learning rate based on entropy weight is introduced into the Q value update formula for optimization, so that the learning rate Entropy weight of the adaptive system Changes: Among them, the algorithm for dynamically adjusting the learning rate based on entropy weight is: ; in, Represents the learning rate dynamically adjusted based on the entropy weight; represents the initial learning rate; represents the average value of entropy weight; represents the standard deviation of the change in entropy weight; The algorithm for dynamically adjusting the learning rate based on entropy weight is introduced into the Q value update formula as follows: ; In traditional Q-Learning, the reward function usually only focuses on a single goal, such as the success rate, latency, bandwidth, etc. of network transmission. However, in reality, network transmission optimization usually involves multiple goals; a single reward function may not be able to take all these goals into account at the same time, which can easily lead to over-optimization of the model in some aspects and ignore other important factors. For example, optimizing the transmission speed may increase the CPU usage, or optimizing the compression efficiency may affect the latency. Therefore, introducing additional reward items allows the model to balance optimization on multiple goals and avoid the emergence of local optimal solutions; increasing the compression rate helps to reduce the amount of transmitted data, thereby improving the transmission efficiency; the compression process usually consumes a certain amount of computing resources. If the compression algorithm is efficient and occupies less CPU resources, it is more conducive to the stability and response speed of the system; the efficiency of the compression algorithm is usually related to the dictionary hit rate. The higher the dictionary hit rate, the better the compression effect of repeated data, thereby reducing the amount of data transmission; by adding compression efficiency rewards, Q-Learning can not only focus on traditional network transmission parameters (such as bandwidth, latency, etc.) when updating the Q value, but also optimize the performance of the compression algorithm; in network transmission, the stability and predictability of the system are very important, especially in high-load or highly variable network environments. Entropy measures the uncertainty or disorder of a system. A higher entropy value usually means that the system is unstable or has large changes, which may cause fluctuations in network performance. The purpose of introducing entropy stability rewards is to encourage the model to maintain the stability of the system as much as possible and avoid excessive fluctuations; In order to balance compression efficiency with computing resources and network environment changes, we add additional bonus items to the Q value update formula for further optimization: Among them, additional rewards include compression efficiency rewards and entropy stability reward : ; in, Represents a tuning parameter used to measure the entropy stability reward , the degree of influence on the overall Q value update; Considering that high compression rate will cause high CPU usage, reward compression efficiency A dynamic synergy factor is introduced to quantify the conflict between compression rate and CPU occupancy (i.e. , where compression rate refers to the dynamic synergy factor introduced to quantify compression rate, CPU occupancy rate refers to the CPU occupancy rate introduced, and compression efficiency reward is adjusted through second-order interaction term (the conflict between targets is quantified through the real-time ratio of compression rate to CPU occupancy rate, and the synergy intensity is dynamically adjusted in combination with bandwidth fluctuation; the hyperbolic tangent function is used to avoid the influence of extreme values, so that the model automatically suppresses the conflict reward term when the bandwidth suddenly changes, and the dynamic synergy factor quantifies the dynamic relationship between targets through nonlinear interaction term, improves the multi-target balance ability, and adjusts the compression efficiency through the second-order interaction term and the real-time negative correlation ( coefficient), which solves the problem that traditional linear superposition cannot quantify target conflicts): ; in, The weight coefficient representing the compression ratio; Indicates the weight coefficient of CPU usage; The weight coefficient representing the dictionary hit rate (the probability of successfully finding a repeated data sequence in the dictionary); It represents the coordination strength coefficient, and dynamically adjusts the negative correlation between compression rate and CPU usage in historical data by sliding window statistics (the stronger the conflict, the higher the conflict strength). the smaller); represents the standard deviation of bandwidth fluctuation; In this embodiment, the weight coefficient is based on: in a typical network environment (bandwidth 1Mbps~100Mbps, end-to-end delay 10ms~500ms) and multi-type data (text, binary, log) scenarios, the system performance indicators are balanced through parameter tuning. The optimization goal is to maximize the compression rate and transmission efficiency while strictly controlling the CPU occupancy rate. Based on multi-objective optimization theory (Pareto optimality), this solution coordinates the conflicts between compression rate, CPU occupancy rate and dictionary hit rate to ensure the optimal compromise among the three; the synergy strength coefficient is adaptively adjusted according to the negative correlation between compression rate and CPU occupancy rate in historical data (sliding window statistics); The weight coefficient value range is: (Compression rate weight coefficient): 0.5~0.7 (high compression rate priority scenario); (Weight coefficient of CPU usage): 0.2~0.4 (low resource consumption priority scenario); (Weight coefficient of dictionary hit rate): 0.1~0.3 (high repetition rate data scenario); (Synergy strength coefficient): 0.05~0.2 (dynamically scaled according to bandwidth fluctuation standard deviation); This formula is a customized improvement for data compression and transmission scenarios under the Q-learning framework. It integrates the multi-objective reward design of compression efficiency and is based on entropy-aware dynamic learning rate and correction terms. Traditional Q-learning uses a fixed learning rate. , and in this formula Introduced dynamic adjustment of entropy dependence and new compression bonus items Used to quantify compression efficiency (including compression rate and processing speed); combined with the entropy change rate in information theory ( ), used to dynamically adjust the compression strategy; S2.67. Dynamically adjust transmission parameters based on the action recommendations output by the machine learning model Q-Learning, including changing the shard size, adjusting the number of concurrent connections, etc.

[0022] The receiving end decompression unit 4 obtains the transmitted compressed data from the network, and decompresses it using the LZ4 decompression algorithm to restore it to the original data format; In this embodiment, the receiving end decompression unit 4 includes a decompression and verification module 41 and a data reassembly module 42; The decompression and verification module 41 calls the LZ4 decompression algorithm to decompress data according to the metadata and verifies the data integrity. The principle of calling the LZ4 decompression algorithm to decompress data is to restore the compressed data to the original data format by reversing the matching and pointer operations used in the compression process. The data reorganization module 42 merges the block data and restores the original structure (the time series data in this embodiment is sorted by timestamp).

[0023] Example 2: The difference between Example 2 of the present invention and Example 1 is that this example introduces the data compression and transmission method used by the dynamic data compression and transmission system based on big data.

[0024] A dynamic data compression and transmission method based on big data, used in any one of the above-mentioned dynamic data compression and transmission systems based on big data, comprises the following steps: S3.1. Obtain and process raw data from various data sources; S3.2, compressing the original data by LZ4 compression algorithm, and introducing the layered matching DEW-LZ4 technology based on dynamic entropy weight into the LZ4 compression algorithm to optimize the compression process; S3.3, the compressed data is transmitted over the network through the transmission protocol HTTPS, and the transmission parameters are dynamically adjusted according to the network status using the machine learning Q-Learning model; S3.4, obtain the transmitted compressed data from the network, and decompress it using the LZ4 decompression algorithm to restore it to the original data format.

[0025] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A dynamic data compression and transmission system based on big data, characterized in that: include: A data acquisition and processing unit (1), wherein the data acquisition and processing unit (1) acquires raw data from a variety of data sources and processes the data; A data compression execution unit (2), wherein the data compression execution unit (2) compresses the original data using an LZ4 compression algorithm, and introduces a layered matching DEW-LZ4 technology based on dynamic entropy weight into the LZ4 compression algorithm to optimize the compression process; A data transmission unit (3), wherein the data transmission unit (3) transmits the compressed data over a network via a transmission protocol HTTPS, and dynamically adjusts transmission parameters using a machine learning Q-Learning model according to a network state; A receiving end decompression unit (4), wherein the receiving end decompression unit (4) obtains the transmitted compressed data from the network, and decompresses the data using an LZ4 decompression algorithm to restore the data to an original data format.

2. The big data based dynamic data compression transmission system according to claim 1, characterized in that: The data acquisition and processing unit (1) comprises a data acquisition module (11) and a data processing module (12); The data acquisition module (11) supports real-time acquisition of multi-source heterogeneous data, and performs preliminary classification and priority marking on the original multi-source heterogeneous data; The data processing module (12) analyzes data features based on the preliminary classified and labeled data, generates metadata tags, and divides the data into blocks of fixed size m.

3. The big data based dynamic data compression transmission system according to claim 2, characterized in that: The data compression execution unit (2) compresses the original data using the LZ4 compression algorithm, and introduces a layered matching DEW-LZ4 technology based on dynamic entropy weight into the LZ4 compression algorithm to optimize the compression process, including the following steps: S1.

1. Use the DEW-LZ4 technology based on dynamic entropy weight to find repeated data sequences in the data block; S1.

2. For the matching part, LZ4 uses a pointer to indicate the position and length information of the matching sequence relative to the current position; S1.3, if the length of the matching sequence found is greater than the threshold n, the original data is replaced by the dictionary pointer; for the part where no match is found, the original data is copied; S1.

4. Concatenate the compressed data blocks and matching information through LZ4 to form compressed output data.

4. The big data based dynamic data compression transmission system according to claim 3 is characterized in that: In S1.1, the DEW-LZ4 technology based on dynamic entropy weight is used to find the repeated data sequence in the data block, including the following steps: S1.

11. Calculate the Shannon entropy of the current data block , and based on metadata tags, establish an entropy value-matching weight model; S1.

12. Calculate the entropy weight of the data block based on the entropy value-matching weight model ; S1.13, if the entropy weight If the entropy weight is greater than the preset threshold A, a convolutional neural network is used to predict matching hotspots. If the entropy weight is greater than the preset threshold B and less than or equal to the preset threshold A, a three-level hash index is constructed for matching; If it is less than or equal to the preset threshold B, the traditional LZ4 algorithm is continued to be used for matching; S1.

14. In all cases, when the Shannon entropy value change rate exceeds the set threshold When the double buffer dictionary structure is updated, the update mechanism is triggered.

5. The big data based dynamic data compression transmission system according to claim 4, characterized in that: In S1.11, the entropy value-matching weight model is: ； in, represents the regulating factor; Represents the benchmark Shannon entropy value.

6. The big data based dynamic data compression transmission system according to claim 5, characterized in that: The data transmission unit (3) transmits the compressed data over the network via the transmission protocol HTTPS and dynamically adjusts the transmission parameters according to the network status, including the following steps: S2.1, receiving a compressed data block from the data compression execution unit (2); S2.

2. Establish a secure HTTPS connection based on the target server information; S2.3, monitor the current network status in real time, and execute adaptive strategies to adjust the data segment size and compression mode according to the real-time network status; S2.4, using a sliding window mechanism to control the sending rate, through ACK confirmation and timeout retransmission, when the network fluctuates, using selective confirmation to retransmit only the lost data segments; S2.

5. Use the built-in security features of the TLS protocol to encrypt and verify the integrity of the transmitted data; S2.

6. Train the machine learning Q-Learning model based on historical data to predict the trend of network status changes and adjust the transmission parameters in advance.

7. The big data based dynamic data compression transmission system according to claim 6, characterized in that: In S2.6, training a machine learning Q-Learning model based on historical data, predicting the trend of network status changes, and adjusting transmission parameters in advance include the following steps: S2.

61. Collect historical data of network status and record associated transmission parameters; S2.62, discretize the network state into a multi-dimensional vector; S2.63, define the state of the network and define adjustable transmission parameters as actions; S2.

64. Design a reward function to evaluate the quality of each action; S2.65, create a state-action matrix, assign initial values ​​to unknown state-action pairs, and initialize Q values; S2.66, use the ε-greedy strategy to balance the relationship between exploring new strategies and using existing knowledge, and update the Q value according to the Q value update formula. In the Q value update formula, an algorithm for dynamically adjusting the learning rate based on entropy weight is introduced to optimize the learning rate. Entropy weight of the adaptive system In order to balance compression efficiency and computing resources and network environment changes, we added additional reward items to the Q value update formula for further optimization. S2.

67. Dynamically adjust transmission parameters based on the action recommendations output by the machine learning Q-Learning model.

8. The big data based dynamic data compression transmission system according to claim 7, characterized in that: In S2.66, the Q value update formula is: ； in, Indicates the current state; Indicates the latter state; Indicates the action taken; Indicates the immediate reward after taking an action; represents the learning rate; represents the discount factor; Indicates status Take action The expected cumulative rewards that can be obtained; represents the time step; The algorithm for dynamically adjusting the learning rate based on entropy weight is introduced into the Q value update formula for optimization, so that the learning rate Entropy weight of the adaptive system Changes: Among them, the algorithm for dynamically adjusting the learning rate based on entropy weight is: ; in, Represents the learning rate dynamically adjusted based on the entropy weight; represents the initial learning rate; represents the average value of entropy weight; represents the standard deviation of the change in entropy weight; The algorithm for dynamically adjusting the learning rate based on entropy weight is introduced into the Q value update formula as follows: ； In order to balance compression efficiency, computing resources and network environment changes, additional reward items are added to the Q value update formula for further optimization: Among them, additional rewards include compression efficiency rewards and entropy stability reward : ； in, represents the adjustment parameter; Considering that high compression rate will cause high CPU usage, reward compression efficiency A dynamic synergy factor is introduced to quantify the conflict between compression rate and CPU usage, and the compression efficiency reward is adjusted through the second-order interaction term: ； in, The weight coefficient representing the compression ratio; Indicates the weight coefficient of CPU usage; The weight coefficient representing the dictionary hit rate; represents the synergy strength coefficient; Indicates the standard deviation of bandwidth fluctuation.

9. The big data based dynamic data compression transmission system according to claim 8, characterized in that: The receiving end decompression unit (4) comprises a decompression and verification module (41) and a data reassembly module (42); The decompression and verification module (41) calls the LZ4 decompression algorithm to decompress data according to the metadata and verifies the data integrity; The data reorganization module (42) merges the block data and restores the original structure.

10. A method for dynamic data compression and transmission based on big data, used in a dynamic data compression and transmission system based on big data as claimed in any one of claims 1 to 9, characterized in that: The steps include: S3.

1. Obtain and process raw data from various data sources; S3.2, compressing the original data by LZ4 compression algorithm, and introducing the layered matching DEW-LZ4 technology based on dynamic entropy weight into the LZ4 compression algorithm to optimize the compression process; S3.3, the compressed data is transmitted over the network through the transmission protocol HTTPS, and the transmission parameters are dynamically adjusted according to the network status using the machine learning Q-Learning model; S3.4, obtain the transmitted compressed data from the network, and decompress it using the LZ4 decompression algorithm to restore it to the original data format.

Images