Data transmission methods and apparatus, electronic devices, storage media

Abstract

This application discloses a data transmission method, apparatus, electronic device, and storage medium, relating to the field of data synchronization. The method includes: when a first node has target data to be sent to a second node, determining the compression ratio of the target data, the network status between the first and second nodes, and the resource load status of the first node; determining a compression strategy for the target data based on the compression ratio, the network status, and the resource load status; and sending the target data to the second node based on the compression strategy. This solution addresses the problem of poor data transmission efficiency.

Description

Technical Field

[0001] This application relates to the field of data synchronization, and more particularly to a data transmission method and apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of information technology, especially the widespread adoption of hybrid cloud architectures in enterprise applications, the transmission and synchronization of data across geographically dispersed sites has become increasingly frequent and crucial. Against this backdrop, object storage systems have become a key bridge connecting various sites. However, despite their powerful functionality, object storage systems exhibit significant efficiency bottlenecks when dealing with multi-site data synchronization, particularly in terms of bandwidth costs for data transmission. Reports indicate that bandwidth costs for cross-regional data replication account for over 35% of total operational costs, a proportion that underscores the cost pressures of data transmission.

[0003] Traditional multi-site data synchronization strategies often employ a "full or static compression strategy," meaning that all types of data are transmitted with the same compression or no compression method. This approach leads to unnecessary and repeated compression of low-entropy data (already compressed video or audio files), wasting computational resources. Furthermore, for highly redundant data, such as log files and plain text data, direct transmission without compression wastes network bandwidth, especially in environments with limited network resources. This inefficient transmission mode severely impacts data synchronization performance.

[0004] Furthermore, traditional data synchronization solutions generally lack the ability to adapt to real-time changes. Whether it's changes in data characteristics, such as the mixing of different file types, fluctuations in network conditions, such as bandwidth jitter, or even changes in local computing resource load, these solutions cannot respond in a timely and appropriate manner. Therefore, the data synchronization process is often inefficient and costly, making it difficult to meet the dual needs of modern enterprises for both timeliness and cost control in data synchronization.

[0005] There is currently no effective solution to the problem of poor data transmission efficiency in related technologies. Summary of the Invention

[0006] This application provides a data transmission method, apparatus, electronic device, and storage medium to at least solve the problem of poor data transmission efficiency.

[0007] This application provides a data transmission method, comprising: when there is target data to be transmitted to a second node at a first node, determining the compression ratio of the target data, the network status between the first node and the second node, and the resource load status of the first node; determining a compression strategy for the target data based on the compression ratio, network status, and resource load status; and transmitting the target data to the second node based on the compression strategy.

[0008] This application also provides a data transmission apparatus, comprising: a first determining module, configured to determine the compression ratio of the target data, the network status between the first node and the second node, and the resource load status of the first node when the first node has target data to be sent to the second node; a second determining module, configured to determine the compression strategy of the target data based on the compression ratio of the target data, the network status, and the resource load status; and a transmission module, configured to send the target data to the second node based on the compression strategy.

[0009] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described data transmission methods.

[0010] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data transmission methods.

[0011] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data transmission methods.

[0012] This application dynamically determines the optimal compression strategy by evaluating the compression ratio of the target data, the real-time network status between the first and second nodes, and the resource load of the first node. This significantly improves the efficiency and economy of data transmission, solves the problem of poor data transmission efficiency, avoids the rigid approach of compressing or not compressing data regardless of its characteristics in traditional schemes, reduces repeated compression of low-entropy data, avoids the waste of computing resources, and also avoids the waste of network bandwidth caused by direct transmission without compression. Attached Figure Description

[0013] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1This is a hardware structure block diagram of a data transmission method according to an embodiment of this application;

[0015] Figure 2 This is a flowchart of a data transmission method according to an embodiment of this application;

[0016] Figure 3 This is an overall flowchart of a data transmission method according to an embodiment of this application;

[0017] Figure 4 This is a structural block diagram of a data transmission device according to an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0019] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0020] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] The specific application environment architecture or specific hardware architecture on which the execution of the data transmission method depends is described here.

[0022] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a data transmission method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1Only one is shown in the image. A processor 102 (which may include, but is not limited to, a central processing unit (CPU), microprocessor (MCU), or programmable logic device (FPGA), etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0023] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the operating system startup method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0024] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0025] For better understanding, the following explanations are provided for the terms appearing in this application:

[0026] Multi-site: Geographically dispersed data centers / edge nodes, interconnected via networks to achieve data synchronization and redundant backup.

[0027] Main site: The source site that supports read and write operations, user management, bucket creation, and object upload.

[0028] From site: A backup site that only supports read operations and object data synchronization (user / bucket management is not supported).

[0029] Pre-compression: A compression operation performed before data is synchronized and transmitted over a network.

[0030] Entropy: Reflects the randomness or information density of data. The lower the entropy, the higher the compressibility.

[0031] Incremental entropy scanning: a technique that analyzes only the first 1MB of data block content to predict the overall compression ratio (error rate <5%).

[0032] To address the aforementioned issues, this embodiment provides a data transmission method, including but not limited to applications on a first node (i.e., the execution entity of the data transmission method in this application is the first node). Figure 2 This is a flowchart of a data transmission method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps S202-S206:

[0033] Step S202: If there is target data to be sent to the second node at the first node, determine the compression rate of the target data, the network status between the first node and the second node, and the resource load status of the first node;

[0034] Optionally, network status includes, but is not limited to, key indicators such as network bandwidth, latency, and packet loss rate. The resource load status of the first node includes the load status of the computing resources executing the compression task in the first node, specifically including but not limited to CPU load rate and memory usage.

[0035] Step S204: Determine the compression strategy for the target data based on the compression ratio, network status, and resource load status of the target data;

[0036] It should be noted that when determining the compression strategy, not only were the characteristics of the data considered, but network conditions and computing resources were also fully evaluated to achieve the best compression effect while avoiding unnecessary pressure on network and computing resources. For example, when network bandwidth is sufficient and computing resources are abundant, a higher compression strategy can be selected to reduce the amount of data transmitted, while when network conditions are poor or computing resources are scarce, a lower compression strategy may be adopted to ensure the smoothness and timeliness of data transmission.

[0037] It should be noted that compression strategies include: not compressing or using a certain compression algorithm for compression.

[0038] Step S206: Send the target data to the second node based on the compression strategy.

[0039] It should be noted that when the compression strategy is no compression, the first node directly sends the target data to the second node. When the compression strategy uses algorithm A, the first node compresses the target data using algorithm A and then sends the compressed data to the second node.

[0040] Steps S202-S206 above dynamically determine the optimal compression strategy by evaluating the compression ratio of the target data, the real-time network status between the first and second nodes, and the resource load of the first node. This significantly improves the efficiency and economy of data transmission, solves the problem of poor data transmission efficiency, avoids the rigid approach of compressing or not compressing data regardless of its characteristics in traditional schemes, reduces repeated compression of low-entropy data, avoids the waste of computing resources, and also avoids the waste of network bandwidth caused by direct transmission without compression.

[0041] In an exemplary embodiment, the determination of the compression ratio of the target data can be achieved through the following steps S11-S12:

[0042] Step S11: Determine the entropy value of the header segment of the target data to obtain the target entropy value. The size of the header segment is a preset size, and the target entropy value is used to reflect the randomness or information density of the header segment data.

[0043] Optionally, the header segment can be the first 1KB or the first 1024 bytes of the target data. This preset size is chosen based on experimental and theoretical analysis to ensure the accuracy and efficiency of entropy calculation.

[0044] It should be noted that the entropy value of data is a mathematical indicator used to reflect the randomness or information density of data. The higher the entropy value, the stronger the randomness and the greater the information density of the data, which usually means that the data is difficult to compress effectively; conversely, the lower the entropy value, the more obvious the regularity and redundancy of the data, and the easier it is to compress.

[0045] In an exemplary embodiment, determining the entropy value of the header segment of the target data to obtain the target entropy value includes: calculating the entropy value of the header segment of the target data using the following formula:

[0046] ;

[0047] in, Let n be the target entropy value, and n represent the dimension of the byte value space. For the byte with ASCII encoding value i, For bytes The probability of occurrence in the head segment data.

[0048] Optional, n=256 (corresponding to all byte values ​​from 0x00 to 0xFF). It is a specific symbol instance in the data symbol set.

[0049] For example, =The letter 'A' (ASCII 65); = The number of times 'A' appears in the header data / the total number of bytes in the header data.

[0050] It should be noted that the Shannon entropy value calculated above can accurately reflect the entropy value of the head segment of the target data.

[0051] Step S12: Calculate the compression ratio of the target data based on the target entropy value and the data type of the target data.

[0052] It's important to note that after determining the target entropy value, the compression ratio of the target data is further calculated by considering its data type (text files, images, audio or video, TXT format data, ZIP format data, etc.). Each data type has its inherent properties and compression characteristics. By analyzing the relationship between compression ratio and data type, the compression effect can be intelligently predicted, avoiding unnecessary processing of unsuitable data, saving computing resources, and ensuring effective compression of data with compression potential, reducing transmission volume and improving network efficiency.

[0053] Optionally, if the CPU load rate of the first node is less than the preset load rate (e.g., 75%), the compression ratio of the target data is calculated using the steps S11-S12 described above. That is, when the CPU load is low, the first node has surplus computing power to perform in-depth analysis and processing of data compression, which means that a more accurate entropy value determination algorithm can be used.

[0054] By using the incremental entropy scanning and compression ratio prediction described above, the compression ratio of the target data can be accurately determined.

[0055] In an exemplary embodiment, the above-mentioned calculation of the compression ratio of the target data based on the target entropy value and the data type of the target data includes: calculating the compression ratio of the target data using the following formula: PCR=k / H_head×FileType_Weight; where PCR is the compression ratio of the target data, k is a preset constant, H_head is the target entropy value, and FileType_Weight is the weighting factor corresponding to the data type of the target data.

[0056] It should be noted that k is a constant factor obtained by fitting a large amount of experimental data (e.g., k≈12.0 to 14.0, which is an empirical range that performs well on many datasets, and k / H_head is a basic compression ratio prediction calculated based on the head entropy value H_head).

[0057] Optionally, the weight factor for DICOM format files is 0.92, the weight factor for TXT format files is 0.85, and the weight factor for ZIP format files is 0.15.

[0058] It should be noted that the compression rate of the target data can be accurately estimated using the above method.

[0059] In an exemplary embodiment, the determination of the compression ratio of the target data described above can also be achieved through the following steps S21-S23:

[0060] Step S21: Collect data blocks of the target size at multiple locations in the target data to obtain multiple data blocks;

[0061] It should be noted that the aforementioned locations include the header, middle, and tail of the target data. The target size is 1KB, meaning three 1KB data blocks can be collected.

[0062] It should be noted that this data collection method avoids the I / O and computational overhead associated with extensive scanning. For large objects, the sample size is sufficient to represent their overall statistical characteristics.

[0063] Step S22: Calculate the entropy values ​​of multiple data blocks in the following manner: ,in, Here, represents the entropy values ​​of multiple data blocks, where n represents the dimension of the byte value space. For the byte with ASCII encoding value i, For bytes Probability of occurrence in multiple data blocks;

[0064] Step S23: Calculate the compression ratio of the target data based on the entropy values ​​of multiple data blocks and the data type of the target data.

[0065] It should be noted that the method for calculating the compression ratio of the target data in step S23 is the same as that in step S12 above, and will not be repeated here.

[0066] It should be noted that steps S21-S23 above use extremely low computational overhead to quickly approximate the information entropy of the data block, thus serving as a proxy indicator of its compressibility. Unlike the huge overhead required to calculate Shannon entropy in steps S11-S12 above, absolute accuracy is abandoned in favor of pursuing the ultimate balance between speed, low overhead, and the correctness of trend judgment.

[0067] Optionally, if the CPU load rate of the first node is greater than or equal to a preset load rate (e.g., 75%), the compression rate of the target data is calculated using the steps S21-S23 described above.

[0068] In an exemplary embodiment, the above-mentioned determination of the compression strategy of the target data based on the compression ratio, network status, and resource load status of the target data includes: calculating a target benefit score based on the compression ratio, network status, and resource load status of the target data, wherein the target benefit score is used to reflect the relationship between the time saved in compression transmission and the time consumed in compression; and determining the compression strategy of the target data based on the target benefit score.

[0069] Optionally, the core of determining the compression strategy for the target data lies in calculating a target benefit score. This score is a comprehensive consideration aimed at balancing the time savings brought by data compression with the time consumed by the compression operation itself. Specifically, the compression ratio of the target data, the network status between the first and second nodes, and the resource load status of the first node are used as input parameters, and the target benefit score is calculated through a preset algorithm model. The calculation result of the target benefit score will guide the first node to select the most suitable compression strategy, ensuring that under the current transmission conditions, the data compression operation can bring the maximum net benefit, that is, to save transmission time to the greatest extent without significantly increasing compression time consumption and computing resource occupation, thereby improving the efficiency and economy of the entire data transmission process. This compression strategy decision-making mechanism based on the target benefit score enables this application to flexibly adapt to different transmission environments and resource statuses, providing a strong guarantee for efficient data transmission.

[0070] In an exemplary embodiment, the above calculation of the target benefit score based on the compression ratio of the target data, network status, and resource load status includes: calculating the target benefit score using the following formula: Benefit_Score=α*(S_orig / S_comp)+β*(T_net_base / T_net_comp)+γ*(1-C_crit);

[0071] Where T_net_base=S_orig / BW, T_net_comp=S_comp / BW;

[0072] Among them, C_crit=max(0,(CPU_Load-Threshold) / (100-Threshold));

[0073] Specifically, when CPU_Load > Threshold (e.g., 75%), C_crit > 0;

[0074] Where Benefit_Score is the target benefit score, S_orig is the target data size, S_comp is the target data size after compression according to the compression ratio, S_orig / S_comp equals the target data compression ratio, BW is the bandwidth, network status includes bandwidth, CPU_Load is the CPU load rate, resource load status includes CPU load rate, Threshold is the preset threshold value, and α, β, and γ are weight values.

[0075] It should be noted that (S_orig / S_comp) represents the space saving factor. (T_net_base / T_net_comp) represents the network transmission time saving factor. The larger this factor, the greater the transmission time saving. (1-C_crit) represents the resource availability factor. C_crit is the normalized threshold of CPU load. The larger this factor, the more idle the CPU is, and the more it can tolerate compression overhead. α, β, and γ (α+β+γ≈1) respectively measure the relative importance of space saving, network transmission saving, and resource overhead in the overall score.

[0076] It should be noted that when the CPU load rate of the first node is greater than or equal to the preset load rate (e.g., 75%), α, β, and γ can use preset conservative values, such as α=0.3, β=0.6, and γ=0.1. When the CPU load rate of the first node is less than the preset load rate, the method for determining α, β, and γ is detailed in steps S41-S44 below.

[0077] In this embodiment, the target revenue score can be accurately calculated using the above method, and then the compression strategy can be accurately determined based on the target revenue score.

[0078] In an exemplary embodiment, the above-described compression strategy for determining the target data based on the target return score includes the following steps S31-S33:

[0079] Step S31: If the target profit score is greater than or equal to the first preset threshold, determine the compression strategy as using the first compression algorithm to compress the target data, wherein the first compression algorithm is a compression algorithm with a compression ratio greater than the preset compression ratio;

[0080] Optionally, the first compression algorithm is the Zstandard / Zstd algorithm. In addition, the compression strategy also includes configuring a high compression level (e.g., level=19) corresponding to the Zstandard / Zstd algorithm. The goal of using the first compression algorithm is to maximize bandwidth savings and tolerate a longer compression time.

[0081] Step S32: If the target return score is greater than or equal to the second preset threshold and less than the first preset threshold, determine the compression strategy as using the second compression algorithm to compress the target data, wherein the second compression algorithm is a fast compression algorithm and the second preset threshold is less than the first preset threshold.

[0082] Optionally, the second compression algorithm is the LZ4 algorithm. In addition, the compression strategy also includes configuration acceleration parameters corresponding to the LZ4 algorithm (such as acceleration=2). The goal of using the second compression algorithm is to balance compression speed and bandwidth saving.

[0083] Step S33: If the target return score is less than the second preset threshold, determine that the compression strategy is not to compress the target data.

[0084] It should be noted that the process of determining the compression strategy based on the target benefit score in this application is designed as a hierarchical decision-making mechanism. Specifically, when the target benefit score reaches or exceeds a first preset threshold, a first compression algorithm is used for data compression. This algorithm has a high compression ratio and can significantly reduce the amount of data transmitted and improve transmission efficiency when data redundancy is high and network conditions permit. When the target benefit score is between a second preset threshold and a first preset threshold, a second compression algorithm is used. This is a fast compression algorithm, although its compression ratio may be lower, it can quickly complete the compression process when computing resources are limited or network conditions are poor, reducing the delay caused by the compression operation and ensuring the timeliness of data transmission. Finally, if the target benefit score is lower than the second preset threshold, it indicates that the potential benefit of the compression operation cannot offset the time and resources consumed. In this case, it is decided not to compress the target data and directly transmit the original data, avoiding unnecessary computational overhead and ensuring the efficiency and stability of data transmission. This multi-level compression strategy decision-making mechanism based on the target benefit score enables this application to flexibly respond to different transmission conditions and resource states, optimize the data transmission process, and improve the overall system performance.

[0085] In an exemplary embodiment, the method further includes the following steps S41-S44:

[0086] Step S41: Determine the compression effect corresponding to the transmission task within the first preset historical time period. The compression effect is determined by a first value and a second value. The first value is used to reflect the transmission time saved after compressing the transmission data, and the second value is used to reflect the bandwidth cost saved after compressing the transmission data.

[0087] Step S42: If the compression effect meets the preset conditions, the α, β, and γ used in the previous transmission task when calculating the target benefit score are determined as the α, β, and γ used in this calculation of the target benefit score.

[0088] Step S43: If the compression effect does not meet the preset conditions, obtain the transmission data corresponding to multiple transmission tasks within the second preset historical time period. The transmission data includes: the actual compression rate, actual transmission time, actual compression time, network status and resource load status corresponding to the transmission data.

[0089] Step S44: Determine α, β, and γ used in calculating the target revenue score based on the transmission data corresponding to multiple transmission tasks.

[0090] Optionally, in this application, the first node reviews the transmission tasks within a first preset historical time period (e.g., the past hour) and analyzes the savings in transmission time (first value) and reduction in bandwidth cost (second value) resulting from the compression operation, thereby evaluating the compression effect. If the compression effect meets preset optimization conditions, such as the saved transmission time and bandwidth cost exceeding a certain threshold, the weight parameters α, β, and γ used in the previous calculation of the target benefit score will be retained. These parameters represent the relative importance of compression time consumption, transmission time savings, and bandwidth cost savings in calculating the compression benefit, respectively, thereby quickly deciding on the compression strategy for this data transmission.

[0091] If the historical compression effect does not meet the preset standard, the first node will analyze multiple transmission tasks within a second preset historical time period (e.g., the past 6 hours), extracting the actual compression ratio, actual transmission time, actual compression time, and network status and resource load during each transmission. Based on this detailed data, deep learning will be performed to adjust the key weight parameters α, β, and γ in the compression strategy to adapt to the changing transmission environment. This process involves training machine learning algorithms (e.g., LSTM networks). By analyzing the relationship between historical compression effects and transmission conditions, parameters are learned and optimized, making the target benefit score more accurately reflect the economic benefits of the current compression strategy. This allows for more informed compression decisions in future data transmission, effectively improving data transmission efficiency and economic benefits.

[0092] In this embodiment, through the above mechanism, it can not only respond to the current transmission conditions in real time, but also adjust and optimize itself based on historical compression effects, thereby realizing intelligent management and dynamic optimization of data compression strategies.

[0093] Optionally, the weight coefficients (α, β, γ) in this application are not fixed, but are periodically adjusted by a lightweight Long Short-Term Memory (LSTM) network to adapt to constantly changing business scenarios and network environments. The adjustment mechanism is as follows:

[0094] I. Adjusting Goals and Principles:

[0095] The weighting coefficients α, β, and γ quantify the relative importance of space savings, network transmission savings, and resource overhead in the decision-making process, respectively. Their adjustment follows a core principle:

[0096] To maximize the overall benefits of the system, that is, to achieve the optimal balance (Pareto optimality) of bandwidth, latency and cost with limited resources.

[0097] α (Space Saving Weight): The higher the weight, the more the engine tends to choose high compression ratio algorithms to save storage costs and cross-site bandwidth.

[0098] β (Network Transmission Saving Weight): The higher the weight, the more the engine tends to prioritize reducing transmission time and the more sensitive it is to network conditions (high latency, low bandwidth).

[0099] γ (Resource overhead weight): The higher the weight, the more conservative the engine is, and the more it tends to avoid compression under high load in order to ensure the overall stability of the system.

[0100] II. Input and Output of LSTM Networks:

[0101] This is a supervised learning process, where the LSTM network acts as a complex nonlinear regression model.

[0102] 1. Input Features (Time Series Window):

[0103] The network receives a time-ordered sequence window (e.g., historical synchronization task data from the past 6 hours). Each data point at each time step is a feature vector containing:

[0104] Environmental characteristics:

[0105] PCR_actual: Actual compression ratio (compressed size / original size);

[0106] Network_BW: Actual network bandwidth (Mbps) during synchronization;

[0107] CPU_Load: Average CPU load (%) during compression task execution.

[0108] Characteristics of decision-making effectiveness:

[0109] T_saved: Actual saved transmission time (theoretical transmission time without compression - compressed transmission time - compression time).

[0110] Cost_saved: Actual bandwidth cost saved (calculated based on the saved traffic and cost per unit of bandwidth);

[0111] Decision_Effect_Score: Overall decision effectiveness score (a normalized score calculated by weighting T_saved and Cost_saved);

[0112] 2. Output:

[0113] The output layer of the LSTM network has three neurons, and its output values ​​[o1, o2, o3] are processed by the Softmax activation function to ensure that the sum of the three values ​​is 1. These three final values ​​are the weight coefficients for the next round.

[0114] [α_new, β_new, γ_new]=Softmax([o1, o2, o3]).

[0115] In an exemplary embodiment, determining the compression strategy of the target data based on the compression ratio of the target data, network status, and resource load status includes: when the CPU load rate of the first node is less than a preset load rate, determining the compression strategy of the target data based on the compression ratio of the target data, network status, and resource load status.

[0116] Specifically, when the CPU load rate of the first node is greater than or equal to the preset load rate, the compression strategy of the target data is determined based on the entropy values ​​of multiple data blocks corresponding to the target data at multiple preset locations.

[0117] Optionally, the compression strategy of the target data is determined based on the entropy values ​​of multiple data blocks corresponding to multiple preset positions, including: if the entropy values ​​of multiple data blocks are less than a preset entropy threshold, the compression strategy of the target data is determined to be no compression; if the entropy values ​​of multiple data blocks are greater than or equal to the preset entropy threshold, the compression strategy of the target data is determined to be compression using a second compression algorithm.

[0118] Optionally, if the CPU load rate of the first node is greater than or equal to the preset load rate, the compression strategy of the target data can also be determined based on the compression rate of the target data, the network status, and the resource load status. However, in this case, the compression rate of the target data is calculated using the above steps S21-S23.

[0119] In this embodiment, the above method can flexibly adapt to different computing resource availability scenarios, intelligently adjust the depth and breadth of data compression, and take into account both data transmission efficiency and economical use of computing resources, providing a more stable and efficient technical solution for big data transmission scenarios.

[0120] In an exemplary embodiment, sending the target data to the second node based on a compression strategy includes the following steps S51-S53:

[0121] Step S51: If the compression strategy is not to compress the target data, send the target data to the second node;

[0122] Step S52: When the compression strategy is to compress the target data using the first compression algorithm, the target data is compressed through the field programmable gate array of the first node, and the compressed data is sent to the second node. The first compression algorithm is a compression algorithm with a compression ratio greater than the preset compression ratio.

[0123] Optionally, the calculation of the quotient and target revenue score can both be performed on the field-programmable gate array of the first node.

[0124] It should be noted that FPGAs, due to their programmability and hardware-level parallel computing capabilities, are particularly suitable for performing computationally intensive compression tasks. They can effectively reduce CPU load and improve data processing speed while ensuring a high compression ratio.

[0125] Step S53: When the compression strategy is to use the second compression algorithm to compress the target data, the target data is compressed by the central processing unit of the first node, and the compressed data is sent to the second node. The second compression algorithm is a fast compression algorithm.

[0126] It should be noted that the advantage of a CPU lies in its versatility and flexibility, making it suitable for quickly completing lightweight compression tasks and ensuring high efficiency in data transmission.

[0127] It should be noted that by intelligently allocating compression tasks to the FPGA or CPU, efficient and flexible data compression and transmission are achieved. The FPGA handles high compression ratio algorithms, reducing CPU load and accelerating data flow; the CPU, on the other hand, quickly executes lightweight compression to ensure transmission efficiency. This strategy significantly improves the economy of data transmission and the rationality of resource utilization.

[0128] In an exemplary embodiment, the method further includes: sending the cyclic redundancy check value of the target data to the second node during the process of sending the target data to the second node; and sending the cyclic redundancy check value of the compressed data to the second node during the process of sending the compressed data to the second node.

[0129] It should be noted that, to ensure the integrity and accuracy of data transmission, regardless of whether the target data is compressed, a Cyclic Redundancy Check (CRC) value is additionally transmitted to the receiving end (second node). The CRC value is a commonly used error detection code, generated through specific mathematical operations on the data, which helps the receiver detect whether errors or corruption have occurred during transmission.

[0130] It should be noted that after receiving the data, the receiver recalculates the CRC value of the received data and compares it with the CRC value provided by the sender. If the two match, it indicates that the data transmission was successful and no error occurred; otherwise, it indicates that the data may have been corrupted during transmission and a retransmission is required.

[0131] It should be noted that this application, by introducing a CRC value verification mechanism, provides a technical solution that is both efficient and reliable for big data transmission, while improving data transmission efficiency and economic benefits and strengthening the integrity check of data transmission.

[0132] In an exemplary embodiment, the method further includes: upon obtaining the synchronization data sent by the third node and the corresponding cyclic redundancy check value, calculating the cyclic redundancy check value of the received synchronization data to obtain a target cyclic redundancy check value; and if it is determined that the target cyclic redundancy check value does not match the received cyclic redundancy check value, sending a data retransmission instruction to the third node.

[0133] In an exemplary embodiment, the method further includes: if it is determined that the target cyclic redundancy check value matches the received cyclic redundancy check value and the received data is not compressed, calculating the hash value of the received data and comparing the calculated hash value with the hash value of the target data; if it is determined that the target cyclic redundancy check value matches the received cyclic redundancy check value and the received data is compressed, decompressing the received data and calculating the hash value of the decompressed data; and comparing the calculated hash value with the hash value of the target data.

[0134] In this embodiment, to enhance the security and accuracy of data transmission, a hash value comparison mechanism is introduced in addition to CRC value verification. Whether the data is before or after compression, as long as the CRC check passes, an additional hash value check will be performed to ensure data integrity and tamper-proofing. When the received synchronization data is uncompressed, the receiver calculates the hash value of the received data and compares it with the hash value of the target data provided by a trusted source or obtained from a third node. As a data fingerprint, even minor changes in the data can lead to drastically different hash values; therefore, hash value consistency confirms the originality and undamaged state of the data.

[0135] For compressed data, the received data is first decompressed to restore it to its original state. Then, the hash value of the decompressed data is calculated and compared with the hash value of the target data. By comparing the hash values ​​after decompression, it is ensured that any potential changes or losses during data transmission can be accurately detected, regardless of whether these changes occur during compression, transmission, or decompression.

[0136] It should be noted that in this embodiment, this dual verification mechanism (first confirming the accuracy of data transmission through CRC value, and then deeply verifying the originality and security of data through hash value) greatly improves the reliability of data transmission. It is suitable for various scenarios with strict requirements for data integrity and security, such as financial transactions and medical information transmission, ensuring the confidentiality and authenticity of data during transmission.

[0137] In an exemplary embodiment, the method further includes: if it is determined that the calculated hash value and the hash value of the target data are consistent, storing the received data or the decompressed data, and sending a success receipt to the third node using the digital signature of the first node.

[0138] It should be noted that, to confirm the data reception status to the third node, the recipient will generate a success receipt using its unique digital signature. A digital signature is a security mechanism based on public-key cryptography, ensuring the sender's identity and the integrity of the information. The success receipt contains the recipient's confirmation of successful data transmission, encrypted with a digital signature to guarantee that the receipt information is not tampered with or forged by a third party during transmission. Upon receiving the success receipt, the third node can use the recipient's public key to decrypt and verify the digital signature, thereby confirming that the data has accurately reached the recipient and that the recipient's confirmation is trustworthy.

[0139] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. To better understand the above method, the following description, in conjunction with embodiments, illustrates the process, but is not intended to limit the technical solutions of the embodiments of this application. Specifically:

[0140] This application provides a method and system for multi-site data synchronization of object storage based on an intelligent pre-compression decision engine. It aims to solve problems in existing technologies such as wasted network bandwidth, excessive computational resource consumption, and uncontrollable synchronization latency caused by static or full compression strategies during cross-site data synchronization. The core of this application lies in real-time and dynamic evaluation of whether pre-compressing a specific data block can bring overall benefits under the current network and computing environment (i.e., whether the transmission time / bandwidth saved by compression can offset the computational overhead of compression itself), and intelligently selecting the optimal compression strategy (including no compression, fast compression, deep compression, or direct transmission) based on the evaluation results.

[0141] Figure 3 This is an overall flowchart of a data transmission method according to an embodiment of this application, which is described below in conjunction with... Figure 3 The technical solution of this application is described in detail below:

[0142] The system architecture of this application mainly includes the following core modules, which are deployed on the main site (source site) of the object storage system:

[0143] (1) Metadata scanner: responds to synchronization instructions and scans the metadata information (such as file type, size, modification time, etc.) of the object (or object block) to be synchronized.

[0144] (2) Data Feature Analysis Module: Receives information from the metadata scanner and performs lightweight feature extraction on the data to be synchronized. The core operation is to perform incremental entropy value scanning.

[0145] (3) Network status monitoring module: Real-time monitoring of network performance indicators between the master site and the target site (one or more slave sites), including but not limited to: available bandwidth, network latency, and packet loss rate.

[0146] (4) Resource monitoring module: Real-time monitoring of the computing resource load of the main site executing compression tasks, especially CPU utilization.

[0147] (5) Intelligent Compression Decision Engine (Core Module): Receives data feature information (predicted compression ratio, file type, etc.) from the data feature analysis module, real-time network status information from the network status monitoring module, and real-time resource load information from the resource monitoring module. This engine has a built-in dynamic benefit scoring model that calculates the pre-compression benefit score (Benefit_Score) for the current data block in real time based on the input parameters, and outputs compression decision instructions (no compression, compression algorithm and parameters) based on the score results.

[0148] (6) Compression Execution Module: Receives and executes instructions issued by the intelligent compression decision engine. Its responsibilities include: if the decision is to compress, it calls the compression library to perform the compression operation according to the compression algorithm (such as LZ4, Zstd) and parameter configuration (such as compression level); if the decision is not to compress, it directly enters the transmission process; manages the transmission of data (compressed or original data) to the target slave station through the network; and triggers the generation of source end verification (such as CRC64).

[0149] (7) Target site processing module (deployed at the slave site): Receives transmitted data. If the data is compressed, it performs decompression. It performs target-side verification (such as SHA256) and compares it with the source-side verification value to achieve dual-channel verification. If the verification passes, it writes the data to the storage system and sends an acknowledgment receipt containing a digital signature to the source site; if the verification fails, it notifies the source site to retransmit.

[0150] Specifically, the core inventive points of this application are described in detail below:

[0151] 1) Incremental entropy scanning and compression ratio prediction (improving efficiency and reducing overhead):

[0152] Technical Description: To avoid the performance loss caused by scanning the entire large data block, the data feature analysis module only reads the first segment (e.g., 1MB) of the data block to be synchronized.

[0153] Core operation: Calculate the Shannon entropy value H(X) of the first segment, using the formula: Entropy reflects the randomness or information density of data; the lower the entropy, the higher the compressibility.

[0154] Prediction Mechanism: Based on the calculated header entropy value H_head, combined with predefined file type weighting factors (e.g., DICOM=0.92, TXT=0.85, ZIP=0.15), the overall compression ratio (Predicted_Compression_Ratio, PCR) of the entire data block is predicted. The goal of this prediction mechanism is to significantly reduce the computational overhead of feature analysis while maintaining a prediction error rate of <5%.

[0155] 2) Dynamic Revenue Scoring Model and Intelligent Decision Engine (Core Innovation):

[0156] Model input:

[0157] PCR: Overall compression rate predicted by the data feature analysis module.

[0158] Network_State: Real-time network status provided by the network status monitoring module (key indicators: bandwidth BW, latency).

[0159] CPU_Load: The CPU load rate provided by the resource monitoring module.

[0160] Benefit_Score Calculation: The core objective of the model is to quantify the comprehensive benefits of pre-compression operations in the current environment. The benefits are mainly reflected in whether the savings in transmission time can significantly exceed the compression time consumption, while considering network bandwidth costs and resource tightness.

[0161] A feasible quantification model formula is as follows:

[0162] Benefit_Score = α * (S_orig / S_comp) + β * (T_net_base / T_net_comp) + γ * (1 - C_crit).

[0163] Dynamic Update of Weight Coefficients: To adapt to different business scenarios and environmental changes (such as nightly log synchronization vs. daytime user uploads), α, β, and γ are not fixed. A machine learning model (such as an LSTM network) can be used to train hourly (or according to the configured period) based on historical synchronization task data (including actual selected decisions, actual compression ratios, actual transmission times, actual compression durations, and the network and CPU status at that time), and automatically adjust the values of α, β, and γ. This enables the decision-making engine to continuously learn and optimize its decision-making strategy.

[0164] Decision Rule: Perform hierarchical decision-making based on the real-time calculated Benefit_Score:

[0165] Score >= Thresh_high (for example: 1.8): Trigger the high compression ratio mode. Select an algorithm with a high compression ratio (such as Zstandard / Zstd), and configure a high compression level (such as level = 19). The goal is to maximize bandwidth savings and tolerate longer compression times.

[0166] Thresh_low <= Score < Thresh_high (for example: 1.2 <= Score < 1.8): Trigger the balanced / fast compression mode. Select an algorithm that is fast and has a moderate compression ratio (such as LZ4), and configure acceleration parameters (such as acceleration = 2). Balance compression speed and bandwidth savings.

[0167] Score < Thresh_low (for example: 1.2): Trigger the direct transmission mode. Skip the compression operation and directly transmit the original data. Applicable to situations where the compression benefit is low or resources are tight.

[0168] 3) Elastic Resource Optimization Mechanism (to ensure system stability):

[0169] Hardware acceleration: To improve the efficiency of core computing in the decision engine (such as entropy calculation, scoring calculation, and model inference) and reduce latency (target latency ≤ 0.7ms), key computing logic can be deployed on FPGA (Field Programmable Gate Array) or dedicated ASIC (Application-Specific Integrated Circuit) chips.

[0170] Load-Sensitive Degradation: When the resource monitoring module detects that the CPU load exceeds a preset high threshold (e.g., 75%), a lightweight degradation mode is triggered. In this mode, the data feature analysis module performs only the minimum feature extraction (e.g., simple type judgment based solely on file extensions + extremely simplified entropy estimation), and the strategy engine may simplify the scoring model or directly adopt a conservative strategy (e.g., favoring no compression or fast compression) to prioritize ensuring the overall system responsiveness and avoid compression operations that would increase the CPU load.

[0171] 4) Dual-channel data integrity verification (ensuring reliability):

[0172] Source verification (channel 1): Before the compression execution module prepares to send data (whether compressed or not), calculate the CRC64 check value of the original data (if the decision is not to compress) or the compressed data.

[0173] Target verification (Channel 2): ​​After the target site processing module receives the data: if the data is compressed, it is first decompressed to obtain the original data. The SHA256 hash value of the decompressed (or received original) data is calculated.

[0174] The detailed comparison and confirmation process is as follows:

[0175] Source end (compression execution module): Calculates the CRC64 value CRC_src of the data to be sent (raw data or compressed data).

[0176] Source end: Send the data plus CRC_src to the target end.

[0177] Target end: Receive data and CRC_src.

[0178] Target end: Calculate the CRC64 value CRC_dst of the received data.

[0179] Target end: Compares CRC_src and CRC_dst. If they do not match, requests a retransmission. (Channel 1: Verifies that the transmission process is error-free)

[0180] Target side: If the data is compressed, decompress the data to obtain the original data.

[0181] Target side: Calculate the SHA256 value Hash_dst of the original data to be stored in the final storage.

[0182] Target end: Compares Hash_dst with the SHA256 value calculated and stored by the source end for the original data to be synchronized (this value needs to be calculated and transmitted by the source end before synchronization begins or provided by a trusted source). If they do not match, it indicates an error in the source data or the decompression process, requiring error handling. (Channel 2: Verifies data integrity and decompression correctness)

[0183] By employing this dual-channel, heterogeneous verification algorithm design, the undetected error rate during data transmission and storage is controlled to an extremely low level (< This meets the requirements for financial-grade data reliability.

[0184] The calculation process for the dual-channel check fault tolerance rate is as follows:

[0185] .

[0186] To better understand, the data synchronization workflow is explained in detail below:

[0187] (1) Trigger: The master site receives an instruction to synchronize a specific object to one or more slave sites.

[0188] (2) Metadata scanning: The metadata scanner extracts the metadata (size, type, etc.) of the object to be synchronized.

[0189] (3) Feature analysis: The data feature analysis module performs incremental entropy value scanning on the object data and predicts the overall compression ratio (PCR) in combination with the file type.

[0190] (4) Status collection: The network status monitoring module obtains the real-time network status (BW, Latency); the resource monitoring module obtains the real-time CPU load.

[0191] (5) Intelligent Decision Making: Intelligent Compression Decision Engine: Obtain PCR, Network_State, and CPU_Load. Apply a dynamic benefit scoring model (using the current weight coefficients α, β, γ) to calculate Benefit_Score. Based on Benefit_Score, apply decision rules to generate decision instructions (compression algorithm / parameters or no compression).

[0192] (6) Execution and source verification: Compression execution module:

[0193] Execute the operation based on the decision instruction: compress (by algorithm parameters) or do not compress.

[0194] Calculate the CRC64 checksum of the data to be sent.

[0195] Send the data (compressed or raw) along with the CRC_src to the target slave station.

[0196] (7) Target processing and verification: The target site receives data and CRC_src.

[0197] Channel 1 verification: Calculate the CRC64 (CRC_dst) of the received data and compare it with CRC_src. If it fails, request a retransmission. If the data is compressed: Perform a decompression operation to restore the original data.

[0198] Channel 2 verification: Calculate the SHA256 (Hash_dst) of the decompressed (or received raw) data and compare it with the expected value (SHA256 of the source raw data). If it fails, perform error handling.

[0199] (8) Storage and Confirmation: If all verifications pass, the target site writes the original data to storage. The target site sends a success confirmation receipt containing a digital signature to the main site.

[0200] For better understanding, the following illustrates specific embodiments of this application:

[0201] Implementation Case 1: Medical Image Synchronization (512GB DICOM Data).

[0202] Scenario: The main site in city A needs to synchronize 512GB of DICOM medical images to the slave site in city B.

[0203] Detailed workflow:

[0204] 1. Feature Analysis:

[0205] The metadata scanner extracts the file type (DICOM) and size (512GB).

[0206] Incremental entropy scan: Read the first 1MB of data and calculate the entropy value H_head = 5.8 bits / byte.

[0207] Compression ratio prediction: Based on the DICOM weighting factor (0.92), the overall compression ratio PCR is predicted to be 1.92 (i.e., the compressed size is approximately 267GB).

[0208] 2. Decision engine calculation:

[0209] Network status: Bandwidth = 410Mbps, latency = 85ms (low jitter);

[0210] CPU load: 30% (idle);

[0211] LSTM dynamic weights (current period): α=0.6 (focusing on compression ratio), β=0.3 (focusing on transmission time), γ=0.1 (focusing on resource overhead).

[0212] Scoring Calculation:

[0213] Space saving factor: S_orig / S_comp = 512 / 267 ≈ 1.92;

[0214] Transmission time saving factor: T_net_base=512GB / (410Mbps)≈170.5 minutes, T_net_comp=267GB / (410Mbps)≈88.9 minutes → 170.5 / 88.9≈1.92;

[0215] Resource availability factor: 1-C_crit=1-0=1 (load <75%).

[0216] Benefit_Score=0.6*1.92+0.3*1.92+0.1*1=1.828;

[0217] 3. Decision Implementation:

[0218] Score=1.828>1.8→ Trigger Zstd hypercompression (level=19).

[0219] The compression execution module called the FPGA to accelerate compression, which took 8 minutes and 17 seconds.

[0220] 4. Transmission and Verification:

[0221] The compressed data (267GB) took approximately 88.9 minutes to transfer (the original transfer took 170.5 minutes, saving 81.6 minutes).

[0222] Target end: Channel 1 checksum (CRC64): Transmitted data integrity verification passed;

[0223] After decompression, Channel 2 verification (SHA256) shows that it matches the pre-stored hash from the source.

[0224] Overall effect:

[0225] Total synchronization time (compression + transmission): 8.28 + 88.9 = 97.18 minutes;

[0226] Raw synchronization time (uncompressed): 170.5 minutes;

[0227] Total time saved: 73.32 minutes (43.0% saving);

[0228] Bandwidth consumption reduced: (512-267) / 512×100%=47.9%.

[0229] Implementation Case 2: Log File Synchronization (1.2TB TXT Data).

[0230] Scenario: The master site synchronizes 1.2TB of real-time log files to the edge slave site.

[0231] Detailed workflow:

[0232] 1. Feature Analysis:

[0233] File type: TXT (weight factor = 0.85), head entropy H_head = 7.2 bits / byte → predicted compression ratio PCR = 1.65.

[0234] 2. Decision engine calculation:

[0235] Network status: Bandwidth = 1Gbps, latency = 15ms;

[0236] CPU load: 65% (medium);

[0237] LSTM weights: α=0.5, β=0.4, γ=0.1 (current cycle focuses on transmission efficiency);

[0238] Scoring Calculation:

[0239] Space saving factor: 1.2TB / 727GB≈1.69;

[0240] Transmission time factor:

[0241] T_net_base = (1.2 × 1024 × 1024 MB) / 125 MB / s ≈ 10,066 seconds ≈ 167.8 minutes; T_net_comp = (727 × 1024 MB) / 125 MB / s ≈ 5,956 seconds ≈ 99.3 minutes → T_net_base / T_net_comp = 167.8 / 99.3 ≈ 1.69;

[0242] Resource factor: 1-max(0,(65-75) / (100-75))=1;

[0243] Benefit_Score=0.5*1.69+0.4*1.69+0.1*1=1.615;

[0244] 3. Decision Implementation:

[0245] Score=1.615∈[1.2,1.8]→LZ4 acceleration mode (acceleration=2).

[0246] The CPU performs compression (without triggering the FPGA) at a speed of 820MB / s. (Compression time: 1.2TB / 820MB / s ≈ 24.9 minutes).

[0247] 4. Comparison of effects:

[0248] Total synchronization time (compression + transmission): 24.9 + 99.3 = 124.2 minutes.

[0249] Raw synchronization time (uncompressed): 167.8 minutes.

[0250] End-to-end latency reduced by 25.9%.

[0251] CPU usage: 32% lower than the Zstd solution (actual test data).

[0252] Implementation Case 3: High Load Degradation Scenario.

[0253] Scenario: Synchronize mixed files (images + text) when the main site's CPU load reaches 85%.

[0254] Key processes:

[0255] 1. The resource monitoring module triggers a lightweight degradation mode:

[0256] Feature analysis relies solely on file extensions (.jpg, .log) + minimal entropy estimation (using 1KB sampling per 128KB).

[0257] The decision engine skips the LSTM model and uses conservative weights (α=0.3, β=0.6, γ=0.1).

[0258] 2. Decision "Do not compress" for low-entropy files (.jpg) (Score=0.9);

[0259] 3. Decision on "LZ4 basic compression" (Score=1.3) for high-entropy logs (.log).

[0260] 4. Results: Peak CPU usage decreased by 40%, and synchronization latency fluctuation was less than 5% (compared to full-featured mode).

[0261] It should be noted that this application achieves the following quantifiable breakthroughs through the collaborative optimization of an intelligent pre-compression decision engine:

[0262] 1. Bandwidth cost reduction:

[0263] Dynamic decision-making avoids invalid compression (such as already compressed files), reducing bandwidth consumption by 38% (industry average, Gartner 2025 benchmark).

[0264] High-entropy data deep compression (Zstd) further reduces traffic by 45%-60%.

[0265] 2. Improved synchronization performance: End-to-end latency reduced by 31%-44% (compared to static compression strategy);

[0266] LZ4 acceleration mode achieves compression throughput of >800MB / s (3 times faster than gzip).

[0267] 3. Enhanced system robustness: Stability is maintained through degradation mechanism under high load scenarios (CPU>75%), and the failure rate is reduced to 1 / 4 of the traditional solution (300% improvement); FPGA hardware acceleration ensures decision latency ≤0.7ms (99.9th percentile).

[0268] 4. Data Reliability: Dual-channel checksum (CRC64 + SHA256) achieves fault tolerance. It meets the financial-grade RTO of <5 minutes; the lightweight edge node agent (memory <15KB) supports tens of thousands of concurrent synchronizations.

[0269] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0270] Embodiments of this application also provide a data transmission device. Figure 4 This is a structural block diagram of a data transmission device according to an embodiment of this application, such as... Figure 4 As shown, the device includes:

[0271] The first determining module 402 is used to determine the compression rate of the target data, the network status between the first node and the second node, and the resource load status of the first node when the first node has target data to be sent to the second node.

[0272] The second determining module 404 is used to determine the compression strategy of the target data based on the compression ratio of the target data, network status, and resource load status.

[0273] The sending module 406 is used to send the target data to the second node based on the compression strategy.

[0274] The aforementioned device dynamically determines the optimal compression strategy by evaluating the compression ratio of the target data, the real-time network status between the first and second nodes, and the resource load of the first node. This significantly improves the efficiency and economy of data transmission, solves the problem of poor data transmission efficiency, avoids the rigid approach of compressing or not compressing data regardless of its characteristics in traditional schemes, reduces repeated compression of low-entropy data, avoids the waste of computing resources, and also avoids the waste of network bandwidth caused by direct transmission without compression.

[0275] In an exemplary embodiment, the first determining module 402 is further configured to determine the entropy value of the header segment of the target data to obtain the target entropy value, wherein the size of the header segment data is a preset size, and the target entropy value is used to reflect the randomness or information density of the header segment data; and to calculate the compression ratio of the target data based on the target entropy value and the data type of the target data.

[0276] In an exemplary embodiment, the first determining module 402 is further configured to calculate the entropy value of the head segment of the target data using the following formula to obtain the target entropy value:

[0277] ;

[0278] in, Let n be the target entropy value, and n represent the dimension of the byte value space. For the byte with ASCII encoding value i, For bytes The probability of occurrence in the head segment data.

[0279] In an exemplary embodiment, the first determining module 402 is further configured to calculate the compression ratio of the target data using the following formula:

[0280] PCR=k / H_head×FileType_Weight;

[0281] Where PCR is the compression ratio of the target data, k is a preset constant, H_head is the target entropy value, and FileType_Weight is the weighting factor corresponding to the data type of the target data.

[0282] In an exemplary embodiment, the first determining module 402 is further configured to collect data blocks of the target size at multiple locations of the target data to obtain multiple data blocks; and calculate the entropy value of the multiple data blocks in the following manner: ,in, Here, represents the entropy values ​​of multiple data blocks, where n represents the dimension of the byte value space. For the byte with ASCII encoding value i, For bytes The probability of occurrence in multiple data blocks; the compression ratio of the target data is calculated based on the entropy values ​​of multiple data blocks and the data type of the target data.

[0283] In an exemplary embodiment, the second determining module 404 is further configured to calculate a target benefit score based on the compression ratio of the target data, network status, and resource load status, wherein the target benefit score is used to reflect the relationship between the time saved by compression transmission and the compression time consumption; and to determine the compression strategy of the target data based on the target benefit score.

[0284] In an exemplary embodiment, the second determining module 404 is further configured to calculate the target benefit score using the following formula:

[0285] Benefit_Score=α (S_orig / S_comp)+β (T_net_base / T_net_comp)+γ (1-C_crit);

[0286] Where T_net_base=S_orig / BW, T_net_comp=S_comp / BW;

[0287] Among them, C_crit=max(0,(CPU_Load-Threshold) / (100-Threshold));

[0288] Where Benefit_Score is the target benefit score, S_orig is the target data size, S_comp is the target data size after compression according to the compression ratio, S_orig / S_comp equals the target data compression ratio, BW is the bandwidth, network status includes bandwidth, CPU_Load is the CPU load rate, resource load status includes CPU load rate, Threshold is the preset threshold value, and α, β, and γ are weight values.

[0289] In an exemplary embodiment, the second determining module 404 is further configured to: determine, when the target revenue score is greater than or equal to a first preset threshold, use a first compression algorithm to compress the target data, wherein the first compression algorithm is a compression algorithm with a compression ratio greater than a preset compression ratio; determine, when the target revenue score is greater than or equal to a second preset threshold and less than the first preset threshold, use a second compression algorithm to compress the target data, wherein the second compression algorithm is a fast compression algorithm and the second preset threshold is less than the first preset threshold; and determine, when the target revenue score is less than the second preset threshold, not to compress the target data.

[0290] In an exemplary embodiment, the second determining module 404 is further configured to determine the compression effect corresponding to the transmission task within a first preset historical time period, wherein the compression effect is determined by a first value and a second value, the first value being used to reflect the transmission time saved after compressing the transmission data, and the second value being used to reflect the bandwidth cost saved after compressing the transmission data; if the compression effect meets preset conditions, the α, β, γ used in the previous transmission task when calculating the target benefit score are determined as the α, β, γ used in the current calculation of the target benefit score.

[0291] In an exemplary embodiment, the second determining module 404 is further configured to, when the compression effect does not meet the preset conditions, acquire transmission data corresponding to multiple transmission tasks within a second preset historical time period, wherein the transmission data includes: the actual compression rate, actual transmission time, actual compression time, network status and resource load status corresponding to the transmission data; and determine α, β, γ used in calculating the target benefit score based on the transmission data corresponding to the multiple transmission tasks.

[0292] In an exemplary embodiment, the second determining module 404 is further configured to determine a compression strategy for the target data based on the compression ratio of the target data, network status, and resource load status when the CPU load rate of the first node is less than a preset load rate; wherein, when the CPU load rate of the first node is greater than or equal to the preset load rate, the compression strategy for the target data is determined based on the entropy values ​​of multiple data blocks corresponding to multiple preset locations of the target data.

[0293] In an exemplary embodiment, the sending module 406 is further configured to: send the target data to the second node if the compression strategy is not to compress the target data; compress the target data using the field-programmable gate array of the first node and send the compressed data to the second node if the compression strategy is to compress the target data using the first compression algorithm, wherein the first compression algorithm is a compression algorithm with a compression ratio greater than a preset compression ratio; and compress the target data using the central processing unit of the first node and send the compressed data to the second node if the compression strategy is to compress the target data using the second compression algorithm, wherein the second compression algorithm is a fast compression algorithm.

[0294] In an exemplary embodiment, the sending module 406 is further configured to send the cyclic redundancy check value of the target data to the second node during the process of sending the target data to the second node; and to send the cyclic redundancy check value of the compressed data to the second node during the process of sending the compressed data to the second node.

[0295] In an exemplary embodiment, the apparatus further includes: a processing module, configured to, upon obtaining the synchronization data sent by the third node and the corresponding cyclic redundancy check value, calculate the cyclic redundancy check value of the received synchronization data to obtain a target cyclic redundancy check value; and, if it is determined that the target cyclic redundancy check value does not match the received cyclic redundancy check value, send a data retransmission instruction to the third node.

[0296] In an exemplary embodiment, the apparatus further includes: a processing module, configured to: calculate the hash value of the received data and compare the calculated hash value with the hash value of the target data when it is determined that the target cyclic redundancy check value matches the received cyclic redundancy check value and the received data is not compressed; decompress the received data and calculate the hash value of the decompressed data when it is determined that the target cyclic redundancy check value matches the received cyclic redundancy check value and the received data is compressed; and compare the calculated hash value with the hash value of the target data.

[0297] In one exemplary embodiment, the apparatus further includes a processing module, configured to, upon determining that the calculated hash value and the hash value of the target data are consistent, store the received data or the decompressed data, and send a success receipt to the third node using the digital signature of the first node.

[0298] For a description of the features in the embodiment corresponding to the data transmission device, please refer to the relevant description in the embodiment corresponding to the data transmission method, which will not be repeated here.

[0299] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data transmission method embodiments.

[0300] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data transmission method embodiments when it is run.

[0301] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0302] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data transmission method embodiments.

[0303] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data transmission method embodiments.

[0304] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0305] The foregoing has provided a detailed description of a data transmission method and apparatus, electronic device, storage medium, and computer program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data transmission method, characterized by, The method comprises: In the case that there is target data to be sent from the first node to the second node, determining a compression rate of the target data, a network state between the first node and the second node, and a resource load state of the first node; determining a compression strategy of the target data according to the compression rate of the target data, the network state, and the resource load state; sending the target data to the second node based on the compression strategy; wherein determining the compression strategy of the target data according to the compression rate of the target data, the network state, and the resource load state comprises: calculating a target benefit score by the following formula, and determining the compression strategy of the target data according to the target benefit score, the target benefit score being used to reflect the relationship between the time saved by compression transmission and the compression time consumption: Benefit_Score = a * (S orig / S comp) + β * (T net_base / T net_comp) + γ * (1 - C crit); wherein T net_base = S orig / BW, and T net_comp = S comp / BW; wherein C crit = max(0, (CPU Load - Threshold) / (100 - Threshold)); wherein Benefit_Score is the target benefit score, S orig is the data size of the target data, S comp is the data size of the target data after compression according to the compression rate, S orig / S comp is equal to the compression rate of the target data, BW is the bandwidth, the network state comprises the bandwidth, CPU Load is the CPU load rate, the resource load state comprises the CPU load rate, Threshold is a preset threshold value, a, β, and γ are weight values; wherein the method further comprises: determining a compression effect corresponding to a transmission task in a first preset historical time period, wherein the compression effect is determined by a first value and a second value, the first value being used to reflect the transmission time saved after compression of transmission data, and the second value being used to reflect the bandwidth cost saved after compression of transmission data; in the case that the compression effect does not satisfy a preset condition, obtaining transmission data corresponding to a plurality of transmission tasks in a second preset historical time period, wherein the transmission data comprises: an actual compression rate of transmission data, an actual transmission time, an actual compression time consumption, a network state corresponding to transmission data, and a resource load state; determining a, β, and γ used in this time for calculating the target benefit score according to the transmission data corresponding to the plurality of transmission tasks.

2. The data sending method according to claim 1, wherein determining the compression rate of the target data comprises: determining an entropy value of data of a header segment of the target data to obtain a target entropy value, wherein the data size of the header segment is a preset size, and the target entropy value is used to reflect the randomness or information density of the data of the header segment. The compression rate of the target data is calculated according to the target entropy value and a data type of the target data.

3. The data sending method of claim 2, wherein The entropy value of the data of the header segment of the target data is determined to obtain a target entropy value, including: The entropy value of the data of the header segment of the target data is calculated according to the following formula to obtain the target entropy value: ； wherein, is the target entropy value, n represents the dimension of the value space in bytes, is a byte with ASCII code value i, is a byte is the probability of occurrence in the data of the header fragment.

4. The data sending method of claim 2, wherein The compression rate of the target data is calculated according to the target entropy value and a data type of the target data, including: The compression rate of the target data is calculated according to the following formula: PCR=k / H_head*FileType_Weight; wherein PCR is the compression rate of the target data, k is a preset constant, H_head is the target entropy value, and FileType_Weight is a weight factor corresponding to the data type of the target data.

5. The data sending method of claim 1, wherein The compression rate of the target data is determined, including: Data blocks of a target size are collected at multiple positions of the target data to obtain a plurality of data blocks; The entropy value of the plurality of data blocks is calculated by the following manner: wherein, is the entropy value of the plurality of data blocks, n represents the value space dimension of byte, is a byte with ASCII code value i, is a byte is the occurrence probability in the plurality of data blocks; The compression rate of the target data is calculated according to the entropy values of the plurality of data blocks and a data type of the target data.

6. The data sending method of claim 1, wherein The compression strategy of the target data is determined according to the target revenue score, including: in a case where the target revenue score is greater than or equal to a first preset threshold, determining that the compression strategy is to compress the target data using a first compression algorithm, wherein the first compression algorithm is a compression algorithm with a compression rate greater than a preset compression rate; in a case where the target revenue score is greater than or equal to a second preset threshold and less than the first preset threshold, determining that the compression strategy is to compress the target data using a second compression algorithm, wherein the second compression algorithm is a fast compression algorithm, and the second preset threshold is less than the first preset threshold; in a case where the target revenue score is less than the second preset threshold, determining that the compression strategy is not to compress the target data.

7. The data sending method of claim 1, wherein The method further includes: in a case where the compression effect meets a preset condition, determining that the a, b, and g used in the calculation of the target revenue score last time are used in the calculation of the target revenue score this time.

8. The data sending method of claim 1, wherein The compression strategy of the target data is determined according to the compression rate of the target data, the network state, and the resource load state, including: in a case where the CPU load rate of the first node is less than a preset load rate, determining the compression strategy of the target data according to the compression rate of the target data, the network state, and the resource load state. In a case where the CPU load rate of the first node is greater than or equal to a preset load rate, a compression strategy of the target data is determined according to entropy values of a plurality of data blocks corresponding to a plurality of preset positions of the target data.

9. The data sending method of claim 1, wherein, sending the target data to the second node based on the compression strategy comprises: in a case where the compression strategy is not to compress the target data, sending the target data to the second node; in a case where the compression strategy is to compress the target data using a first compression algorithm, compressing the target data by a field programmable gate array of the first node and sending the compressed data to the second node, wherein the first compression algorithm is a compression algorithm with a compression rate greater than a preset compression rate; in a case where the compression strategy is to compress the target data using a second compression algorithm, compressing the target data by a central processing unit of the first node and sending the compressed data to the second node, wherein the second compression algorithm is a fast compression algorithm.

10. The data sending method of claim 9, wherein, the method further comprises: sending a cyclic redundancy check value of the target data to the second node in the process of sending the target data to the second node; and sending a cyclic redundancy check value of the compressed data to the second node in the process of sending the compressed data to the second node.

11. The data sending method of claim 1, wherein, the method further comprises: in a case where the synchronization data and a cyclic redundancy check value corresponding to the synchronization data sent by a third node are obtained, calculating a cyclic redundancy check value of the received synchronization data to obtain a target cyclic redundancy check value; in a case where it is determined that the target cyclic redundancy check value does not match the received cyclic redundancy check value, sending a data retransmission indication to the third node.

12. The data sending method of claim 11, wherein, the method further comprises: in a case where it is determined that the target cyclic redundancy check value matches the received cyclic redundancy check value and the received data is not compressed, calculating a hash value of the received data and comparing the calculated hash value with a hash value of the target data; and in a case where it is determined that the target cyclic redundancy check value matches the received cyclic redundancy check value and the received data is compressed, decompressing the received data, calculating a hash value of the decompressed data, and comparing the calculated hash value with the hash value of the target data.

13. The data sending method of claim 12, wherein, the method further comprises: in a case where it is determined that the calculated hash value is consistent with the hash value of the target data, storing the received data or the decompressed data and sending a success acknowledgement to the third node using a digital signature of the first node.

14. A data transmission device, characterized in that, ​ The first determining module is configured to determine a compression rate of target data, a network state between the first node and the second node, and a resource load state of the first node when the target data exists in the first node and is to be transmitted to the second node. The second determining module is configured to determine a compression strategy of the target data according to the compression rate of the target data, the network state, and the resource load state. The sending module is configured to transmit the target data to the second node based on the compression strategy. The second determining module is further configured to calculate a target benefit score according to the following formula: Benefit_Score=α*(S_orig / S_comp)+β*(T_net_base / T_net_comp)+γ*(1-C_crit), and determine the compression strategy of the target data according to the target benefit score, wherein the target benefit score reflects a relationship between time saved by compression transmission and compression time consumption. The second determining module is further configured to determine a compression effect of transmission tasks in a first preset historical time period, wherein the compression effect is determined by a first value and a second value, the first value reflects transmission time saved after compression of transmission data, and the second value reflects bandwidth cost saved after compression of transmission data; when the compression effect does not satisfy a preset condition, the second determining module is further configured to acquire transmission data corresponding to a plurality of transmission tasks in a second preset historical time period, wherein the transmission data includes an actual compression rate, an actual transmission time, an actual compression time consumption, a network state, and a resource load state corresponding to the transmission data; and the second determining module is further configured to determine α, β, and γ used for calculating the target benefit score this time according to the transmission data corresponding to the plurality of transmission tasks. The memory is configured to store a computer program. The processor is configured to implement steps of the data transmission method in any one of claims 1 to 13 when the computer program is executed. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement steps of the data transmission method in any one of claims 1 to 13. ​ 15. An electronic device, comprising: ​ ​ ​ 16. A computer-readable storage medium, characterized in that, ​ 17. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the data sending method of any one of claims 1 to 13.

Images