Methods, apparatus, equipment, storage media, and software products for data transmission compression between primary and backup nodes.

Abstract

This application provides a method, apparatus, device, storage medium, and program product for compressing data transmission between primary and backup nodes, relating to the field of computer technology. The method includes: receiving a compression flag sent by a backup node, the compression flag indicating whether to compress transmitted data; parsing the resource load status of the primary node based on the compression flag; and performing data compression processing based on the resource load status and the compression flag. This method aims to improve resource utilization and data transmission efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium, and program product for compressing data transmission between primary and backup nodes. Background Technology

[0002] In a database cluster system, the master node transmits the generated write-ahead logs to the standby nodes in real time to ensure data synchronization between the standby and master nodes. If cluster nodes are distributed in different geographical locations or experience network fluctuations, the efficiency of data transmission between the master and standby nodes and server performance can be affected. Therefore, how data is compressed during transmission between the master and standby nodes is crucial.

[0003] Currently, existing methods for compressing data transmission between primary and backup nodes primarily rely on globally unified configuration. This method uses the same compression mode on all backup nodes. However, because these methods depend on the same compression mode, they result in wasted performance resources and poor data transmission efficiency. Summary of the Invention

[0004] This application provides a method, apparatus, device, storage medium, and program product for compressing data transmission between primary and backup nodes, in order to improve resource utilization and data transmission efficiency.

[0005] Firstly, this application provides a method for compressing data transmission between primary and backup nodes, including:

[0006] Receive a compression flag sent by the backup node, the compression flag being used to indicate whether the transmitted data should be compressed;

[0007] The resource load status of the master node is parsed based on the compression tags;

[0008] Data compression processing is performed based on the resource load status and the compression flag.

[0009] In one possible implementation, before receiving the compression tag sent by the backup node, the process includes: reading compression configuration parameters when the backup node is started, the compression configuration parameters including at least one of the following: disabling compression, enabling compression, and adaptive compression.

[0010] In one possible implementation, parsing the resource load status of the master node based on the compression tag includes: detecting the processor utilization and memory utilization of the master node; and determining whether the processor utilization and memory utilization exceed a preset resource load threshold.

[0011] In one possible implementation, performing data compression based on the resource load state and the compression flag includes: when the compression flag is adaptive compression and the resource load state is not exceeded, invoking a compression algorithm to compress the data; and when the compression flag is adaptive compression and the resource load exceeds the limit, not performing data compression.

[0012] In one possible implementation, after receiving the compressed tag sent by the backup node, the network status of the backup node is detected, the network status including at least one of the following: bandwidth, latency, and packet loss rate.

[0013] In one possible implementation, after receiving the compressed tag sent by the backup node, the method further includes:

[0014] Maintain historical transmission behavior records for backup nodes, wherein the historical transmission behavior records include at least one of the following: compression gain ratio and decompression efficiency.

[0015] In one possible implementation, after performing data compression processing based on the resource load status and the compression mark, the process includes: sending the compressed data to the backup node, wherein the data includes at least one of the following: compressed data and original data.

[0016] In one possible implementation, the method further includes: after the backup node receives the data, performing decompression processing based on the compression mark in the data header.

[0017] Secondly, this application provides a data transmission compression device for primary and backup nodes, comprising:

[0018] The receiving module is used to receive a compression flag sent by the backup node, the compression flag being used to indicate whether the transmitted data is compressed;

[0019] The parsing module is used to parse the resource load status of the master node based on the compression tags;

[0020] An execution module is used to perform data compression processing based on the resource load status and the compression flag.

[0021] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0022] The memory stores computer-executed instructions;

[0023] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method described in any of the first aspects above.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any of the first aspects above.

[0025] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0026] This application provides a method, apparatus, device, storage medium, and program product for compressing data between primary and backup nodes. It receives a compression flag from the backup node, indicating whether to compress the transmitted data, thus clarifying the backup node's compression requirements, avoiding indiscriminate compression or no compression, and ensuring consistency between the data transmission strategy and the backup node. Based on the compression flag, the resource load status of the primary node is analyzed, laying the foundation for subsequent data compression processing. This enables dynamic adaptation of the compression strategy to the primary node's resource status, improving resource utilization. Data compression processing is performed based on the resource load status and the compression flag, ensuring data consistency and reliability, and improving resource utilization and data transmission efficiency. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0028] Figure 1 A flowchart illustrating a primary / backup node data transmission compression method provided in this application;

[0029] Figure 2 This application provides an architectural diagram of an adaptive data transmission compression system based on master and backup nodes in a cluster.

[0030] Figure 3 A flowchart illustrating another method for compressing data transmission between primary and backup nodes provided in this application;

[0031] Figure 4 A schematic diagram of a primary / backup node data transmission compression device provided in this application;

[0032] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application.

[0033] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0035] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0036] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0037] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0038] First, let's explain some of the terms used in this application:

[0039] Write-Ahead Logging (WAL) refers to writing log records before a transaction is committed.

[0040] In a database cluster system, the master node transmits the generated write-ahead logs to the standby nodes in real time to ensure data synchronization between the standby and master nodes. If cluster nodes are distributed in different geographical locations or experience network fluctuations, the efficiency of data transmission between the master and standby nodes and server performance can be affected. Therefore, how data is compressed during transmission between the master and standby nodes is crucial.

[0041] Currently, existing methods for compressing data transmission between primary and standby nodes primarily rely on globally unified configuration. This method applies the same compression mode to all standby nodes. The primary node, after generating WAL logs, determines whether to compress the logs based on its configuration before transmitting them to all standby nodes via streaming replication. However, enabling compression requires setting global parameters on the primary node during startup, and any changes to these parameters require a database instance restart to take effect. Therefore, existing methods for compressing data transmission between primary and standby nodes result in wasted performance resources and poor data transmission efficiency.

[0042] Considering the aforementioned problems with existing primary / backup node data transmission compression methods, this application proposes a data transmission compression method where the primary node dynamically decides whether to compress data based on the backup node's needs and its own resource load. This method improves resource utilization and data transmission efficiency.

[0043] The execution entity of this primary / backup node data transmission compression method can be, for example, a transmission compression system. Optionally, the transmission compression system can be any existing electronic device with processing capabilities, such as a terminal or a server. In some embodiments, the transmission compression system can also be deployed in a server cluster or cloud environment. This application does not limit the deployment environment of the transmission compression system.

[0044] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments.

[0045] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0046] Figure 1 This is a flowchart illustrating a primary / backup node data transmission compression method provided in this application. Figure 1 As shown, the method includes:

[0047] S101, Receive a compression flag sent by the backup node, which is used to indicate whether the transmitted data is compressed.

[0048] For example, a backup node can be a slave node in a database cluster that receives data synchronization from the master node and is used to store or back up data.

[0049] For example, the master node can be the core node in the database cluster responsible for the distribution of core business processing.

[0050] In some embodiments, before receiving a compression tag sent by the backup node, the transmission compression system may read compression configuration parameters upon startup of the backup node. These compression configuration parameters include at least one of the following: disabled compression, enabled compression, and adaptive compression.

[0051] For example, compression configuration parameters can be a set of parameters defining a data compression strategy. For example, if the compression flag is 0, compression can be disabled, meaning no compression algorithm is executed during data transmission, and the original data is transmitted directly. If the compression flag is 1, compression can be enabled, meaning a preset compression algorithm is forcibly executed during data transmission, compressing all transmitted data. If the compression flag is 2, adaptive compression can be used, meaning the compression strategy can be dynamically adjusted according to the real-time transmission scenario, such as determining whether to compress based on data type, network bandwidth, node load, etc.

[0052] Optionally, the transmission compression system can read compression configuration parameters from the local configuration file through the input / output interface when the standby node is started, and can also query the database by executing query statements to obtain the standby node name and corresponding compression configuration parameters.

[0053] By using the above method, before receiving the compression tag sent by the backup node, the compression configuration parameters are read when the backup node is started. The compression configuration parameters include at least one of the following: disable compression, enable compression, and adaptive compression, which can reduce transmission latency and improve data transmission efficiency.

[0054] Optionally, the transmission compression system can receive compression tags generated by the backup node according to the compression configuration parameters via the Transmission Control Protocol (TCP). The backup node can encapsulate the compression tags into messages and send them to a message queue. Then, the transmission compression system can also consume messages and extract compression tags in real time by subscribing to the message queue containing compression tags.

[0055] As one possible implementation, after receiving the compression tag sent by the backup node, the transmission compression system can detect the network status of the backup node, which includes at least one of the following: bandwidth, latency, and packet loss rate.

[0056] For example, network status can be the transmission channel status between the backup node and the master node.

[0057] Optionally, after receiving the compression tag, the transmission compression system can send a test data packet of a preset size to the backup node. It calculates the bandwidth by counting the total number of successfully transmitted data packets per unit time, the latency by the difference between the data packet transmission time and the reception response time, and the packet loss rate by the ratio of the number of lost data packets to the total number of transmitted packets. The calculated bandwidth, latency, and packet loss rate values ​​are then written into the system cache.

[0058] By using the above method, after receiving the compression tag sent by the backup node, the network status of the backup node can be detected, which can provide a basis for adjusting the compression strategy and improve the utilization of performance resources. The network status includes at least one of the following: bandwidth, latency, and packet loss rate.

[0059] As one possible implementation, after receiving the compression tag sent by the backup node, the transmission compression system can also maintain the historical transmission behavior record of the backup node, which includes at least one of the following: compression gain ratio and decompression efficiency.

[0060] For example, historical transmission behavior records can be compression and decompression related data recorded by the backup node during transmission.

[0061] For example, the compression gain ratio can be the ratio of the original data volume before compression to the data volume after compression during a single data transmission. For instance, a higher compression gain ratio indicates a more significant compression effect.

[0062] For example, decompression efficiency can be defined as the amount of data successfully decompressed per unit time.

[0063] Optionally, the transmission compression system can deploy a lightweight database locally, creating a data table with fields including the backup node identifier, compression flag, compression gain ratio, decompression efficiency, and a timestamp. After each data transmission and decompression cycle, the transmission compression system can calculate the compression gain ratio and decompression efficiency for that transmission and write them to the data table using Structured Query Language (SQL).

[0064] Optionally, the transmission compression system can also write the latest historical transmission behavior records into a distributed cache in the form of key-value pairs, using the standby node identifier as the key and a combination of compression gain ratio and decompression efficiency as the value. Simultaneously, the transmission compression system can start a scheduled task to batch synchronize the data in the cache to the backend database for persistent storage every preset period (e.g., 1 hour). Through the above method, after receiving the compression tag sent by the standby node, the system maintains the historical transmission behavior records of the standby node. These records include at least one of the following: compression gain ratio and decompression efficiency, avoiding misjudgments in compression decisions, improving the accuracy of the compression strategy, and thus reducing resource consumption.

[0065] S102, parse the resource load status of the master node based on the compression tag.

[0066] As one possible implementation, the transmission compression system can detect the processor and memory usage of the master node. Then, it determines whether the processor and memory usage exceed preset resource load thresholds.

[0067] For example, processor utilization can be the percentage of the master node's current processor usage. Processor utilization can be used to measure the consumption of computing resources.

[0068] For example, memory utilization can be the percentage of memory currently used by the master node. Memory utilization can be used to measure storage resource consumption.

[0069] Optionally, the transmission compression system can call the Application Programming Interface (API) to obtain the cumulative processor runtime and calculate the processor utilization rate based on the percentage of time used. It can also calculate memory utilization based on total memory and free memory data. Furthermore, the transmission compression system can compare the collected processor utilization and memory utilization with preset resource load thresholds in a local configuration file. If the processor utilization exceeds the preset processor threshold, or the memory utilization exceeds the preset memory threshold, the transmission compression system can determine that the resource load has exceeded the limit.

[0070] Optionally, the transmission compression system can deploy distributed monitoring nodes to collect and store the processor and memory usage of the master node in real time, and obtain the average resource usage of the master node over the past 5 minutes based on the data transmission protocol interface. The transmission compression system can then dynamically adjust the resource load threshold based on the aforementioned average resource usage.

[0071] The above methods are used to detect the processor and memory utilization of the master node, providing a basis for subsequent resource decisions. By determining whether the processor and memory utilization exceed preset resource load thresholds, the compression strategy is dynamically adapted to the master node's resource status, thereby improving resource utilization.

[0072] S103, perform data compression processing based on the resource load status and the compression flag.

[0073] As one possible implementation, the transmission compression system can invoke a compression algorithm to compress data if the compression is marked as adaptive and the resource load is not exceeded. If the compression is marked as adaptive and the resource load is exceeded, data compression is not performed.

[0074] For example, a compression algorithm can be an algorithm used to reduce the size of data.

[0075] Optionally, if the compression is marked as adaptive compression and the resource load of the primary node is not exceeded, the transmission compression system can retrieve historical transmission behavior records, select the compression algorithm that yields the highest historical compression gain ratio for the backup node, compress the data, generate a compressed packet, and transmit it to the backup node. If the compression is marked as adaptive compression and the resource load is exceeded, the transmission compression system can skip the compression algorithm call process, encapsulate the data into a transmission message, send it to the backup node via the network link, and simultaneously record log information indicating that compression was skipped due to resource overload.

[0076] Using the above method, when the compression is marked as adaptive compression and the resource load is not exceeded, the compression algorithm is invoked to compress the data. When the compression is marked as adaptive compression and the resource load exceeds the limit, data compression is not performed. By judging the conditions of adaptive compression mode and resource load status, fine-grained control of the compression strategy is achieved, enhancing the flexibility and stability of the compression strategy.

[0077] As one possible implementation, after performing data compression processing based on the resource load status and the compression mark, the transmission compression system can send the compressed data to the backup node. The data includes at least one of the following: compressed data and original data.

[0078] For example, the raw data can be uncompressed data.

[0079] Optionally, the transmission compression system can pre-define compressed data packets and original data packets based on a compression identifier. The compressed data packet may include a compression identifier, compression algorithm type, and original data volume, while the original data packet contains the original data. Based on the compression flag and the resource load status of the master node, after performing data compression, the transmission compression system can send the compressed data packet to the backup node via a TCP link. If compression is disabled based on the compression flag and the resource load status of the master node, the transmission compression system can fast-transmit the original data packet to the backup node via TCP.

[0080] Using the above method, after performing data compression based on the resource load status and the compression mark, the compressed data is sent to the backup node. The data includes at least one of the following: compressed data and original data. This can complete the data synchronization between the primary and backup nodes, improve data transmission efficiency, and avoid resource waste.

[0081] As one possible implementation, after the backup node receives the data, the transmission compression system can perform decompression processing based on the compression mark in the data header.

[0082] Optionally, the transmission compression system can parse the compression flag in the packet header after the backup node receives the data. If the compression flag is 1, the algorithm type field is extracted, the corresponding local decompression algorithm is called to decompress the packet, restore it to the original data, and write it to local storage. If the compression flag is 0, the decompression step is skipped, the original data in the packet is written to storage, and a "successfully received uncompressed data" log is recorded.

[0083] Using the above method, after the standby node receives the data, it performs decompression processing based on the compression mark in the data header to restore the compressed data, thus ensuring the accuracy of data synchronization between the primary and standby nodes.

[0084] In this embodiment, a compression flag is received from the backup node. This flag indicates whether the transmitted data should be compressed, clarifying the backup node's compression requirements and avoiding blind compression or no compression, ensuring that the data transmission strategy is consistent with the backup node. The resource load status of the master node is analyzed based on this compression flag, laying the foundation for subsequent data compression processing. This enables dynamic adaptation of the compression strategy to the master node's resource status, improving resource utilization. Data compression processing is performed based on this resource load status and the compression flag, ensuring data consistency and reliability, and improving resource utilization and data transmission efficiency.

[0085] Figure 2 This application provides a schematic diagram of an architecture for an adaptive data transmission compression system based on master and backup nodes in a cluster. Figure 2 As shown, the system includes:

[0086] Optionally, the cluster may include two regions, such as Region 1 and Region 2. Region 1 may contain one master node and two backup nodes (backup node 1 and backup node 2). The master node is directly connected to backup node 1 and backup node 2 in Region 1. Region 2 may contain one backup node (backup node 3). The master node and backup node 3 are connected across regions.

[0087] Optionally, cross-regional communication between the primary node and backup node 3 requires authentication, backup node 3 sending a request to enable streaming replication, and the primary node also needs to determine whether to enable compressed transmission capability to backup node 3 based on the compression flag sent by backup node 3. For example, if the compression flag value is 0, it is determined that no compression is needed; if the compression flag value is 1, it is determined that compression is needed; and if the compression flag value is 2, compression needs to be automatically calculated.

[0088] Figure 3 This is a flowchart illustrating another method for compressing data transmission between primary and backup nodes provided in this application. Figure 3 As shown, the method includes:

[0089] The method flow is divided into two parts: a backup node and a parent node. For example, the backup node includes a WAL receiving process, and the parent node includes a WAL sending process.

[0090] Optionally, the compression flag can be changed from "compressed" or "uncompressed" to "adaptive compression." Adaptive compression can calculate whether compression is needed based on transmission speed and data pressure.

[0091] Optionally, processor and memory overload thresholds can be added. When the master node or memory usage exceeds the corresponding threshold, data compression can be canceled.

[0092] Optionally, before starting the backup node database that requires compressed transmission, configure the compression flag to adaptive compression. Here, the compression flag value can be 2 to indicate whether the primary node needs to enable compressed transmission capability to the backup node. When this backup node starts, it sends a connection request to the primary node with the compression flag. For example, the request sent by the backup node can include a 1-byte type identifier, a 4-byte timestamp, an 8-byte log sequence number (LSN), a 1-byte compression flag, and an extension field of undefined length. A compression flag value of 0 indicates no compression is needed, a compression flag value of 1 indicates compression is needed, and a compression flag value of 2 indicates automatic calculation of whether compression is needed.

[0093] Optionally, after receiving a request from the standby node, the master node parses the request information, records the compression mark to the master node process connected to the standby node, uses the LSN information sent by the standby node to locate the master node's WAL file and reads the write-ahead log data from it.

[0094] Optionally, the master node can determine whether the compression flag is 2, and use the server processor and memory usage in the data cache. If it does not exceed the set threshold, it calls the configured compression algorithm to compress the data block and then modifies the data header compression flag to 1. If the set threshold value is exceeded, the data will no longer be compressed, the data header compression flag will be set to 0, and the data will be transmitted.

[0095] Optionally, after receiving data, the backup node checks if the compression flag in the data header is 1. If the compression flag is 1, it calls the decompression algorithm to decompress the data. The backup node can also report the transmission status to the master node, and this process continues until the data transmission is complete.

[0096] The above are the method embodiments provided in this application. The apparatus provided in this application will be described below.

[0097] Figure 4 A schematic diagram of a primary / backup node data transmission compression device provided in this application is shown below. Figure 4 As shown, the primary / backup node data transmission compression device 400 provided in this embodiment includes: a receiving module 401, a parsing module 402, and an execution module 403. Wherein,

[0098] The receiving module 401 is used to receive a compression flag sent by the backup node, which is used to indicate whether the transmitted data is compressed;

[0099] Parsing module 402 is used to parse the resource load status of the master node based on the compression mark;

[0100] Execution module 403 is used to perform data compression processing based on the resource load status and the compression mark.

[0101] Optionally, the receiving module 401 is further configured to read compression configuration parameters when the standby node is started before receiving the compression tag sent by the standby node. The compression configuration parameters include at least one of the following: disable compression, enable compression, and adaptive compression.

[0102] Optionally, the parsing module 402 is also used to detect the processor utilization and memory utilization of the master node; and to determine whether the processor utilization and memory utilization exceed a preset resource load threshold.

[0103] Optionally, the execution module 403 is further configured to call a compression algorithm to compress the data when the compression is marked as adaptive compression and the resource load is not exceeded; and not to perform data compression when the compression is marked as adaptive compression and the resource load is exceeded.

[0104] Optionally, the receiving module 401 is further configured to detect the network status of the backup node after receiving the compressed tag sent by the backup node, the network status including at least one of the following: bandwidth, latency, and packet loss rate.

[0105] Optionally, the receiving module 401 is further configured to maintain a record of the historical transmission behavior of the backup node after receiving the compression tag sent by the backup node. The record of historical transmission behavior includes at least one of the following: compression gain ratio and decompression efficiency.

[0106] Optionally, the execution module 403 is further configured to perform data compression processing based on the resource load status and the compression mark, and then send the compressed data to the backup node. The data includes at least one of the following: compressed data and original data.

[0107] Optionally, the execution module 403 is also configured to perform decompression processing based on the compression mark in the data header after the backup node receives the data.

[0108] The primary and backup node data transmission compression device provided in this embodiment can execute the methods provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0109] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 5 As shown, the electronic device 500 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 500 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0110] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0111] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0112] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0113] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0114] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0115] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0116] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0117] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0118] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0119] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0120] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0121] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0122] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0123] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0124] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0125] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for compressing data transmission between primary and backup nodes, characterized in that, The method includes: Receive a compression flag sent by the backup node, the compression flag being used to indicate whether the transmitted data should be compressed; The resource load status of the master node is parsed based on the compression tags; Data compression processing is performed based on the resource load status and the compression flag.

2. The method according to claim 1, characterized in that, Before receiving the compressed tag sent by the backup node, the process includes: When the standby node is started, read the compression configuration parameters, which include at least one of the following: disable compression, enable compression, and adaptive compression.

3. The method according to claim 1, characterized in that, The step of parsing the resource load status of the master node based on the compression tag includes: Detect the processor and memory usage of the master node; Determine whether the processor utilization and memory utilization exceed the preset resource load threshold.

4. The method according to any one of claims 1-3, characterized in that, The data compression process based on the resource load status and the compression flag includes: If the compression flag is set to adaptive compression and the resource load status is not exceeded, the compression algorithm is invoked to compress the data. If the compression flag is set to adaptive compression and the resource load exceeds the limit, data compression will not be performed.

5. The method according to claim 1 or 2, characterized in that, After receiving the compressed tag sent by the backup node, the network status of the backup node is detected. The network status includes at least one of the following: bandwidth, latency, and packet loss rate.

6. The method according to claim 5, characterized in that, After receiving the compressed tag sent by the backup node, the method further includes: Maintain historical transmission behavior records for backup nodes, wherein the historical transmission behavior records include at least one of the following: compression gain ratio and decompression efficiency.

7. The method according to any one of claims 1-6, characterized in that, After performing data compression processing based on the resource load status and the compression flag, the process includes: The compressed data is sent to the backup node, and the data includes at least one of the following: compressed data and original data.

8. The method according to claim 7, characterized in that, The method further includes: After receiving the data, the backup node performs decompression processing based on the compression mark in the data header.

9. A data transmission compression device for primary and backup nodes, characterized in that, include: The receiving module is used to receive a compression flag sent by the backup node, the compression flag being used to indicate whether the transmitted data is compressed; The parsing module is used to parse the resource load status of the master node based on the compression tags; An execution module is used to perform data compression processing based on the resource load status and the compression flag.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images