Data transmission method, device and computer storage medium

Abstract

The present application discloses a data transmission method, device and computer storage medium, and relates to the field of network communication technology. The method is applied to a data transmission proxy module on the sending side, and includes: receiving a data transmission request, determining the service priority corresponding to the data transmission request; obtaining the current utilization rate of the transmission link between the export route and the export route on the receiving side; if the current utilization rate is greater than or equal to a first threshold, when there is an idle TCP connection in the TCP connection pool, and there is a data transmission request with the service priority of the first priority, and there is no ongoing data transmission, the data packet corresponding to the data transmission request with the service priority of the first priority is compressed according to the first compression level; and the compressed data packet is transmitted using the idle TCP connection. The present application effectively improves the data transmission efficiency and reduces the packet loss caused by the competition for bandwidth resources.

Description

Technical Field

[0001] The present application relates to the field of network communication technology, and in particular to a data transmission method, device and computer storage medium. Background Art

[0002] In the data transmission scenario between data centers across regions, there are problems with limited dedicated line bandwidth resources and high costs. With limited dedicated line bandwidth resources, it is impossible to dynamically adjust according to real-time bandwidth conditions and business needs. When multiple business systems synchronize a large amount of data at the same time during peak hours, it is easy to cause bandwidth resources to be quickly occupied, which in turn causes router packet loss and ultimately causes data synchronization failure. Summary of the invention

[0003] The main purpose of this application is to provide a data transmission method, device and computer storage medium, aiming to solve the technical problem of how to achieve efficient and stable data transmission under limited dedicated line bandwidth resources.

[0004] To achieve the above object, an embodiment of the present application provides a data transmission method, which is applied to a data transmission agent module on the sending side, and the data transmission agent module is deployed between the service system and the egress router. The data transmission method includes: receiving a data transmission request, and determining a service priority corresponding to the data transmission request; Acquire a current usage rate of a transmission link between the egress route and a receiving-side egress route; If the current usage rate is greater than or equal to the first threshold, when there is an idle TCP connection in the TCP connection pool, and there is a data transmission request with the service priority of the first priority, and there is no ongoing data transmission, compressing the data packet corresponding to the data transmission request with the service priority of the first priority according to the first compression level; The idle TCP connection is used to perform data transmission on the compressed data packet.

[0005] In one embodiment, before the step of receiving a data transmission request and determining a service priority corresponding to the data transmission request, the following steps are included: Pre-configure business priority rules according to different business types and / or business identifiers; When receiving a data transmission request, identifying a service type and / or a service identifier in the data transmission request, and retrieving the service priority rule based on the service type and / or the service identifier; According to the search result, the service priority corresponding to the data transmission request is determined.

[0006] In one embodiment, after the step of determining the service priority corresponding to the data transmission request according to the search result, the method further includes: Allocate a cache queue for each different service priority; When the current usage rate of the transmission link between the egress route and the receiving-side egress route is greater than or equal to the first threshold, determining whether there is ongoing data transmission on the transmission link; If there is ongoing data transmission, the data packet corresponding to the data transmission request is compressed at a first compression level according to the service priority corresponding to the data transmission request, and the data transmission request is placed in the corresponding cache queue.

[0007] In one embodiment, before the step of receiving a data transmission request and determining a service priority corresponding to the data transmission request, the following steps are included: Establishing a TCP connection pool between the data transmission proxy module on the sending side and the data transmission proxy module on the receiving side, wherein the TCP connection pool includes at least one pre-established TCP connection; The connection status of the TCP connection is monitored, and for the invalid or abnormal TCP connection, an operation of reestablishing or removing the connection from the TCP connection pool is performed.

[0008] In one embodiment, after the step of obtaining the current usage rate of the transmission link between the egress route and the receiving side egress route, the following steps are included: When the current usage rate is less than the first threshold, compressing the data packet corresponding to the data transmission request according to a preset compression level; The idle TCP connection in the TCP connection pool is used to perform data transmission on the compressed data packet.

[0009] In one embodiment, after the step of using the idle TCP connection in the TCP connection pool to perform data transmission on the compressed data packet, the following steps are included: When the current usage rate is less than a second threshold, if the idle TCP connection does not exist in the TCP connection pool, a new TCP connection is established for transmitting the compressed data packet; After the compressed data packet transmission is completed, the new TCP connection is closed.

[0010] In one embodiment, after the step of using the idle TCP connection in the TCP connection pool to perform data transmission on the compressed data packet, the method further includes: When the current usage rate is greater than or equal to the second threshold and less than the first threshold, if the TCP connection pool does not have the idle TCP connection, the compressed data packet is placed in the corresponding cache queue according to the service priority corresponding to the data transmission request; When the idle TCP connection appears in the TCP connection pool, the compressed data packet is taken out from the cache queue to perform data transmission.

[0011] The embodiment of the present application further provides a data transmission method, which is applied to a data transmission agent module on a receiving side. The data transmission method includes: Receiving the compressed data packet transmitted by the data transmission agent module on the sending side through the pre-established TCP connection; Calculating a hash value for the compressed data packet, and comparing the hash value with the original compressed hash value provided by the data transmission agent module on the sending side; If the comparison results are inconsistent, informing the data transmission agent module on the sending side to resend the compressed data packet; If the comparison results are consistent, the compressed data packet is decompressed to obtain the decompressed original data, and a hash value of the original data is calculated; Compare the hash value of the original data with the original hash value provided by the data transmission agent module on the sending side, and if the comparison results are consistent, repackage the original data and distribute it to the target business system; If the distribution to the target service system fails, the repackaged original data is placed in a cache queue corresponding to the service priority according to the service priority of the data transmission request corresponding to the compressed data packet; The state of the target business system is detected, and when it is confirmed that the target business system is in a live state, the repackaged original data is taken out from the cache queue and distributed to the target business system.

[0012] An embodiment of the present application also provides a data transmission device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data transmission method described above.

[0013] An embodiment of the present application further provides a computer storage medium, which is a computer-readable storage medium. A computer program is stored on the computer storage medium. When the computer program is executed by a processor, the steps of the data transmission method described above are implemented.

[0014] The embodiment of the present application discloses a data transmission method, which is applied to a data transmission proxy module on the sending side, and the data transmission proxy module is deployed between the business system and the export route. The data transmission method includes: receiving a data transmission request, determining the business priority corresponding to the data transmission request; obtaining the current usage rate of the transmission link between the export route and the receiving side export route; if the current usage rate is greater than or equal to a first threshold, when there is an idle TCP connection in the TCP connection pool, and there is a data transmission request with the business priority of the first priority, and there is no ongoing data transmission, the data packet corresponding to the data transmission request with the business priority of the first priority is compressed according to the first compression level; and the compressed data packet is performed using the idle TCP connection. The present application effectively improves the data transmission efficiency and stability in a low-bandwidth environment by deploying a data transmission proxy module before the data center, and dynamically manages the TCP connection pool using business priority and real-time load status, reduces the packet loss caused by bandwidth resource competition, and ensures the priority transmission of critical business data in limited bandwidth resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of a first embodiment of a data transmission method according to an embodiment of the present application; Figure 2 This is a schematic diagram of the data transmission architecture between different data centers in the first embodiment of the data transmission method involved in the embodiment of the present application; Figure 3 This is a schematic diagram of a data transmission architecture between different data centers after adding a data transmission proxy module to the first embodiment of the data transmission method involved in the embodiment of the present application; Figure 4 This is a schematic diagram of the structure of the transmission protocol of the first embodiment of the data transmission method involved in the embodiment of the present application; Figure 5 This is a flow chart of a second embodiment of a data transmission method according to an embodiment of the present application; Figure 6 This is a flow chart of a third embodiment of a data transmission method according to an embodiment of the present application; Figure 7 This is a flow chart of a fourth embodiment of a data transmission method according to an embodiment of the present application; Figure 8 This is a flowchart of a fifth embodiment of a data transmission method according to an embodiment of the present application; Fig. 9 This is a schematic diagram of the structure of the data transmission equipment of this application.

[0016] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0017] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] Data centers are large, centralized server warehouses that store, process, and distribute large amounts of data. In most data center architectures, in order to ensure real-time data synchronization and business continuity, data transmission is often required between data centers located in different geographical locations. Communication between data centers is usually achieved through a dedicated network connection, which is called a dedicated line. Dedicated lines provide a secure and stable way to transmit data, but dedicated line bandwidth resources are usually limited and expensive. Due to the small bandwidth of dedicated lines, it is impossible to dynamically adjust according to real-time bandwidth conditions and business needs. Therefore, during certain periods, such as the evening rush hour, when multiple business systems may try to synchronize a large amount of data at the same time, it is easy to cause the dedicated line bandwidth to be quickly occupied, causing router packet loss, and ultimately causing data synchronization failure.

[0019] In order to solve the above-mentioned defects existing in the related art, the embodiment of the present application proposes a data transmission method, which is applied to the data transmission agent module on the sending side, and the data transmission agent module is deployed between the business system and the export route. The data transmission method includes: receiving a data transmission request, determining the business priority corresponding to the data transmission request; obtaining the current utilization rate of the transmission link between the export route and the export route on the receiving side; if the current utilization rate is greater than or equal to the first threshold, when there is an idle TCP connection in the TCP connection pool, and there is a data transmission request with the business priority of the first priority, and there is no ongoing data transmission, the data packet corresponding to the data transmission request with the business priority of the first priority is compressed according to the first compression level; and the idle TCP connection is used to perform data transmission on the compressed data packet. The present application effectively improves the data transmission efficiency and stability in a low-bandwidth environment, reduces the packet loss caused by bandwidth resource competition, and ensures the priority transmission of key business data in limited bandwidth resources by deploying a data transmission agent module between the business system and the export route, and dynamically manages the TCP connection pool using business priority and dedicated line utilization rate.

[0020] It should be noted that the execution subject of this embodiment may be a transmission agent module, or a computing service device with data processing, network communication and program running functions, or a data transmission device capable of realizing the above functions, etc. The transmission agent module is taken as an example to illustrate this embodiment and the following embodiments.

[0021] The data transmission method of the first embodiment proposed in this application, please refer to Figure 1 The method is applied to a data transmission agent module on the sending side. The data transmission agent module is deployed between the business system and the export router to manage data transmission across data centers. The method includes steps S10 to S40: Step S10: receiving a data transmission request, and determining a service priority corresponding to the data transmission request.

[0022] In the communication between data centers, the data transmission agent module plays the role of intelligent scheduling. The data transmission agent module is responsible for managing the flow of data from one data center to another, ensuring that data can be transmitted efficiently, securely, and reliably. The data transmission agent module is usually deployed on the network boundary of the data center, or as part of the internal network of the data center. The data transmission agent module can be a physical device or a software program.

[0023] Please refer to Figure 2 , Figure 2 The following is a diagram of the current data transmission architecture between different data centers. In the current dedicated line scenario, multiple business systems (business system 1, business system 2, etc.) communicate with the external network through egress routes. Each data center is equipped with one or more egress routes, which are responsible for forwarding data packets from the internal network to the external network. This data transmission architecture allows data exchange between multiple business systems and between business systems and external networks.

[0024] It should be noted that the egress router is the connection point between the data center network and the external network (such as the Internet or another data center). The egress router can receive data packets from the internal network and determine the best path based on the destination address of the data packet to forward the data packet to the external network.

[0025] In this embodiment, the data transmission agent module is deployed between the business system and the export router, and the data transmission agent module is deployed on both the sending side and the receiving side. Figure 3 , Figure 3 Schematic diagram of data transmission architecture between different data centers after adding the data transmission proxy module ( Figure 3 The transport agent module is referred to as the "transport agent" in the text file.

[0026] The data transmission agent module on the sending side (hereinafter referred to as the "data transmission agent module") can receive data transmission requests from various business systems by establishing a connection with the business system, and determine the business priority of the data transmission request based on the specific nature of the data transmission request and the specific needs of the business. This means that the data transmission agent module needs to be able to identify and understand the importance and urgency of the business data corresponding to different data transmission requests in order to reasonably allocate transmission resources.

[0027] The communication connection between the business system and the data transmission agent module is established using the http (HyperText Transfer Protocol) protocol.

[0028] Optionally, to pass business-related metadata, insert the following custom fields into the HTTP protocol header: X-App-Id: 123 X-App-Priority: 1 X-Data-Sync: 1 In the above fields, X-App-Id is a business identifier jointly defined by the business party and the operation and maintenance party, and is used to uniquely identify the business corresponding to the data transmission request. X-App-Priority is also jointly defined by the business party and the operation and maintenance party, and comes from the business system. It is used to indicate the priority of the business and indicates the importance that the data transmission agent module should give when processing data transmission requests. The X-Data-Sync identifier is used to indicate whether data transmission is currently in progress.

[0029] When the X-Data-Sync field is 0, the data packet corresponding to the data transmission request is directly transmitted or cached. When the X-Data-Sync field is 1, it means that data transmission is required. At this time, the body part of the http of the data transmission request is extracted, and the data is compressed according to the pre-configured compression rules before data transmission or caching.

[0030] It should be noted that the body part of HTTP refers to the actual data part contained in the data transmission request, that is, the data that the business system needs to synchronize or transmit.

[0031] By inserting custom fields into the header of the http protocol, the data transfer agent module can quickly identify the business priority and synchronization status of different business data, so as to better process and schedule data transfer requests.

[0032] The data transmission agent module continuously monitors and listens to connection requests from business systems inside or outside the data center. In this way, the data transmission agent module can intervene at the initial stage of data transmission, and then effectively manage and schedule the upcoming data transmission. When a business system needs to transmit or synchronize business data to another data center, the business system first initiates a data transmission request to the data transmission agent module, and the data transmission request includes information such as data volume, data type, source data center, and target data center.

[0033] Step S20: obtaining the current usage rate of the transmission link between the egress route and the receiving-side egress route.

[0034] In the network environment of the data center, the transmission link between the egress route and the receiving-side egress route is the dedicated line between the two data centers. The amount of data passing through the dedicated line within a certain period of time can be understood as the data transmission volume of the dedicated line within that time range. This amount of data includes all data packets passing through the dedicated line, whether it is a data transmission request from a business system or other types of network communication. The current utilization rate of the transmission link refers to the proportional relationship between the data transmission volume currently undertaken by the dedicated line and its maximum transmission capacity (i.e., bandwidth).

[0035] It should be noted that the current utilization rate of the transmission link is a key indicator for measuring the utilization of the dedicated line bandwidth, and can indicate the current load status of the dedicated line. Specifically, the load status of the dedicated line directly reflects the busyness of the dedicated line and the remaining available bandwidth. The load status can be expressed as a percentage. For example, if the total bandwidth of the dedicated line is 100Mbps, and the actual usage flow of the dedicated line at a certain moment is 50Mbps, then the current load status of the dedicated line can be considered to be 50%.

[0036] By comparing the ratio of data transmission volume at a certain moment to the maximum transmission capacity of the dedicated line, it is possible to determine whether the dedicated line is in a high load, medium load or low load state.

[0037] Optionally, a network traffic monitoring tool is installed on the network device where the data transmission proxy module is located to monitor the network traffic. The network monitoring tool can collect the traffic information of each data packet flowing through the dedicated line, including the number of data packets, data transmission size, data transmission rate, and data transmission time. Based on the data packet traffic information, the utilization rate of the dedicated line at the current time point is calculated. Then the utilization rate is compared with the preset load threshold to determine the load status of the current dedicated line.

[0038] For example, when the usage rate of the dedicated line is greater than or equal to 90%, the dedicated line is under high load; when the usage rate of the dedicated line is greater than or equal to 70% and less than 90%, the dedicated line is under medium load; when the usage rate of the dedicated line is less than 70%, the dedicated line is under low load.

[0039] Optionally, the built-in monitoring function of the network device where the dedicated line is located can be used to monitor network traffic. Current network devices usually have the ability to monitor and count traffic, and can directly provide real-time bandwidth usage. The data transmission agent module can regularly query the traffic statistics of the network device through SNMP (Simple Network Management Protocol) or other management interfaces to obtain the current load status of the dedicated line. The advantage of this method is that it does not require additional monitoring tools, and can directly use the functions of existing network devices to obtain data, effectively reducing potential monitoring delays.

[0040] Optionally, after step S20, after receiving the data transmission request, the data transmission proxy module determines the service priority of the data transmission request and the value of the X-Data-Sync field by parsing the header information of the http protocol corresponding to the data transmission request. When the X-Data-Sync field is 1, it indicates that data transmission is required. At this time, the body part of the http of the data transmission request is extracted, and the business data is compressed according to the pre-configured compression rules before data transmission or caching is performed.

[0041] The purpose of compression processing is to reduce the size of business data in data transmission requests, thereby reducing the demand for dedicated line bandwidth and improving data transmission efficiency.

[0042] During the data compression process, the data transmission agent module will select the appropriate compression algorithm based on the type and characteristics of the data. For example, for text data, compression algorithms such as gzip or deflate can be used; for binary data, different compression strategies may be used. The compression level will also be dynamically adjusted according to the load status of the current dedicated line, such as using a higher compression level to reduce the size of the data when the network is congested. After the compression operation is completed, the data transmission agent module will encode the compressed data according to the transmission protocol, and then select a TCP connection for data transmission.

[0043] This embodiment takes the gzip compression algorithm as an example. The compression level is 1-9 according to the gzip algorithm standard. The default compression level is level 6 and the highest compression level is level 9. Service priorities are divided into three levels: high, medium and low. The load status is set according to the following standards: if the dedicated line usage rate is greater than or equal to 90%, the dedicated line is highly loaded; if the dedicated line usage rate is greater than or equal to 70% and less than 90%, the dedicated line is medium loaded; if the dedicated line usage rate is less than 70%, the dedicated line is lightly loaded.

[0044] Optionally, in order to ensure orderly and efficient data transmission, based on data integrity, compression efficiency and service priority considerations, this embodiment defines a structured transmission protocol that ensures that critical services can be transmitted preferentially and securely even when bandwidth is limited.

[0045] In this embodiment, after receiving a data transmission request and compressing the business data corresponding to the data transmission request (hereinafter referred to as "data"), the data transmission agent module will encapsulate the data in a predefined format (i.e., according to the transmission protocol encoding). The format consists of multiple fields, each of which has its specific function and length, and is used to describe the characteristics of the data and the details of the transmission. Please refer to Figure 4 , Figure 4 It is a schematic diagram of the structure of the transmission protocol. Specifically: Start is the protocol header field, 1 Byte in length, marking the beginning of the protocol, such as ASCII code 0x28.

[0046] pkg head length is the header length, which is 2 bytes. The header includes fields from pkg head length to reserved.

[0047] pkg body length is the data length, which is 4 bytes. The data only includes the compressed data field, that is, the length of the compressed data.

[0048] version is the version number of the transmission protocol, 1 Byte in length.

[0049] App ID is a predefined global service ID, 1 Byte in length, used to uniquely identify the service.

[0050] priority is the service priority, with a length of 1 Byte.

[0051] protocol is the protocol used for business transmission data, with a length of 1 Byte. This application uses the http protocol.

[0052] compression algorithm is the compression algorithm, length 1 Byte.

[0053] compression level is the compression level, length 1 Byte.

[0054] inital data md5 is the MD5 value of the original HTTP body data, with a length of 16 bytes.

[0055] compressed data md5 is the MD5 value of the compressed HTTP body data, with a length of 16 bytes.

[0056] reseved is a reserved field with a length of 64 bytes and is used for extension.

[0057] Compressed data is the compressed HTTP body data.

[0058] end is the protocol tail field, marking the end of the protocol, with a length of 1 Byte, such as the ASCII code 0x29.

[0059] It should be noted that the two check fields, inital data md5 and compressed data md5, in the transmission protocol correspond to the MD5 values ​​of the original data and compressed data of the data packet in the data transmission request, respectively. These two check fields can help the data transmission agent module on the receiving side to verify whether the data remains intact during the transmission process, thereby achieving the purpose of ensuring data integrity. The HTTP body (also known as the HTTP message body or subject) is a part of the HTTP request (i.e., the data transmission request). The HTTP body contains the actual business data of the HTTP request, such as web page content, pictures, videos, or any other type of data.

[0060] Step S30: If the current usage rate is greater than or equal to the first threshold, when there is an idle TCP connection in the TCP connection pool, and there is a data transmission request with the business priority of the first priority, and there is no ongoing data transmission, the data packet corresponding to the data transmission request with the business priority of the first priority is compressed according to the first compression level.

[0061] In this embodiment, after completing the determination of the service priority and monitoring the load status of the dedicated line (ie, the current utilization rate of the transmission link), the next step is to execute the key link of data transmission according to the service priority and the load status of the dedicated line.

[0062] The data transmission agent module pre-establishes and maintains a TCP connection pool, which contains multiple established and active TCP connections that are ready to be used for data transmission at any time.

[0063] TCP (Transmission Control Protocol) is a connection-oriented, reliable, byte-stream-based transport layer communication protocol. In a TCP connection, the data sender and receiver must first establish a logical connection before data can be transmitted. This connection is established through a three-way handshake process, ensuring that the communication link between the two parties is reliable and orderly.

[0064] The TCP connection pool is a technology used to manage multiple TCP connections. It provides a container for storing established TCP connections so that these TCP connections can be reused efficiently. The TCP connections in the TCP connection pool are usually long connections, that is, TCP connections that have been established through a three-way handshake and can be used for data transmission at any time when needed.

[0065] When the data transmission agent module on the sending side and the data transmission agent module on the receiving side need to transmit data frequently, by reusing the established TCP connection, the overhead of three-way handshake and four-way handshake is avoided every time data is transmitted, thereby improving the efficiency of data transmission. In addition, the TCP connection pool can manage the life cycle of TCP connections, including the establishment, use and release of TCP connections. Through reasonable connection management, problems such as connection leakage and connection exhaustion can be avoided, thereby enhancing the stability of the system.

[0066] When the data transmission agent module receives a data transmission request and determines its service priority, the data transmission agent module will compress the data corresponding to the data transmission request according to the load status of the current dedicated line, and use it to select the most appropriate TCP connection for data transmission.

[0067] Optionally, if the current utilization rate is greater than or equal to a first threshold, when there is an idle TCP connection in the TCP connection pool, and there is a data transmission request with a business priority of the first priority, and there is no ongoing data transmission, the data packet corresponding to the data transmission request with a business priority of the first priority (indicating the highest business priority) is compressed according to the first compression level (indicating the highest compression level, corresponding to level 9 in this embodiment).

[0068] It should be noted that when the current usage rate of the transmission link (i.e., dedicated line) between the egress route and the receiving side egress route reaches or exceeds the threshold indicating the highest load state (in this embodiment, the first threshold corresponds to the critical point of the high load state), it indicates that the current network bandwidth usage is close to saturation. In this case, in order to more efficiently utilize limited bandwidth resources and ensure that high-priority data can be transmitted first, the data transmission agent module will take specific optimization measures for these high-priority data transmission requests.

[0069] It should be noted that no ongoing data transmission means that when the load status of the dedicated line is high, the data transmission proxy module will ensure that there is only one TCP connection in the TCP connection pool for data transmission at the same time. This design is intended to prevent multiple TCP connections from competing for limited bandwidth at the same time, which may cause network congestion or reduced data transmission efficiency. By limiting the number of simultaneous data transmission connections, more bandwidth resources can be provided for a single data transmission request, thereby improving its transmission rate and reliability.

[0070] Specifically, the data transmission proxy module first checks the status of each TCP connection in the TCP connection pool to determine whether there is an idle TCP connection, that is, there is no TCP connection in which data transmission is in progress. If there is an idle TCP connection, and there is a data transmission request with a high business priority mark waiting to be processed. Then for the data packets corresponding to these high business priority data transmission requests, the data transmission proxy module will use the highest level of compression algorithm to compress the data packets, that is, the first compression level, that is, the gzip algorithm with a level of 9, to minimize the size of the data packets.

[0071] In this embodiment, when the data in the data transmission request is compressed according to the first compression level, the data transmission agent module calculates the MD5 value of the original data, the MD5 value of the compressed data, etc., and then encapsulates the compressed data into a format suitable for data transmission according to the predefined transmission protocol. This includes adding necessary metadata to the data packet, such as the compression algorithm type, the compression level, and possible other service-related information.

[0072] It should be noted that in order to ensure the integrity and consistency of the data packet, the calculated MD5 value of the data before and after compression, the compression algorithm used, the compression level and other data will be attached to the data packet during the encapsulation process so that it can be verified after the data packet reaches the data transmission agent module on the receiving side.

[0073] For example, assuming that the current usage rate of the dedicated line at a certain moment reaches 92%, according to the preset load threshold standard, the dedicated line is in a high load state. Under this network condition, the data transmission agent module will check the status of the TCP connection pool to confirm whether there is an idle TCP connection. If there is an idle TCP connection and no other TCP connection is currently transmitting data, then the idle TCP connection will be used to process data transmission requests with high business priority.

[0074] It should be noted that if the current utilization rate is greater than or equal to the first threshold, there is no idle TCP connection in the TCP connection pool, or there is no data transmission request with the first business priority, or there is ongoing data transmission, then all current data transmission requests are compressed according to the first compression level and encoded according to the transmission protocol, and then the data corresponding to the data transmission request is placed in the cache queue, waiting for the dedicated line resources to be released, and then the data transmission strategy is executed according to the load status of the dedicated line.

[0075] Step S40: using the idle TCP connection to perform data transmission on the compressed data packet.

[0076] After the data packet in the data transmission request of the high service priority is compressed, the compressed data packet is encapsulated according to the predefined transmission protocol, the encapsulated data packet is bound to the selected idle TCP connection, and data transmission begins.

[0077] During the data transmission process, the data transmission agent module will continuously monitor the data transmission status, including the load status of the dedicated line. If it is found that the load status of the dedicated line decreases during the data transmission process, the data transmission agent module will adjust the transmission strategy in time.

[0078] Optionally, according to the real-time bandwidth margin, a flow control algorithm (such as a token bucket algorithm) is used to dynamically adjust the sending rate to ensure that the sending rate does not exceed a specific proportion of the real-time bandwidth margin.

[0079] Specifically, the sending rate can be controlled within 20% of the real-time bandwidth margin, or the ratio can be adjusted according to actual conditions. For example, the sending rate can be controlled between 10% and 30% of the real-time bandwidth margin to avoid excessive occupation of bandwidth resources while maintaining data transmission efficiency.

[0080] It should be noted that the "token bucket algorithm" is a common flow control algorithm. It controls the data sending rate through a virtual token bucket, allowing network traffic to be sent out at a relatively uniform speed. At the same time, tokens can be accumulated in the token bucket to support burst data transmission without exceeding the preset bandwidth limit.

[0081] The basic principle of the token bucket algorithm is: add tokens to the token bucket at a fixed rate, and each token represents a certain amount of data. When data needs to be sent, the corresponding number of tokens must be taken out of the token bucket. If there are not enough tokens in the token bucket, data transmission will be temporarily blocked until enough tokens are available. By adjusting the token addition rate and the size of the bucket, precise control of the sending rate can be achieved. This algorithm helps smooth data traffic when the network is congested and prevents network overload.

[0082] Optionally, if a data transmission request encounters a problem during transmission, such as connection interruption or transmission delay, the data transmission agent module will reselect an idle TCP connection to perform data transmission on the data transmission request to ensure data integrity and transmission reliability.

[0083] Through the above data transmission strategy, the data transmission agent module can effectively utilize limited network resources to ensure that key business data can be given priority in low-bandwidth dedicated line scenarios, while also optimizing the performance and efficiency of data transmission as a whole.

[0084] Please refer to Figure 5 The data transmission method of the second embodiment proposed in the present application includes steps S11 to S13 before step S10: Step S11: pre-configure service priority rules according to different service types and / or service identifiers.

[0085] In this embodiment, the data transmission agent module pre-configures a service priority rule, and determines the service priority corresponding to the data transmission request according to the service priority rule.

[0086] Specifically, according to business requirements or network policies, business priorities are set for business data of different business types and / or business identifiers. Business priority rules can be static configuration files or dynamically adjusted algorithms, which define the correspondence between different business types and / or business identifiers and business priorities. For example, business priorities are divided into three types: high, medium, and low. Transaction data may be marked as high priority, while email communications may be assigned low priority.

[0087] Step S12: When a data transmission request is received, the service type and / or service identifier in the data transmission request is identified, and the service priority rule is retrieved based on the service type and / or service identifier.

[0088] In this embodiment, when the data transmission agent module receives a data transmission request, it first parses the data transmission request and extracts the service type and / or service identification information corresponding to the data transmission request. This information can be part of the data transmission request or metadata associated with the data transmission request. Based on this information, the data transmission agent module retrieves the service priority corresponding to the service type and / or service identification in the pre-configured service priority rules.

[0089] Step S13: Determine the service priority corresponding to the data transmission request according to the search result.

[0090] In this embodiment, the determination of the service priority of the data transmission request directly affects the scheduling and execution of the data transmission. The data transmission agent module will assign the service priority to each data transmission request according to the search result. When data transmission is performed, data transmission can be performed according to the preset data transmission strategy based on the service priority of each data transmission request.

[0091] Optionally, after step S13, steps S131 to S133 may also be included: Step S131: Allocate a cache queue to each different service priority.

[0092] In this embodiment, in order to better manage data transmission requests of different business priorities, the data transmission proxy module allocates an independent cache queue for each different business priority and establishes corresponding local file mappings. These cache queues and local file mappings act as temporary storage space for managing data transmission requests to be transmitted according to business priorities. For example, if the business priorities are high, medium, and low, three different cache queues are set up respectively, and three local file mappings are established in the data transmission proxy module to store data that has been compressed and encoded according to the transmission protocol, and the data transmission proxy module on the receiving side also establishes three local file mappings to store data that has been decompressed and re-encoded and encapsulated according to the transmission protocol.

[0093] Specifically, when a data transmission request arrives at the data transmission agent module, if there is no idle TCP connection in the current TCP connection pool, the corresponding data transmission request can be placed in the corresponding cache queue according to the business priority of the data transmission request, and the data encoded according to the transmission protocol can be temporarily stored in the local file system through local file mapping, waiting for scheduling by the data transmission agent module.

[0094] The data transfer agent module creates a cache queue instance in memory for each business priority and creates a corresponding file mapping instance in the local file system. The management of these cache queues can be dynamic to adapt to changes in network load and fluctuations in data transfer requests.

[0095] It should be noted that the change of network load refers to the change of network data traffic over time. For example, the network load may increase during business peak hours, while the network load may decrease at night or during non-peak hours. The fluctuation of data transmission requests refers to the fluctuation of the number and frequency of data transmission requests due to various reasons, such as user activities, system updates or emergencies.

[0096] In summary, dynamic management ensures that when data transmission requests increase, the data transmission agent module can effectively handle these requests, including creating idle TCP connections to increase data transmission channels and thus improve transmission efficiency; caching data through the local file mapping mechanism; when data transmission requests decrease, dynamic management can also ensure that resources are used reasonably, such as reducing resource consumption by reducing file mapping instances, closing unnecessary idle TCP connections, reducing the resources required to maintain connections, or adjusting transmission strategies, such as reducing the frequency of data packet sending to reduce network bandwidth usage, etc. Such a mechanism improves the overall efficiency and stability of data transmission.

[0097] Step S132: when the current usage rate of the transmission link between the egress route and the receiving-side egress route is greater than or equal to the first threshold, determine whether there is any ongoing data transmission on the transmission link.

[0098] Step S133: If there is ongoing data transmission, then according to the service priority corresponding to the data transmission request, the data packet corresponding to the data transmission request is compressed at a first compression level, and the data transmission request is placed in the corresponding cache queue.

[0099] It should be noted that when the current usage rate of the transmission link between the egress routing and the receiving side egress routing is greater than or equal to the first threshold, before placing the data transmission request into the corresponding cache queue, it is necessary to first compress the data corresponding to the data transmission request according to the first compression level to reduce the amount of data, thereby reducing the burden on the transmission link.

[0100] In this embodiment, when the current utilization rate of the transmission link between the egress route and the receiving side egress route is greater than or equal to the first threshold, it is first determined whether there is ongoing data transmission on the current transmission link. If there is ongoing data transmission, for all currently received data transmission requests, the data transmission agent module will first compress the data corresponding to these data transmission requests, and then put the compressed data into the corresponding cache queue according to the business priority of these data transmission requests, so as to better handle these requests when the utilization rate of the transmission link decreases. This process ensures that when the link is highly loaded, only one TCP connection is transmitting data, thereby avoiding transmission delays caused by link congestion.

[0101] When the transmission link is congested and TCP connection resources are insufficient, the data transmission agent module compresses the data and sorts it into the cache queue according to service priority. This can maximize the needs of different priority services without sacrificing link performance, thereby achieving optimal allocation and effective management of resources.

[0102] It should be noted that, in this embodiment, the first threshold is defined as a critical point of a high load state.

[0103] Exemplarily, this embodiment divides the load status of the dedicated line into three levels: high load, medium load and low load. When the current usage rate of the transmission link (i.e., the dedicated line) between the egress route and the receiving side egress route is less than 70%, the dedicated line is determined to be in a low load state. When the current usage rate of the transmission link (i.e., the dedicated line) between the egress route and the receiving side egress route is greater than or equal to 90%, the dedicated line is determined to be in a high load state. When the current usage rate of the transmission link (i.e., the dedicated line) between the egress route and the receiving side egress route is greater than or equal to 70% and less than 90%, the dedicated line is determined to be in a medium load state.

[0104] If the dedicated line is currently in a high-load state, the data transmission agent module will determine whether there is a connection currently performing data transmission on the dedicated line. If there is a connection currently performing data transmission, the data transmission agent module will compress the data corresponding to all received data transmission requests and put the compressed data into the cache queue according to the service priority.

[0105] If there is no ongoing data transmission and there is an idle TCP connection, when a data transmission request receives a high-priority data transmission request, the data is compressed according to the first compression level and transmitted directly, or the compressed data is taken out from the high-priority cache queue for transmission, ensuring that the transmission of critical business data will not be delayed.

[0106] It should be noted that the above scheme is only an optional implementation method in this embodiment. When the current usage rate of the transmission link between the egress route and the receiving side egress route is greater than or equal to the first threshold, the data transmission agent module may also preferentially determine whether the currently received data transmission request contains a high-priority data transmission request. Then further confirm whether there is ongoing data transmission on the current transmission link. This embodiment does not impose any restrictions on the order of these inspection or confirmation steps.

[0107] Please refer to Figure 6 The data transmission method of the third embodiment proposed in the present application includes steps S301 to S302 before step S30: Step S301: A TCP connection pool is established between a data transmission agent module on a sending side and a data transmission agent module on a receiving side, wherein the TCP connection pool includes at least one pre-established TCP connection.

[0108] It should be noted that the main purpose of TCP connection is to provide a stable data transmission channel between data centers. Pre-establishing and maintaining a TCP connection pool can reduce the time required to establish a connection between the data transmission agent module and the target data center, thereby improving the efficiency of data transmission.

[0109] Specifically, the implementation method of establishing a TCP connection pool is as follows: During the initialization phase, the data transmission agent module pre-establishes a certain number of TCP connections based on the network configuration of the data center and the expected data transmission requirements.

[0110] A TCP connection can be a persistent connection, which means that it remains connected after being established until a connection exception occurs or it is explicitly closed.

[0111] The size of the TCP connection pool and the number of connections can be adjusted according to the actual needs of the data center and the network resources to achieve optimal data transmission performance.

[0112] Step S302: monitor the connection status of the TCP connection, and for the invalid or abnormal TCP connection, perform an operation of re-establishing or removing it from the TCP connection pool.

[0113] In this embodiment, the connection status of the TCP connection is monitored to ensure that all TCP connections are active and reliable so that they can be used for data transmission immediately when needed. Specifically, the implementation method of monitoring the connection status of the TCP connection is as follows: The data transfer agent module periodically sends heartbeat packets or uses other mechanisms, such as SNMP (Simple Network Management Protocol), Syslog (system log or system record, a standard for transmitting log file messages on the Internet Protocol (TCP / IP) network), etc. to detect the activity of the TCP connection.

[0114] If a TCP connection is detected to be invalid or abnormal, such as connection timeout, network interruption or error response, the data transmission agent module will try to re-establish the connection to restore the data transmission channel.

[0115] If re-establishing the connection fails or the connection is abnormal and cannot be quickly restored, the data transmission agent module will remove the TCP connection from the TCP connection pool to avoid affecting the overall efficiency of data transmission.

[0116] At the same time, the data transmission agent module can also create new TCP connections according to business needs to supplement the TCP connection pool, ensuring that there are sufficient connection resources to meet data transmission needs.

[0117] It should be noted that the heartbeat packet is a mechanism used in computer networks to periodically send data packets between the two ends of a network connection to confirm the survival status of the connection.

[0118] Please refer to Figure 7 The data transmission method of the fourth embodiment proposed in the present application further includes steps S210 to S220 after step S20: Step S210: When the current usage rate is less than the first threshold, compressing the data packet corresponding to the data transmission request according to a preset compression level.

[0119] Step S220: Utilize the idle TCP connection in the TCP connection pool to perform data transmission on the compressed data packet.

[0120] It should be noted that the preset compression level is the default compression level, and in this embodiment, level 6 is used.

[0121] In this embodiment, when the current usage rate of the transmission link between the egress routing and the receiving side egress routing is less than the first threshold (high load threshold), that is, the dedicated line is in a low load and medium load state, the data packet corresponding to the data transmission request is compressed according to the default compression level.

[0122] After compression is completed, the data transmission agent module will encapsulate the compressed data packet into a format suitable for data transmission according to the predefined transmission protocol. This includes adding necessary metadata to the data packet, such as the compression algorithm type, compression level, MD5 checksum of the data before and after compression, and other possible business-related information.

[0123] Finally, the compressed data packets are transmitted using idle TCP connections in the TCP connection pool.

[0124] Furthermore, step S220 further includes steps S221-S222: Step S221: When the current usage rate is less than a second threshold, if the idle TCP connection does not exist in the TCP connection pool, a new TCP connection is established for transmitting the compressed data packet.

[0125] It should be noted that the second threshold is defined as a critical point of a low load state.

[0126] Step S222: After the compressed data packet transmission is completed, close the new TCP connection.

[0127] If the current usage rate is less than the second threshold (low load threshold), that is, the dedicated line is under low load, the data transmission agent module checks whether there is an idle TCP connection in the TCP connection pool. If there is an idle TCP connection, it directly takes out an idle TCP connection to transmit the encapsulated compressed data packets.

[0128] If the TCP connections in the TCP connection pool are insufficient, a new TCP connection is created with the data transmission agent module on the receiving side, and then the encapsulated compressed data packet is sent using the newly created TCP connection.

[0129] After data transmission is completed, the TCP connection is closed and the newly created TCP connection is not added to the TCP connection pool.

[0130] Furthermore, step S220 further includes steps S223-S224: Step S223: When the current usage rate is greater than or equal to the second threshold and less than the first threshold, if the TCP connection pool does not have the idle TCP connection, the compressed data packet is placed in the corresponding cache queue according to the service priority corresponding to the data transmission request.

[0131] Step S224: when the idle TCP connection appears in the TCP connection pool, the compressed data packet is taken out from the cache queue to perform data transmission.

[0132] In this embodiment, if the current usage rate is greater than or equal to the second threshold and less than the first threshold, that is, the dedicated line is under medium load, if there is an idle TCP connection in the TCP connection pool, the extracted idle TCP connection is directly used to send the encapsulated compressed data packet.

[0133] If there are insufficient TCP connections in the TCP connection pool, the encapsulated compressed data packets are placed in a cache queue corresponding to the service priority. When there are idle TCP connections in the TCP connection pool, the encapsulated compressed data packets are taken out from the cache queue for sending.

[0134] Please refer to Figure 8 The data transmission method of the fifth embodiment proposed in the present application is applied to the data transmission agent module on the receiving side, and specifically includes steps S60 to S120: Step S60: receiving the compressed data packet transmitted by the data transmission agent module on the sending side through the pre-established TCP connection.

[0135] Step S70: Calculate a hash value for the compressed data packet, and compare the hash value with the original compressed hash value provided by the data transmission agent module on the sending side.

[0136] Step S80: If the comparison result is inconsistent, the data transmission agent module on the sending side is notified to resend the compressed data packet.

[0137] Step S90: If the comparison results are consistent, decompress the compressed data packet to obtain the decompressed original data, and calculate the hash value of the original data.

[0138] Step S100: Compare the hash value of the original data with the original hash value provided by the data transmission agent module on the sending side. If the comparison result is consistent, repackage the original data and distribute it to the target business system.

[0139] Step S110: If the distribution to the target service system fails, then according to the service priority of the data transmission request corresponding to the compressed data packet, the re-encapsulated original data is placed in a cache queue corresponding to the service priority.

[0140] Step S120: Detect the state of the target business system, and when it is confirmed that the target business system is in a live state, take out the repackaged original data from the cache queue and distribute it to the target business system.

[0141] In the data transmission agent module on the receiving side, once the data transmission agent module on the receiving side receives the data transmission request sent by the data transmission agent module on the sending side through the http protocol, it will immediately decompress the compressed data packet corresponding to the data transmission request to determine the integrity of the data packet. The process of decompressing the data packet involves checking the header information of the data packet, such as comparing the MD5 value.

[0142] Specifically, the data transmission agent module on the receiving side first performs a hash value check on the compressed data packet. The hash value of the compressed data packet is calculated and compared with the original compressed hash value (corresponding to the compressed data md5 field data in the transmission protocol) provided by the data transmission agent module on the sending side. If the check fails, that is, the calculated hash value of the compressed data packet is inconsistent with the original compressed hash value in the transmission protocol provided by the data transmission agent module on the sending side, the data transmission agent module on the sending side is requested to resend the compressed data packet.

[0143] It should be noted that the MD5 (Message-Digest Algorithm 5) value is a specific hash value. The hash value is a fixed-length string obtained by performing a hash function calculation on the data, which is usually used to verify the integrity of the data. MD5 is a widely used hash algorithm that can generate a 128-bit hash value, usually expressed as 32 hexadecimal digits.

[0144] If the MD5 value verification passes, the compressed data packet (compressed data in the transmission protocol) is decompressed to obtain the original data. Then the MD5 value of the original data is calculated, and the MD5 value of the original data is compared with the original MD5 value (corresponding to the inital data md5 field data in the transmission protocol). If the comparison results are inconsistent, the data transmission agent module on the sending side is also requested to resend the compressed data packet.

[0145] If the MD5 value of the decompressed original data is consistent with the original MD5 value, the business system address, data transmission path, http parameters and other information are obtained from the database according to the business identifier (such as appid), the original data is repackaged, and the repackaged original data is directly distributed to the target business system through the connection established with the target business system.

[0146] If the repackaged original data cannot be sent temporarily due to network problems or target business system failures, or the data distribution fails, the data transmission agent module on the receiving side will write the repackaged original data into the cache queue corresponding to the business priority, and periodically detect whether the status of the target business system is alive. After the target business system returns to normal, that is, when the target business system is in a alive state, the repackaged original data will be taken out from the cache queue and sent to the target business system.

[0147] The present application provides a data transmission device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data transmission method in the above-mentioned embodiment one.

[0148] Reference below Fig. 9 , which shows a schematic diagram of the structure of a data transmission device suitable for implementing the embodiment of the present application. The data transmission device in the embodiment of the present application may include various hardware and software components for implementing the scheduling method of the inspection task. Fig. 9 The data transmission device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0149] like Fig. 9As shown, the data transmission device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data transmission device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data transmission device to communicate with other devices wirelessly or by wire to exchange data. Although the data transmission device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0150] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0151] The data transmission device provided by the present application adopts the data transmission device method in the above embodiment, which can solve the technical problem of how to achieve efficient and stable data transmission under limited dedicated line bandwidth resources. Compared with the prior art, the beneficial effects of the data transmission device provided by the present application are the same as the beneficial effects of the data transmission method provided by the above embodiment, and the other technical features in the data transmission device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0152] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0153] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0154] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data transmission method in the above-mentioned embodiment.

[0155] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0156] The computer-readable storage medium may be included in the data transmission device; or may exist independently without being assembled into the data transmission device.

[0157] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a data transmission device, the data transmission device: receives a data transmission request, determines the service priority corresponding to the data transmission request; obtains the current utilization rate of the transmission link between the exit route and the receiving side exit route; if the current utilization rate is greater than or equal to a first threshold, when there is an idle TCP connection in the TCP connection pool, and there is a data transmission request with the service priority of the first priority, and there is no ongoing data transmission, compresses the data packet corresponding to the data transmission request with the service priority of the first priority according to the first compression level; and uses the idle TCP connection to perform data transmission on the compressed data packet.

[0158] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0160] The modules involved in the embodiments described in the present application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0161] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data transmission method, and can solve the technical problem of how to achieve efficient and stable data transmission under limited dedicated line bandwidth resources. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data transmission method provided in the above-mentioned embodiment, and will not be elaborated here.

[0162] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above-mentioned data transmission method when executed by a processor.

[0163] The computer program product provided by the present application can solve the technical problem of how to achieve efficient and stable data transmission under limited dedicated line bandwidth resources. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as the beneficial effects of the data transmission method provided by the above embodiment, which will not be repeated here.

[0164] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

[0165] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0167] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data transmission method, characterized in that: A data transmission agent module applied to the sending side, wherein the data transmission agent module is deployed between the service system and the egress router, and the data transmission method comprises: receiving a data transmission request, and determining a service priority corresponding to the data transmission request; Acquire a current usage rate of a transmission link between the egress route and a receiving-side egress route; If the current usage rate is greater than or equal to the first threshold, when there is an idle TCP connection in the TCP connection pool, and there is a data transmission request with the service priority of the first priority, and there is no ongoing data transmission, compressing the data packet corresponding to the data transmission request with the service priority of the first priority according to the first compression level; The idle TCP connection is used to perform data transmission on the compressed data packet.

2. The data transmission method according to claim 1, characterized in that: Before the step of receiving a data transmission request and determining a service priority corresponding to the data transmission request, the method further comprises: Pre-configure business priority rules according to different business types and / or business identifiers; When receiving a data transmission request, identifying a service type and / or a service identifier in the data transmission request, and retrieving the service priority rule based on the service type and / or the service identifier; According to the search result, the service priority corresponding to the data transmission request is determined.

3. The data transmission method according to claim 2, characterized in that: After the step of determining the service priority corresponding to the data transmission request according to the search result, the method further includes: Allocate a cache queue for each different service priority; When the current usage rate of the transmission link between the egress route and the receiving-side egress route is greater than or equal to the first threshold, determining whether there is ongoing data transmission on the transmission link; If there is ongoing data transmission, the data packet corresponding to the data transmission request is compressed at a first compression level according to the service priority corresponding to the data transmission request, and the data transmission request is placed in the corresponding cache queue.

4. The data transmission method according to claim 1, characterized in that: Before the step of receiving a data transmission request and determining a service priority corresponding to the data transmission request, the method further comprises: Establishing a TCP connection pool between the data transmission proxy module on the sending side and the data transmission proxy module on the receiving side, wherein the TCP connection pool includes at least one pre-established TCP connection; The connection status of the TCP connection is monitored, and for the invalid or abnormal TCP connection, an operation of reestablishing or removing the connection from the TCP connection pool is performed.

5. The data transmission method according to claim 1, characterized in that: After the step of obtaining the current usage rate of the transmission link between the egress route and the receiving side egress route, the method further comprises: When the current usage rate is less than the first threshold, compressing the data packet corresponding to the data transmission request according to a preset compression level; The idle TCP connection in the TCP connection pool is used to perform data transmission on the compressed data packet.

6. The data transmission method according to claim 5, characterized in that: After the step of using the idle TCP connection in the TCP connection pool to perform data transmission on the compressed data packet, the method further comprises: When the current usage rate is less than a second threshold, if the idle TCP connection does not exist in the TCP connection pool, a new TCP connection is established for transmitting the compressed data packet; After the compressed data packet transmission is completed, the new TCP connection is closed.

7. The data transmission method according to claim 5, characterized in that: After the step of using the idle TCP connection in the TCP connection pool to perform data transmission on the compressed data packet, the method further includes: When the current usage rate is greater than or equal to the second threshold and less than the first threshold, if the TCP connection pool does not have the idle TCP connection, the compressed data packet is placed in the corresponding cache queue according to the service priority corresponding to the data transmission request; When the idle TCP connection appears in the TCP connection pool, the compressed data packet is taken out from the cache queue to perform data transmission.

8. A data transmission method, characterized in that: A data transmission agent module applied to the receiving side, the data transmission method includes: Receiving the compressed data packet transmitted by the data transmission agent module on the sending side through the pre-established TCP connection; Calculating a hash value for the compressed data packet, and comparing the hash value with the original compressed hash value provided by the data transmission agent module on the sending side; If the comparison results are inconsistent, informing the data transmission agent module on the sending side to resend the compressed data packet; If the comparison results are consistent, the compressed data packet is decompressed to obtain the decompressed original data, and a hash value of the original data is calculated; Compare the hash value of the original data with the original hash value provided by the data transmission agent module on the sending side, and if the comparison results are consistent, repackage the original data and distribute it to the target business system; If the distribution to the target service system fails, the repackaged original data is placed in a cache queue corresponding to the service priority according to the service priority of the data transmission request corresponding to the compressed data packet; The state of the target business system is detected, and when it is confirmed that the target business system is in a live state, the repackaged original data is taken out from the cache queue and distributed to the target business system.

9. A data transmission device, characterized in that: The data transmission device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data transmission method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that: The computer storage medium is a computer-readable storage medium, and a computer program is stored on the computer storage medium. When the computer program is executed by a processor, the steps of the data transmission method according to any one of claims 1 to 8 are implemented.

Images