Data transmission method and device, electronic equipment and storage medium

Abstract

The invention provides a data transmission method and device, electronic equipment and a storage medium, the method comprises the following steps: caching data read from a computing node in a memory to obtain target data in the memory, the target data being data not sent to a server; under the condition that the target data cached in the memory meets a preset threshold value, compressing the target data in the memory to obtain a compressed block; and sending the compressed block to a server. Through the method and the device, the problem that a large file transmission scheme capable of occupying a small amount of bandwidth and ensuring transmission real-time performance is lacked in related technologies is solved.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data transmission method and apparatus, electronic device and storage medium. Background Technology

[0002] In the ever-growing financial system, as data volume continues to expand, the stability and reliability of distributed databases face severe challenges in high-concurrency scenarios. Full SQL data generates a large number of local cache files. Traditional disk compression and transmission can lead to excessive disk I / O pressure, causing excessive consumption of database computing layer resources, resulting in system performance bottlenecks and affecting database access for other normal business operations.

[0003] Existing large file transfer solutions for distributed databases generally use raw data packets or direct JSON transmission, resulting in bandwidth waste of 60%–90%. When relying on offline compression methods such as gzip or zstd, the data must first be written to disk in blocks (i.e., compression cannot begin until the entire data block is written), leading to high transmission latency, large bandwidth peaks, and overall low efficiency. Therefore, there is an urgent need in this field for a method that can efficiently transfer files while consuming minimal bandwidth, and possesses a precise rate-limiting mechanism to ensure overall database throughput while effectively preventing service cascading failures caused by isolated abnormal requests.

[0004] Therefore, there is a lack of large file transfer solutions in related technologies that can consume a small amount of bandwidth and guarantee real-time transmission. Summary of the Invention

[0005] This application provides a data transmission method and apparatus, electronic device and storage medium to at least solve the problem in the related art of lacking a large file transmission scheme that can occupy a small amount of bandwidth and guarantee real-time transmission.

[0006] According to one aspect of the embodiments of this application, a data transmission method is provided, including: The data read from the computing node is cached in memory to obtain the target data located in the memory, wherein the target data is the data that has not been sent to the server; If the target data cached in the memory meets a preset threshold, the target data is compressed in memory to obtain a compressed block; The compressed block is sent to the server.

[0007] Optionally, as described above, before caching the data read from the computing node in memory to obtain the target data located in the memory, the method further includes: Establish a TCP connection with the server to obtain the target TCP connection; The metadata information of the computing node is sent to the server through the target TCP connection, so that the server can authenticate the computing node based on the metadata information. The authentication is used to enable the server to store the compressed block in the storage area corresponding to the computing node after receiving the compressed block.

[0008] Optionally, as described above, if the target data cached in the memory meets a preset threshold, compressing the target data in memory to obtain a compressed block includes: The target memory space is pre-allocated in the aforementioned memory. If the target data cached in the memory meets a preset threshold, the target data is compressed in memory to obtain a compressed block; The compressed block is stored in the target memory space using a contiguous memory layout.

[0009] Optionally, as described above, sending the compressed block to the server includes: The target memory space is pre-attached to the TCP write buffer so that it can be used as the data source for TCP transmission; The compressed block is read from the target memory space according to the memory address of the target memory space, and the compressed block is sent to the server.

[0010] Optionally, as described above, sending the compressed block to the server includes: Monitor the data traffic sending data; If the data traffic is greater than or equal to a preset traffic threshold, the sending speed of the compressed block is reduced.

[0011] Optionally, as described above, before caching the data read from the computing node in memory to obtain the target data located in the memory, the method further includes: Monitor the computing nodes; If the computing node generates a target file, a jump operation is performed to proceed to the following steps: the data read from the computing node is cached in memory to obtain the target data located in the memory, wherein the target file is used to instruct the computing node to fully acquire the full amount of SQL data to be transmitted to the server, and the target data is a part of the full amount of SQL data.

[0012] Optionally, as described above, the preset threshold is a preset number of data rows.

[0013] According to another aspect of the embodiments of this application, a data transmission apparatus is also provided, comprising: The reading module is used to cache the data read from the computing node in memory to obtain the target data located in the memory, wherein the target data is data that has not been sent to the server. A compression module is used to compress the target data cached in the memory to obtain a compressed block when the target data meets a preset threshold. The sending module is used to send the compressed block to the server.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the method steps of any of the above embodiments by running the computer program stored in the memory.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the method steps of any of the above embodiments when it is run.

[0016] In this embodiment, a method of compressing and transmitting real-time acquired data is adopted. Data read from the computing node is cached in memory to obtain target data located in the memory, where the target data is data not yet sent to the server. If the target data cached in memory meets a preset threshold, it is compressed in memory to obtain a compressed block. The compressed block is then sent to the server. Since the target data can be compressed to obtain a compressed block as long as the read data meets the preset threshold, and then the compressed block in memory is sent to the server, real-time transmission can be achieved even with a small amount of data (i.e., the preset threshold of data). Furthermore, transmission can be performed without writing data to disk, thus avoiding the need for full data compression before a concentrated burst of transmission, preventing instantaneous bandwidth spikes that could affect network stability. This achieves the technical effect of effectively reducing bandwidth consumption while ensuring real-time transmission, thereby solving the problem in related technologies of lacking a large file transmission solution that can consume minimal bandwidth and guarantee real-time transmission. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the hardware environment for an optional data transmission method according to an embodiment of this application; Figure 2 This is a flowchart illustrating an optional data transmission method according to an embodiment of this application; Figure 3 This is a flowchart illustrating another optional data transmission method according to an embodiment of this application; Figure 4 This is a timing diagram of an optional data transmission method according to an application example of this application; Figure 5 This is a structural block diagram of an optional data transmission device according to an embodiment of this application; Figure 6 This is a structural block diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] According to one aspect of the embodiments of this application, a data transmission method is provided. Optionally, in this embodiment, the above-described data transmission method can be applied to, for example... Figure 1 The system shown consists of a Computing Node (CN), a Client, a Server, and a third-party RDB, where the Server and the third-party RDB are included in the server-side component. Figure 1 As shown, the CN connects to the Client via the network, the Client connects to the server via the network, the server can be used to provide data storage services to the Client, and the Client connects to a third-party RDB via the network, the third-party RDB is used to provide data storage services.

[0023] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal may not be limited to PC, mobile phone, tablet computer, etc.

[0024] The data transmission method in this application embodiment can be executed by a client.

[0025] Taking the data transmission method in this embodiment as an example, which is executed by the client, Figure 2 A data transmission method provided in this application includes the following steps: Step S202 involves caching the data read from the computing node in memory to obtain the target data located in memory, where the target data is the data that has not been sent to the server.

[0026] The data transmission method in this embodiment can be applied to scenarios where data is transmitted from computing nodes to a database for storage. Each client can connect to one or more settlement nodes.

[0027] Specifically, the client can continuously read data from the compute node, for example, by reading binary logs. The read data is then cached in memory, and this cached data is designated as the target data. It's important to note that this target data is the raw data currently in the cache, awaiting compression, that has not yet been sent to the server. In other words, the data in memory can contain the target data as well as other data; that is, the other data can be data already sent to the server, and the target data and other data can be a portion of the full SQL data.

[0028] Step S204: If the target data cached in memory meets the preset threshold, the target data is compressed in memory to obtain a compressed block.

[0029] In other words, the target data changes as the computing nodes read more data; the more data is read, the larger the target data becomes. When the target data meets a preset threshold—that is, when the target data volume reaches the preset threshold—the target data is directly compressed in memory to obtain a compressed block. Furthermore, this compressed block also resides in memory, meaning there is no need to write the compressed block to disk. The preset threshold represents a balance between performance and latency: too small a threshold results in low compression efficiency and too many network packets; too large a threshold increases memory usage and first-packet latency.

[0030] Step S206: Send the compressed block to the server.

[0031] In other words, after obtaining the compressed block, it can be sent to the server so that the server can obtain the target data according to the compressed block and write the target data into a third-party library. Furthermore, after receiving the compressed block, the server can decode it according to the decoding method corresponding to the compression encoding method described above, restore the target data, and then write the target data into the third-party library.

[0032] In this embodiment, a method of compressing and transmitting real-time acquired data is adopted. Data read from the computing node is cached in memory to obtain target data, which is data not yet sent to the server. When the cached target data in memory meets a preset threshold, it is compressed in memory to obtain a compressed block. The compressed block is then sent to the server. Since the target data can be compressed to obtain a compressed block as long as the read data meets the preset threshold, and then sent to the server, real-time transmission can be achieved even with a small amount of data (i.e., the preset threshold amount). Furthermore, transmission can be performed without writing data to disk, thus avoiding the need for full data compression before a concentrated burst of transmission. This prevents instantaneous bandwidth spikes that could affect network stability, effectively reducing bandwidth consumption while ensuring real-time transmission. This solves the problem in related technologies of lacking a large file transmission solution that can consume minimal bandwidth while guaranteeing real-time transmission.

[0033] like Figure 3 As shown, as an optional implementation, the method described above, before executing step S202 to read the data from the computing node and cache it in memory to obtain the target data located in memory, the method further includes the following steps: Step S302: Establish a TCP connection with the server to obtain the target TCP connection.

[0034] In other words, the client requests to establish a TCP connection with the server in order to obtain the target TCP connection.

[0035] Step S304: Send the metadata information of the computing node to the server through the target TCP connection so that the server can authenticate the computing node based on the metadata information. The authentication is used to enable the server to store the compressed block in the storage area corresponding to the computing node after receiving the compressed block.

[0036] Specifically, the metadata information of a compute node may include, but is not limited to: a unique node identifier (such as Node ID, Hostname, UUID), a digital certificate or public key, an access token or API key, cluster / tenant information (Tenant ID), a timestamp (for preventing replay attacks), and a signature (a digital signature of the above information based on the private key). Then, after sending the metadata information to the server through the established target TCP connection, the server can authenticate the compute node based on this metadata information. The purpose of authentication includes, but is not limited to: ensuring that the compute node initiating the connection is indeed the legitimate compute node it claims to be (and not a forged or malicious node); and ensuring permission compliance: that the node is allowed to perform subsequent operations (such as uploading logs, requesting data, participating in computation, etc.). Furthermore, after successful authentication, the server can store the compressed blocks from the compute node in the storage area corresponding to that compute node, which is the server's corresponding third-party RDB.

[0037] The method described in this embodiment can effectively ensure the legality of the data and protect the security of the database where the storage area is located.

[0038] As an optional implementation, the method described above can be implemented through the following steps: If the target data cached in memory meets a preset threshold, the target data is compressed in memory to obtain a compressed block: A target memory space is pre-allocated in memory; if the target data cached in memory meets a preset threshold, the target data is compressed in memory to obtain a compressed block; the compressed block is stored in the target memory space using a contiguous memory layout. In other words, before the data processing flow starts, the client actively allocates a memory region of fixed size or a predictable range as a dedicated storage area for subsequent compressed data, called the "target memory space." This avoids frequent calls to dynamic memory allocation (such as multiple new / malloc calls) during high-frequency compression, thereby reducing memory fragmentation, lowering allocation latency, and improving cache locality; the lifecycle of this target memory space can be managed by the application and remains valid until the transmission task ends. The client continuously accumulates raw data to be processed (such as binary log events, database change records, etc.) in memory, forming a memory cache. When the amount of cached target data reaches a preset threshold (e.g., ≥1000 rows, ≥64KB bytes), compression is triggered. The compression algorithm (e.g., zstd, LZ4) operates directly on the original target data in memory without disk writes, and the output is a compressed block. The entire process is completed in memory, achieving a streaming, low-latency compression pipeline. The method of storing compressed blocks in the target memory space using a contiguous memory layout is as follows: After compression, the system writes the generated compressed blocks into the pre-allocated target memory space in a contiguous and compact manner. A contiguous memory layout means that the data of the compressed blocks occupy a contiguous, gapless region in physical memory. This layout facilitates CPU cache prefetching and improves subsequent processing efficiency. The starting address and length of this memory region can be directly passed to high-performance I / O interfaces (such as io_uring and sendmsg with MSG_ZEROCOPY), allowing the kernel or network card to directly read via DMA without additional copying. Each compressed block can include a header (such as sequence number, length, and checksum), stored adjacent to the data body to form a self-describing structure, thus facilitating metadata management. For example, if the total size of the target memory space is 1MB, and the first compressed block occupies 64KB, it is written starting from offset 0; the next compressed block is written immediately after at offset 64KB, and so on, forming a compact block sequence.

[0039] The method described in this embodiment enables controllable memory usage (avoiding dynamic allocation jitter); the absence of disk transfer allows for low-latency compression and transmission; contiguous memory and zero-copy enable efficient network transmission; and pipelined processing and resource reuse improve system throughput. In summary, this approach is suitable for distributed system scenarios such as high-throughput, low-latency log synchronization, database replication, and real-time backup.

[0040] As an optional implementation, the method described above can be used to send the compressed block to the server through the following steps: The target memory space is pre-attached to the TCP write buffer so that it can be used as the data source for TCP transmission. Specifically, this attachment can be achieved through the zero-copy mechanism provided by the operating system, which registers or maps the target memory space (i.e., the contiguous memory region storing compressed blocks) already allocated in user space as a data source that can be directly accessed by the TCP protocol stack.

[0041] The compressed block is read from the target memory space according to its memory address and then sent to the server. Specifically, since the compressed block is written to a specific offset position in the target memory space during generation (as described in the previous contiguous layout), the system can directly locate the compressed block using its starting virtual address and length. The base address of the target memory space is 0x7f0000000000; the first compressed block starts at offset 0 and has a length of 65536 bytes; therefore, its memory address range is [0x7f0000000000, 0x7f0000010000). During transmission, the application or I / O framework directly passes this address and length to the network transmission interface without requiring additional memory copying or data reassembly. After completing the above preparations, the system calls efficient send primitives (such as `io_uring`'s `IORING_OP_SEND` or `sendmsg` with `MSG_ZEROCOPY`) to push the compressed block directly from the target memory space to the network protocol stack: the kernel (or user-space network driver) locks the memory page to prevent it from being released or modified before the transmission is complete; the network subsystem (or smart NIC) reads data from this user memory address via DMA (Direct Memory Access), encapsulates it into a TCP packet, and sends it. Furthermore, after the transmission is complete, the kernel notifies the target memory space that it can be safely released via events (such as `io_uring CQE` or `SO_ZEROCOPY` notifications).

[0042] In this embodiment, zero-copy optimization is achieved by pre-attaching the target memory space to the TCP write buffer. Through the high-level interface of the operating system or network framework, user-space memory can be directly used as a data source by the TCP protocol stack. The mechanism of reading compressed blocks by memory address and sending them utilizes this mechanism to directly extract compressed data from the pre-allocated contiguous memory and send it to the server. There is no disk I / O or kernel copy throughout the process, achieving high-throughput and low-latency efficient transmission.

[0043] As an optional implementation, the method described above can be implemented by sending compressed blocks to the server through the following steps: monitoring the data traffic of transmitted data; reducing the sending speed of compressed blocks when the data traffic is greater than or equal to a preset traffic threshold. Specifically, during data transmission, the client collects and counts the amount of data sent through the network per unit time in real time, i.e., "data traffic". Optional implementation methods include: time window-based rate calculation: for example, counting the number of bytes sent every 100 milliseconds and converting it into instantaneous bandwidth (such as MB / s or Gbps); using kernel or I / O framework feedback information: such as monitoring the socket sending rate through the io_uring completion event, SO_SNDBUF status, or eBPF program; application layer counter: after each successful transmission of a compressed block, accumulating its original or compressed byte count and calculating the moving average rate by combining it with the timestamp. Through the above methods, the actual load of the current network egress can be dynamically perceived, providing a basis for subsequent rate adjustment. The client can pre-configure a preset traffic threshold (e.g., 800 Mbps, 1 GB / s, or 80% of the network card bandwidth) as the upper limit for network resource usage. This preset traffic threshold can be determined based on the following information: physical network card capacity (e.g., 9 Gbps for a 10 GbE network); QoS quotas in a multi-tenant environment; and avoiding triggering network congestion or interfering with other critical services on the same machine. When the monitoring module detects that the current transmission rate (i.e., data traffic) is greater than or equal to the preset traffic threshold, it is determined as "excessive traffic" and a rate limiting mechanism needs to be activated. To avoid continuous high bandwidth usage leading to network congestion, packet loss, or impact on other services, the system actively slows down the transmission rate of compressed blocks.

[0044] By using the method in this embodiment, the transmission rate is actively reduced once the rate reaches or exceeds a preset traffic threshold, thereby controlling the network bandwidth usage within a reasonable range. This mechanism ensures high throughput while taking into account network stability and multi-task collaboration, and is a key link in achieving controllable, reliable, and efficient data transmission in high-performance distributed systems.

[0045] As an optional implementation, the method described above, before caching the data read from the computing node in memory and obtaining the target data located in memory, further includes: Monitoring compute nodes is crucial. Specifically, clients can deploy a monitoring agent or utilize kernel / user-space hooks to continuously observe the compute nodes. Monitoring methods may include, but are not limited to, the following: file system event listening (e.g., inotify, fanotify on Linux) to monitor the creation of new files in specific directories; process behavior tracing (e.g., eBPF, ptrace) to capture the behavior of target processes writing to specific files; and task scheduling status polling to check whether compute tasks have entered the "data ready" phase. The monitoring objective is to identify whether the compute node has generated a target file with specific semantics.

[0046] Upon detecting that a compute node has generated a target file, a jump operation is executed to proceed to the following steps: Data read from the compute node is cached in memory, resulting in the target data located in memory. The target file instructs the compute node to fully acquire the complete SQL data to be transmitted to the server; the target data is a portion of the complete SQL data. Specifically, when the monitoring agent detects a target file (e.g., an IOCACHE file) indicating that the complete SQL data has been successfully created and written (e.g., file handle closed, size stable, presence flags, etc.), it determines that "the complete SQL data has been generated." At this point, a control flow jump is triggered. This jump operation serves as the start signal for subsequent data processing, ensuring that transmission only begins after the data is complete, avoiding the reading of incomplete or intermediate temporary files.

[0047] like Figure 4 As shown, an application example of any of the aforementioned data transmission methods is provided: The system implementing this application example mainly includes a computing node (i.e., Figure 4 CN), client (i.e., Figure 4 Client), separate server and third-party libraries (i.e., Figure 4The binary log is compressed line by line in the client's memory: once a compressed block is generated, it immediately enters the network transmission queue without disk transfer. The compressed block adopts a contiguous memory layout and can be directly attached to the TCP write buffer to achieve true zero copy; the receiving end operates based on a streaming state machine, supporting out-of-order block caching and decompression while receiving, achieving low-bandwidth, high-efficiency file transmission. That is, the server uses a streaming state machine for data processing. This state machine continuously listens to the network input stream and dynamically updates its internal state based on the metadata carried in the header of the received data block (such as sequence number, compression identifier, block length, etc.), driving the subsequent processing flow.

[0048] The client first establishes a TCP connection with the server, sending relevant metadata information about the CN (Network Provider) to the server for authentication. When the client detects that the CN has generated an IOCACHE file (i.e., the target file), it compresses the target data while reading the file from the CN, provided the read target data meets a preset threshold, and sends the compressed block to the server. The server receives the compressed block, decodes it, and writes the restored target data to a third-party RDB. While sending data blocks, the client monitors data traffic and dynamically adjusts the sending speed to achieve rate limiting.

[0049] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0050] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0051] According to another aspect of the embodiments of this application, a data transmission apparatus for implementing the above-described data transmission method is also provided. Figure 5 This is a structural block diagram of an optional data transmission device according to an embodiment of this application, such as... Figure 5 As shown, the device may include: The reading module 51 is used to cache the data read from the computing node in memory to obtain the target data located in memory, wherein the target data is the data that has not been sent to the server. Compression module 52 is used to compress the target data in memory to obtain a compressed block when the target data cached in memory meets a preset threshold. The sending module 53 is used to send the compressed block to the server.

[0052] It should be noted that the reading module 51 in this embodiment can be used to perform the above step S202, the compression module 52 in this embodiment can be used to perform the above step S204, and the sending module 53 in this embodiment can be used to perform the above step S206.

[0053] In addition to the modules described above, the apparatus in this embodiment may also include modules that execute any method in any of the aforementioned data transmission methods.

[0054] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown can be implemented through software or hardware, and the hardware environment includes the network environment.

[0055] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described data transmission method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0056] According to another embodiment of this application, an electronic device is also provided, comprising: Figure 6 As shown, the electronic device may include: a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504, wherein the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.

[0057] Memory 1503 is used to store computer programs; When processor 1501 executes the program stored in memory 1503, it performs the following steps: Step S202 involves caching the data read from the computing node in memory to obtain the target data located in memory, where the target data is the data that has not been sent to the server.

[0058] Step S204: If the target data cached in memory meets the preset threshold, the target data is compressed in memory to obtain a compressed block.

[0059] Step S206: Send the compressed block to the server.

[0060] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic device and other devices.

[0061] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0062] As an example, the memory 1503 described above may include, but is not limited to, the read module 51, the compression module 52, and the send module 53 of the data transmission device described above. Furthermore, it may include, but is not limited to, other module units of the data transmission device described above, which will not be elaborated upon in this example.

[0063] The processor mentioned above can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0064] This application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the method steps of the above method embodiments when it runs.

[0065] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0066] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0067] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0068] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0069] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0070] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the solution provided in this embodiment, depending on actual needs.

[0071] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0072] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data transmission method, characterized in that, include: The data read from the computing node is cached in memory to obtain the target data located in the memory, wherein the target data is the data that has not been sent to the server; If the target data cached in the memory meets a preset threshold, the target data is compressed in memory to obtain a compressed block; The compressed block is sent to the server.

2. The method according to claim 1, characterized in that, Before the step of caching the data read from the computing node in memory and obtaining the target data located in the memory, the method further includes: Establish a TCP connection with the server to obtain the target TCP connection; The metadata information of the computing node is sent to the server through the target TCP connection, so that the server can authenticate the computing node based on the metadata information. The authentication is used to enable the server to store the compressed block in the storage area corresponding to the computing node after receiving the compressed block.

3. The method according to claim 1, characterized in that, When the target data cached in the memory meets a preset threshold, the target data is compressed in memory to obtain a compressed block, including: The target memory space is pre-allocated in the aforementioned memory. If the target data cached in the memory meets a preset threshold, the target data is compressed in memory to obtain a compressed block; The compressed block is stored in the target memory space using a contiguous memory layout.

4. The method according to claim 3, characterized in that, Sending the compressed block to the server includes: The target memory space is pre-attached to the TCP write buffer so that it can be used as the data source for TCP transmission; The compressed block is read from the target memory space according to the memory address of the target memory space, and the compressed block is sent to the server.

5. The method according to claim 1, characterized in that, Sending the compressed block to the server includes: Monitor the data traffic sending data; If the data traffic is greater than or equal to a preset traffic threshold, the sending speed of the compressed block is reduced.

6. The method according to claim 1, characterized in that, Before the step of caching the data read from the computing node in memory and obtaining the target data located in the memory, the method further includes: Monitor the computing nodes; If the computing node generates a target file, a jump operation is performed to proceed to the following steps: the data read from the computing node is cached in memory to obtain the target data located in the memory, wherein the target file is used to instruct the computing node to fully acquire the full amount of SQL data to be transmitted to the server, and the target data is a part of the full amount of SQL data.

7. The method according to any one of claims 1 to 6, characterized in that, The preset threshold is the preset number of data rows.

8. A data transmission device, characterized in that, include: The reading module is used to cache the data read from the computing node in memory to obtain the target data located in the memory, wherein the target data is data that has not been sent to the server. A compression module is used to compress the target data cached in the memory to obtain a compressed block when the target data meets a preset threshold. The sending module is used to send the compressed block to the server.

9. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, the communication interface, and the memory communicate with each other via the communication bus, characterized in that... The memory is used to store computer programs; The processor is configured to perform the method of any one of claims 1 to 7 by running the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 7 when run on a processor.

Images