Data transmission methods, equipment, media and software products

Abstract

This application provides a data transmission method, device, medium, and program product, relating to the field of data processing technology. The method is applied at the source end and includes: establishing a data transmission channel with a receiving end, and setting an identifier field in the communication protocol of the data transmission channel to distinguish transmission modes; encoding the transmitted data according to the identifier field to obtain encoded transmitted data, the encoding process including metadata encapsulation of chunked transmitted data and direct transmission interface calls for zero-copy transmitted data; sending the encoded transmitted data to the receiving end through the data transmission channel, so that the receiving end receives the encoded transmitted data, and decodes the encoded transmitted data according to the identifier field, the decoding process including metadata parsing of chunked transmitted data and direct writing of the byte stream of zero-copy transmitted data. Based on this method, multiple transmission modes can be used for coordinated transmission within a single channel, reducing resource consumption at both the source and receiving ends.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data transmission method, device, medium and program product. Background Technology

[0002] In enterprise-level database scenarios that require real-time data synchronization, efficient transmission of database logs is a crucial step in ensuring data consistency, enabling disaster recovery, and facilitating real-time analysis.

[0003] In some databases, log files are divided into two categories: online logs and archived logs. Online logs record real-time database operations and require continuous appending; archived logs are archived backups of online logs, with fixed content that must be transmitted completely. In real-time data synchronization scenarios, the proxy server needs to read log files from the source database server and transmit them over the network to the target (receiving) server for parsing and application. This process places high demands on transmission efficiency, stability, and resource consumption.

[0004] For example, in high-concurrency business scenarios such as finance and telecommunications, log files may grow to gigabyte (GB) scale. If transmission latency or resource consumption is too high, it will lead to synchronization delays, data loss, or even database system crashes. In addition, proxy servers may be deployed in resource-constrained environments (such as edge computing nodes), requiring high-throughput transmission under limited hardware conditions.

[0005] The Netty framework is a high-performance network programming framework. While existing file transfer solutions based on Netty offer high performance advantages, they are limited by data transmission protocol compatibility issues, requiring separate data transmission channels for online logs and archived logs. This significantly increases system complexity and operational difficulty, making it difficult to meet the needs of enterprise databases for efficient, low-resource-consumption real-time data synchronization. Summary of the Invention

[0006] This application provides a data transmission method, device, medium, and program product for implementing multi-transmission mode data transmission within a single channel in scenarios where data transmission is based on the Netty framework, thereby reducing resource consumption during data transmission.

[0007] In a first aspect, embodiments of this application provide a data transmission method, which is applied at a source end and includes:

[0008] Establish a data transmission channel with the receiving end, and set an identification field in the communication protocol of the data transmission channel to distinguish the transmission mode;

[0009] The transmitted data is encoded based on the identifier field to obtain encoded transmitted data. The encoding process includes metadata encapsulation of chunked transmitted data and direct transmission interface calls for zero-copy transmitted data.

[0010] Encoded transmission data is sent to the receiving end through the data transmission channel so that the receiving end can receive the encoded transmission data and decode the encoded transmission data according to the identification field. The decoding process includes parsing the metadata of the chunked transmission data and directly writing the byte stream of the zero-copy transmission data.

[0011] In one possible implementation, the transmitted data is encoded according to the identification field to obtain encoded transmitted data, including:

[0012] If the type of transmitted data is determined to be chunked data, the transmitted data is encapsulated into structured data based on the file name and block sequence number in the identifier field, and a byte buffer object is generated.

[0013] And / or, if it is determined that the type of data to be transferred is zero-copy data transfer, call the file region interface to generate a file region object.

[0014] In one possible implementation, before encoding the transmitted data according to the identification field, the method further includes:

[0015] Custom codecs are dynamically inserted into the Netty pipeline of the data transmission channel. The Netty pipeline is used to handle network events in the Netty framework, and the custom codec is used to switch the encoding processing logic based on the value of the identifier field.

[0016] In one possible implementation, encoding the transmitted data according to the identification field includes:

[0017] If the transmitted data is determined to be online log, obtain the real-time update frequency of the online log;

[0018] The block size for online log segmentation and transmission is adjusted according to the real-time update frequency.

[0019] In one possible implementation, encoding the transmitted data according to the identification field includes:

[0020] If the type of data to be transferred is determined to be zero-copy data transfer, the memory-mapped file interface is called to generate a memory-mapped file object.

[0021] Secondly, embodiments of this application provide a data transmission method, which is applied at a receiving end and includes:

[0022] A data transmission channel is established with the source end, and the source end sets an identification field in the communication protocol of the data transmission channel to distinguish the transmission mode;

[0023] It receives encoded transmission data sent by the source and decodes the encoded transmission data according to the identification field. The encoded transmission data is obtained by the source after encoding the transmission data according to the identification field. The encoding process includes metadata encapsulation of the chunked transmission data and direct transmission interface calls for zero-copy transmission data. The decoding process includes metadata parsing of the chunked transmission data and direct writing of the byte stream of zero-copy transmission data.

[0024] In one possible implementation, decoding the encoded transmission data based on the identifier field includes:

[0025] Based on the identifier field, if the type of encoded transmission data is determined to be chunked transmission data, the file name and block number in the structured data of the encoded transmission data are parsed, and the data is concatenated according to the file name and block number.

[0026] And / or, if the identifier field determines that the type of encoded transmission data is zero-copy transmission data, directly receive the byte stream and write it to a local file.

[0027] Thirdly, embodiments of this application provide a data transmission apparatus, which is applied at a source end, and the apparatus includes:

[0028] The configuration module is used to establish a data transmission channel with the receiving end and to set an identification field in the communication protocol of the data transmission channel to distinguish the transmission mode.

[0029] The encoding module is used to encode the transmitted data according to the identifier field to obtain encoded transmitted data. The encoding process includes metadata encapsulation of chunked transmitted data and direct transmission interface calls for zero-copy transmitted data.

[0030] The sending module is used to send encoded transmission data to the receiving end through the data transmission channel, so that the receiving end can receive the encoded transmission data and decode the encoded transmission data according to the identification field. The decoding process includes parsing the metadata of the chunked transmission data and directly writing the byte stream of the zero-copy transmission data.

[0031] In one possible implementation, the encoding module is specifically used for:

[0032] If the type of transmitted data is determined to be chunked data, the transmitted data is encapsulated into structured data based on the file name and block sequence number in the identifier field, and a byte buffer object is generated.

[0033] And / or, if it is determined that the type of data to be transferred is zero-copy data transfer, call the file region interface to generate a file region object.

[0034] In one possible implementation, the setting module is also used for:

[0035] Custom codecs are dynamically inserted into the Netty pipeline of the data transmission channel. The Netty pipeline is used to handle network events in the Netty framework, and the custom codec is used to switch the encoding processing logic based on the value of the identifier field.

[0036] In one possible implementation, the encoding module is specifically used for:

[0037] If the transmitted data is determined to be online log, obtain the real-time update frequency of the online log;

[0038] The block size for online log segmentation and transmission is adjusted according to the real-time update frequency.

[0039] In one possible implementation, the encoding module is specifically used for:

[0040] If the type of data to be transferred is determined to be zero-copy data transfer, the memory-mapped file interface is called to generate a memory-mapped file object.

[0041] Fourthly, embodiments of this application provide a data transmission apparatus, which is applied at a receiving end, and the apparatus includes:

[0042] The connection module is used to establish a data transmission channel with the source end. The source end sets an identification field in the communication protocol of the data transmission channel to distinguish the transmission mode.

[0043] The decoding module is used to receive encoded transmission data sent by the source end and decode the encoded transmission data according to the identification field. The encoded transmission data is obtained by the source end after encoding the transmission data according to the identification field. The encoding process includes metadata encapsulation of the chunked transmission data and direct transmission interface calls for zero-copy transmission data. The decoding process includes metadata parsing of the chunked transmission data and direct writing of the byte stream of zero-copy transmission data.

[0044] In one possible implementation, the decoding module is specifically used for:

[0045] Based on the identifier field, if the type of encoded transmission data is determined to be chunked transmission data, the file name and block number in the structured data of the encoded transmission data are parsed, and the data is concatenated according to the file name and block number.

[0046] And / or, if the identifier field determines that the type of encoded transmission data is zero-copy transmission data, directly receive the byte stream and write it to a local file.

[0047] Fifthly, embodiments of this application provide an electronic device, including: a memory and a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the processor to perform the first aspect, various possible implementations of the first aspect, the second aspect and / or various possible implementations of the second aspect as described above.

[0048] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect, various possible implementations of the first aspect, the second aspect, and / or various possible implementations of the second aspect as described above.

[0049] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect, various possible implementations of the first aspect, the second aspect, and / or various possible implementations of the second aspect as described above.

[0050] The data transmission method, device, medium, and program products provided in this application are applied at the source end. By setting an identification field in the communication protocol and dynamically switching the encoding / decoding logic based on this identification field, the technical problem of single-channel incompatibility between chunked transmission and zero-copy transmission in the Netty framework can be solved. The identification field allows the encoder to determine the transmission mode and choose whether to encode the data using the identification field based on the log file type. The decoder can then decide whether to parse the data or write it directly to the file based on the identification field. Based on this, multiple transmission modes can be used for coordinated transmission within a single channel, reducing resource consumption such as threads and memory at the source and receiving ends, improving resource utilization, and reducing data transmission complexity and operational difficulty. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0052] Figure 1 This is a schematic diagram of the existing database log transmission process based on the Netty framework;

[0053] Figure 2 A flowchart illustrating the data transmission method provided in the embodiments of this application. Figure 1 ;

[0054] Figure 3A flowchart illustrating the data transmission method provided in the embodiments of this application. Figure 2 ;

[0055] Figure 4 A schematic diagram of a communication protocol for transmitting online logs and archived logs through the same data transmission channel, as provided in the embodiments of this application;

[0056] Figure 5 A schematic diagram of a Netty-based single-channel log transmission process provided for an embodiment of this application;

[0057] Figure 6 Schematic diagram of the data transmission device provided in the embodiments of this application Figure 1 ;

[0058] Figure 7 Schematic diagram of the data transmission device provided in the embodiments of this application Figure 2 ;

[0059] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0060] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0061] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0062] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0063] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0064] In the embodiments of this application, the use of terms such as "first" and "second" is to distinguish between identical or similar items that have essentially the same function and effect. For example, "first electronic device" and "second electronic device" are merely used to distinguish different electronic devices and do not limit their order of execution. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0065] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0066] For example, in a database application scenario involving real-time data synchronization, data synchronization typically involves four stages. The first stage is extracting data from the source database, the second stage is filtering the extracted data, the third stage is transforming the data, and the fourth stage is writing the data to the target database.

[0067] For the first stage of data extraction, in a typical low-intrusion data synchronization solution, the log files on the source database server need to be transferred to the server where the synchronization tool is located. This step usually uses a proxy server to read and send the files, and the receiving end needs a proxy client to receive the files.

[0068] In some databases, log files are divided into online logs and archived logs. Online logs change in real time, and proxy servers send online log data in chunks. Archived logs are complete and static log files, and they can also be transmitted in chunks.

[0069] The proxy client receives log file chunks from the proxy server and writes them sequentially to the target's local machine. If this transmission process is based on a communication flow implemented using the Netty framework, then to improve log file transmission performance, archived logs will use the zero-copy transmission capabilities provided by the Netty framework for sending and receiving data.

[0070] However, the communication protocol using zero-copy transmission is incompatible with the communication protocol for sending file data in chunks. Therefore, if we want to use chunked transmission to transmit online logs and zero-copy transmission to transmit archived files, we need to establish two independent data transmission channels to send online logs and archived logs respectively in order to avoid communication protocol conflicts.

[0071] Figure 1 This is a schematic diagram of the existing database log transmission process based on the Netty framework, such as... Figure 1 As shown, the source side includes a source database and a proxy server, while the target side includes a proxy client, synchronization tools, and a target database. On the source side, the proxy server can be used to read log files and query logs. Two independent data transmission channels are established between the proxy server and the proxy client: an online log sending channel and an archived log sending channel.

[0072] Since the online log sending channel and the archived log sending channel are two independent data transmission channels, the problem of communication protocol conflict can be overcome. In this way, online log data can be transmitted in a chunked manner through the online log sending channel, and archived log data can be transmitted in a zero-copy manner through the archived log sending channel.

[0073] On the target side, the synchronization tool can query the log writing progress of the proxy client, and the proxy client can also write the received log file. The synchronization tool can read the written log file, thus enabling the synchronization tool to write data from the source database into the target database.

[0074] For example, in the transmission of online logs, the proxy server reads the online log file in chunks, encapsulates the data into message blocks (containing metadata such as filename and block sequence number), and sends these chunks to the proxy client via a Netty channel. Upon receiving the data, the proxy client can concatenate the data according to the block sequence number and append it to the local online log file. This solution relies on Netty's codecs (such as the length-field-based frame decoder LengthFieldBasedFrameDecoder) to parse the chunked data, but it is not compatible with zero-copy transmission.

[0075] For the transmission of archived logs, the proxy server directly sends the archived log files using Netty's zero-copy capabilities (such as the FileRegion interface), avoiding multiple copies of data between user space and kernel space, which can significantly improve transmission efficiency. However, zero-copy transmission requires a dedicated channel, and the data is not processed by the codec, which means that the proxy client needs to receive and write the archived logs directly through a separate channel.

[0076] Since online logs need to be transmitted in chunks (and rely on codecs), while archived logs need to be transmitted with zero copies (bypassing codecs), existing solutions require establishing separate channels for the two different log file types. As can be seen, this will lead to channel redundancy.

[0077] Existing log transmission methods rely on dual-channel transmission, which not only increases system resource consumption (such as threads and memory) but also leads to increased protocol management complexity, such as the need to maintain multiple channel states and handle channel reconnection issues. Furthermore, the dual-channel architecture may require proxy clients to monitor multiple connections simultaneously, increasing development and maintenance costs, and is prone to channel blocking or resource contention in high-concurrency scenarios.

[0078] Specifically, in existing technologies, databases based on the Netty framework need to establish separate channels for online logs (chunked transmission) and archived logs (zero-copy transmission) during log transmission. For example, in real-time transaction synchronization scenarios in the financial industry, the proxy server needs to maintain two Netty connections simultaneously: one channel for sending online logs in chunks (requiring an encoder to parse block sequence numbers and filenames), and the other channel for sending archived logs with zero copies (transmitted directly through the FileRegion interface). This dual-channel architecture not only suffers from channel redundancy but also leads to numerous problems such as high resource consumption, compatibility defects due to protocol conflicts, high operational complexity, and transmission efficiency bottlenecks.

[0079] For example, to be compatible with chunked transmission and zero-copy transmission, two independent channels need to be established. Each channel needs to maintain its own thread pool and buffer, which means that the proxy server and proxy client need to maintain multiple threads and multiple connections, consuming additional central processing unit (CPU) and memory resources. Especially in high-concurrency scenarios, resource consumption increases significantly.

[0080] In the Netty framework, there is a logical conflict between the codec and the zero-copy interface. Chunked transmission relies on the codec to parse the message structure, while zero-copy transmission needs to bypass Netty's codec to send files directly; the two cannot coexist in a single channel. Existing solutions address this issue through channel separation, but at the expense of transmission efficiency and system simplicity.

[0081] Furthermore, the dual-channel architecture requires additional network policy configuration (such as port opening and firewall rules), handling of multi-channel anomalies (such as needing to re-establish two connections during disconnection and reconnection, and data out-of-order processing), and the proxy client's log writing logic needs to dynamically distinguish between channel types, which increases code complexity and the difficulty of system debugging and maintenance. Moreover, the chunked transmission of online logs requires an encoding-decoding process, introducing additional latency; while zero-copy transmission of archived logs is efficient, the establishment and management process of independent channels reduces overall transmission efficiency.

[0082] This application provides a solution that aims to achieve the coordination of block transmission and zero-copy transmission within a single channel through a preset communication protocol, thereby solving the above-mentioned technical problems, reducing system complexity, optimizing resource consumption, and improving transmission efficiency.

[0083] Starting with the shortcomings of existing dual-channel technologies, the inventors first analyzed the compatibility issues between the codec and the zero-copy interface in the Netty framework. For example, zero-copy transmission requires directly sending FileRegion objects, while the codec relies on byte buffers (ByteBuf) for message parsing; the two cannot coexist.

[0084] To address this, a technical concept for the method in this application is proposed: a transmission mode is distinguished by a preset protocol header, enabling the encoder / decoder to dynamically switch processing logic based on the protocol header. For example, a "transmission mode identifier" field (e.g., 1 byte to identify chunked transmission or zero-copy) is added to the protocol header. Before sending data, the proxy server selects the transmission mode based on the log type, and the proxy client decoder determines whether to parse the data or write it directly to the file based on the identifier.

[0085] Furthermore, by optimizing the protocol design, ensuring protocol compatibility based on message serialization rules (such as fixed message header length and field alignment), and dynamically inserting custom codecs into the Netty pipeline, seamless switching between two transmission modes within a single channel is achieved, thus solving the shortcomings of the existing dual-channel architecture.

[0086] In view of this, embodiments of this application provide a data transmission method. This method, through a designed adaptive multi-mode transmission protocol, achieves coordinated control of chunked transmission and zero-copy transmission in single-channel communication, resolving the resource redundancy and protocol conflict issues caused by existing dual-channel architectures. This protocol, based on differentiated message command design, enables the encoder / decoder to dynamically identify the transmission mode (chunked transmission or zero-copy transmission), and achieves collaborative processing between the proxy client and proxy server through metadata in the message header (such as file type and transmission mode identifier). This allows for compatibility with both transmission mechanisms in a single channel while maintaining the high-performance characteristics of the Netty framework.

[0087] The method provided in this application is applicable to various database application scenarios, such as distributed real-time data synchronization systems, including real-time transmission of database logs and log synchronization between edge computing nodes and a central server. In the network architecture, a proxy server can be deployed on the source database server, responsible for reading online logs (which change in real time) and archived logs (with fixed content); the proxy client can be deployed on the target server, requiring dynamic handling of the reception and storage of both log types. The database system can meet the requirements of high throughput and low latency transmission, and the resource consumption of the proxy server (such as CPU and memory) can be precisely controlled.

[0088] The technical solutions of this application will be described in detail below with reference to specific embodiments. The specific embodiments described below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0089] Figure 2 A flowchart illustrating the data transmission method provided in the embodiments of this application. Figure 1 This method is applied to the source end, and the executing entity can be an electronic device with corresponding data storage and computing capabilities at the source end, such as a computer, server, or server cluster. Figure 2 As shown, the method includes:

[0090] S201, establish a data transmission channel with the receiving end, and set an identification field in the communication protocol of the data transmission channel to distinguish the transmission mode.

[0091] For example, a communication protocol can be understood as a standardized structure that defines data transmission formats and interaction rules, used to regulate the data exchange methods between communicating parties. For instance, the Transmission Control Protocol (TCP) / Internet Protocol Address (IP) protocol defines the header field and data body structure of data packets. The communication protocol set in this embodiment may include identifier fields such as identifiers that distinguish transmission modes, file names, and block sequence numbers.

[0092] The identification field can be understood as a field in the communication protocol used to distinguish different transmission modes. For example, a 1-byte value indicates chunked transmission or zero-copy transmission. For instance, the transmission mode identifier 0x01 indicates chunked transmission mode, and 0x02 indicates zero-copy transmission mode.

[0093] For example, the electronic device executing the method of this application embodiment can be a proxy server at the source end. The proxy server can establish a data transmission channel for data communication and interaction with the electronic device at the target end (proxy client). When establishing the data transmission channel, the proxy server can configure the communication protocol of the data transmission channel. For example, it can be configured to use the identifier 0x01 to represent data transmitted in chunks.

[0094] S202, the transmitted data is encoded according to the identifier field to obtain encoded transmitted data. The encoding process includes metadata encapsulation of the chunked transmitted data and direct transmission interface calls for zero-copy transmitted data.

[0095] For example, metadata encapsulation can be understood as encapsulating additional information such as filenames and block sequence numbers into the transmitted data to guide data processing at the receiving end. For instance, in chunked transmission, information such as filenames and block sequence numbers are encapsulated into ByteBuf objects and sent via the Netty framework.

[0096] Direct transport interface calls can be understood as bypassing the codec and directly calling the underlying transport interface (such as FileRegion) to send file data. For example, in archived log transport, the proxy server sends file data directly through the FileRegion interface without codec processing.

[0097] For example, when reading logs, the proxy server can identify whether a log file is an online log or an archived log. For instance, if log file A is identified as an online log, each block of data in log file A can be encoded. For example, the identifier field 0x01 can be encapsulated into each block of data in log file A. Conversely, if log file B is identified as an archived log, the FileRegion interface can be directly called to transmit data to log file B using a zero-copy transfer method.

[0098] S203, send encoded transmission data to the receiving end through the data transmission channel so that the receiving end can receive the encoded transmission data and decode the encoded transmission data according to the identification field. The decoding process includes parsing the metadata of the chunked transmission data and directly writing the byte stream of the zero-copy transmission data.

[0099] For example, direct byte stream writing can be understood as the receiving end directly writing the received byte stream to a local file without parsing the structured data generated by the codec. For instance, after receiving zero-copy data from archived logs, the proxy client directly writes the byte stream to a local file without parsing the block sequence number or filename.

[0100] This method enables coordinated processing of block transmission and zero-copy transmission within a single data transmission channel by setting an identifier field in the communication protocol.

[0101] For example, firstly, during the data transmission channel establishment phase, an identification field (such as a 1-byte transmission mode identifier) ​​is set for the communication protocol to distinguish the data transmission mode. Secondly, during the encoding process, the proxy server can select the corresponding encoding logic based on the value of the identification field: if it is a chunked transmission mode, metadata such as filename and block sequence number can be encapsulated into a ByteBuf object; if it is a zero-copy transmission mode, the FileRegion interface can be directly called to send the transmission data.

[0102] After receiving the encoded transmission data at the receiving end (proxy client), decoding can be performed. The proxy client can select the corresponding decoding logic based on the value of the identifier field: if it is a chunked transmission mode, it parses the metadata and concatenates the data according to the block sequence number; if it is a zero-copy transmission mode, it directly receives the byte stream and writes it to a local file.

[0103] As can be seen, the entire process described above can achieve seamless switching between two transmission modes within a single channel through dynamic identification of the identifier field, without the need to maintain multiple channels.

[0104] The data transmission method provided in this application, by setting an identifier field in the communication protocol and dynamically switching the encoding / decoding logic based on the identifier field, can solve the technical problem that single-channel transmission cannot be compatible with block transmission and zero-copy transmission in the Netty framework.

[0105] Specifically, the setting of the identifier field enables the encoder to determine the transmission mode based on the type of the log file itself and choose whether to encode the data using the identifier field, while the decoder can decide whether to parse the data or write it directly to the file based on the identifier field.

[0106] For example, in online log transmission, the proxy server sends chunked data containing block sequence numbers, and the proxy client parses and concatenates them in order. In archived log transmission, the proxy server sends files directly through a zero-copy interface, and the proxy client writes directly to the file without parsing. This solution achieves coordination between the two transmission modes through a single channel, reducing the resource consumption of the proxy server and proxy client in terms of threads, memory, etc., and can improve resource utilization, especially in high-concurrency scenarios.

[0107] Meanwhile, the complexity of communication protocol management is reduced, operational complexity decreases, and system stability is improved. Furthermore, the dynamic switching logic avoids data transmission interruptions caused by protocol conflicts in existing dual-channel architectures, making log file transmission more efficient and reliable.

[0108] In one possible implementation, encoding the transmitted data according to the identifier field to obtain encoded transmitted data includes: if the type of transmitted data is determined to be chunked transmitted data, encapsulating the transmitted data into structured data based on the file name and block number in the identifier field, and generating a byte buffer object; and / or, if the type of transmitted data is determined to be zero-copy transmitted data, calling the file region interface to generate a file region object.

[0109] For example, structured data can be understood as data objects containing fixed field formats used to transmit metadata (such as filenames and block numbers). For instance, in chunked transmission, filenames and / or block numbers are individually or combined into fixed-format byte buffer objects, i.e., ByteBuf objects.

[0110] A ByteBuf object can be understood as a buffer object in the Netty framework used to encapsulate byte data, supporting dynamic read and write operations. For example, during chunked transmission, the filename and chunk sequence number are written to the ByteBuf and sent to the proxy client.

[0111] A FileRegion object can be understood as an interface object in the Netty framework used to implement zero-copy data transfer, directly manipulating the data transfer channel to transmit data. For example, when transmitting archived logs, the proxy server directly sends the archived log file data by calling the FileRegion interface, without the need for codec processing.

[0112] During the encoding process, the proxy server selects the corresponding encoding logic based on the value of the identifier field: for chunked transmission mode, it encapsulates metadata such as filename and block sequence number into structured data and generates a ByteBuf object; for zero-copy transmission mode, it calls the file region interface to generate a FileRegion object. The ByteBuf object is used to support metadata parsing for chunked transmission, while the FileRegion object directly operates the data transmission channel, avoiding multiple copies of data between user space and kernel space. This process, through dynamic identification of the identifier field, enables a single channel to be compatible with both transmission modes, eliminating the need to maintain multiple channels.

[0113] In this embodiment, precise control over different transmission modes is achieved through structured data encapsulation for chunked transmission and file region interface calls for zero-copy transmission. For example, in online log transmission, the ByteBuf object enables the receiving end to parse filenames and block numbers and concatenate data in order; in archived log transmission, the FileRegion interface directly operates the data transmission channel, significantly reducing transmission latency. This solution dynamically adapts transmission modes within a single channel, avoiding resource waste caused by protocol conflicts in existing dual-channel architectures, while improving transmission efficiency and system stability.

[0114] In existing application scenarios, because proxy servers need to maintain multiple threads and connections simultaneously to support dual-channel transmission, resource contention and channel blocking issues are prone to occur under high concurrency. To address this, the method provided in this application dynamically inserts a custom codec into the Netty pipeline, achieving coordinated processing of chunked transmission and zero-copy transmission through a single channel, without the need for additional maintenance of multiple channels.

[0115] In one possible implementation, before encoding the transmitted data according to the identifier field, the method further includes: dynamically inserting a custom codec into the Netty pipeline of the data transmission channel, the Netty pipeline being used to handle network events in the Netty framework, and the custom codec being used to switch the encoding processing logic according to the value of the identifier field.

[0116] For example, a Netty pipeline can be understood as a chained structure within the Netty framework used to process network events, including components such as codecs and processors. For instance, a proxy server's Netty pipeline includes a custom codec and a log processing processor.

[0117] A custom codec can be understood as a component that dynamically switches encoding logic based on an identifier field, supporting collaborative processing of chunked and zero-copy transmissions. For example, a custom codec can encapsulate metadata during chunked transmission and call the FileRegion interface during zero-copy transmission.

[0118] For example, before encoding processing, a custom codec can be dynamically inserted into the Netty pipeline in the proxy server. This custom codec can switch encoding processing logic based on the value of the identifier field: if it is a chunked transfer mode, it can encapsulate metadata; if it is a zero-copy transfer mode, it can call the FileRegion interface. This process, through a dynamic insertion mechanism, enables the Netty pipeline to adapt to different transfer modes without the need to maintain multiple channels.

[0119] By dynamically processing two transmission modes through a single channel, the proxy server does not need to establish separate data transmission channels for online logs and archived logs, which can significantly reduce the consumption of resources such as thread pools and buffers.

[0120] For example, in real-time transaction synchronization scenarios in the financial industry, the proxy server can simultaneously handle the chunked transmission of high-frequency online logs and the zero-copy transmission of low-frequency archived logs through a single channel, avoiding resource contention and blocking issues between multiple channels. This solution can reduce the CPU and memory load of the proxy server, improve the stability and scalability of the database system, and also reduce the complexity of network configuration (such as eliminating the need to open multiple ports or configure multiple firewall rules).

[0121] In this embodiment, Netty pipelines achieve flexible support for different transmission modes by dynamically inserting custom codecs. For example, the proxy server does not need to configure separate channels for chunked transmission and zero-copy transmission, but can dynamically switch encoding logic through custom codecs. This reduces the configuration complexity of Netty pipelines while improving transmission efficiency and system scalability.

[0122] For example, when transmitting online logs in chunks, if chunks are divided based on a fixed chunk size, it may be difficult to balance transmission latency and throughput. For instance, if a small chunk size is set for chunking, although transmitting smaller chunks can reduce latency, it will increase the total number of chunks, leading to increased throughput; if a large chunk size is set for chunking, although it can reduce throughput, it will increase transmission latency.

[0123] To address this, the method provided in this application employs a dynamic block size adjustment mechanism. This mechanism dynamically selects the appropriate block size based on the real-time update frequency of the online logs, thus balancing transmission latency and throughput. For example, small blocks (e.g., 4 kilobytes, 4KB) are used for high-frequency updates, while large blocks (e.g., 64KB) are used for low-frequency updates. This dynamic switching is achieved through the "block size identifier" field in the protocol header.

[0124] In one possible implementation, encoding the transmitted data according to the identification field includes: if the transmitted data is determined to be an online log, obtaining the real-time update frequency of the online log; and adjusting the block size when transmitting the online log in blocks according to the real-time update frequency.

[0125] For example, the real-time update frequency can be understood as the update rate of the online logs, used to dynamically adjust the block size to optimize transmission efficiency. For instance, in high-frequency trading scenarios, where the real-time update frequency is high, 4KB small block transmission can be used to reduce latency. The proxy server can obtain the real-time update frequency of the online logs by querying the online log write rate of the source database or by switching the frequency.

[0126] Block size can be understood as the size of a single data block in a chunked transmission, and it can be used to balance latency and throughput. For example, in low-frequency update scenarios, 64KB large blocks are used for transmission to improve throughput.

[0127] During the encoding process, the proxy server can dynamically select the block size based on the real-time update frequency of the online logs. For example, when the online logs are updated frequently, small blocks of 4KB or 8KB are used; when the online logs are updated infrequently, large blocks of 64KB are used. It should be understood that in practical application scenarios, the block size is not limited to 4KB, 8KB, or 64KB, and can be set to any size according to the needs of the application scenario.

[0128] When adjusting the block size for online log chunked transmission based on the real-time update frequency, this can be achieved dynamically by switching the block size using the "block size identifier" field in the protocol header. This allows chunked transmission to flexibly adapt to different load scenarios.

[0129] For example, by dynamically adapting the block size, the real-time performance and throughput of online log transmission can be maximized. In high-concurrency database operation scenarios, when the proxy server detects frequent updates to the online logs, it automatically switches to small-block transmission by adjusting the block size value of the "block size identifier" in the protocol header, reducing latency for single-block data. Conversely, during low-load periods at night, it switches to large-block transmission by adjusting the protocol header, improving overall throughput. This solution reduces the total number of blocks in the chunked transmission, lowers the splicing overhead at the receiving end, and avoids resource waste or latency issues caused by fixed block sizes, enabling the database system to flexibly adapt to transmission needs under different load scenarios.

[0130] In this embodiment, a balance between real-time performance and throughput in online log transmission is achieved by dynamically adjusting the block size. For example, in high-concurrency scenarios, small block transmission can reduce single-block latency; in low-load scenarios, large block transmission can improve overall throughput. Based on this solution, the total number of blocks for segmented transmission can be reduced, lowering the splicing overhead at the receiving end, while avoiding resource waste or latency issues caused by fixed block sizes.

[0131] For example, in zero-copy archive log transmission, the existing file channel (FileChannel) reads files block by block, resulting in high CPU and input / output (I / O) overhead, affecting transmission efficiency. Therefore, the method provided in this application adopts a memory-mapped ByteBuffer processing method to directly map archive files to the memory address space, and combines this with the FileRegion interface to achieve zero-copy transmission.

[0132] In one possible implementation, encoding the transmitted data based on the identifier field includes: if it is determined that the type of transmitted data is zero-copy transmitted data, calling the memory-mapped file interface to generate a memory-mapped file object.

[0133] For example, a memory-mapped file interface can be understood as an interface that directly maps a file to a memory address space to achieve zero-copy transfer. For instance, a proxy server uses the FileChannel.map() interface to map archived logs to memory and directly transfers them as a MappedByteBuffer.

[0134] A memory-mapped file object, also known as a MappedByteBuffer object, can be understood as a byte buffer object for a memory-mapped file, supporting direct reading and writing of file content. For example, a proxy server can directly send archived file data through a MappedByteBuffer, without needing to read it block by block. During the zero-copy transfer phase, the proxy server can call the memory-mapped file interface to generate a MappedByteBuffer object, directly mapping the file to the memory address space. This process can be implemented through the FileChannel.map() interface, avoiding block-by-block read operations based on FileChannel, thus reducing CPU and I / O overhead.

[0135] By using memory mapping, the proxy server avoids reading archived log files block by block and encapsulating them into ByteBuf objects. Instead, it directly transmits the memory-mapped region via the FileRegion interface. The proxy client also avoids piecing together files block by block, directly writing to the corresponding location in the memory-mapped file. For example, in archived log transmission scenarios, the proxy server maps the entire archived log at once using the FileChannel.map() interface, significantly reducing CPU and I / O overhead and decreasing the number of database system calls. This solution improves the efficiency of zero-copy transmission while reducing resource consumption, making archived log transmission faster and more stable, especially suitable for large archived log transmission scenarios.

[0136] In this embodiment of the application, the efficiency of zero-copy transmission is optimized by calling the memory-mapped file interface, reducing the CPU and I / O overhead of zero-copy transmission and improving the transmission efficiency of archived logs.

[0137] In one possible implementation, encoding the transmitted data according to the identifier field includes: if the transmitted data is determined to be an online log, obtaining the real-time update frequency of the online log and adjusting the block size when transmitting the online log in blocks according to the real-time update frequency; if the transmission mode of the transmitted data is determined to be zero-copy transmission, calling the memory-mapped file interface to generate a memory-mapped file object.

[0138] For example, dynamically adjusting the block size can be understood as adjusting the block size for chunked transmission based on the real-time update frequency of the online logs. For instance, using 4KB small blocks for high-frequency updates and 64KB large blocks for low-frequency updates. The memory-mapped file interface can be understood as an interface that directly maps a file to a memory address space, used to achieve zero-copy transmission. For example, the proxy server generates a MappedByteBuffer object through the FileChannel.map() interface.

[0139] During the encoding process, the proxy server can dynamically adjust the block size or invoke the memory-mapped file interface based on the value of the identifier field. During chunked transmission, the block size is dynamically adjusted to optimize transmission efficiency. During zero-copy transmission, the memory-mapped file interface is invoked to reduce CPU and I / O overhead. This process, through the dynamic identification of the identifier field, enables a single channel to flexibly adapt to different transmission requirements.

[0140] In this embodiment, the optimal configuration of block transfer and zero-copy transfer is achieved by dynamically adjusting the block size and the memory-mapped file interface. For example, in high-frequency update scenarios, small block transfers reduce latency; in large file transfer scenarios, the memory-mapped file interface improves transfer efficiency. This solution comprehensively optimizes the performance of single-channel transfer, enabling the system to flexibly adapt to the transfer requirements of different application scenarios.

[0141] Based on the method provided in the above embodiments of this application, this method can also realize dynamic transmission mode switching based on network status.

[0142] For example, a network status monitoring module can be deployed at the source end (such as a proxy server) to collect metrics such as bandwidth, latency, and packet loss rate in real time. Based on the type of file being transmitted (online logs / archived logs), the system can dynamically determine whether to use chunked or zero-copy transmission.

[0143] For example, when high bandwidth and low latency are detected, zero-copy transmission of archived logs is prioritized; when bandwidth is limited or there is a high packet loss rate, chunked transmission is switched to ensure reliability. The dynamic switching logic is implemented through the "Transmission Mode Identifier" field in the protocol header, and the proxy server can automatically generate corresponding message instructions based on the network status.

[0144] Based on this, by adapting to the network environment in real time, the system maximizes the use of available bandwidth and reduces the risk of transmission interruption. For example, in edge computing scenarios, when the proxy server detects network fluctuations, it automatically switches to chunked transmission mode to avoid file corruption due to packet loss during zero-copy transmission; while when the network is stable, zero-copy mode significantly improves the transmission efficiency of archived logs.

[0145] Based on the method provided in the above embodiments of this application, this method can also achieve transmission integrity assurance based on hash verification.

[0146] For example, a "hash checksum field" can be added to the communication protocol. Before sending file data, the source calculates its hash value (e.g., SHA-256) and appends it to the file start message (SYNC_ONLINE_FILE_START or SEND_ARCHIVE_FILE_START). After receiving the file, the receiving end recalculates the hash value of the local file and compares it with the hash value sent by the source. If they do not match, a retransmission mechanism is triggered (e.g., sending a RETRANSMISSION_REQUEST message).

[0147] Therefore, end-to-end data integrity verification avoids file errors caused by network packet loss or data corruption. For example, in archived log transmission, if the agent client detects a hash mismatch, it can immediately request the retransmission of specific file blocks instead of retransmitting the entire file, thereby reducing redundant traffic and improving transmission reliability.

[0148] Based on the method provided in the above embodiments of this application, the method can also implement a transmission recovery mechanism based on breakpoint resumption.

[0149] For example, in chunked transmission mode, the source records the sequence number of each sent block for an online log file, and the receiving end maintains a reception status table. When transmission is interrupted, the receiving end sends a RECOVERY_REQUEST message to the source, containing a list of currently received block sequence numbers. The source skips the sent blocks according to the status table and only retransmits the incomplete blocks. Meanwhile, in zero-copy transmission, the source records the number of bytes sent and retransmits the remaining data from the point of interruption after an interruption.

[0150] Based on this, the retransmission overhead after transmission interruption can be reduced, and transmission failures caused by network fluctuations can be avoided. For example, in the synchronization scenario between edge computing nodes and central servers, if transmission is interrupted, the proxy client can quickly locate the position of incomplete blocks or bytes and request only the retransmission of the missing parts, rather than retransmitting the entire file, thereby improving system robustness and resource utilization.

[0151] This application also provides a data transmission method. Figure 3 A flowchart illustrating the data transmission method provided in the embodiments of this application. Figure 2 This method is applied at the receiving end (i.e., the target end). The executing entity of this method can be an electronic device with corresponding data storage and computing capabilities at the receiving end, such as a computer, server, or server cluster. Figure 3 As shown, the method includes:

[0152] S301, establish a data transmission channel with the source end, and the source end sets an identification field in the communication protocol of the data transmission channel to distinguish the transmission mode.

[0153] For example, a communication protocol can be understood as a standardized structure that defines data transmission formats and interaction rules, which can be used to regulate the data exchange methods between communicating parties. The identification field can be understood as a field in the communication protocol used to distinguish different transmission modes. The specific implementation of this step can be found in the description of any of the above embodiments.

[0154] S302, receive the encoded transmission data sent by the source end, and decode the encoded transmission data according to the identification field. The encoded transmission data is obtained by the source end after encoding the transmission data according to the identification field. The encoding process includes metadata encapsulation of the block transmission data and direct transmission interface call for zero-copy transmission data. The decoding process includes metadata parsing of the block transmission data and direct writing of the byte stream of zero-copy transmission data.

[0155] For example, encoded transmission data is data obtained by encoding the transmission data of online logs or archived logs at the source end. The encoding and decoding processes are similar to the specific implementation methods described in any of the above embodiments, and can be referred to the descriptions in the above embodiments.

[0156] The data transmission method provided in this application embodiment is similar to the data transmission method provided in any of the above embodiments in terms of implementation principle and technical effect. For details, please refer to the description of any of the above embodiments. This application embodiment will not be repeated here.

[0157] As can be seen from the above analysis, since online log block transmission relies on the codec to parse the block sequence number and file name, while archived log zero-copy transmission needs to bypass the codec and send the file directly, there is a protocol conflict, and they cannot coexist in a single channel.

[0158] The method provided in this application can overcome protocol conflicts in communication protocols by using differentiated message body structures. These differentiated message body structures include, for example: chunked transmission message bodies contain metadata such as filenames and block sequence numbers, requiring codec processing; zero-copy transmission message bodies only contain file size and filenames, requiring no codec processing.

[0159] In one possible implementation, decoding the encoded transmission data based on the identifier field includes: if the identifier field determines that the type of the encoded transmission data is chunked transmission data, parsing the file name and block number in the structured data of the encoded transmission data, and concatenating the data according to the file name and block number; and / or, if the identifier field determines that the type of the encoded transmission data is zero-copy transmission data, directly receiving the byte stream and writing it to a local file.

[0160] For example, structured data can be understood as data objects containing fixed field formats used to transmit metadata (such as filenames and block numbers). For instance, in chunked transmission, filenames and / or block numbers are individually or combined into fixed-format byte buffer objects, i.e., ByteBuf objects.

[0161] Parsing structured data can be understood as extracting metadata such as filenames and block numbers from ByteBuf objects to guide file concatenation. For example, a proxy client parses the filenames and block numbers from a ByteBuf object and concatenates online log files according to these filenames and block numbers.

[0162] Byte stream writing can be understood as directly writing the received byte stream to a local file without parsing the structured data generated by the codec. For example, after receiving zero-copy data from archived logs, the proxy client directly writes the byte stream to a local file.

[0163] During the decoding phase, the proxy client selects the corresponding decoding logic based on the value of the identifier field. If the identifier field indicates that the current transmission mode is chunked, the filename and block number in the ByteBuf object are parsed, and data is concatenated according to the filename and block number. If the identifier field indicates that the current transmission mode is zero-copy, the byte stream can be directly received and written to a local file. This process, through dynamic identification of the identifier field, allows the receiving end to flexibly handle data from different transmission modes without maintaining multiple channels.

[0164] By differentiating the message body settings, the codec can process data in different transmission modes as needed. For example, in online log transmission, when the proxy server sends an ONLINE_FILE_DATA message, the encoder encapsulates metadata such as filename and block sequence number into a ByteBuf object. The proxy client receives the message and parses it through the decoder, then concatenates the data according to the filename and block sequence number to obtain the complete online log.

[0165] In archived log transmission, the proxy server directly sends log file data by calling the FileRegion interface. The proxy client's decoder can skip the encoding / decoding process, directly receive the byte stream, and write it to the file. This solution resolves the protocol conflict issue in a dual-channel architecture, improves transmission efficiency, ensures data integrity, and enables a single channel to flexibly adapt to different types of file transmission needs.

[0166] In this embodiment, efficient processing of different transmission modes is achieved through structured data parsing in chunked transmission and direct writing of byte streams in zero-copy transmission. For example, in online log transmission, the proxy client can ensure data integrity by parsing filenames and block sequence numbers; in archived log transmission, direct writing of byte streams avoids encoding and decoding overhead. This scheme can reduce the processing complexity of the receiving end while improving transmission efficiency and resource utilization.

[0167] In one possible implementation, decoding the encoded transmission data based on the identifier field includes: if the transmission mode of the encoded transmission data is determined to be zero-copy transmission, receiving the memory-mapped file object and writing it directly to a local file.

[0168] For example, a memory-mapped file object, or MappedByteBuffer object, can be understood as a byte buffer object for a memory-mapped file, supporting direct reading and writing of file content. For instance, after a proxy client receives a MappedByteBuffer object, it can directly write it to the receiving end's local file.

[0169] During the zero-copy transfer phase, the proxy client receives the MappedByteBuffer object and writes it directly to the local file. This process avoids the system calls of traditional file reading by directly reading and writing memory-mapped files, thus reducing I / O overhead.

[0170] In this embodiment, end-to-end optimization for zero-copy transmission is achieved through direct writing of MappedByteBuffer objects. For example, the proxy client does not need to parse the structured data generated by the codec; it can directly write to the memory-mapped file, reducing transmission latency and resource consumption. This scheme makes the transmission of archived logs faster and more stable, and is particularly suitable for scenarios involving the transmission of large archived logs.

[0171] Figure 4 This is a schematic diagram illustrating the communication protocol for transmitting online logs and archived logs through the same data transmission channel, as provided in an embodiment of this application. Figure 5 This is a schematic diagram of the Netty-based single-channel log transmission process provided in the embodiments of this application. The following is in conjunction with... Figure 4 and Figure 5 The data transmission method provided in the embodiments of this application will be further described.

[0172] For example, existing log file transmission schemes based on the Netty framework cannot simultaneously use zero-copy transmission and chunked transmission capabilities in a single channel. The data transmission method provided in this application embodiment can use both file transmission methods to transmit log data in a single channel.

[0173] Regarding the reading mechanism of online logs and archived logs, the method of this application embodiment designs a customized communication protocol. Through custom encoding and decoding, the receiving end can distinguish between zero-copy data and chunked file data sent by the source end in a single connection channel (data transmission channel), thereby correctly writing the file data into the local online log file and archived log file respectively. Figure 4 and Figure 5 As shown, the specific data transmission steps are as follows:

[0174] Step 1: Based on the reading mechanisms of online and archived logs, a custom communication protocol is defined. This protocol specifies the sending order of message commands for online and archived logs, as well as the format and content of each message command. The format and content of the message commands are shown in Table 1 below. The customized encoder and decoder need to encode and decode the data according to the defined format for sending and parsing received data.

[0175] Table 1. Examples of message command formats and content.

[0176]

[0177] Step 2: The source retrieves the file data to be sent from the cache queue. If it is online file data, it is configured according to a custom communication protocol, and the start, data, and end message commands for the online file are sent sequentially. The encoder encodes the data according to the custom message structure. If it is an archive file, it is configured according to a custom communication protocol, first sending the corresponding start message command, then sending the archive file data using the zero-copy interface, and finally sending the archive file end command. When sending file data in the zero-copy mode, the archive file data is not encoded by the encoder.

[0178] Step 3: The target device needs to decode the received data using a custom decoder according to the message format set in the communication protocol. If it is online file data, the data is appended to a local online log file with the same name based on the filename.

[0179] Step 4: If the message is a start message for sending an archive file, the archive file data acquisition process needs to be initiated in the decoder. Since the source archive file is sent using zero-copy, the data is not encoded by the encoder, so there is no need to decode it at the target end. After receiving the archive file sending start message, the target end writes the subsequently received byte stream data directly into the locally specified archive file according to the archive file size and filename in the start message. Writing to the archive file stops when the received data size equals the file size.

[0180] The following example illustrates data transmission through a data transmission channel established between the source proxy server and the target proxy client. Figure 4 and Figure 5 As shown, the proxy client sends a request to the proxy server to synchronize log information. The request includes the database type and the starting position number of the target log. Upon receiving the request, the proxy server initiates the log reading process. When sending online logs, if it is the first record in the online log, the proxy server first sends a SYNC_ONLINE_FILE_START message to the proxy client, containing the online log filename.

[0181] After receiving the online log start message, the proxy client creates a file locally using the filename specified in the message. The proxy server then sends the current log chunk data to the proxy client. The proxy client receives the data and writes it to the local online log file. After the online log data transmission is complete, the proxy server sends an ONLINE_FILE_END message. Upon receiving this message, the proxy client performs a flush operation, closing the write file stream. Finally, it sends a receive complete message to the proxy server.

[0182] When the proxy server sends an archive file, it first sends a `SEND_ARCHIVE_FILE_START` message. Upon receiving the archive log sending start message, the proxy client initiates a zero-copy data acquisition process in its decoder, including creating a local archive file and initializing received byte statistics. The proxy server uses a zero-copy mechanism to send file data. The proxy client executes the zero-copy data acquisition process, including receiving byte stream data, counting the number of bytes received, writing to the local archive file, and stopping data reception when the counted number of received bytes equals the file size. After sending the archive file, the proxy server sends an `ARCHIVE_FILE_END` message. Upon receiving the archive sending end message, the proxy client sends a `GET_ARCHIVE_FILE_SUCCESS` message.

[0183] The data transmission method provided in this application implements a customized encoder and decoder for database log file transmission through a custom communication protocol. This integrates file chunk transmission and zero-copy transmission in a single channel, significantly reducing the complexity of the file transmission module, improving log file transmission efficiency, and indirectly reducing real-time data synchronization latency. Simultaneously, it reduces resource consumption on the server hosting the log agent, lowering the system's operational and maintenance complexity.

[0184] For example, this application also provides a single-channel transmission method based on a custom codec, the method comprising: inserting a custom codec into a Netty pipeline, wherein the custom codec dynamically switches encoding logic according to the value of an identifier field; in chunked transmission mode, encapsulating filenames and block sequence numbers with metadata through the custom codec; and in zero-copy transmission mode, generating a FileRegion object by calling a file region interface through the custom codec.

[0185] The single-channel transmission method based on a custom codec provided in this application is similar in principle and technical effect to the data transmission method provided in any of the above embodiments. The specific implementation can be found in the descriptions in the above embodiments, and will not be repeated here.

[0186] This application also provides a Netty pipeline configuration structure, including: a custom codec configured in the Netty pipeline for dynamically switching encoding logic based on the value of an identifier field; a file transfer interface calling module connected to the custom codec for generating a FileRegion object by calling the file region interface in zero-copy transfer mode; and a metadata processing module connected to the custom codec for encapsulating filenames and block sequence numbers in chunked transfer mode.

[0187] For example, a Netty pipeline can be understood as a chained structure within the Netty framework used to process network events, including components such as codecs and processors. For instance, a proxy server's Netty pipeline includes a custom codec and a log processing processor.

[0188] A custom codec can be understood as a component that dynamically switches encoding logic based on an identifier field, supporting coordinated processing of chunked and zero-copy transmissions. For example, a custom codec encapsulates metadata during chunked transmission and calls the FileRegion interface during zero-copy transmission.

[0189] The FileRegion object can be understood as an interface in the Netty framework used to implement zero-copy data transfer, directly manipulating the file channel (FileChannel) to transmit data. For example, when transmitting archived logs, the proxy server sends file data directly through the FileRegion interface without the need for codec processing.

[0190] The metadata processing module can be understood as a component responsible for encapsulating and parsing metadata such as filenames and block numbers. For example, in chunked transmission, the metadata processing module writes the filename and block number into a ByteBuf object.

[0191] The Netty pipeline configuration structure enables coordinated processing of chunked and zero-copy transfers within a single channel by inserting a custom codec. The custom codec dynamically switches encoding logic based on the value of an identifier field: in chunked transfer mode, the metadata processing module encapsulates the filename and block sequence number; in zero-copy transfer mode, the file transfer interface calling module calls the FileRegion interface to generate a file data transfer object. This structure eliminates the need to maintain multiple channels and adapts to different transfer modes through dynamic encoding and decoding logic, significantly reducing system complexity.

[0192] In this embodiment, by using a custom codec's dynamic encoding logic, the Netty pipeline configuration structure achieves efficient integration of chunked transmission and zero-copy transmission within a single channel. For example, in online log transmission, the metadata processing module ensures that the agent client can parse the filename and block sequence number and concatenate the data in order; in archived log transmission, the file transmission interface call module directly operates the file channel, avoiding multiple copies of data between user space and kernel space. This structure significantly reduces the thread and memory resource consumption at both the source and receiver ends, improves transmission efficiency, and reduces the complexity of system operation and maintenance.

[0193] The Netty pipeline configuration structure provided in this application embodiment is similar in implementation principle and technical effect to the data transmission method provided in any of the above embodiments. Specifically, it can be described in the above embodiments, and will not be repeated here.

[0194] Figure 6 Schematic diagram of the data transmission device provided in the embodiments of this application Figure 1 ,like Figure 6 As shown in the figure, this application provides a data transmission device applied at a source end, the device comprising:

[0195] The setting module 601 is used to establish a data transmission channel with the receiving end and to set an identification field in the communication protocol of the data transmission channel to distinguish the transmission mode.

[0196] Encoding module 602 is used to encode the transmitted data according to the identification field to obtain encoded transmitted data. The encoding process includes metadata encapsulation of chunked transmitted data and direct transmission interface calls for zero-copy transmitted data.

[0197] The sending module 603 is used to send encoded transmission data to the receiving end through the data transmission channel, so that the receiving end can receive the encoded transmission data and decode the encoded transmission data according to the identification field. The decoding process includes parsing the metadata of the block transmission data and directly writing the byte stream of the zero-copy transmission data.

[0198] In one possible implementation, the encoding module 602 is specifically used for:

[0199] If the type of transmitted data is determined to be chunked data, the transmitted data is encapsulated into structured data based on the file name and block sequence number in the identifier field, and a byte buffer object is generated.

[0200] And / or, if it is determined that the type of data to be transferred is zero-copy data transfer, call the file region interface to generate a file region object.

[0201] In one possible implementation, the setting module 601 is further configured to:

[0202] Custom codecs are dynamically inserted into the Netty pipeline of the data transmission channel. The Netty pipeline is used to handle network events in the Netty framework, and the custom codec is used to switch the encoding processing logic based on the value of the identifier field.

[0203] In one possible implementation, the encoding module 602 is specifically used for:

[0204] If the transmitted data is determined to be online log, obtain the real-time update frequency of the online log;

[0205] The block size for online log segmentation and transmission is adjusted according to the real-time update frequency.

[0206] In one possible implementation, the encoding module 602 is specifically used for:

[0207] If the type of data to be transferred is determined to be zero-copy data transfer, the memory-mapped file interface is called to generate a memory-mapped file object.

[0208] The data transmission device provided in this application can be used to execute the data transmission method in any of the above-described source-side execution embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0209] Figure 7 Schematic diagram of the data transmission device provided in the embodiments of this application Figure 2 ,like Figure 7 As shown, this application embodiment provides a data transmission device, which is applied at a receiving end, and the device includes:

[0210] The connection module 701 is used to establish a data transmission channel with the source end, and the source end sets an identification field in the communication protocol of the data transmission channel to distinguish the transmission mode.

[0211] The decoding module 702 is used to receive the encoded transmission data sent by the source end and decode the encoded transmission data according to the identification field. The encoded transmission data is obtained by the source end after encoding the transmission data according to the identification field. The encoding process includes metadata encapsulation of the block transmission data and direct transmission interface call for zero-copy transmission data. The decoding process includes metadata parsing of the block transmission data and direct writing of the byte stream of zero-copy transmission data.

[0212] In one possible implementation, the decoding module 702 is specifically used for:

[0213] Based on the identifier field, if the type of encoded transmission data is determined to be chunked transmission data, the file name and block number in the structured data of the encoded transmission data are parsed, and the data is concatenated according to the file name and block number.

[0214] And / or, if the identifier field determines that the type of encoded transmission data is zero-copy transmission data, directly receive the byte stream and write it to a local file.

[0215] The data transmission device provided in this application can be used to execute the data transmission method in any of the embodiments executed by the receiving end of this application. Its implementation principle and technical effect are similar, and will not be described again here.

[0216] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 8 As shown, the electronic device of this embodiment may include: at least one processor 801; and a memory 802 communicatively connected to the at least one processor; wherein the memory 802 stores instructions that can be executed by the at least one processor 801, and the instructions are executed by the at least one processor 801 to cause the electronic device to perform the method as described in any of the above embodiments.

[0217] Optionally, the memory 802 can be either standalone or integrated with the processor 801.

[0218] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0219] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method of any of the foregoing embodiments.

[0220] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the foregoing embodiments.

[0221] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0222] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0223] It should be understood that the aforementioned processor can be a central processing unit (CPU) or other general-purpose processors. The processor can also be a digital signal processor (DSP) or an application-specific integrated circuit (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0224] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be various media that can store program code, such as USB flash drives, portable hard drives, read-only memory (ROM), disks or optical discs.

[0225] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof. Examples of storage media include Static Random-Access Memory (SRAM) or Electrically Erasable Programmable Read Only Memory (EEPROM).

[0226] Storage media can be, for example, erasable programmable read-only memory (EPROM) or programmable read-only memory (PROM). Storage media can also be read-only memory (ROM), magnetic storage, flash memory, magnetic disks, or optical disks. Storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0227] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside within an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components within an electronic device or host device.

[0228] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0229] The sequence numbers of the embodiments in this application are merely for description and do not represent the superiority or inferiority of the embodiments. Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0230] Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0231] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0232] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0233] It should be further noted that although the steps in the flowchart are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated in this document, there is no strict order requirement for the execution of these steps, and they can be executed in other orders.

[0234] Furthermore, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0235] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0236] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0237] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data transmission method, characterized in that, The method is applied to the source end, and the method includes: Establish a data transmission channel with the receiving end, and set an identification field in the communication protocol of the data transmission channel to distinguish the transmission mode; The transmitted data is encoded according to the identification field to obtain encoded transmitted data. The encoding process includes metadata encapsulation of chunked transmitted data and direct transmission interface calls for zero-copy transmitted data. The encoded transmission data is sent to the receiving end through the data transmission channel, so that the receiving end receives the encoded transmission data and performs decoding processing on the encoded transmission data according to the identification field. The decoding processing includes parsing the metadata of the block transmission data and directly writing the byte stream of the zero-copy transmission data.

2. The method according to claim 1, characterized in that, The step of encoding the transmitted data according to the identifier field to obtain encoded transmitted data includes: If the type of the transmitted data is determined to be chunked data, the transmitted data is encapsulated into structured data based on the file name and block number in the identifier field, and a byte buffer object is generated. And / or, if it is determined that the type of the transmitted data is zero-copy transmitted data, the file region interface is invoked to generate a file region object.

3. The method according to claim 1 or 2, characterized in that, Before encoding the transmitted data according to the identifier field, the method further includes: A custom codec is dynamically inserted into the Netty pipeline of the data transmission channel. The Netty pipeline is used to process network events in the Netty framework, and the custom codec is used to switch the encoding processing logic according to the value of the identifier field.

4. The method according to claim 1 or 2, characterized in that, The encoding process of the transmitted data according to the identifier field includes: If the transmitted data is determined to be an online log, the real-time update frequency of the online log is obtained; The block size for transmitting the online log in blocks is adjusted according to the real-time update frequency.

5. The method according to claim 1, characterized in that, The encoding process of the transmitted data according to the identifier field includes: If the type of the transmitted data is determined to be zero-copy transmission data, the memory-mapped file interface is called to generate a memory-mapped file object.

6. A data transmission method, characterized in that, The method is applied at the receiving end, and the method includes: A data transmission channel is established with the source end, wherein the source end sets an identification field in the communication protocol of the data transmission channel to distinguish the transmission mode; The system receives encoded transmission data sent by the source end and decodes the encoded transmission data according to the identification field. The encoded transmission data is obtained by the source end after encoding the transmission data according to the identification field. The encoding process includes metadata encapsulation of the chunked transmission data and direct transmission interface call for zero-copy transmission data. The decoding process includes metadata parsing of the chunked transmission data and direct writing of the byte stream of the zero-copy transmission data.

7. The method according to claim 6, characterized in that, The decoding process of the encoded transmission data based on the identifier field includes: Based on the identification field, if the type of the encoded transmission data is determined to be chunked transmission data, the file name and block number in the structured data of the encoded transmission data are parsed, and the data is concatenated according to the file name and the block number; And / or, if the type of the encoded transmission data is determined to be zero-copy transmission data based on the identification field, the byte stream is directly received and written to a local file.

8. An electronic device, characterized in that, include: Memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images