Cross-protocol data writing method and device, server and medium

Abstract

The embodiment of the invention provides a cross-protocol data writing method and device, a server and a medium. The method comprises the following steps: a database server receives a data format declaration request sent by a client, and converts the data format declaration request into a data stream receiving preparation instruction following a kernel protocol of the database server; and executing the data flow receiving preparation instruction to send a confirmation message to the client according to the client communication protocol. And receiving a data message sent by the client in response to the confirmation message, analyzing and extracting a data stream from the data message through a pre-registered data acquisition function, and writing the data stream into the target table. The data format declaration request follows a client communication protocol and is used for declaring a data flow format to be written into the target table. The acknowledgement message is used for indicating the client to start to send the data stream, the data message follows a client communication protocol and packages the data stream, the kernel protocol is different from the client communication protocol, and the method reduces the design and implementation complexity of the database protocol plug-in.

Description

Technical Field

[0001] This application relates to data processing, and more particularly to a cross-protocol data writing method, apparatus, server, and medium. Background Technology

[0002] In enterprise database applications, cross-platform data migration and batch writing are common requirements. However, differences in protocol processing exist between different databases. This protocol incompatibility not only limits the feasibility of cross-platform data migration but also increases development and maintenance costs. Therefore, how to achieve cross-protocol data writing is an urgent problem to be solved.

[0003] Currently, the main focus is on developing a dedicated module within the target database's protocol plugin layer to parse and simulate the batch write behavior of the source database. This module needs to fully recognize and be compatible with the specific request syntax in the source database protocol, and simulate the interaction logic and state machine of the source database within the plugin layer to complete data reception.

[0004] However, the above solution requires adding a lot of code to the protocol plugin layer, which significantly increases the complexity of the protocol plugin. Summary of the Invention

[0005] This application provides a method, apparatus, server, and medium for writing cross-protocol data, which aims to reduce the complexity of protocol plugins.

[0006] In a first aspect, embodiments of this application provide a cross-protocol data writing method applied to a database server, the method comprising:

[0007] Receive a data format declaration request sent by the client; wherein the data format declaration request follows the client communication protocol and is used to declare the data stream format to be written to the target table;

[0008] The data format declaration request is converted into a data stream receive preparation instruction that conforms to the kernel protocol of the database server; wherein the kernel protocol is different from the client communication protocol;

[0009] The data stream reception preparation instruction is executed to send an acknowledgment message to the client in accordance with the client communication protocol. The acknowledgment message is used to instruct the client to start sending the data stream.

[0010] Receive a data packet sent by the client in response to the acknowledgment message, the data packet conforming to the client's communication protocol and encapsulating the data stream;

[0011] The data stream is parsed and extracted from the data packet using a pre-registered data acquisition function, and then written to the target table.

[0012] In one possible implementation, when the data input format corresponding to the kernel protocol is text format, the step of parsing and extracting the data stream from the data packet through the pre-registered data acquisition function and writing it to the target table includes:

[0013] Based on the data stream format, the data stream is parsed and extracted from the data packet;

[0014] The data stream is converted into text format to obtain a text-formatted data stream;

[0015] Write the text-formatted data stream to the buffer;

[0016] When the data acquisition function is called, the text-formatted data stream in the buffer is written to the target table.

[0017] In one possible implementation, the method further includes:

[0018] Monitor the amount of data in the buffer;

[0019] When the amount of data in the buffer reaches a preset data volume threshold, the writing of the text-formatted data stream to the buffer is paused, and a pause sending instruction is sent to the client. The pause sending instruction is used to instruct the client to pause sending the data stream.

[0020] In one possible implementation, writing the text-formatted data stream in the buffer to the target table includes:

[0021] The target data block is obtained from the buffer using the data acquisition function. The target data block is a data block of a predetermined size that is segmented from the data stream.

[0022] Write the target data block to the target table, and repeat the above steps until the entire text-formatted data stream is written to the target table.

[0023] In one possible implementation, the method further includes:

[0024] When the write operation of the target data block fails, the corresponding error log is stored and the write operation of the remaining data is suspended.

[0025] In one possible implementation, prior to receiving the data format declaration request from the client, the method further includes:

[0026] The system obtains a table information retrieval request sent by the client. The table information retrieval request is used to obtain the metadata of the target table. The data stream format is determined based on the metadata of the target table.

[0027] The metadata of the target table is sent to the client.

[0028] In one possible implementation, the method further includes:

[0029] After the data stream is written to the target table, a result message is generated;

[0030] The result message is sent to the client in accordance with the client communication protocol.

[0031] Secondly, embodiments of this application provide a cross-protocol data writing device applied to a database server, the device comprising:

[0032] The receiving module is used to receive a data format declaration request sent by the client; wherein the data format declaration request follows the client communication protocol and is used to declare the data stream format to be written to the target table;

[0033] A conversion module is used to convert the data format declaration request into a data stream receive preparation instruction that conforms to the kernel protocol of the database server; wherein the kernel protocol is different from the client communication protocol;

[0034] An execution module is configured to execute the data stream reception preparation instruction to send an acknowledgment message to the client in accordance with the client communication protocol. The acknowledgment message is used to instruct the client to start sending the data stream.

[0035] The receiving module is further configured to receive a data packet sent by the client in response to the acknowledgment message, the data packet conforming to the client communication protocol and encapsulating the data stream;

[0036] The writing module is used to parse and extract the data stream from the data packet using a pre-registered data acquisition function, and write it to the target table.

[0037] In one possible implementation, when the data input format corresponding to the kernel protocol is text format, the writing module is specifically used for:

[0038] Based on the data stream format, the data stream is parsed and extracted from the data packet;

[0039] The data stream is converted into text format to obtain a text-formatted data stream;

[0040] Write the text-formatted data stream to the buffer;

[0041] When the data acquisition function is called, the text-formatted data stream in the buffer is written to the target table.

[0042] In one possible implementation, the cross-protocol data writing device further includes a monitoring module for:

[0043] Monitor the amount of data in the buffer;

[0044] When the amount of data in the buffer reaches a preset data volume threshold, the writing of the text-formatted data stream to the buffer is paused, and a pause sending instruction is sent to the client. The pause sending instruction is used to instruct the client to pause sending the data stream.

[0045] In one possible implementation, the writing module is specifically used for:

[0046] The target data block is obtained from the buffer using the data acquisition function. The target data block is a data block of a predetermined size that is segmented from the data stream.

[0047] Write the target data block to the target table, and repeat the above steps until the entire text-formatted data stream is written to the target table.

[0048] In one possible implementation, the cross-protocol data writing device further includes an error handling module, which stores the corresponding error log and suspends the writing operation of the remaining data when the target data block writing fails.

[0049] In one possible implementation, the cross-protocol data writing device further includes a sending module. Before receiving the data format declaration request sent by the client, the receiving module is further configured to obtain a table information acquisition request sent by the client. The table information acquisition request is used to obtain the metadata of the target table, and the data stream format is determined based on the metadata of the target table.

[0050] The sending module is used to send the metadata of the target table to the client.

[0051] In one possible implementation, the sending module is further configured to:

[0052] After the data stream is written to the target table, a result message is generated;

[0053] The result message is sent to the client in accordance with the client communication protocol.

[0054] Thirdly, embodiments of this application provide a database server, including: a communication component, a memory, and a processor;

[0055] The communication component is used for communication and interaction with the client;

[0056] The memory stores computer-executed instructions;

[0057] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0058] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0059] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0060] The cross-protocol data writing method, apparatus, server, and medium provided in this application embodiment involve a database server first receiving a data format declaration request from a client, converting the request into a data stream reception preparation instruction conforming to the database server's kernel protocol. The server then executes the data stream reception preparation instruction to send an acknowledgment message to the client, following the client's communication protocol. Next, the server receives the data packet sent by the client in response to the acknowledgment message, parses and extracts the data stream from the data packet using a pre-registered data acquisition function, and writes it to the target table. The database server does not need to understand or simulate the entire client interaction process; it only needs to convert the key request (data format declaration request) into a data stream reception preparation instruction executable by its own kernel protocol to guide the client into the data transmission phase. Subsequently, by parsing and extracting the data stream from the client protocol packet using the pre-registered data acquisition function, the server successfully delivers batch data conforming to the external protocol to the database kernel and writes it to the target table. This significantly reduces the design and implementation complexity of the database protocol plugin while ensuring the complete implementation of the batch write function, achieving highly efficient cross-protocol compatibility with minimal kernel modifications. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0062] Figure 1 Flowchart of the cross-protocol data writing method provided in this application Figure 1 ;

[0063] Figure 2 Flowchart of the cross-protocol data writing method provided in this application Figure 2 ;

[0064] Figure 3 Flowchart of the cross-protocol data writing method provided in this application Figure 3 ;

[0065] Figure 4 A schematic diagram of the cross-protocol data writing device provided in this application;

[0066] Figure 5 A schematic diagram of the database server provided in this application.

[0067] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0068] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0069] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0070] First, the application scenarios involved in this application will be explained:

[0071] In practical applications, when the .NET SqlBulkCopy method (SQL bulk write class) performs bulk data writing with the SQL Server based on the Tabular Data Stream (TDS) protocol, it requires three message exchanges:

[0072] (1) First SQL Batch request.

[0073] The SQL Batch request is a batch query sent by the client to retrieve complete metadata for the target table [BulkInsertTest].

[0074] For example, an SQL Batch request can be represented as follows:

[0075]

[0076] Among them, select @@trancount is used to obtain the number of active transactions in the current session and determine the transaction context state; SET FMTONLY ON means to enable the metadata-only return mode (English: FMTONLY); This means executing a query in FMTONLY mode to obtain the column structure information of the table [BulkInsertTest]; SET FMTONLY OFF means turning off FMTONLY mode; exec ..sp_tablecollations_100N'.[BulkInsertTest]' means calling the SQL Server internal stored procedure to obtain the detailed collation rules (in English, Collation) and character set configuration of each column of the table [BulkInsertTest].

[0077] (2) Second SQL Batch request.

[0078] The SQL Batch request is followed by an INSERT BULK statement sent by the client, which declares the format of the subsequent binary data stream.

[0079] For example, the SQL Batch request can be represented as follows:

[0080] insert bulk BulkInsertTest ([Name] VarChar(50) COLLATE bbf_unicode_cp1_ci_as, [Val] Float)

[0081] Here, INSERT BULK is the keyword used to indicate the start of a batch insert operation; BulkInsertTest is the target table name into which data will be inserted; ([Name] VarChar(50) COLLATE bbf_unicode_cp1_ci_as, [Val] Float) is used to describe the structure of each column in the binary data stream to be sent. Specifically, [Name] VarChar(50) COLLATE bbf_unicode_cp1_ci_as indicates that the first column is named Name, the data type is a variable-length string (maximum 50 characters), and the sorting rule is specified as bbf_unicode_cp1_ci_as, which is a case-insensitive and accent-sensitive Unicode sorting rule; [Val] Float indicates that the second column is named Val, and the data type is a floating-point number.

[0082] (3) Bulk Load Data message.

[0083] After the format declaration is accepted by the server, the client immediately sends a Bulk Load Data message. This message encapsulates the actual data to be inserted in batches as a binary stream, strictly adhering to the column structure declared in the previous step. The server receives and parses this data stream, ultimately writing it into the target table.

[0084] In enterprise database applications, cross-platform data migration and bulk writes are common requirements. For example, in heterogeneous database environments, users may need to seamlessly migrate SQL Server-based TDS protocol applications to a target database to take advantage of its high availability, security, or performance benefits.

[0085] In the process of achieving protocol compatibility, traditional databases need to mimic the behavior of the native protocol at the front-end and back-end interaction layer and send corresponding response messages. However, there are some differences in protocol processing between different databases. For example, the three-way interaction process of TDS batch write is not fully supported by the target database. When a user attempts to run a batch write operation that relies on the TDS protocol on the target database, due to protocol incompatibility, the client cannot recognize the messages sent by the server and cannot return the corresponding messages, resulting in interruption due to syntax errors or protocol errors.

[0086] This protocol incompatibility not only limits the feasibility of cross-platform data migration but also increases development and maintenance costs, especially in enterprise applications that need to support multiple database protocols simultaneously. Therefore, achieving compatibility with TDS protocol batch writes while avoiding large-scale modifications to the database kernel has become a critical technical problem that urgently needs to be solved.

[0087] Currently, to address the aforementioned technical issues, the main approach is to develop a separate TDS protocol Bulk Load processing module within the target database's protocol plugin. This module is syntactically compatible with the SQLBatch request syntax and simulates SQL Server behavior. For example, it parses the syntax of the INSERT BULK command, constructs SQL Server-compatible response messages, and directly processes the data stream on the server side.

[0088] However, existing technologies require adding a large amount of code to the protocol plugin layer, including full support for the three-way interaction process of the TDS protocol, which significantly increases the complexity of the protocol plugin.

[0089] Based on the aforementioned technical problems, the technical concept of this application is as follows: While researching database protocol compatibility solutions, the inventors discovered that for complex client communication protocols like TDS, attempting to fully simulate the entire batch writing interaction process at the protocol plugin layer necessitates maintaining a massive protocol state machine and message parsing logic, leading to a rapid increase in code size and maintenance difficulties. Furthermore, the inventors found that although the communication protocols used by the client and server differ, the two core operations—preparing to receive data streams and writing data to the target table—are functionally interconnected. Based on this, the inventors realized that instead of mechanically mimicking client behavior at the protocol layer, they should focus on semantically converting key instructions. Specifically, this involves stripping the outer protocol encapsulation of data format declaration requests in the client communication protocol, extracting the semantic information of the core defined data stream format, and reorganizing it into data stream receiving preparation instructions that the kernel protocol can directly understand and execute. Through this key conversion, coupled with the targeted extraction of subsequent data streams, the complex simulation of the entire client interaction process can be bypassed, thereby significantly reducing the implementation complexity of the protocol plugin.

[0090] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0091] Figure 1 Flowchart of the cross-protocol data writing method provided in this application Figure 1 ,like Figure 1As shown, the method includes:

[0092] S11. Receive the data format declaration request sent by the client.

[0093] The execution entity in this embodiment is a database server. The specific type of the database server can be determined according to the actual situation, and there are no specific restrictions on it.

[0094] In this step, the client and database server have established a connection, the client has identified the target table for the bulk write operation, and the client is ready to declare the format of the subsequent data stream to the database server. At this point, the client needs to send a request to the database server according to its own client communication protocol. Correspondingly, the database server receives the data format declaration request sent by the client.

[0095] In this context, the client refers to the application or driver that initiates the bulk write operation, such as a .NET application using the SqlBulkCopy class.

[0096] The data format declaration request follows the client communication protocol and is used to declare the data stream format to be written to the target table.

[0097] For example, in the TDS protocol, a data format declaration request is specifically represented by an INSERT BULK statement, which details the target table name, column names, data types of each column, and collation rules.

[0098] S12. Convert the data format declaration request into a data stream receive preparation instruction that conforms to the kernel protocol of the database server.

[0099] In this step, after successfully receiving the client's data format declaration request, the database server's kernel cannot directly understand or execute the native instructions in the client's communication protocol. Therefore, it must be converted into data stream reception preparation instructions that conform to the database server's kernel protocol.

[0100] The kernel protocol is the internal language used by the database server to implement its functions. It should be understood that the kernel protocol is different from the client communication protocol.

[0101] The data stream receive preparation instruction initializes a batch write session, preparing the kernel to read data from a specific channel and write it to the target table. For example, the data stream receive preparation instruction can be the COPY command supported by the kernel protocol in the KingbaseES database.

[0102] In one possible implementation, the protocol plugin parses the received data format declaration request (such as the INSERT BULK statement in TDS) to extract key semantic information such as the target table name and column definitions. Then, based on this information, the protocol plugin constructs a new command that is semantically equivalent but conforms to the database kernel protocol, namely the data stream receiving preparation instruction (such as the COPY FROM STDIN command).

[0103] S13. Execute the data stream reception preparation instruction to send an acknowledgment message to the client in accordance with the client communication protocol.

[0104] In this step, after successfully converting the external request into an internal instruction and submitting it for execution, the database kernel has entered a state ready to receive data based on the data stream reception preparation instruction. The client, however, is still waiting for a response conforming to its own protocol specifications to confirm that it can begin sending actual data. Therefore, an acknowledgment message needs to be sent to the client to instruct it to begin sending the data stream.

[0105] The acknowledgment message is used to instruct the client to begin sending a data stream.

[0106] For example, in the TDS protocol, this confirmation message is a DONE token message containing a specific status flag.

[0107] In one possible implementation, after the protocol plugin drives the kernel to execute the data stream reception preparation instruction, it needs to simulate the interactive behavior expected by the client. Instead of directly using the kernel's native response (such as the G message in the PostgreSQL / KingbaseES protocol), it actively constructs an acknowledgment message (such as a DONE token) indicating successful operation according to the specifications of the client communication protocol (such as TDS) and sends it to the client via the network connection.

[0108] S14. Receive the data packet sent by the client in response to the acknowledgment message.

[0109] In this step, after receiving an acknowledgment message conforming to its protocol specifications, the client can determine that the database server is ready and can begin sending the actual data packets encapsulated with the data stream. Correspondingly, the database server receives the data packets sent by the client in response to the acknowledgment message.

[0110] The data packets follow the client communication protocol and encapsulate a data stream. For example, in the TDS protocol, the data packets are called Bulk Load Data packets.

[0111] It should be understood that the data stream has been pre-encoded by the client according to the data stream format declared in the data format declaration request.

[0112] In one possible implementation, after sending an acknowledgment message, the database server's protocol plugin immediately begins to wait for and receive subsequent messages sent by the client.

[0113] S15. Using the pre-registered data acquisition function, parse and extract the data stream from the data packet, and write it to the target table.

[0114] In this step, since the data stream still exists in the form of binary data packets of the client communication protocol and has not yet been obtained by the database kernel, it is necessary to parse and extract the data stream from the data packets so that it can be written to the target table later.

[0115] The pre-registered data acquisition function refers to a callback function pre-registered by the protocol plugin when the database kernel executes the data stream reception preparation instruction. When the database kernel needs to read data, it will automatically call this function instead of reading directly from the network socket.

[0116] In one possible implementation, the protocol plugin parses and extracts the data stream from the received data packets. It then converts the binary data types in the data stream into a format required by the database kernel (such as text format). Simultaneously, when the database kernel needs to read data during a batch write context, it calls a previously pre-registered data retrieval function. This function returns the converted data stream from the protocol plugin's buffer to the kernel, which then writes this data to the target table.

[0117] It should be understood that the specific implementation process and principles of this step will be explained later. Figure 2 The embodiments shown are described in detail here, and will not be repeated here.

[0118] The cross-protocol data writing method provided in this application involves a database server first receiving a data format declaration request from a client, converting the request into a data stream reception preparation instruction conforming to the database server's kernel protocol. The server then executes the data stream reception preparation instruction to send an acknowledgment message to the client, following the client's communication protocol. Next, the server receives the data packet sent by the client in response to the acknowledgment message, parses and extracts the data stream from the data packet using a pre-registered data acquisition function, and writes it to the target table. The data format declaration request, conforming to the client's communication protocol, declares the format of the data stream to be written to the target table. The acknowledgment message instructs the client to begin sending the data stream; the data packet conforms to the client's communication protocol and encapsulates the data stream, while the kernel protocol differs from the client's communication protocol. In this technical solution, the database server does not need to understand or simulate the entire client interaction process; it only needs to convert the key request (data format declaration request) into a data stream reception preparation instruction executable by its own kernel protocol to guide the client into the data transmission phase. Subsequently, by parsing and extracting the data stream from the client protocol message through the pre-registered data acquisition function, the batch data conforming to the external protocol was successfully delivered to the database kernel and written to the target table. This significantly reduced the design and implementation complexity of the database protocol plugin while ensuring the complete implementation of the batch write function, and achieved relatively efficient cross-protocol compatibility at the cost of minor kernel modifications.

[0119] Optionally, in some embodiments, after the data stream is written to the target table, a result message can be generated and sent to the client in accordance with the client communication protocol.

[0120] The result message contains a status identifier indicating success or failure. Upon success, it includes information such as the number of rows of data written; upon failure, it includes an error code and a brief error description.

[0121] In one possible implementation, after the database kernel completes the data stream write or terminates at some point due to an error, the protocol plugin or its cooperating state management module obtains the final operation status from the kernel. Based on this operation status, the protocol plugin or its cooperating state management module constructs the result message content according to a predefined format. Further, the protocol plugin uses the result message content as the payload, encoding and encapsulating it according to the detailed specifications of the client communication protocol. This includes adding necessary protocol headers, setting the correct message type (such as the DONE message stream in the TDS protocol, with a flag indicating the end of the session), calculating the checksum, etc. Subsequently, the encapsulated result message conforming to the client communication protocol format is sent to the client through the established network connection. After receiving and parsing the result message, the client can determine the result of this batch write operation.

[0122] For example, if the write is completely successful, a result message containing a success status code and the written row number field is generated; if it fails due to a constraint violation, a result message containing a failure status code, a specific error code (such as a violation of a unique key constraint), and information about the row number where the error may have occurred is generated.

[0123] In this embodiment, by generating and sending a result message in accordance with the client communication protocol after the write operation is completed, the client can clearly know whether the write operation was successful or not or the final state, thereby performing corresponding application logic processing (such as committing transactions, logging, or triggering subsequent processes), which improves the completeness of the entire solution as a service and the user experience, and makes it easier to integrate into standard client application frameworks.

[0124] Furthermore, when the data input format corresponding to the kernel protocol is text format, the above-mentioned S15 can be... Figure 2 The embodiments shown are explained in detail below.

[0125] Figure 2 Flowchart of the cross-protocol data writing method provided in this application Figure 2 ,like Figure 2 As shown, S15 includes:

[0126] S21. Based on the data stream format, parse and extract the data stream from the data packet.

[0127] In this step, after the database server has learned the data stream format through S11-S14 and has received the data packets that are encoded and encapsulated according to the format, it is necessary to parse and extract the data stream since it is wrapped in the message structure of the client protocol.

[0128] In one possible implementation, the protocol plugin decodes the received data packets according to the data stream format. Specifically, it can parse the packet header to determine the data area boundaries, then read the binary data column by column and row by row according to the format definition, and restore this data to a sequence of raw values ​​with a clear data type meaning. This sequence of raw values ​​is the extracted data stream.

[0129] For example, for a column defined as INT, a 4-byte binary value is extracted from the message and interpreted as an integer; for a column defined as VARCHAR, the corresponding byte sequence is extracted based on the length information.

[0130] S22. Convert the data stream to text format to obtain a data stream in text format.

[0131] In this step, since the default data receiving channel of the database kernel (such as an engine that supports COPY FROM STDIN) expects to receive data in text format, it is also necessary to convert the data stream to text format.

[0132] For example, the binary representation of the integer 1 is converted to the string "1", the floating-point number 3.14 is converted to the string "3.14", and the date is converted to a string in the format "year-month-day".

[0133] In one possible implementation, the protocol plugin calls the corresponding conversion function based on the data type definition of each column in the data stream format, converting each value of the extracted binary data stream into text format. For numeric types, number base conversion is performed; for string types, character set encoding conversion is handled to conform to the encoding required by the database kernel (such as UTF-8); for complex types such as date and time, they are formatted according to standard formats. Ultimately, the original binary data stream is converted into a sequence of text lines separated by specific delimiters (such as tabs), i.e., a text-formatted data stream.

[0134] S23. Write the text-formatted data stream to the buffer.

[0135] In this step, after converting the data stream into text format, the text data needs to be temporarily stored so that the database kernel can read it in a controlled, batch manner, thus solving problems such as excessive kernel processing pressure caused by direct push or mismatch between client and server processing speeds.

[0136] The buffer refers to a temporary storage area in the memory of the database server, used to store text-formatted data streams in an orderly manner.

[0137] In one possible implementation, the protocol plugin continuously writes the converted text-formatted data stream into a memory buffer (i.e., a buffer). This writing process is streaming, meaning it's written while converting. The buffer is typically designed using a First-In, First-Out (FIFO) queue structure. When the buffer has enough space, new text data is appended to the end; when the database kernel retrieves data via a data retrieval function, the data at the head of the buffer is released, and the space is reused.

[0138] Optionally, the amount of data in the buffer can also be monitored; when the amount of data in the buffer reaches a preset data volume threshold, the writing of text-formatted data streams to the buffer is paused, and a pause sending command is sent to the client.

[0139] The pause sending command is used to instruct the client to pause sending data streams.

[0140] The preset data volume threshold is a pre-configured value used to identify the buffer capacity alarm line. This threshold is usually set to be less than the total buffer capacity to provide a safety margin.

[0141] For example, the preset data volume threshold can be 80% of the total buffer capacity or a fixed byte size (such as 10MB).

[0142] It should be understood that the preset data volume threshold can be preset based on empirical or experimental values, and this application embodiment does not impose specific restrictions on it.

[0143] In practical applications, the protocol plugin periodically or event-drivenly queries the buffer's status during operation. For example, the plugin maintains a counter associated with the buffer to calculate the amount of data stored in it, and also maintains a state variable associated with the buffer. This state variable increases as data is written to the buffer and decreases as data is read and removed by the data acquisition function. When the protocol plugin determines that the amount of data in the buffer has reached or exceeded a preset data volume threshold, it first interrupts the data path for writing newly generated text-formatted data streams into the buffer, thus pausing the write operation locally. Secondly, based on the message format of the client communication protocol, it generates a pause transmission command and sends it to the client via the network connection. Upon receiving this pause transmission command, the client temporarily suspends the transmission of subsequent data packets, thereby slowing down the data flow into the database server. This allows the kernel to gradually consume the data in the buffer, thus reducing the amount of data in the buffer.

[0144] Furthermore, when the amount of data in the buffer drops below a preset data volume threshold, the protocol plugin reopens the data path for writing newly generated text-formatted data streams into the buffer and sends a continue sending instruction to the client, instructing the client to continue sending the remaining data packets.

[0145] By monitoring the amount of data in the buffer and pausing writing and notifying the client to pause sending when the preset data volume threshold is reached, the risk of buffer overflow and data loss is effectively prevented. The client's data sending rate can be dynamically adjusted according to the server's real-time processing capability, thereby ensuring that system memory resources are protected under high load or instantaneous data influx. The entire writing process can run stably under controllable resource consumption, enhancing the adaptability to abnormal scenarios.

[0146] S24. When the data acquisition function is called, the text-formatted data stream in the buffer is written to the target table.

[0147] In this step, once the text-formatted data stream is ready in the buffer, the database kernel, after executing the data stream receiving preparation instruction, will actively call the data acquisition function pre-registered by the protocol plugin during the initialization phase to periodically attempt to read the text-formatted data stream for subsequent write operations.

[0148] In one possible implementation, when the data acquisition function is called, the target data block is retrieved from the buffer through the data acquisition function, and the target data block is written to the target table. The above steps are repeated until the entire text-formatted data stream is written to the target table.

[0149] The target data block is a data block of a predetermined amount that is cut from the data stream.

[0150] Specifically, when the database kernel triggers a call to the data fetch function, this function reads a certain amount of text-formatted data (e.g., one or more lines) from the beginning of the buffer and provides this text-formatted data stream to the kernel as the function's return value. Upon receiving this text data, the kernel parses it according to the semantics of the COPY command and writes it to the specified target table. This process repeats, with each call to the data fetch function reading and transferring a portion of the data from the buffer until all data in the buffer has been read.

[0151] In the above implementation, a one-time massive data write operation is broken down into multiple controllable data block-level write tasks. The independent writing of each target data block significantly reduces the pressure and holding time on system resources such as database transaction logs and locks during a single operation, which helps improve system concurrency performance and reduces the impact on online services. At the same time, writing in units of target data blocks also enables more granular error control and breakpoint resumption, making the write process easier to manage and monitor.

[0152] Specifically, further, when writing to the target data block fails, the corresponding error log is stored and the writing operation of the remaining data is suspended.

[0153] Optionally, the error log includes a failure timestamp, the identifier or offset of the failed data block, the specific error code and description, and related contextual data.

[0154] The remaining data refers to the data in the text-formatted data stream that has not yet been processed by the database kernel.

[0155] Specifically, when the database kernel performs a write operation on a data block and returns an error status, the protocol plugin or its cooperating exception handling module collects detailed information about the failure, including the sequence number or range of the failed data block, the error message obtained from the kernel, and the database error code that may have been triggered. This information is then formatted into a structured record and stored in a designated error log via the system log interface, such as being written to the server's log file or system event table, to ensure the problem is traceable. Furthermore, the protocol plugin or its cooperating exception handling module sets a global failure flag and interrupts the loop or process of submitting subsequent data blocks to the database kernel, thus pausing the write operation of the remaining data.

[0156] By pausing subsequent operations and logging errors when a target data block write fails, the spread of errors can be promptly prevented, thus preventing the target table from being in an inconsistent intermediate state due to partial data errors. Storing error logs provides precise information for quickly locating and diagnosing problems, enabling operations personnel to intervene based on clear error information (such as rolling back or retrying after repair), greatly improving the reliability and maintainability of batch data operations.

[0157] In the above embodiments, by converting the data stream into text format and writing it to a buffer, the compatibility of the data stream with the kernel batch receive interface (such as the COPY command) is ensured, bridging the format gap between binary protocol data and the kernel text interface. Introducing a buffer decouples the data reception and parsing from the kernel write processes, which are at different speeds, thus serving as a flow shaper and temporary buffer, ensuring the smoothness and continuity of data transmission, and thereby improving the stability and resilience of the entire batch write process.

[0158] Optionally, in some embodiments, before S11, a table information retrieval request sent by the client may be obtained, and the metadata of the target table may be sent to the client.

[0159] Among them, the table information retrieval request is used to obtain the metadata of the target table, and the data stream format is determined based on the metadata of the target table.

[0160] For example, in the TDS protocol, a table information retrieval request is a compound batch SQL statement containing a SELECT statement, a SET FMTONLY setting, and a call to the sp_tablecollations stored procedure, used to securely retrieve table metadata without touching the actual data.

[0161] Metadata refers to data that describes the structure of the target table, including the column names, data types of each column (such as INT, VARCHAR(255)), whether null values ​​are allowed (NULL constraint), default values, primary key information, and collation and character set information that are crucial for character type columns.

[0162] In one possible implementation, the database server's protocol plugin first listens for and receives table information retrieval requests from clients. The plugin then submits the request to the database kernel for execution. After processing the request, the kernel generates detailed structural information about the target table as a result. The protocol plugin layer then captures these results and, according to the message format specified by the client communication protocol, encapsulates the target table's metadata into a response message and sends it back to the client.

[0163] In this embodiment, by first obtaining the table information retrieval request sent by the client and returning the metadata of the target table, it is ensured that the client can strictly follow the actual structure definition of the target table on the database server when constructing the data format declaration request and data stream, thus avoiding data format errors, conversion failures, or write rejections caused by inconsistent understanding of the table structure.

[0164] To further illustrate the cross-protocol data writing method shown in the embodiments of this application, a practical example will be provided below. In this example, the client communication protocol is the TDS communication protocol of SQL Server database, and the database server is KingbaseES.

[0165] Figure 3 Flowchart of the cross-protocol data writing method provided in this application Figure 3 ,like Figure 3 As shown, the method includes:

[0166] S301. Based on the first SQLBatch request initiated by the client using the TDS protocol, send the metadata of the target table to the client.

[0167] When a client using the TDS protocol initiates a batch write, it first sends two SQLBatch requests. The first SQLBatch request, used to query the target table metadata, can be processed and responded to by the database server as a regular SQLBatch request.

[0168] It should be understood that the first SQLBatch request is the table information retrieval request mentioned above.

[0169] S302. Convert the second SQLBatch request initiated by the client into a COPY FROM STDIN statement.

[0170] For the second SQLBatch request, which is a data format declaration request containing an INSERT BULK statement, the protocol plugin needs to intercept it and rewrite it into a COPY FROM STDIN statement (i.e., a Copy In command) that the database kernel can recognize, and then submit it to the kernel for execution.

[0171] It should be understood that the second SQLBatch request is the data format declaration request mentioned above, and the COPY FROM STDIN statement is the data stream reception preparation instruction mentioned above.

[0172] S303. Send the Done token to the client and initialize the buffer.

[0173] When the kernel begins executing the copy operation, native database protocols (such as libkci) register and wait for data streams by sending G messages. To handle clients using different protocols uniformly, this application abstracts a general protocol interface (such as fn_bulk_copy). For native clients, G messages are sent according to the original logic; for TDS clients, this interface is used to construct and send the Done token expected by the TDS protocol as an acknowledgment message, and after sending this acknowledgment message, the application waits for the client to send subsequent data.

[0174] It should be understood that a protocol interface is a function, method, or callback point within a protocol plugin.

[0175] S304. Set the data acquisition method to a callback function for registering data reading.

[0176] While the copy-in data stream from the native client can be read directly from the network socket, data sent by the TDS client cannot be directly used by the database kernel and requires conversion. This application changes the source of data acquisition for the kernel from directly reading the socket to calling a pre-registered callback function.

[0177] The callback function is designed to read the processed data stream from the global data buffer maintained by the protocol plugin.

[0178] S305. Obtain the Bulk Load Data message returned by the user.

[0179] It should be understood that the Bulk Load Data message is the same as the data message mentioned above.

[0180] S306. Determine whether the Bulk Load Data message is a bulk load request.

[0181] If yes, then execute S307; otherwise, execute S311.

[0182] S307. Parse the Bulk Load Data message.

[0183] S308. Determine whether the request type inside the message is consistent with the external type identifier.

[0184] If yes, then execute S309; ​​otherwise, end.

[0185] S309. Convert the data stream carried by the Bulk Load Data message into text format and store it in a buffer for use by the callback function.

[0186] The system reads data values ​​stored in their original data types based on their metadata and converts the values ​​of various basic data types defined by the TDS protocol into text formats required by the database kernel (such as KingbaseES's COPY command). The converted text data is then written to the protocol plugin's internal buffer in predefined block sizes. If the data volume exceeds a preset data volume threshold, an overflow detection and fragmented retransmission process is triggered to ensure the reliability of data transmission.

[0187] When the database kernel executes the callback function that needs to read data, it will automatically call the registered data acquisition function mentioned above. This data acquisition function reads data from the buffer in batches according to the specified data volume until all the data parsed and converted from the Bulk Load Data message is written to the target table.

[0188] Optionally, after the data writing is complete, the execution result of the kernel Copy In command will be encapsulated into a response message conforming to the TDS protocol specification, serving as the final response to this Bulk Load Data interaction. Throughout the data transmission process, the integrity and consistency of the messages are monitored in real time, and any error events are recorded in the system log and trigger the corresponding recovery process.

[0189] S310, Reset environment status.

[0190] S311 Error: The current request type is not supported.

[0191] In summary, based on the above embodiments, the cross-protocol data writing method provided in this application brings substantial benefits in multiple aspects by introducing core mechanisms such as request conversion, data stream bridging, and protocol abstraction interfaces. This method significantly improves the efficiency of batch data processing under heterogeneous protocols by accurately converting client protocol requests into kernel instructions and efficiently extracting and delivering data using pre-registered callback functions. Built-in buffer management, flow control, and rapid error isolation and logging mechanisms effectively enhance the system's reliability and robustness. Simultaneously, this solution centralizes protocol adaptation logic in a pluggable plugin layer, significantly reducing the intrusiveness and modification costs to the database kernel. This not only optimizes system resource utilization and maintainability but also provides a clear expansion path for supporting more client protocols in the future. These improvements collectively ensure that the database system can provide high-performance, highly compatible, and stable and reliable data writing services in complex multi-protocol environments.

[0192] Figure 4 A schematic diagram of the cross-protocol data writing device provided in this application is shown below. Figure 4 As shown, the cross-protocol data writing device 40 provided in this embodiment is applied to a database server. The cross-protocol data writing device 40 includes:

[0193] The receiving module 401 is used to receive a data format declaration request sent by the client; wherein, the data format declaration request follows the client communication protocol and is used to declare the data stream format to be written to the target table;

[0194] The conversion module 402 is used to convert the data format declaration request into a data stream receive preparation instruction that conforms to the kernel protocol of the database server; wherein, the kernel protocol and the client communication protocol are different;

[0195] The execution module 403 is used to execute the data stream reception preparation instruction to send an acknowledgment message to the client in accordance with the client communication protocol. The acknowledgment message is used to instruct the client to start sending the data stream.

[0196] The receiving module 401 is also used to receive a data packet sent by the client in response to the acknowledgment message. The data packet follows the client communication protocol and encapsulates a data stream.

[0197] The write module 404 is used to parse and extract the data stream from the data packet through the pre-registered data acquisition function and write it to the target table.

[0198] In one possible implementation, when the data input format corresponding to the kernel protocol is text format, the writing module 404 is specifically used for:

[0199] Based on the data stream format, parse and extract the data stream from the data packet;

[0200] Convert the data stream to text format to obtain a text-formatted data stream;

[0201] Write the text-formatted data stream to the buffer;

[0202] When the data acquisition function is called, the text-formatted data stream in the buffer is written to the target table.

[0203] In one possible implementation, the cross-protocol data writing device 40 further includes a monitoring module for:

[0204] Monitor the amount of data in the buffer;

[0205] When the amount of data in the buffer reaches a preset data volume threshold, the process of writing text-formatted data streams to the buffer is paused, and a pause sending command is sent to the client. The pause sending command is used to instruct the client to pause sending the data stream.

[0206] In one possible implementation, the writing module 404 is specifically used for:

[0207] The target data block is retrieved from the buffer using a data acquisition function. The target data block is a data block of a predetermined size that is segmented from the data stream.

[0208] Write the target data block to the target table, and repeat the above steps until the entire text-formatted data stream is written to the target table.

[0209] In one possible implementation, the cross-protocol data writing device 40 further includes an error handling module for storing the corresponding error log and suspending the writing operation of the remaining data when the target data block writing fails.

[0210] In one possible implementation, the cross-protocol data writing device 40 further includes a sending module. Before receiving the data format declaration request sent by the client, the receiving module 401 is also used to obtain a table information acquisition request sent by the client. The table information acquisition request is used to obtain the metadata of the target table, and the data stream format is determined based on the metadata of the target table.

[0211] The sending module is used to send the metadata of the target table to the client.

[0212] In one possible implementation, the sending module is further configured to:

[0213] After writing the data stream to the target table, a result message is generated;

[0214] The result message is sent to the client in accordance with the client communication protocol.

[0215] The cross-protocol data writing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0216] Figure 5 This is a schematic diagram of the database server structure provided in this application. Figure 5 As shown, the database server 50 provided in this embodiment includes at least one processor 501, a memory 502, and a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0217] The communication component 503 is used for communication and interaction with the client.

[0218] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0219] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0220] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0221] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0222] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0223] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0224] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0225] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0226] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0227] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0228] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0229] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0230] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0231] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0232] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for writing data across protocols, characterized in that, Applied to a database server, the method includes: Receive a data format declaration request sent by the client; wherein the data format declaration request follows the client communication protocol and is used to declare the data stream format to be written to the target table; The data format declaration request is converted into a data stream receive preparation instruction that conforms to the kernel protocol of the database server; wherein the kernel protocol is different from the client communication protocol; The data stream reception preparation instruction is executed to send an acknowledgment message to the client in accordance with the client communication protocol. The acknowledgment message is used to instruct the client to start sending the data stream. Receive a data packet sent by the client in response to the acknowledgment message, the data packet conforming to the client's communication protocol and encapsulating the data stream; The data stream is parsed and extracted from the data packet using a pre-registered data acquisition function, and then written to the target table.

2. The method according to claim 1, characterized in that, When the data input format corresponding to the kernel protocol is text format, the step of parsing and extracting the data stream from the data packet through the pre-registered data acquisition function and writing it to the target table includes: Based on the data stream format, the data stream is parsed and extracted from the data packet; The data stream is converted into text format to obtain a text-formatted data stream; Write the text-formatted data stream to the buffer; When the data acquisition function is called, the text-formatted data stream in the buffer is written to the target table.

3. The method according to claim 2, characterized in that, The method further includes: Monitor the amount of data in the buffer; When the amount of data in the buffer reaches a preset data volume threshold, the writing of the text-formatted data stream to the buffer is paused, and a pause sending instruction is sent to the client. The pause sending instruction is used to instruct the client to pause sending the data stream.

4. The method according to any one of claims 2 or 3, characterized in that, The step of writing the text-formatted data stream in the buffer to the target table includes: The target data block is obtained from the buffer using the data acquisition function. The target data block is a data block of a predetermined size that is segmented from the data stream. Write the target data block to the target table, and repeat the above steps until the entire text-formatted data stream is written to the target table.

5. The method according to claim 4, characterized in that, The method further includes: When the write operation of the target data block fails, the corresponding error log is stored and the write operation of the remaining data is suspended.

6. The method according to any one of claims 1-3 or 5, characterized in that, Before receiving the data format declaration request sent by the client, the method further includes: The system obtains a table information retrieval request sent by the client. The table information retrieval request is used to obtain the metadata of the target table. The data stream format is determined based on the metadata of the target table. The metadata of the target table is sent to the client.

7. The method according to any one of claims 1-3 or 5, characterized in that, The method further includes: After the data stream is written to the target table, a result message is generated; The result message is sent to the client in accordance with the client communication protocol.

8. A cross-protocol data writing device, characterized in that, The device, applied to a database server, includes: The receiving module is used to receive a data format declaration request sent by the client; wherein the data format declaration request follows the client communication protocol and is used to declare the data stream format to be written to the target table; A conversion module is used to convert the data format declaration request into a data stream receive preparation instruction that conforms to the kernel protocol of the database server; wherein the kernel protocol is different from the client communication protocol; An execution module is configured to execute the data stream reception preparation instruction to send an acknowledgment message to the client in accordance with the client communication protocol. The acknowledgment message is used to instruct the client to start sending the data stream. The receiving module is further configured to receive a data packet sent by the client in response to the acknowledgment message, the data packet conforming to the client communication protocol and encapsulating the data stream; The writing module is used to parse and extract the data stream from the data packet using a pre-registered data acquisition function, and write it to the target table.

9. A database server, characterized in that, include: Communication components, memory, and processor; The communication component is used for communication and interaction with the client; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images