Batch data exporting method, device and equipment based on stream load protocol, medium and product

Abstract

The invention discloses a batch data exporting method, device and equipment based on a stream load protocol, a medium and a product. The method comprises the following steps: starting a hypertext transfer protocol (HTTP) service process, and initiating a communication connection establishment request carrying HTTP connection related parameters to a database server, so that the database server performs communication connection according to the HTTP connection related parameters of the communication connection establishment request, and generates and feeds back a connection establishment success message after the connection is successful; sending a data query statement to the database server, determining target query data and a data query amount, generating and feeding back a batch data to-be-pushed message when it is determined that a preset stream load batch pushing judgment condition is met according to the data query amount, performing data packaging on the target query data based on a stream load protocol and HTTP connection related parameters, and sending the packaged data to the database server. Generating and feeding back an HTTP push request; and determining target query data according to the HTTP push request, generating a push request response, feeding back the push request response to the database server, and closing the HTTP service process.

Description

Technical Field

[0001] This invention relates to the field of batch data export from databases, and more particularly to a method, apparatus, device, medium, and product for batch data export based on the stream load protocol. Background Technology

[0002] As the core carrier of data storage and management, databases are crucial for the stable operation of various data transfer, analysis, and backup services by ensuring the efficient and secure export of large volumes of data. With the expansion of business scale, the need to export batches of data from databases is becoming increasingly urgent, placing higher demands on the timeliness and data integrity of the export process.

[0003] Existing database batch export solutions have significant limitations: they lack optimized transmission protocols and flexible trigger control mechanisms, which can easily lead to data lag, packet corruption, and other problems, affecting the integrity of the exported data; improper management of the timing of starting and stopping the HTTP (Hypertext Transfer Protocol) service process can result in resource waste or transmission interruption; non-standard parameter interaction between the database driver and the database server can easily cause connection failures or data security risks; and they do not fully adapt to the streaming advantages of the stream load protocol, resulting in low transmission efficiency for massive amounts of data and making it difficult to meet the needs of large-scale data processing scenarios. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and product for batch data export based on the stream load protocol, in order to optimize the process of batch exporting database data, thereby ensuring the execution efficiency and stability of batch exporting database data.

[0005] According to one aspect of the present invention, a batch data export method based on the stream load protocol is provided, applied to a database driver, the method comprising:

[0006] Start the Hypertext Transfer Protocol (HTTP) service process, initiate a communication connection establishment request to the database server carrying HTTP connection-related parameters, so that the database server can establish a communication connection based on the HTTP connection-related parameters in the communication connection establishment request, and generate and send back a connection establishment success message after the connection is successfully established.

[0007] After receiving the connection establishment success message, a data query statement is sent to the database server so that the database server can determine the target query data and the data query volume based on the data query statement. When the preset stream load batch push judgment condition is met based on the data query volume, a batch data to be pushed message is generated and fed back. At the same time, the target query data is packaged based on the stream load protocol and HTTP connection related parameters, and an HTTP push request is generated and fed back.

[0008] Based on the HTTP push request, determine the target query data, generate a push request response and send it back to the database server, and then close the HTTP service process.

[0009] According to another aspect of the present invention, a batch data export method based on the stream load protocol is provided, applied to a database server, the method comprising:

[0010] Get the communication connection establishment request sent by the database driver, which carries HTTP connection-related parameters; establish a communication connection with the data driver based on the HTTP connection-related parameters in the communication connection establishment request; and generate a connection establishment success message after the connection is successfully established.

[0011] A connection establishment success message is sent back to the database driver so that the database driver can send a data query statement to the database server after receiving the connection establishment success message;

[0012] Based on the data query statement, determine the target query data and the data query volume. When the preset stream load batch push judgment condition is met based on the data query volume, generate and return a batch data to be pushed message. At the same time, based on the stream load protocol and HTTP connection related parameters, package the target query data and generate an HTTP push request.

[0013] The system sends an HTTP push request back to the database driver, which then determines the target query data, generates a push request response, sends it back to the database server, and closes the HTTP service process.

[0014] According to another aspect of the present invention, a batch data export device based on the stream load protocol is provided, configured on a database driver, the device comprising:

[0015] The success message feedback module is used to start the Hypertext Transfer Protocol (HTTP) service process, initiate a communication connection establishment request carrying HTTP connection-related parameters to the database server, so that the database server can establish a communication connection according to the HTTP connection-related parameters of the communication connection establishment request, and generate and send back a connection establishment success message after the connection is successful.

[0016] The HTTP push request feedback module is used to send a data query statement to the database server after receiving the connection establishment success message, so that the database server can determine the target query data and the data query volume based on the data query statement. When the data query volume determines that the preset stream load batch push judgment condition is met, the module generates and feeds back a batch data to be pushed message. At the same time, the module packages the target query data based on the stream load protocol and the HTTP connection related parameters, and generates and feeds back an HTTP push request.

[0017] The HTTP service process shutdown module is used to determine the target query data based on the HTTP push request, generate a push request response and send it back to the database server, and shut down the HTTP service process.

[0018] According to one aspect of the present invention, a batch data export device based on the stream load protocol is provided, configured on a database server, the device comprising:

[0019] The success message generation module is used to obtain a communication connection establishment request sent by the database driver, which carries HTTP connection-related parameters, establish a communication connection with the database driver according to the HTTP connection-related parameters of the communication connection establishment request, and generate a connection establishment success message after the connection is successful.

[0020] The query statement sending module is used to send a connection establishment success message back to the database driver so that the database driver can send a data query statement to the database server after receiving the connection establishment success message.

[0021] The HTTP push request generation module is used to determine the target query data and the data query volume based on the data query statement, and when the preset stream load batch push judgment condition is met based on the data query volume, generate and return a batch data to be pushed message. At the same time, the target query data is packaged based on the stream load protocol and the HTTP connection related parameters to generate an HTTP push request.

[0022] The HTTP service process shutdown module is used to respond to the HTTP push request, so that the database driver can determine the target query data based on the HTTP push request, generate a push request response and send it back to the database server, and shut down the HTTP service process.

[0023] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0024] At least one processor; and

[0025] A memory that is communicatively connected to at least one processor; wherein,

[0026] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to execute a batch data export method based on the stream load protocol according to any embodiment of the present invention.

[0027] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute a batch data export method based on a stream load protocol according to any embodiment of the present invention.

[0028] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements a batch data export method based on a stream load protocol according to any embodiment of the present invention.

[0029] This invention's embodiment initiates a Hypertext Transfer Protocol (HTTP) service process to send a communication connection establishment request carrying HTTP connection-related parameters to a database server. The database server then establishes a communication connection based on these parameters and generates and sends a connection establishment success message upon successful connection establishment. Upon receiving the connection establishment success message, a data query statement is sent to the database server. The database server determines the target query data and the query volume based on the query statement. If the query volume meets a preset stream load batch push condition, a batch data push message is generated and sent back. Simultaneously, the target query data is packaged based on the stream load protocol and the HTTP connection-related parameters, and an HTTP push request is generated and sent back. Based on the HTTP push request, the target query data is determined, a push request response is generated and sent back to the database server, and the HTTP service process is closed. Leveraging the streaming transmission characteristics of streaming loading protocols and combining HTTP connection parameters, the system can achieve fragmented and packaged transmission of target query data. This effectively improves the export efficiency of massive amounts of data and avoids problems such as link blocking and excessive memory consumption caused by traditional one-time transmission modes. It reduces connection failures, data loss, or packet corruption caused by non-standard parameter interaction, ensuring the integrity and reliability of data transmission. By adopting an HTTP service process management logic that starts on demand and closes after transmission is complete, it avoids the waste of system resources caused by long-term process resident, prevents transmission interruptions caused by premature process termination, optimizes system resource utilization, and can adapt to the data export needs between multiple nodes of distributed databases.

[0030] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart of a batch data export method based on the stream load protocol provided in Embodiment 1 of the present invention;

[0033] Figure 2This is a flowchart of a batch data export method based on the stream load protocol provided in Embodiment 2 of the present invention;

[0034] Figure 3A This is an interaction diagram of the database server and database driver for a batch data export method based on the stream load protocol provided in Embodiment 3 of the present invention.

[0035] Figure 3B This is an interaction diagram between the database server and the database driver for another batch data export method based on the stream load protocol provided in Embodiment 3 of the present invention.

[0036] Figure 4 This is a schematic diagram of the structure of a batch data export device based on the stream load protocol that can be deployed on the database driver side, according to Embodiment 4 of the present invention.

[0037] Figure 5 This is a schematic diagram of a batch data export device based on the stream load protocol that can be deployed on a database server, according to Embodiment 5 of the present invention.

[0038] Figure 6 This is a schematic diagram of the structure of an electronic device that implements a batch data export method based on the stream load protocol according to an embodiment of the present invention. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0041] Example 1

[0042] Figure 1 This is a flowchart illustrating a batch data export method based on the stream load protocol provided in Embodiment 1 of the present invention. It can be applied to a database driver and is suitable for scenarios involving the export of large amounts of data from a database. The method can be executed by a batch data export device based on the stream load protocol, applicable to the database driver. This batch data export device can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0043] S101. Start the Hypertext Transfer Protocol (HTTP) service process, and send a communication connection establishment request carrying HTTP connection-related parameters to the database server so that the database server can establish a communication connection based on the HTTP connection-related parameters in the communication connection establishment request, and generate and send back a connection establishment success message after the connection is successfully established.

[0044] S102. After receiving the connection establishment success message, a data query statement is sent to the database server so that the database server can determine the target query data and the data query volume based on the data query statement. When the preset stream load batch push judgment condition is met based on the data query volume, a batch data to be pushed message is generated and fed back. At the same time, the target query data is packaged based on the stream load protocol and HTTP connection related parameters, and an HTTP push request is generated and fed back.

[0045] S103. Based on the HTTP push request, determine the target query data, generate a push request response and send it back to the database server, and close the HTTP service process.

[0046] The database server can be a server-side system that provides database storage services and receives connection requests from the driver. The HTTP service process can be an HTTP service started by the database driver. The HTTP service can be a service started by the database driver to receive data pushed from the database server. The database driver can be a client system that initiates a communication connection establishment request to the database server. HTTP connection-related parameters can be the configuration information of the driver's HTTP service, such as the port number and communication address. The communication connection establishment request can be a database connection establishment message initiated by the database driver to the database server, carrying the HTTP connection-related parameters. The connection establishment success message can be a connection success confirmation message sent by the database server to the database driver after completing connection initialization.

[0047] For example, the database driver first starts an HTTP service process, which includes HTTP connection-related parameters such as port 8888. Then, it sends a connection establishment request to the database server with port number 8888. The database server records the parameter, completes the connection on the corresponding port, and sends a successful database connection notification to the database driver.

[0048] Here, the data query statement can be a query statement sent from the database driver to the database server, such as a DQL (Data Query Language) statement. The target query data can be the specific data retrieved from the database by the database server based on the data query statement. The data query volume can be the total number of target query data. The stream load protocol can be a protocol used for large-scale data transmission. The preset stream load batch push judgment condition can be a standard pre-set by relevant technical personnel on the database server to determine whether data needs to be pushed via the stream load protocol, such as a data query volume exceeding 100,000 records. The batch data to be pushed message can be a notification message sent by the database server to the database driver after determining that the stream load batch push judgment condition is met. The HTTP push request can be an HTTP PUT request sent by the database server to the driver after packaging the target query data based on the stream load protocol and using the HTTP connection parameters provided by the database driver.

[0049] For example, the database driver sends a data query statement, such as a DQL statement, to the server. The database server retrieves 600,000 user data records, which meets the condition set by relevant technical personnel that the number of data records exceeds 100,000. Therefore, the server sends a message to the database driver that it will push data via the stream load protocol and that the batch data is pending push. At the same time, it packages the 600,000 data records based on the stream load protocol and generates an HTTP push request pointing to the corresponding port of the database driver.

[0050] The push request response can be a confirmation message sent back to the server by the HTTP service on the database driver side after receiving the HTTP push request from the server and processing the data, in the format of the stream load protocol.

[0051] For example, the database driver receives the HTTP push request from the server through the HTTP service process, parses the data packaged by the stream load protocol, obtains the data corresponding to the query volume, then sends the HTTP push request response back to the database server, and finally closes the HTTP service process.

[0052] Furthermore, to clarify data transmission link management and ensure database connection security, in one optional embodiment, based on the HTTP push request, the target query data is determined, a push request response is generated and sent back to the database server, and the HTTP service process is closed, including:

[0053] Step a1: Based on the HTTP push request, determine the target query data, generate a push request response, and send it back to the database server.

[0054] Step a2: Generate a request to close the database connection and send it to the database server, so that the database server can generate and return a connection closure response based on the request.

[0055] Step a3: If the connection closure response indicates that the connection was closed successfully, then close the HTTP service process.

[0056] For example, the database driver parses the stream load protocol message in the HTTP push request, extracts the target query data and stores it, and then sends the data reception result back to the database server.

[0057] The database connection close request can be a request initiated by the database driver to the database server to terminate the database connection after completing data processing. The connection close response result can be the result sent back by the database server after receiving the database connection close request, performing the connection close operation, and sending it back to the database driver, such as connection close successful or connection close failed.

[0058] For example, after the database driver sends a push request response, it sends a request to the database server to close the database connection. After the database server releases its resources, it returns a connection closure response indicating that the database connection was successfully closed.

[0059] For example, if the database driver confirms that the connection closure response result is successful, it will shut down the HTTP service process on the corresponding port and release the port corresponding to its HTTP parameters.

[0060] The above solution ensures complete data transmission, avoids prolonged HTTP service downtime, and reduces waste of client host resources by processing HTTP push requests, handling target query data, initiating a connection closure request, and finally closing the HTTP service upon successful confirmation.

[0061] This invention's embodiment initiates a Hypertext Transfer Protocol (HTTP) service process to send a communication connection establishment request carrying HTTP connection-related parameters to a database server. The database server then establishes a communication connection based on these parameters and generates and sends a connection establishment success message upon successful connection establishment. Upon receiving the connection establishment success message, a data query statement is sent to the database server. The database server determines the target query data and the query volume based on the query statement. If the query volume meets a preset stream load batch push condition, a batch data push message is generated and sent back. Simultaneously, the target query data is packaged based on the stream load protocol and the HTTP connection-related parameters, and an HTTP push request is generated and sent back. Based on the HTTP push request, the target query data is determined, a push request response is generated and sent back to the database server, and the HTTP service process is closed. Leveraging the streaming transmission characteristics of streaming loading protocols and combining HTTP connection parameters, the system can achieve fragmented and packaged transmission of target query data. This effectively improves the export efficiency of massive amounts of data and avoids problems such as link blocking and excessive memory consumption caused by traditional one-time transmission modes. It reduces connection failures, data loss, or packet corruption caused by non-standard parameter interaction, ensuring the integrity and reliability of data transmission. By adopting an HTTP service process management logic that starts on demand and closes after transmission is complete, it avoids the waste of system resources caused by long-term process resident, prevents transmission interruptions caused by premature process termination, optimizes system resource utilization, and can adapt to the data export needs between multiple nodes of distributed databases.

[0062] Example 2

[0063] Figure 2 This is a flowchart illustrating a batch data export method based on the stream load protocol, provided in Embodiment 2 of the present invention. It can be applied to a database server. This embodiment is suitable for applications involving the efficient export of large volumes of data. The method can be executed by a batch data export device based on the stream load protocol, applicable to a database server. This batch data export device can be implemented in hardware and / or software and can be configured in an electronic device. Figure 2 As shown, the method includes:

[0064] S201. Obtain the communication connection establishment request sent by the database driver, which carries HTTP connection-related parameters. Establish a communication connection with the data driver based on the HTTP connection-related parameters in the communication connection establishment request, and generate a connection establishment success message after the connection is successfully established.

[0065] S202. Send a connection establishment success message to the database driver so that the database driver can send a data query statement to the database server after receiving the connection establishment success message.

[0066] S203. Based on the data query statement, determine the target query data and the data query volume. When the preset stream load batch push judgment condition is met based on the data query volume, generate and return a batch data to be pushed message. At the same time, based on the stream load protocol and HTTP connection related parameters, package the target query data and generate an HTTP push request.

[0067] S204. Feedback HTTP push request so that the database driver can determine the target query data based on the HTTP push request, generate a push request response and send it back to the database server, and close the HTTP service process.

[0068] For example, the database server obtains the connection establishment request sent by the database driver, establishes a connection based on the port corresponding to the connection establishment request, and generates an internal identifier indicating successful connection establishment if the connection is successful.

[0069] For example, the database server sends a message to the database driver indicating that the database connection has been successfully established and that the HTTP parameters have been recorded. After receiving the message, the database driver sends the corresponding data query statement, such as a DQL statement to query the target data.

[0070] For example, after receiving a data query statement, if the preset batch push conditions are met, the database server will send a batch data push message indicating that the data will be pushed via the stream load protocol. At the same time, the server will package the data based on the stream load protocol and generate an HTTP push request.

[0071] For example, the database server sends the generated HTTP push request to the database driver. After receiving the HTTP push request, the database driver parses the request content to determine the target query data, processes the data, generates a push request response, sends it back to the database server, and finally shuts down the local HTTP service process.

[0072] Furthermore, to ensure the release of database connections and service resources, avoid resource occupation issues, and improve the stability and resource utilization of large-volume data transmission, in an optional embodiment, an HTTP push request is fed back. This allows the database driver to determine the target query data based on the HTTP push request, generate a push request response and send it back to the database server, and then close the HTTP service process, including:

[0073] Step b1: Feedback HTTP push request so that the database driver can determine the target query data based on the HTTP push request, generate a push request response and feed it back to the database server, and generate a database connection closure request and send it to the database server.

[0074] Step b2: Generate a connection closure response based on the database connection closure request.

[0075] Step b3: Feedback the connection closure response result so that the database driver can close the HTTP service process when it determines that the connection closure response result is successful.

[0076] For example, the database server sends an HTTP push request to the driver. The driver parses the request to obtain the target query data, processes the data, and generates a push request response to send back to the database server. At the same time as sending back the response, the database driver simultaneously generates a request to close the database connection and sends the request to the database server.

[0077] For example, after receiving a request from the database driver to close the database connection, the database server performs database communication link release and connection resource reclamation operations. After the operations are completed, a connection closure response result is generated, which includes whether the connection was closed successfully or failed.

[0078] For example, the database server sends the generated connection closure response to the database driver. The database driver verifies the result, and if it determines that the connection was successfully closed, it immediately executes the shutdown operation of the HTTP service process, releasing the port and related resources.

[0079] The above solution pushes data from the database server, receives a response from the driver and initiates a request to close the database connection, generates a closing result from the database server, and finally shuts down the HTTP service after successful verification by the database driver. This achieves the orderly termination of the database communication connection and the HTTP service on the driver side, significantly improving the reliability of the interaction between the database server and the driver side, and further optimizing the overall resource utilization efficiency of the system.

[0080] Furthermore, to adapt to cross-node data synchronization scenarios in database clusters and improve the efficiency of large-volume cross-node data transmission, in an optional embodiment, the above method further includes:

[0081] Step c1: In response to the data synchronization request, determine the amount of data to be synchronized. If the amount of data to be synchronized reaches the preset data amount threshold, generate an HTTP push synchronization request based on the stream load protocol according to the data to be synchronized.

[0082] Step c2: Based on the data synchronization request, determine the target data node and send the HTTP push synchronization request to the target data node so that the target data node can determine the data to be synchronized based on the HTTP push synchronization request and perform data export operation on the data to be synchronized.

[0083] The data to be synchronized can be a batch of data that needs to be synchronized between the database server and the target data node. The preset data volume threshold can be a value pre-set by relevant technical personnel. The HTTP push synchronization request can be an HTTPPUT request sent by the database server to the target data node after packaging the data to be synchronized based on the stream load protocol.

[0084] For example, after receiving a data synchronization request, the database server first filters out the data to be synchronized and counts its actual data volume. If the data volume of the data to be synchronized reaches the preset data volume threshold, then all the data to be synchronized is packaged and encapsulated based on the stream load protocol, and finally an HTTP push synchronization request for cross-node transmission is generated.

[0085] The target data node can be a database node that needs to receive the data to be synchronized.

[0086] For example, the database server extracts the identification information of the target data node from the data synchronization request, accurately locates the corresponding target data node, and then sends the generated HTTP push synchronization request to the target data node. After receiving the request, the target data node parses the request content to determine the data to be synchronized and performs the data export operation.

[0087] The above solution applies the stream load protocol to cross-node data synchronization scenarios. For data that has reached the threshold and needs to be synchronized, a standardized HTTP push request is used to complete the targeted push and export, which greatly reduces the number of interactions in cross-node data synchronization, significantly improves the synchronization efficiency of large amounts of data, and effectively ensures data consistency between the database server and the target data node.

[0088] The technical solution of this invention can be applied to a database server that receives a communication connection establishment request carrying HTTP connection-related parameters sent by a database driver. Based on these parameters, the server establishes a communication connection with the database driver. Upon successful connection establishment, a connection establishment success message is generated and sent back to the database driver. Upon receiving this message, the database driver sends a data query statement to the database server. Based on the query statement, the server determines the target query data and the query volume. When the query volume meets a preset stream load batch push condition, the server generates and sends back a batch data to be pushed message. Simultaneously, based on the stream load protocol and HTTP connection-related parameters, the server packages the target query data, generates an HTTP push request, and sends it back. The database driver then determines the target query data, generates a push request response, sends it back to the database server, and closes the HTTP service process. It ensures the stability of the communication link between the driver and the server, reduces connection failures caused by parameter mismatch, determines the target query data and data query volume based on the data query statement sent by the driver, and accurately triggers the data packaging process of the streaming loading protocol through preset batch push judgment conditions, realizing efficient transmission of massive data, avoiding problems such as excessive server memory consumption and transmission lag caused by one-time data processing, and completes the closed-loop management of the whole process by feeding back batch data to be pushed and HTTP push requests to the driver, which not only ensures the integrity and reliability of data export, but also avoids the waste of system resources caused by invalid process resident.

[0089] Example 3

[0090] Figure 3A This is an interaction diagram of the database server and database driver for a batch data export method based on the stream load protocol provided in Embodiment 3 of the present invention. This embodiment provides a preferred example based on the above embodiments.

[0091] S301. The database driver first starts the HTTP server, and then sends a communication connection establishment request carrying HTTP connection-related parameters to the database server to request the establishment of a data transmission link with the database server.

[0092] S302. After receiving the connection establishment request, the database server establishes a communication connection with the database driver, generates a connection establishment success message, and sends it back to the driver.

[0093] S303. After receiving the connection establishment success message, the database driver sends a DQL statement to the server.

[0094] S304. After receiving the DQL statement, the database server checks the amount of data obtained from the query and determines whether to enable stream load batch push. If it is not required, the query results are directly sent back to the driver in the response message. If it is required, the server notifies the driver through a specific message field that data will be pushed through the stream load protocol, and the response message does not contain detailed data.

[0095] S305. The database server packages the target query data into a stream load protocol message and pushes the data to the driver through an HTTP PUT request based on the HTTP connection parameters.

[0096] S306. After receiving and processing the data pushed by the server, the database driver sends an HTTP PUT request response back to the server in the format of the stream load protocol.

[0097] S307. After the data push is completed, the database server notifies the driver that the push has ended through a specific control message.

[0098] S308. The database driver sends a communication connection closure request to the server. After receiving the request, the database server performs the connection closure operation and sends a confirmation message to the driver confirming the successful closure. After receiving the message, the database driver closes the HTTP server.

[0099] Figure 3B This is an interaction diagram of the database server and database driver for another batch data export method based on the stream load protocol provided in Embodiment 3 of the present invention.

[0100] S411, The database driver sends a communication connection establishment request to the database server.

[0101] S412. The database server receives and verifies the request. If successful, it establishes a communication connection with the database driver, generates a connection establishment success message, and sends it back to the driver.

[0102] S413. After receiving the connection establishment success message, the database driver sends a DQL statement carrying HTTP connection-related parameters to the server.

[0103] S414. After receiving the DQL statement, the database server checks the amount of data obtained from the query and determines whether to enable stream load batch push. If it is not required, the query results are directly sent to the driver in the response message. If it is required, the server notifies the driver through a specific message field that data will be pushed through the stream load protocol. The response message does not contain detailed data, and a preparation completion message is generated and sent to the database driver.

[0104] S415. After receiving the preparation completion message, the database driver starts the HTTP server and stores the relevant configuration information returned by the storage server.

[0105] S416. The database server packages the target query data into a stream load protocol message based on the HTTP connection parameters and pushes the data to the driver through an HTTP PUT request.

[0106] S417. After receiving and processing the data pushed by the server, the database driver sends an HTTP PUT request response to the server in the format of the stream load protocol; after confirming that the data reception is complete, it sends a batch push completion message to the server.

[0107] S418. After receiving the batch push completion message, the database server generates a completion response message and sends it back to the database driver. After receiving the message, the database driver closes the HTTP server.

[0108] Example 4

[0109] Figure 4 This is a schematic diagram of a batch data export device based on the stream load protocol, which can be deployed on a database driver, according to Embodiment 4 of the present invention. The batch data export device based on the stream load protocol, which can be deployed on a database driver, is suitable for scenarios involving the export of large amounts of data. This batch data export device based on the stream load protocol can be implemented in hardware and / or software. It can be applied to a batch data export method based on the stream load protocol and can be specifically configured in a controller, such as... Figure 4 As shown, the device includes: a success message feedback module 401, an HTTP push request feedback module 402, and an HTTP service process shutdown module 403. Wherein:

[0110] The success message feedback module 401 is used to start the Hypertext Transfer Protocol (HTTP) service process, initiate a communication connection establishment request carrying HTTP connection-related parameters to the database server, so that the database server can establish a communication connection according to the HTTP connection-related parameters of the communication connection establishment request, and generate and send back a connection establishment success message after the connection is successful.

[0111] The HTTP push request feedback module 402 is used to send a data query statement to the database server after receiving a successful connection establishment message. The database server can determine the target query data and the data query volume based on the data query statement. When the data query volume determines that the preset stream load batch push judgment condition is met, the module generates and sends back a batch data to be pushed message. At the same time, the module packages the target query data based on the stream load protocol and HTTP connection-related parameters, generates and sends back an HTTP push request.

[0112] The HTTP service process shutdown module 403 is used to determine the target query data based on the HTTP push request, generate a push request response and send it back to the database server, and shut down the HTTP service process.

[0113] This invention's embodiment initiates a Hypertext Transfer Protocol (HTTP) service process to send a communication connection establishment request carrying HTTP connection-related parameters to a database server. The database server then establishes a communication connection based on these parameters and generates and sends a connection establishment success message upon successful connection establishment. Upon receiving the connection establishment success message, a data query statement is sent to the database server. The database server determines the target query data and the query volume based on the query statement. If the query volume meets a preset stream load batch push condition, a batch data push message is generated and sent back. Simultaneously, the target query data is packaged based on the stream load protocol and the HTTP connection-related parameters, and an HTTP push request is generated and sent back. Based on the HTTP push request, the target query data is determined, a push request response is generated and sent back to the database server, and the HTTP service process is closed. Leveraging the streaming transmission characteristics of streaming loading protocols and combining HTTP connection parameters, this solution enables the segmented and packaged transmission of target query data. This effectively improves the efficiency of exporting massive amounts of data, avoiding problems such as link congestion and excessive memory consumption caused by traditional one-time transmission modes. It reduces connection failures, data loss, or packet corruption caused by non-standard parameter interaction, ensuring the integrity and reliability of data transmission. By adopting an HTTP service process management logic that starts on demand and closes upon completion of transmission, it avoids the waste of system resources caused by long-term process resident status and prevents transmission interruptions caused by premature process termination, optimizing system resource utilization. Furthermore, it can adapt to the data export requirements between multiple nodes in distributed databases, enhancing the adaptability and scalability of the solution and providing efficient and stable technical support for various large-scale data processing businesses.

[0114] Optionally, the HTTP service process shutdown module 403 is used for:

[0115] Based on the HTTP push request, determine the target query data, generate a push request response, and send it back to the database server.

[0116] Generate a request to close the database connection and send it to the database server, so that the database server can generate and return a connection closure response based on the request.

[0117] If the connection closure response indicates that the connection was closed successfully, then the HTTP service process will be shut down.

[0118] The batch data export device provided in this embodiment of the invention can execute a batch data export method based on the stream load protocol provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0119] Example 5

[0120] Figure 5 This is a schematic diagram of a batch data export device based on the stream load protocol, applicable to a database server, provided in Embodiment 4 of the present invention. The batch data export device based on the stream load protocol, deployable on a database server, provided in this embodiment of the invention, is suitable for scenarios involving the export of large volumes of data. This batch data export device based on the stream load protocol can be implemented in hardware and / or software. This batch data export device based on the stream load protocol, deployable on a database server, can be applied to a batch data export method based on the stream load protocol, and can be specifically configured in a controller, such as... Figure 5 As shown, the device includes: a success message generation module 501, a query statement sending module 502, an HTTP push request generation module 503, and an HTTP service process shutdown module 504. Wherein:

[0121] The success message generation module 501 obtains the communication connection establishment request sent by the database driver, which carries HTTP connection-related parameters. Based on the HTTP connection-related parameters in the communication connection establishment request, it establishes a communication connection with the data driver and generates a connection establishment success message after the connection is successfully established.

[0122] The query statement sending module 502 sends a connection establishment success message to the database driver, so that the database driver can send a data query statement to the database server after receiving the connection establishment success message.

[0123] The HTTP push request generation module 503 determines the target query data and the data query volume based on the data query statement. When the preset stream load batch push judgment condition is met based on the data query volume, it generates and sends back a batch data to be pushed message. At the same time, it packages the target query data based on the stream load protocol and HTTP connection-related parameters to generate an HTTP push request.

[0124] The HTTP service process shutdown module 504 sends an HTTP push request so that the database driver can determine the target query data based on the HTTP push request, generate a push request response and send it back to the database server, and then shut down the HTTP service process.

[0125] The technical solution of this invention can be applied to a database server that receives a communication connection establishment request carrying HTTP connection-related parameters sent by a database driver. Based on these parameters, the server establishes a communication connection with the database driver. Upon successful connection establishment, a connection establishment success message is generated and sent back to the database driver. Upon receiving this message, the database driver sends a data query statement to the database server. Based on the query statement, the server determines the target query data and the query volume. When the query volume meets a preset stream load batch push condition, the server generates and sends back a batch data to be pushed message. Simultaneously, based on the stream load protocol and HTTP connection-related parameters, the server packages the target query data, generates an HTTP push request, and sends it back. The database driver then determines the target query data, generates a push request response, sends it back to the database server, and closes the HTTP service process. It ensures the stability of the communication link between the driver and the server, reduces connection failures caused by parameter mismatch, determines the target query data and data query volume based on the data query statement sent by the driver, and accurately triggers the data packaging process of the streaming loading protocol through preset batch push judgment conditions, realizing efficient transmission of massive data, avoiding problems such as excessive server memory consumption and transmission lag caused by one-time data processing, and completes the closed-loop management of the whole process by feeding back batch data to be pushed and HTTP push requests to the driver, which not only ensures the integrity and reliability of data export, but also avoids the waste of system resources caused by invalid process resident.

[0126] Optionally, the HTTP service process closes module 504, specifically for:

[0127] It sends an HTTP push request to the database driver to determine the target query data, generate a push request response and send it back to the database server, and also generates a database connection closure request and sends it to the database server.

[0128] Generate a connection closure response based on the database connection closure request.

[0129] The system provides a response indicating that the connection was successfully closed, allowing the database driver to shut down the HTTP service process upon confirming that the connection was closed successfully.

[0130] Optionally, the device further includes:

[0131] The HTTP push synchronization request generation module is used to respond to data synchronization requests, determine the amount of data to be synchronized, and if the amount of data to be synchronized reaches a preset data amount threshold, then generate an HTTP push synchronization request based on the stream load protocol according to the data to be synchronized.

[0132] The data export push module is used to determine the target data node based on the data synchronization request, and send the HTTP push synchronization request to the target data node so that the target data node can determine the data to be synchronized based on the HTTP push synchronization request and perform data export operation on the data to be synchronized.

[0133] The batch data export device provided in this embodiment of the invention can execute a batch data export method based on the stream load protocol provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0134] Example 6

[0135] Figure 6 A schematic diagram of an electronic device 60 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0136] like Figure 6As shown, the electronic device 60 includes at least one processor 61 and a memory, such as a read-only memory (ROM) 62 and a random access memory (RAM) 63, communicatively connected to the at least one processor 61. The memory stores computer programs executable by the at least one processor. The processor 61 can perform various appropriate actions and processes based on the computer program stored in the ROM 62 or loaded from storage unit 68 into the RAM 63. The RAM 63 can also store various programs and data required for the operation of the electronic device 60. The processor 61, ROM 62, and RAM 63 are interconnected via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0137] Multiple components in electronic device 60 are connected to I / O interface 55, including: input unit 66, such as keyboard, mouse, etc.; output unit 67, such as various types of monitors, speakers, etc.; storage unit 68, such as disk, optical disk, etc.; and communication unit 69, such as network card, modem, wireless transceiver, etc. Communication unit 69 allows electronic device 60 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0138] Processor 61 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 61 performs the various methods and processes described above, such as a batch data export method based on a stream load protocol.

[0139] In some embodiments, a batch data export method based on a stream load protocol may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 68. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 60 via ROM 62 and / or communication unit 69. When the computer program is loaded into RAM 63 and executed by processor 61, one or more steps of the batch data export method based on a stream load protocol described above may be performed. Alternatively, in other embodiments, processor 61 may be configured as a batch data export method based on a stream load protocol by any other suitable means (e.g., by means of firmware).

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0145] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0146] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for batch data export based on the stream load protocol, characterized in that, Applied to the database driver side, including: Start the Hypertext Transfer Protocol (HTTP) service process, initiate a communication connection establishment request carrying HTTP connection-related parameters to the database server, so that the database server can establish a communication connection according to the HTTP connection-related parameters of the communication connection establishment request, and generate and send back a connection establishment success message after the connection is successfully established. After receiving the connection establishment success message, a data query statement is sent to the database server so that the database server can determine the target query data and the data query volume based on the data query statement. When the preset stream load batch push judgment condition is met based on the data query volume, a batch data to be pushed message is generated and fed back. At the same time, the target query data is packaged based on the stream load protocol and the HTTP connection related parameters, and an HTTP push request is generated and fed back. Based on the HTTP push request, the target query data is determined, a push request response is generated and sent back to the database server, and the HTTP service process is closed.

2. The method according to claim 1, characterized in that, The step of determining the target query data based on the HTTP push request, generating a push request response and sending it back to the database server, and closing the HTTP service process includes: Based on the HTTP push request, the target query data is determined, a push request response is generated, and the response is sent back to the database server. A request to close the database connection is generated and sent to the database server, so that the database server can generate and return a connection closure response result based on the request. If the connection closure response indicates that the connection was closed successfully, then the HTTP service process is closed.

3. A method for batch data export based on the stream load protocol, characterized in that, Applied to the database server, including: Obtain a communication connection establishment request sent by the database driver, which carries HTTP connection-related parameters; establish a communication connection with the database driver based on the HTTP connection-related parameters of the communication connection establishment request; and generate a connection establishment success message after the connection is successfully established. The connection establishment success message is sent back to the database driver so that the database driver can send a data query statement to the database server after receiving the connection establishment success message; Based on the data query statement, the target query data and the data query volume are determined. When the preset stream load batch push judgment condition is met based on the data query volume, a batch data push message is generated and fed back. At the same time, the target query data is packaged based on the stream load protocol and the HTTP connection related parameters to generate an HTTP push request. The HTTP push request is fed back so that the database driver can determine the target query data based on the HTTP push request, generate a push request response and send it back to the database server, and then close the HTTP service process.

4. The method according to claim 3, characterized in that, The step of feeding back the HTTP push request, so that the database driver can determine the target query data based on the HTTP push request, generate a push request response and feed it back to the database server, and close the HTTP service process, includes: The HTTP push request is fed back so that the database driver can determine the target query data based on the HTTP push request, generate a push request response and feed it back to the database server, and generate a database connection close request and send it to the database server. Based on the request to close the database connection, a connection closure response result is generated; The connection closure response result is fed back so that the database driver can close the HTTP service process when it determines that the connection closure response result is successful.

5. The method according to claim 3, characterized in that, The method further includes: In response to a data synchronization request, the amount of data to be synchronized is determined. If the amount of data to be synchronized reaches a preset data amount threshold, an HTTP push synchronization request is generated based on the stream load protocol according to the data to be synchronized. Based on the data synchronization request, a target data node is determined, and the HTTP push synchronization request is sent to the target data node so that the target data node can determine the data to be synchronized based on the HTTP push synchronization request and perform a data export operation on the data to be synchronized.

6. A batch data export device based on the stream load protocol, characterized in that, Configured on the database driver side, including: The successful message feedback module starts the Hypertext Transfer Protocol (HTTP) service process, initiates a communication connection establishment request carrying HTTP connection-related parameters to the database server, so that the database server can establish a communication connection according to the HTTP connection-related parameters of the communication connection establishment request, and generates and sends back a connection establishment success message after the connection is successfully established. After receiving the connection establishment success message, the HTTP push request feedback module sends a data query statement to the database server so that the database server can determine the target query data and the data query volume based on the data query statement. When the data query volume determines that the preset stream load batch push judgment condition is met, the module generates and feeds back a batch data to be pushed message. At the same time, the module packages the target query data based on the stream load protocol and the HTTP connection related parameters, and generates and feeds back an HTTP push request. The HTTP service process shutdown module determines the target query data based on the HTTP push request, generates a push request response and sends it back to the database server, and shuts down the HTTP service process.

7. A batch data export device based on the stream load protocol, characterized in that, Configured on the database server, including: The success message generation module obtains a communication connection establishment request sent by the database driver, which carries HTTP connection-related parameters. Based on the HTTP connection-related parameters of the communication connection establishment request, it establishes a communication connection with the database driver and generates a connection establishment success message after the connection is successful. The query statement sending module sends a connection establishment success message back to the database driver, so that the database driver can send a data query statement to the database server after receiving the connection establishment success message. The HTTP push request generation module determines the target query data and the data query volume based on the data query statement. When the preset stream load batch push judgment condition is met based on the data query volume, it generates and sends back a batch data to be pushed message. At the same time, it packages the target query data based on the stream load protocol and the HTTP connection-related parameters to generate an HTTP push request. The HTTP service process shutdown module sends back the HTTP push request so that the database driver can determine the target query data based on the HTTP push request, generate a push request response and send it back to the database server, and shut down the HTTP service process.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a batch data export method based on the stream load protocol according to any one of claims 1-2 and / or 3-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute a batch data export method based on any one of claims 1-2 and / or 3-5.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements a batch data export method based on the stream load protocol according to any one of claims 1-2 and / or 3-5.

Images