Data exporting method, device, system and equipment for pure memory database and medium

Abstract

The invention discloses a data exporting method, device, system and equipment for a pure memory database and a medium, and the method comprises the steps that to-be-processed data of the pure memory database is determined, and the to-be-processed data is streaming business data collected by the pure memory database from a business system; determining a serial number of the to-be-processed data according to the partition ID and the table name of the to-be-processed data, and performing filling processing on the serial number to obtain an update number; and extracting corresponding to-be-transmitted data according to the update number, and transmitting the to-be-transmitted data to a client in a single-thread transmission mode. According to the invention, the serial number is filled, so that the serial number can be matched with the serial number sent by the previous time node, and the serial numbers of the data sent before and after can be connected, thereby ensuring that the data received by the client is consistent before and after, and ensuring the consistency of the data. The data deviation is reduced; and the data transmission precision is improved.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to a method, apparatus, system, device, and medium for exporting data from a pure in-memory database. Background Technology

[0002] With the development of the digital economy, more and more companies are launching online business services and providing services to users online. This online processing generates various business data. To properly store and manage this data, different companies set up and implemented databases to manage their business data.

[0003] One commonly used type of database is the traditional relational database, which uses a relational model to organize data. In a relational database, these rows and columns are called tables, and a group of tables makes up the database. In modern finance, telecommunications, and the Internet of Things, relational databases can store data in the form of minimal relational tables, making data management very clear and easy for users to view. However, in practice, relational databases heavily emphasize data consistency, and their efficiency becomes very poor when dealing with massive amounts of data, especially under high-concurrency read / write operations, where performance degrades drastically. To improve read / write performance and concurrency capabilities, pure in-memory databases (also known as memory databases, such as ShuangzhaoDB) are used for data management.

[0004] However, pure in-memory databases have the following technical problems when performing data read and write operations: Although pure in-memory databases can perform data read and write operations in parallel to improve data export efficiency, the transmission efficiency of different nodes is different when reading and writing data in parallel, resulting in a time difference in exported data, which in turn leads to inconsistencies in the data. Data transmitted earlier and data transmitted later cannot be connected, resulting in deviations or garbled characters in the exported data, making it difficult to guarantee data consistency. Summary of the Invention

[0005] This invention provides a method, apparatus, system, device, and medium for exporting data from a pure in-memory database, which can solve the technical problem that the exported data from existing pure in-memory databases has deviations and it is difficult to guarantee data consistency.

[0006] A first aspect of this invention provides a data export method for a pure in-memory database, the method comprising: Identify the data to be processed in the pure in-memory database, wherein the data to be processed is streaming business data collected by the pure in-memory database from the business system; The sequence number of the data to be processed is determined based on the partition ID and table name of the data to be processed, and the sequence number is filled to obtain an updated number. The filling process is the process of updating the sequence number when there is a gap between the sequence number and the sequence number of the previous time node. Extract the corresponding data to be transmitted based on the update number, and transmit the data to the client using a single-threaded transmission method.

[0007] A second aspect of the present invention provides a data export apparatus for a pure in-memory database, the apparatus comprising: The determination module is used to determine the data to be processed in the pure memory database, wherein the data to be processed is streaming business data collected by the pure memory database from the business system. The filling module is used to determine the sequence number of the data to be processed based on the partition ID and table name of the data to be processed, and to fill the sequence number to obtain an updated number. The filling process is the process of updating the sequence number when there is a gap between the sequence number and the sequence number of the previous time node. The transmission module is used to extract the corresponding data to be transmitted according to the update number, and transmit the data to be transmitted to the client through a single-threaded transmission method.

[0008] A third aspect of this invention provides a data export system for a pure in-memory database, the system comprising: a pure in-memory database, a business terminal, and a client; The pure memory database is connected to both the business terminal and the client. The pure memory database has several data storage node units, and each node unit has a data transmission module. The data transmission module is adapted to the data export method of the pure memory database described above.

[0009] A fourth aspect of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data export method of the pure memory database as described above.

[0010] A fifth aspect of the present invention provides a computer-readable storage medium storing a computer-executable program for causing a computer to perform the data export method for a pure in-memory database as described above.

[0011] Compared to existing technologies, the data export method, apparatus, system, device, and medium provided by this invention for a pure in-memory database have the following advantages: This invention can determine the data to be processed in a pure in-memory database; determine the sequence number of the data to be processed based on the partition ID and table name of the data to be processed, and fill in the sequence number to obtain an update number; extract the corresponding data to be transmitted based on the update number, and transmit the data to be transmitted to the client through a single-threaded transmission method. By filling in the sequence number, this invention ensures that the sequence number can be aligned with the sequence number sent at a previous time node, allowing the sequence numbers of data sent before and after to be connected, thereby ensuring that the data received by the client remains consistent, thus ensuring data consistency, reducing data deviation, and improving the accuracy of data transmission. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the structure of a data export system for a pure in-memory database according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the combined structure of node units provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating a data export method for a pure in-memory database according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the sequence numbers of different nodes provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a data export device for a pure memory database provided in an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] With the development of the digital economy, more and more companies are launching online business services and providing services to users online. This online processing generates various business data. To properly store and manage this data, different companies set up and implemented databases to manage their business data.

[0015] One commonly used type of database is the traditional relational database, which uses a relational model to organize data. In a relational database, these rows and columns are called tables, and a group of tables makes up the database. In modern finance, telecommunications, and the Internet of Things, relational databases can store data in the form of minimal relational tables, making data management very clear and easy for users to view. However, in practice, relational databases heavily emphasize data consistency, and their efficiency becomes very poor when dealing with massive amounts of data, especially under high-concurrency read / write operations, where performance degrades drastically. To improve read / write performance and concurrency capabilities, pure in-memory databases (also known as memory databases, such as ShuangzhaoDB) are used for data management.

[0016] However, pure in-memory databases have the following technical problems when performing data read and write operations: Although pure in-memory databases can perform data read and write operations in parallel to improve data export efficiency, the transmission efficiency of different nodes is different when reading and writing data in parallel, resulting in a time difference in exported data, which in turn leads to inconsistencies in the data. Data transmitted earlier and data transmitted later cannot be connected, resulting in deviations or garbled characters in the exported data, making it difficult to guarantee data consistency.

[0017] To address the aforementioned issues, the following detailed embodiments will be used to describe and explain a data export method, apparatus, system, device, and medium for a pure in-memory database provided in this application.

[0018] To address the technical issue of data inconsistency and deviations in data exported from existing pure in-memory databases, this paper refers to... Figure 1 The diagram shows a schematic of the structure of a data export system for a pure in-memory database according to an embodiment of the present invention.

[0019] As an example, the data export system for the pure in-memory database may include: a pure in-memory database, a business terminal, and a client. The pure memory database is connected to the business terminal and the client respectively. The pure memory database has several data storage node units, and each node unit has a data transmission module. The data transmission module is applicable to the data export method of the pure memory database provided by the present invention.

[0020] Reference Figure 1In one embodiment, the business side can be a database business system, and the client side can be an external data system. A pure in-memory database (also known as an in-memory database) is a database that directly manipulates data within memory. Specifically, business data from the business side can be stored in a pure in-memory database. Compared to disk, the data read / write speed of a pure in-memory database is several orders of magnitude faster. Storing data in memory significantly improves application performance compared to accessing it from disk.

[0021] Reference Figure 2 The diagram illustrates a combined structure of a node unit according to an embodiment of the present invention. In one embodiment, the node unit may include an export manager, an export generator, an export processor, an export coordinator, and an export client.

[0022] The export manager can connect to the export generator, export processor, and export coordinator respectively, and the export coordinator can communicate with the export client.

[0023] In practice, the exported data can be processed through node units to ensure consistency, avoid data deviation, and improve the accuracy of the exported data.

[0024] In this embodiment, the present invention provides a data export system for a pure in-memory database. Its advantages are: the present invention can process the exported data through node units, which can keep the exported data consistent, avoid deviations in the exported data, and improve the accuracy of the exported data.

[0025] To address the technical issue of data inconsistency and deviations in data exported from existing pure in-memory databases, this paper refers to... Figure 3 The diagram shows a flowchart of a data export method for a pure in-memory database according to an embodiment of the present invention.

[0026] As an example, the data export method for the pure in-memory database may include: S11. Determine the data to be processed in the pure memory database, wherein the data to be processed is streaming business data collected by the pure memory database from the business system.

[0027] In one embodiment, the pure in-memory database may be shuangzhaoDB. Before exporting data, in order for shuangzhaoDB to support and use the relevant node units, the following preparatory work needs to be performed: The first item is to configure the relevant export configuration items for ShuangzhaoDB, including setting the export type. The ShuangzhaoDB database supports exporting the following data objects: exporting to the file system; exporting via HTTP, which can be used to export to Hadoop databases; exporting to JDBC, which can be used to export to any database system that supports the JDBC protocol; exporting to Kafka system; exporting to Elasticsearch system, etc.

[0028] Taking exporting to Kafka as an example, the configuration includes a list of Kafka servers, message communication mode, response timeout, the correspondence between Kafka topic information and the corresponding tables in the shuangzhaoDB database, and the relevant attributes required for Kafka producers.

[0029] In the database configuration, there are several important options that need to be configured: exportoverflow: path: This configuration option specifies the disk path for persistently saving data when export speed is matched.

[0030] The second step involves creating relevant streaming data tables in the ShuangzhaoDB database to store the data exported from these tables. In the ShuangzhaoDB database system, this export can be created in two ways: The first method: Create the corresponding export using a flow table, whose DDL is shown below: CREATE STREAM alerts; EXPORT TO TARGET systemlog; ( {column-definition} [,...] ).

[0031] Among them, the Stream table is a data structure specifically introduced by ShuangzhaoDB to support streaming data processing. Streams only support INSERT operations, do not store historical data (i.e., only support data in transit), and do not support SELECT, UPDATE, or DELETE operations; otherwise, an error will occur.

[0032] The second method: Specify the corresponding export for a general database table, but during the definition process, you need to specify which operations are performed on the table to perform streaming export, as shown below: CREATE TABLE orders; EXPORT TO TARGET orderprocessing ON INSERT, UPDATE; ( {column-definition} [,...] ); It should be noted that the statement above defines that the ShuangzhaoDB execution engine will stream the data to the target system when performing inserts and updates on the orders table.

[0033] In one embodiment, the export process is handled by node units. Specifically, in ShuangzhaoDB, streaming data export can be managed by the export manager. The export manager is responsible for creating export processors, processing exported data, processing export data sources, and maintaining distributed consistency in the export system based on changes to the ShuangzhaoDB data cluster. Its basic structure is as follows: Figure 2 As shown: In actual operation, each device in the node unit can be initialized, and then the data export process can be performed after initialization.

[0034] Specifically, based on the deployment information of ShuangzhaoDB, a corresponding export connector can be created, and the data tables to be exported can be determined by reading the Catalog information of ShuangzhaoDB. The Catalog contains the schema information of all databases, and the exported database tables will have the export keyword defined when the data table is defined.

[0035] Next, the corresponding export processor can be initialized using reflection, followed by the export generator to manage the export data source. An export data source is set for each configured export table, and the data source is associated with the corresponding export client. The data source definition is determined based on the table definition in the Catalog, with each export table corresponding to a specific data source. Furthermore, a ZooKeeper coordination mailbox can be created for export confirmation messages to receive confirmation messages from the export client, and Kafka export client information can be initialized.

[0036] After completing the above initialization, the data to be exported can be determined. This data consists of streaming business data collected by the pure in-memory database from the business system. After collecting the streaming business data from the business side, the pure in-memory database can store it on different nodes. When data needs to be exported and transmitted, an export request can be obtained from the client, and the data to be exported can be determined based on this request.

[0037] S12. Determine the sequence number of the data to be processed based on the partition ID and table name of the data to be processed, and perform a filling process on the sequence number to obtain an updated number. The filling process is a process of updating the sequence number when there is a gap between the sequence number and the sequence number of the previous time node.

[0038] Since streaming business data is stored on different nodes of a pure in-memory database, the sequence number of the data to be processed on a node can be determined based on the partition ID and table name of the data to be processed. Thus, the node where the data to be processed is located can be determined based on the sequence number, and the data source to be exported can be determined within the node.

[0039] When storing data, business data may be stored on different nodes. The client can extract data from different nodes, so that the sequence number of the data recorded on each node is different. After determining one of the sequence numbers of the data to be processed, the sequence number can be filled to obtain an updated number. The filling process is the process of updating the sequence number when there is a gap between the sequence number of the data to be processed and other sequence numbers.

[0040] In an optional embodiment, the step of determining the sequence number of the data to be processed based on the partition ID and table name of the data to be processed, and filling the sequence number to obtain an update number, may include the following sub-steps: S121. Obtain the partition ID and table name from the data to be processed, and determine the data source of the streaming based on the partition ID and the table name.

[0041] S122. Obtain the sequence number contained in the data source, and determine whether the sequence number is consistent with the node number according to the node tracker, wherein the node number is the sequence number recorded after the earlier time node sends data to the client.

[0042] S123. If the serial number is inconsistent with the node number, the serial number is updated and replaced with the node number to obtain the updated number.

[0043] In one embodiment, data to be processed can be received from the execution engine of ShuangzhaoDB, processed, and then passed to the corresponding client. Upon receiving a client request, the data can be synchronized to the entire distributed system using ZooKeeper's distributed consistency algorithm. The export manager receives data to be processed from the execution engine of ShuangzhaoDB. The data to be processed can be data blocks. It analyzes the content of the data blocks to obtain the partition ID and table name to determine the data source. It then uses the query export generator to find the data source corresponding to the corresponding data block (determined by the partition ID and table name) and obtains the sequence number of the streaming data contained in the data.

[0044] It's important to note that the data source is streaming data, continuously generated, and data from different points in time is distinguished by sequence numbers. The data source updates its own sequence number tracker, which needs to be updated based on data processing, including both the tracker's sequence numbers and the sequence numbers that have already been processed. Each data block includes a sequence number, and the data source records: currently unacknowledged sequence numbers, sequence numbers for which acknowledgments have been received, and any potential gaps in the sequence count. (See reference...) Figure 4 This diagram illustrates the sequence numbers of different nodes according to an embodiment of the present invention. When the data source receives the data, it updates the unconfirmed sequence numbers based on the sequence numbers of the data blocks. The sequence numbers are primarily used to ensure the sequential nature of streaming data.

[0045] Since ShuangzhaoDB is a distributed database, the exported tables will create data sources in different sites and be exported. The export client can obtain the exported data from different sites. In order to ensure data consistency, different sites need to synchronize the export progress. The export progress is identified by the sequence number. Therefore, different sites or different nodes need to synchronize and update the current sequence number.

[0046] It should be noted that a node may have multiple sites. For a certain exported table, the application will be executed on multiple sites, but generally these multiple sites will be distributed on different nodes, and each site will apply to different partitions.

[0047] In practice, the export coordinator collects the export sequence number trackers from each node and uses the trackers to determine if there are gaps in the sequence numbers of the data sources of different nodes. If there are gaps in the sequence numbers of different nodes, it determines which node can fill the corresponding gap.

[0048] In practice, the sequence number contained in the data source can be obtained, and the consistency between the sequence number and the node number can be determined based on the node tracker. The node number is the sequence number recorded by the node where the data source is located when it receives the data source, corresponding to the acknowledgment sequence number currently recorded by the sequence number tracker. Specifically, the node number is the sequence number recorded after the earlier node sends data to the client.

[0049] If the serial number and the node number are inconsistent, the node number is used to update and replace the serial number to obtain the updated number.

[0050] Reference Figure 4 The client retrieves data from node 1 (or site 1). The sequence number tracked by node 1 (or site 1) is as follows: Figure 4 As shown, the client may have also retrieved data from node 2 (or site 2) at a earlier time. The sequence number tracked by node 2 (or site 2) is as follows. Figure 4As shown, the sequence numbers of the two nodes are different. Therefore, there is a gap between the sequence number of the current node and the node number. The sequence number of node 1 can be replaced with the sequence number of node 2 to obtain the updated number.

[0051] In one operating mode, if there are multiple nodes, the export coordinator will select one node as the master node according to an algorithm. The master node is the reference node, and other nodes will update their own sequence numbers based on the master node's sequence number. Since streaming data processing requires ensuring that data is processed only once, and data stream processing can be handled by multiple nodes processing the data source export, with these nodes distributively processing different streaming data blocks, gaps may exist between sequences for a given node. If a node is the master node, then the other nodes are slave nodes, and their sequence numbers will be padded according to the master node's information sequence number.

[0052] In an optional embodiment, the step of determining the sequence number of the data to be processed based on the partition ID and table name of the data to be processed, and filling the sequence number to obtain an update number, may further include the following sub-steps: S124. Eliminate the cached data of the node where the data source is located.

[0053] As explained above, when there is a discrepancy between the pending confirmation sequence number and the received confirmation sequence number of the current node, it indicates that the exporting client has already obtained relevant data from other nodes. In order to release resources, shuangzhaoDB will obtain relevant confirmation information from the corresponding node to eliminate the data cached in the current node and update its own sequence number tracker.

[0054] S13. Extract the corresponding data to be transmitted according to the update number, and transmit the data to be transmitted to the client through a single-threaded transmission method.

[0055] After determining the update number, the corresponding data to be transmitted can be retrieved from the node based on the update number, and then transmitted to the client using a single-threaded transmission method. This padding process avoids sending duplicate or identical data to the client, ensuring consistency in the data received by the client. All node state changes are executed in a single thread, avoiding thread concurrency issues. All coordination operations in the export coordinator are executed in a single thread to guarantee the correctness and consistency of data transmission. ShuangzhaoDB employs a lock-free design; without single-threaded operation, inconsistencies in state could occur.

[0056] It should be noted that after processing the sequence code tracker, the export coordinator can re-determine the leader in the distributed system. If the current node changes from a non-leader to a leader, it will notify the remaining replica nodes and send the next batch of data.

[0057] In one embodiment, the step of extracting the corresponding data to be transmitted based on the update number and transmitting the data to be transmitted to the client via a single-threaded transmission method may include the following sub-steps: S131. Determine the node corresponding to the update number as the master node, and obtain the data to be transmitted from the master node.

[0058] S132. Based on the amount of data to be transmitted, the data to be transmitted is divided into storage data and queue data, wherein the storage data is the data stored in the corresponding directory on the hard disk.

[0059] S133. The stored data and the queue data are transmitted to the client respectively through a single-threaded transmission method, and the sequence number tracked by the sequence tracker is updated according to the client's response method, wherein the response method is the response method for communicating with the client.

[0060] As analyzed above, there may be multiple nodes. We can determine the node corresponding to the update number as the master node, and refer to it. Figure 4 Node 2 can be used as the master node, and the data to be transmitted can be obtained from the master node. Since the amount of data may be large, the data to be transmitted can be divided into storage data and queue data according to the amount of data to be transmitted.

[0061] Finally, the stored data and queued data are transmitted to the client using a single-threaded transmission method, and the sequence number tracked by the sequence tracker is updated according to the client's response method, wherein the response method is the response method used in communication with the client.

[0062] Stored data refers to data that overflows from the directory and is stored in a hard disk directory. Queued data refers to streaming data from business processes.

[0063] In one embodiment, communication can be achieved between exporting clients. Upon exporting a data source, a response message can be received, distinguishing between a local response and a remote response. If it's a local response, the exported data will update the local acknowledged sequence number tracker and notify the relevant remote copy to update its acknowledged sequence number tracker. If a remote ACK response is received, the exported data source will update its local sequence number tracker. Simultaneously, the acknowledged data resources will be released.

[0064] As an example, the step of dividing the data to be transmitted into storage data and queue data based on the amount of data to be transmitted may include the following sub-steps: S1321. If the amount of data to be transmitted is greater than a preset threshold, scan the overflow directory of the data to be transmitted, extract the data corresponding to the overflow directory from the data to be transmitted to obtain overflow data, store the overflow data in the hard disk directory corresponding to the overflow directory to obtain storage data, and extract data not corresponding to the overflow directory from the data to be transmitted and push it to the streaming blocking queue to obtain queue data.

[0065] S1322. If the amount of data to be transmitted is less than or equal to a preset threshold, the data to be transmitted is pushed to a streaming blocking queue to obtain queue data.

[0066] Nodes can push exported data to a streaming blocking queue, waiting for the export data processor to retrieve the data, generate a corresponding ACK response, and transmit it to the client. While pushing data to the streaming blocking queue, the node checks the data in the queue. If the data volume exceeds a preset threshold (i.e., exceeds the set data limit), it scans the overflow directory of the data to be transmitted, saving the data in the corresponding files within the overflow directory, thus obtaining the stored data. The remaining data is pushed to the streaming blocking queue, resulting in queued data.

[0067] If the amount of data to be transmitted is less than or equal to a preset threshold, the data to be transmitted can be directly pushed to a streaming blocking queue to obtain queue data.

[0068] When data needs to be transmitted to the client, the data stored in the overflow directory should be transmitted first, followed by the data in the queue.

[0069] It should be noted that the export overflow directory will be scanned. If .pbd and .ad files exist in this directory, indicating that the previously exported data was retained on the hard drive for some reason, the export generator will create a data source based on these files. This is mainly for recovering previously interrupted data exports. If this directory exists, it means that the data export was abnormally terminated or overflowed, so ShuangzhaoDB will continue the export to achieve fault recovery. If it does not exist, the relevant data source will be created directly, that is, a corresponding data source will be created for each partition of each flow table.

[0070] In one embodiment, the client may modify the directory when transmitting data. To ensure that the data remains consistent after the directory modification, the method, as an example, further includes: S14. Respond to the directory update request, stop obtaining the current data source, determine the target data source according to the directory update request, and perform transmission update processing according to the target data source.

[0071] During the export process, if the user creates a new streaming data export, the export manager will update accordingly to accommodate the Catalog update.

[0072] Upon receiving a Catalog update message, the export processor determines whether a new export generator is needed. If so, it closes the existing export generator, thereby stopping the acquisition of data from the current data source or the current node. Simultaneously, a new export generator can be generated to match the new directory. During this process, the relevant data sources are closed synchronously, and any data that the export system has not yet processed is saved in relevant files in the overflow data directory.

[0073] It should be noted that closing the relevant data source can be done synchronously or asynchronously.

[0074] You can then start new data sources as needed based on the new directory. The export generator stores data sources corresponding to partition and table information. If the directory is updated, the tables or partitions may change, which could lead to changes in the data sources. Therefore, the export generator needs to be updated again.

[0075] In addition, the ShuangzhaoDB database also allows users to manually control the output of data streams.

[0076] In this embodiment, the present invention provides a data export method for a pure in-memory database. Its advantages are as follows: The present invention can determine the data to be processed in the pure in-memory database; determine the sequence number of the data to be processed based on the partition ID and table name of the data to be processed, and fill in the sequence number to obtain an update number; extract the corresponding data to be transmitted based on the update number, and transmit the data to be transmitted to the client through a single-threaded transmission method. By filling in the sequence number, the present invention ensures that the sequence number can be aligned with the sequence number sent at a previous time node, allowing the sequence numbers of data sent before and after to be connected, thereby ensuring that the data received by the client is consistent, thus ensuring data consistency, reducing data deviation, and improving the accuracy of data transmission.

[0077] This invention also provides a data export device for a pure in-memory database, see [link to documentation]. Figure 5 The diagram shows a structural schematic of a data export device for a pure memory database according to an embodiment of the present invention.

[0078] As an example, the data export device for the pure in-memory database may include: The determination module 201 is used to determine the data to be processed in the pure memory database, wherein the data to be processed is the streaming business data collected by the pure memory database from the business system. The filling module 202 is used to determine the sequence number of the data to be processed based on the partition ID and table name of the data to be processed, and to fill the sequence number to obtain an updated number. The filling process is a process of updating the sequence number when there is a gap between the sequence number and the sequence number of the previous time node. The transmission module 203 is used to extract the corresponding data to be transmitted according to the update number and transmit the data to be transmitted to the client through a single-threaded transmission method.

[0079] Optionally, the filling module is further configured to: Obtain the partition ID and table name from the data to be processed, and determine the data source of the streaming based on the partition ID and the table name; Obtain the sequence number contained in the data source, and determine whether the sequence number is consistent with the node number according to the node tracker, wherein the node number is the sequence number recorded after the earlier time node sends data to the client; If the serial number and the node number are inconsistent, the serial number is replaced by the node number to obtain the updated number.

[0080] Optionally, the filling module is further configured to: Eliminate the cached data on the node where the data source is located.

[0081] Optionally, the transmission module is further configured to: Determine the node corresponding to the update number as the master node, and obtain the data to be transmitted from the master node; Based on the amount of data to be transmitted, the data to be transmitted is divided into storage data and queue data. The stored data and the queue data are transmitted to the client using a single-threaded transmission method, and the sequence number tracked by the sequence tracker is updated according to the client's response method, wherein the response method is the response method used in communication with the client.

[0082] Optionally, the transmission module is further configured to: If the amount of data to be transmitted is greater than a preset threshold, scan the overflow directory of the data to be transmitted, extract the data corresponding to the overflow directory from the data to be transmitted to obtain the stored data, and extract the data not corresponding to the overflow directory from the data to be transmitted and push it to the streaming blocking queue to obtain the queue data. If the amount of data to be transmitted is less than or equal to a preset threshold, the data to be transmitted is pushed to a streaming blocking queue to obtain queue data.

[0083] Optionally, the device further includes: The update module is used to stop obtaining the current data source in response to the directory update request, determine the target data source according to the directory update request, and perform transmission update processing according to the target data source.

[0084] Those skilled in the art will understand that, for ease of description and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0085] Furthermore, this application also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data export method for a pure memory database as described in the above embodiments.

[0086] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer-executable program, the computer-executable program being used to cause a computer to execute the data export method for a pure in-memory database as described in the above embodiments.

[0087] In the description of the embodiments of the present invention, it should be noted that the terms "above," "below," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. When an element such as a layer, region, or substrate is referred to as being "above" or "on top of" another element, it may be directly on the other element, or there may be an intermediate element. Conversely, when an element is referred to as being "directly on" or "above" another element, there is no intermediate element. It should also be understood that when an element is referred to as being "below" or "under" another element, it may be directly below or under the other element, or there may be an intermediate element. Conversely, when an element is referred to as being "directly below" or "under" another element, there is no intermediate element. Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0088] Those skilled in the art will understand that embodiments of this application may also include computer program products. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), devices, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for exporting data from a pure in-memory database, characterized in that, The method includes: Identify the data to be processed in the pure in-memory database, wherein the data to be processed is streaming business data collected by the pure in-memory database from the business system; The sequence number of the data to be processed is determined based on the partition ID and table name of the data to be processed, and the sequence number is filled to obtain an updated number. The filling process is the process of updating the sequence number when there is a gap between the sequence number and the sequence number of the previous time node. Extract the corresponding data to be transmitted based on the update number, and transmit the data to the client using a single-threaded transmission method.

2. The data export method for a pure in-memory database according to claim 1, characterized in that, The step of determining the sequence number of the data to be processed based on the partition ID and table name of the data to be processed, and filling the sequence number to obtain an update number, includes: Obtain the partition ID and table name from the data to be processed, and determine the data source of the streaming based on the partition ID and the table name; Obtain the sequence number contained in the data source, and determine whether the sequence number is consistent with the node number according to the node tracker, wherein the node number is the sequence number recorded after the earlier time node sends data to the client; If the serial number and the node number are inconsistent, the serial number is replaced by the node number to obtain the updated number.

3. The data export method for a pure in-memory database according to claim 2, characterized in that, The step of determining the sequence number of the data to be processed based on the partition ID and table name of the data to be processed, and filling the sequence number to obtain an update number, includes: Eliminate the cached data on the node where the data source is located.

4. The data export method for a pure in-memory database according to claim 1, characterized in that, The step of extracting the corresponding data to be transmitted based on the update number and transmitting the data to be transmitted to the client using a single-threaded transmission method includes: Determine the node corresponding to the update number as the master node, and obtain the data to be transmitted from the master node; Based on the amount of data to be transmitted, the data to be transmitted is divided into storage data and queue data, wherein the storage data is the data stored in the corresponding directory on the hard disk. The stored data and the queue data are transmitted to the client using a single-threaded transmission method, and the sequence number tracked by the sequence tracker is updated according to the client's response method, wherein the response method is the response method used in communication with the client.

5. The data export method for a pure in-memory database according to claim 4, characterized in that, The step of dividing the data to be transmitted into storage data and queue data based on the data volume includes: If the amount of data to be transmitted is greater than a preset threshold, scan the overflow directory of the data to be transmitted, extract the data corresponding to the overflow directory from the data to be transmitted to obtain overflow data, store the overflow data in the hard disk directory corresponding to the overflow directory to obtain storage data, and extract data not corresponding to the overflow directory from the data to be transmitted and push it to the streaming blocking queue to obtain queue data. If the amount of data to be transmitted is less than or equal to a preset threshold, the data to be transmitted is pushed to a streaming blocking queue to obtain queue data.

6. The data export method for a pure in-memory database according to any one of claims 1-5, characterized in that, The method further includes: In response to a directory update request, the system stops acquiring the current data source, determines the target data source based on the directory update request, and performs transmission update processing based on the target data source.

7. A data export device for a pure in-memory database, characterized in that, The device includes: The determination module is used to determine the data to be processed in the pure memory database, wherein the data to be processed is the streaming business data collected by the pure memory database from the business system. The filling module is used to determine the sequence number of the data to be processed based on the partition ID and table name of the data to be processed, and to fill the sequence number to obtain an updated number. The filling process is the process of updating the sequence number when there is a gap between the sequence number and the sequence number of the previous time node. The transmission module is used to extract the corresponding data to be transmitted according to the update number, and transmit the data to be transmitted to the client through a single-threaded transmission method.

8. A data export system for a pure in-memory database, characterized in that, The system includes: a pure in-memory database, a business terminal, and a client terminal; The pure memory database is connected to the business terminal and the client respectively. The pure memory database has a number of data storage node units, and each node unit has a data transmission module. The data transmission module is applicable to the data export method of the pure memory database according to any one of claims 1-6.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the data export method for a pure memory-based database as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer-executable program, which is used to cause a computer to perform the data export method for a pure in-memory database as described in any one of claims 1-6.

Images