A data export method, device, export node, medium and system

Abstract

Embodiments of the present application disclose a data export method and device, an export node, a medium and a system. The method comprises: obtaining a target global transaction version number when an export instruction is received; determining at least one target shard according to the export instruction; creating a corresponding query task for the target shard based on the target global transaction version number, and performing a data export operation based on the query task. The method creates a query task containing the target global transaction version number for the determined target shard, and performs a data export operation on this basis, which can export data corresponding to the target global transaction version number, solves the problem of inconsistent exported data in an online distributed database scenario, and improves the reliability of data export.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and in particular, to a data export method, device, export node, medium and system. BACKGROUND

[0002] A distributed database is composed of multiple physically independent database servers, and each database server is a database node.

[0003] In many data processing scenarios, a user often needs to export required data from a distributed database that stores a large amount of data. However, in the process of online exporting required data in a distributed database scenario, since the data in the distributed database is real-time changing, the data exported from the distributed database based on the existing export method may have the problem of inconsistency of exported data, thereby affecting the reliability of data export. SUMMARY

[0004] Embodiments of the present application provide a data export method, device, export node, medium and system to solve the problem of inconsistency of exported data in a distributed database scenario and improve the reliability of data export.

[0005] According to an aspect of embodiments of the present application, a data export method is provided, comprising:

[0006] receiving an export instruction, and obtaining a target global transaction version number when the export instruction is received;

[0007] determining at least one target shard according to the export instruction;

[0008] creating a corresponding query task for the target shard based on the target global transaction version number, and performing a data export operation based on the query task.

[0009] According to another aspect of embodiments of the present application, a data export device is provided, comprising:

[0010] an obtaining module configured to obtain a target global transaction version number when an export instruction is received;

[0011] a determining module configured to determine at least one target shard according to the export instruction;

[0012] an executing module configured to create a corresponding query task for the target shard based on the target global transaction version number, and perform a data export operation based on the query task.

[0013] According to another aspect of embodiments of the present application, an export node is provided, comprising:

[0014] at least one processor; and

[0015] a memory in communication with the at least one processor; wherein

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the data exporting method according to any one of the embodiments of the present application.

[0017] According to another aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the data exporting method according to any one of the embodiments of the present application when executed by the processor.

[0018] The technical solution of the embodiments of the present application firstly acquires a target global transaction version number when receiving an exporting instruction; then determines at least one target shard according to the exporting instruction; and finally creates a corresponding query task for the target shard based on the target global transaction version number, and performs a data exporting operation based on the query task. The present solution creates a query task containing the target global transaction version number for the determined target shard based on the acquired target global transaction version number, and performs a data exporting operation on this basis, which can export data corresponding to the target global transaction version number, solves the problem of inconsistent exported data in the online scenario of a distributed database, and improves the reliability of data exporting.

[0019] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0021] Figure 1 A flowchart of a data exporting method is provided for the first embodiment of the present application;

[0022] Figure 2 A flowchart of a data exporting method is provided for the second embodiment of the present application;

[0023] Figure 3 An implementation schematic diagram of a data exporting method is provided for the second embodiment of the present application;

[0024] Figure 4A structure schematic view of a data export device provided for the third embodiment of the present application is shown in the figure;

[0025] Figure 5 A structure schematic view of an export node provided for the fourth embodiment of the present application is shown in the figure;

[0026] Figure 6 A structure schematic view of a data export system provided for the fifth embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0027] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by the personnel in the field without creative labor should belong to the protection scope of the present application.

[0028] It should be noted that the terms "first", "second", "target" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] The export of a data table can go through multiple batch query operations, and the data at each export is real-time. After the completion of the export task, the data in the data file is not at the same time, because it is possible that the previously exported data has been modified during the export process, that is, it is not finally consistent. When performing a batch export query operation, it is possible that the data on each database shard is not globally consistent, and the data in the export file will also appear inconsistent export data.

[0030] The embodiments of the present application propose a distributed transaction strong consistency scheme based on a global transaction version number. The scheme can use a unique and increasing global transaction version number as a transaction mark, and use a two-phase commit method of a database to make the global transaction of a distributed database consistent.

[0031] Specifically, the embodiment provides a two-phase commit protocol method based on a global transaction version number, allocates a global transaction version number for a transaction when the transaction starts, enters a preparation phase (i.e., XAPREPARE) when the transaction meets a commit state, enters a commit phase (i.e., XA COMMIT) when the transaction is committed, and carries the committed global transaction version number, so as to guarantee the read-write consistency between transactions. The distributed database after using the method can also be understood as a distributed database with a global transaction version number.

[0032] On this basis, the embodiment of the application provides a data export method, which is used for exporting some databases or some data tables in an online manner in a distributed database scenario, so that the data in the exported file is globally transaction-consistent. The global consistency of the data file after import is laid a foundation, that is, the data of each shard is globally consistent at a certain moment after import.

[0033] Embodiment one

[0034] Figure 1 A flowchart of a data export method is provided for the embodiment one of the application, the embodiment can be applicable to the case of exporting data in a distributed database online scenario, the method can be executed by a data export device, the data export device can be realized in the form of hardware and / or software, and the data export device can be configured in an export node, wherein the export node includes but is not limited to a server, a computer, a notebook computer, or a tablet computer, and the like. As shown in the figure, the method includes the following steps. Figure 1

[0035] S110, when a export instruction is received, a target global transaction version number is acquired.

[0036] In the embodiment, the export node can be understood as a node used for receiving an export instruction and processing data export and the like. The export instruction can be understood as an instruction used for triggering data export. The target global transaction version number can be understood as a global transaction version number to be used corresponding to the export instruction.

[0037] When the export node receives the export instruction, it indicates that the export of corresponding data is needed at present, at this time, the target global transaction version number can be acquired, so as to be used for subsequent data export. How to acquire the target global transaction version number is not limited here, for example, the corresponding target global transaction version number can be acquired by requesting a global transaction version management node responsible for managing the global transaction version number.

[0038] ​Optionally, the target global transaction version number is acquired by: sending an acquisition request to the global transaction version management node, the acquisition request being a request for acquiring the target global transaction version number; and receiving the target global transaction version number returned by the global transaction version management node, wherein the target global transaction version number is determined by the global transaction version management node according to the current global transaction version number.

[0039] The global transaction version management node can be understood as a management node for maintaining and distributing the global transaction version number. The acquisition request can be understood as a request for acquiring the target global transaction version number. The global transaction version number (Snapshot-Transaction Sequence, STS) can refer to a unique increasing sequence issued by the global transaction version management node, used to represent a timestamp snapshot of the distributed database at a certain time. The global transaction version number can be composed of a physical time and a logical time, wherein the physical time is the millisecond difference time from January 1, 1970, 00:00:00, and the logical time is a sequence value distinguishing different transactions at the same physical time. The specific composition of the global transaction version number is not limited here.

[0040] Specifically, when the export node receives the export instruction, it sends an acquisition request for acquiring the target global transaction version number to the global transaction version management node. The global transaction version management node receives the acquisition request sent by the export node, determines the target global transaction version number according to the current global transaction version number, and returns the target global transaction version number to the export node. The export node receives the target global transaction version number returned by the global transaction version management node. The embodiment does not limit how to determine the target global transaction version number according to the current global transaction version number. For example, the current global transaction version number can be taken as the target global transaction version number when the global transaction version management node receives the acquisition request sent by the export node. The current global transaction version number can be understood as a timestamp snapshot at the current time.

[0041] S120, determining at least one target shard according to the export instruction.

[0042] In this embodiment, the distributed database can include multiple database shards. Through certain specific conditions, the data stored in the same database table can be dispersed and stored in multiple database shards, that is, each database shard can store part of the data in a database table. How the data in a database table is dispersed and stored in multiple database shards is not limited here. For example, the data in a database table can be dispersed and stored in all database shards in the distributed database, or the data in a database table can be dispersed and stored in part of the database shards in the distributed database.

[0043] The target shard can be understood as a database shard on which the data corresponding to the export instruction is distributed. How to determine at least one target shard according to the export instruction is not specifically limited here; for example, if the data corresponding to the export instruction is distributed on each database shard in the distributed database, the target shard can be all database shards in the distributed database; if the data corresponding to the export instruction is distributed on part of the database shards in the distributed database, the part of the database shards is the target shard. The data to be exported can be understood as the data to be exported corresponding to the export instruction. It can be understood that the data to be exported on each target shard can be different.

[0044] In S130, a corresponding query task is created for the target shard based on the target global transaction version number, and a data export operation is performed based on the query task.

[0045] In this embodiment, the query task can be understood as a task for querying the data to be exported distributed on the target shard. The target global transaction version number can be included in the created query task, so as to export the data corresponding to the target global transaction version number when the data export operation is subsequently performed according to the query task. The data export operation can be understood as an operation of exporting the data to be exported corresponding to the export instruction.

[0046] How the export node creates a corresponding query task for the target shard based on the target global transaction version number is not specifically limited here. For example, a corresponding query task can be created for each target shard determined based on the target global transaction version number, so as to query the data to be exported corresponding to the export instruction distributed on each target shard. On this basis, for each target shard, if the data amount of the data to be exported distributed on the target shard is large (such as exceeding a pre-set data amount threshold), the query task corresponding to the target shard can be divided into multiple batches of queries (i.e., multiple sub-query tasks), wherein the multiple sub-query tasks can be executed in parallel, and the target global transaction version number can be included in each sub-query task. Correspondingly, if the data amount of the data to be exported distributed on the target shard is not large (such as within the pre-set data amount threshold), the query task can be directly executed without batched query, and it can also be understood that the sub-query task divided is the query task itself.

[0047] After creating the corresponding query task for each target shard, if a query task is divided into multiple sub-query tasks, a corresponding query command can be created for each sub-query task; if a query task is not queried in batches, the sub-query task for the query task itself can be considered, and a corresponding query command is directly created for the query task. On this basis, the export node sends the created query command to the corresponding target shard. The query command can be understood as a query statement command containing the target global transaction version number generated based on the sub-query task, that is, each query command contains the target global transaction version number. After receiving the query command of the export node, the target shard executes the query on the data to be exported stored by itself, and returns the query result to the export node. After receiving the query result of the target shard, the export node processes the query result and exports the processed data result. Herein, how the export node processes the query result is not specifically limited, such as processing the query result according to a unified data storage format, so that the query result is converted into a unified data storage format for export.

[0048] It should be noted that in the case of multiple sub-query tasks, each sub-query task can be executed in parallel. Among them, all sub-query tasks can be executed in parallel, or part of the sub-query tasks can be executed in parallel first, and then part of the sub-query tasks in the remaining sub-query tasks are executed in parallel after the execution of this part of the sub-query tasks, and so on, until all sub-query tasks are executed.

[0049] The data export method provided by the embodiment one of the application first acquires a target global transaction version number when receiving an export instruction; then determines at least one target shard according to the export instruction; and finally creates a corresponding query task for the target shard based on the target global transaction version number, and performs a data export operation based on the query task. This method creates a query task containing the target global transaction version number for the determined target shard based on the acquired target global transaction version number, and performs a data export operation on this basis, which can export data corresponding to the target global transaction version number, solves the problem of inconsistent exported data in the online scenario of a distributed database, and improves the reliability of data export.

[0050] Embodiment two

[0051] Figure 2 The flowchart of the data export method provided by the embodiment two of the application is based on the above embodiments. In this embodiment, the process of determining at least one target shard according to the export instruction, creating a corresponding query task for the target shard based on the target global transaction version number, and performing a data export operation based on the query task is specifically described. It should be noted that the technical details not specifically described in this embodiment can be referred to the above embodiments. For example,Figure 2 The method comprises the following steps of:

[0052] S210, obtaining a target global transaction version number when the export instruction is received.

[0053] In this embodiment, when the export instruction is received, the export node can send a request for obtaining the target global transaction version number to the global transaction version management node, and receive the target global transaction version number returned by the global transaction version management node, wherein the target global transaction version number can be determined by the global transaction version management node according to the current global transaction version number.

[0054] S220, parsing the export instruction to obtain a shard where the data to be exported is located.

[0055] In this embodiment, the export node parses the obtained export instruction to obtain a shard where the data to be exported corresponding to the export instruction is located. The way of parsing the export instruction is not limited here.

[0056] S230, taking the shard where the data to be exported is located as a target shard, and the number of the target shards is at least one.

[0057] In this embodiment, the export node can take the shard where the data to be exported is located as a target shard, and the number of the target shards can be at least one.

[0058] S240, creating a query task containing the target global transaction version number for the target shard.

[0059] In this embodiment, the export node can create a corresponding query task containing the target global transaction version number for each target shard.

[0060] S250, dividing the query task into at least one sub-query task according to distribution information of the data to be exported on the target shard.

[0061] In this embodiment, the distribution information can be understood as data volume information of the data to be exported distributed on the target shard. The sub-query task can be understood as dividing the query task into multiple batch queries, and each batch query corresponds to a task. According to the distribution information of the data to be exported on the target shard, the query task can be divided into at least one sub-query task. The sub-query task can contain the target global transaction version number, and each sub-query task can be executed in parallel.

[0062] S260, creating a corresponding query command for each sub-query task in the query task, and sending the query command to the corresponding target shard.

[0063] In this embodiment, the query command can be understood as a query statement command containing the target global transaction version number generated based on the subquery task. The export node creates a corresponding query command for each subquery task in the query task and sends the query command to the corresponding target shard.

[0064] S270, receiving the query result returned by the target shard.

[0065] In this embodiment, the query result can be understood as the target to-be-exported data corresponding to the target global transaction version number determined by the target shard according to the query command. The target to-be-exported data can be understood as the to-be-exported data obtained by the target shard according to the corresponding query command. The export node receives the query result returned by each target shard.

[0066] Optionally, the query result includes the target to-be-exported data corresponding to the target global transaction version number determined by the target shard according to the query command.

[0067] Here, how to determine the target to-be-exported data corresponding to the target global transaction version number according to the query command is not limited. For example, the target to-be-exported data can be the data result corresponding to the target global transaction version number obtained by the target shard by executing the query command.

[0068] S280, processing the query result and exporting the processing result.

[0069] In this embodiment, the processing result can be understood as the data result obtained by processing the query result. The export node processes the query result, for example, the data storage format of the query result can be uniformly processed according to a set format, and the obtained result is the processing result. On this basis, after obtaining the query results of all target shards and processing them, the export node exports the processing result, for example, exports the processing result to a corresponding export file, which is not limited here.

[0070] The data export method provided in Embodiment Two of the application specifically implements the process of determining at least one target shard according to the export instruction, creating a corresponding query task for the target shard based on the target global transaction version number, and executing a data export operation based on the query task. This method creates a query task containing the target global transaction version number for the determined target shard based on the target global transaction version number, divides the query task into at least one subquery task, and creates a corresponding query command for each subquery task to execute a data export operation based on the query command, which can export data corresponding to the target global transaction version number, solves the problem of inconsistent exported data online in a distributed database scenario, and improves the reliability of data export. In addition, by processing the query result returned by the target shard, the data storage format of the processed query result can be unified.

[0071] The application is exemplarily described below.

[0072] Figure 3 An implementation schematic diagram of a data export method provided for Embodiment Two of the application is shown in FIG. 2. As shown in the figure, taking two shards as an example, the specific implementation process of the data export method is as follows: Figure 3

[0073] 1. After receiving an export instruction, the export tool sends an acquisition request to the global transaction version manager (i.e., the global transaction version management node);

[0074] 2. After receiving the acquisition request, the global transaction version manager responds to the current global transaction version number (TS1, i.e., the target global transaction version number) to the export tool;

[0075] 3. The export tool creates a query task for each shard according to the number of shards of the distributed database;

[0076] 4. The export tool divides the query tasks into multiple query batches (i.e., at least one sub-query task) according to the distribution of data in the table;

[0077] 5. The export tool creates query commands in batches and sends query requests (i.e., query commands) carrying the current global transaction version number information (i.e., / *STS TS1 * / ) to each shard (i.e., the target shard);

[0078] 6. Each shard responds to the data result set (i.e., the query result) found by the export tool;

[0079] 7. The export tool processes the data result set and formats the data and writes it into a file (the formatting function can be to export the data in a unified data storage format);

[0080] 8. Repeat the operations of steps 5, 6 and 7 until the entire data table is completely traversed and completely written into the corresponding export file.

[0081] Among them, the export tool: can be the export tool of the distributed database, which is responsible for the export function of the database and formats the data of each database shard and outputs it to the corresponding export file. The export tool can be deployed on the export node.

[0082] Global transaction version manager: can be a management node responsible for maintaining the distributed global transaction version number.

[0083] ​Sharding: can refer to database sharding, by a certain condition, the data stored in the same database table is stored on multiple database hosts (i.e. database shards) to achieve the effect of dispersing the load of a single host. A distributed database system can have multiple shards, and each shard often has a host and several standby machines, and the data stored by the host and the standby machine is consistent to achieve backup effect.

[0084] Global transaction version number (Snapshot-Transaction Sequence, STS): can refer to a unique incremental sequence issued by a global transaction version manager to represent a snapshot of a distributed database at a certain time. The global transaction version number can be composed of physical time and logical time, wherein the physical time is the millisecond difference time from January 1, 1970, 00:00:00, and the logical time is the sequence value to distinguish different transactions at the same physical time.

[0085] Batch query: can refer to splitting the query command by index splitting or paging, etc. to split batch query to the data table.

[0086] The embodiment can ensure that the data queried by each export of a single shard is at the same time by specifying the global transaction version number corresponding to the single shard at the current time during export. By specifying each shard to be the same global transaction version number during export, the data exported by multiple shards can be globally consistent at a certain time. Through the data export method provided by the embodiment, even if there is online business in the distributed database, the global consistency of the exported data can be ensured by the global transaction version number corresponding to a certain time during data export.

[0087] In an embodiment, the external interface of the distributed database stops serving. When the distributed database is in an offline scenario, i.e. there is no online business, the data during export can be ensured to be consistent when the distributed database is in a globally consistent state.

[0088] In an embodiment, the timing of the lock table operation (which can be understood as the operation of locking the exported data table) is selected. After locking the table, it is similar to the offline scenario, but only the table level is locked. In the case of a globally consistent table, the data during export is also consistent.

[0089] Embodiment three

[0090] Figure 4 A structure diagram of a data export device provided by embodiment three of the application is shown in FIG. 3. As shown in FIG. 3, the device includes: Figure 4

[0091] The acquisition module 310 is configured to acquire a target global transaction version number when receiving an export instruction.​

[0092] The determining module 320 is configured to determine at least one target shard according to the export instruction.

[0093] The executing module 330 is configured to create a corresponding query task for the target shard based on the target global transaction version number, and perform a data export operation based on the query task.

[0094] The data export device provided in the third embodiment of the present application first acquires a target global transaction version number when receiving an export instruction through the acquiring module 310, then determines at least one target shard according to an export instruction through the determining module 320, and finally creates a corresponding query task for the target shard based on the target global transaction version number through the executing module 330, and performs a data export operation based on the query task. The device creates a query task containing the target global transaction version number for the determined target shard based on the acquired target global transaction version number, and performs a data export operation on this basis, which can export data corresponding to the target global transaction version number, solves the problem of inconsistent exported data in the online scenario of a distributed database, and improves the reliability of data export.

[0095] Optionally, the acquiring module 310 comprises:

[0096] The request sending unit is configured to send an acquisition request to a global transaction version management node, the acquisition request being a request for acquiring the target global transaction version number;

[0097] The version number receiving unit is configured to receive the target global transaction version number returned by the global transaction version management node, wherein the target global transaction version number is determined by the global transaction version management node according to the current global transaction version number.

[0098] Optionally, the determining module 320 comprises:

[0099] The parsing unit is configured to parse the export instruction to obtain a shard where the data to be exported is located;

[0100] The node determining unit is configured to take the shard where the data to be exported is located as a target shard, and the number of the target shards is at least one.

[0101] Optionally, the executing module 330 comprises:

[0102] The task creating unit is configured to create a query task containing the target global transaction version number for the target shard;

[0103] The dividing unit is configured to divide the query task into at least one sub-query task according to distribution information of the data to be exported on the target shard;

[0104] The subquery task contains the target global transaction version number, and each of the subquery tasks is executed in parallel.

[0105] Optionally, the execution module 33 further comprises:

[0106] The command creation unit creates a corresponding query command for each subquery task in the query task, and sends the query command to the corresponding target shard, wherein the query command is a query statement command generated based on the subquery task and containing the target global transaction version number.

[0107] The result receiving unit receives the query result returned by the target shard.

[0108] The processing unit processes the query result and exports the processing result.

[0109] Optionally, the query result includes target to-be-exported data corresponding to the target global transaction version number determined by the target shard according to the query command.

[0110] The data export apparatus provided by the embodiments of the present application can execute the data export method provided by any of the embodiments of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0111] Embodiment four

[0112] Figure 5 A structure diagram of an export node provided by Embodiment Four of the present application. The export node 10 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The export node can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (such as headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0113] As Figure 5As shown, the export node 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the export node 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0114] Various components in the export node 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, speakers, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the export node 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0115] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the data export method.

[0116] In some embodiments, the data export method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the export node 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the data export method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the data export method by any other appropriate means, such as by means of firmware.

[0117] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0118] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program

[0119] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] To provide user interaction, the systems and techniques described herein can be implemented on an export node having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the export node. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including sound input, voice input, or haptic input).

[0121] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0122] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0123] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0124] Example 5

[0125] Figure 6 This is a schematic diagram of the structure of a data export system provided in Embodiment 5 of the present invention. Figure 6As shown, the data export system comprises the export node 410 provided by the embodiment of the present application and at least one target shard 420.

[0126] The export node 410 is connected with each target shard 420.

[0127] On the basis of the above-mentioned embodiment, the data export system further comprises:

[0128] The global transaction version management node is configured to receive an acquisition request sent by the export node 410, determine a target global transaction version number according to a current global transaction version number, and return the target global transaction version number to the export node 410, wherein the acquisition request is a request for acquiring the target global transaction version number.

[0129] The global transaction version management node is connected with the export node 410.

[0130] The global transaction version management node can be connected with each target shard 420 through a computing node in a distributed database, for example, the global transaction version management node is connected with at least one computing node in a distributed database, and the computing node is connected with each target shard 420, so as to realize indirect connection between the global transaction version management node and each target shard 420.

[0131] The data export system provided by the fifth embodiment can be used to execute the data export method provided by any of the above-mentioned embodiments, and has corresponding functions and beneficial effects.

[0132] The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement and improvement within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data export method, characterized in that, Applied to exported nodes, the method includes: Upon receiving an export command, obtain the target global transaction version number; At least one target shard is determined according to the export instruction; the target shard is the database shard where the data to be exported corresponding to the export instruction is distributed; A corresponding query task is created for the target shard based on the target global transaction version number, and a data export operation is performed based on the query task; The step of creating a corresponding query task for the target shard based on the target global transaction version number includes: Create a query task for the target shard that contains the target global transaction version number; Based on the distribution information of the data to be exported on the target shard, the query task is divided into at least one sub-query task; the sub-query task is to divide the query task into multiple batch queries, with each batch query corresponding to a task. The subquery task includes the target global transaction version number, and each subquery task is executed in parallel. The data export operation based on the query task includes: Create a corresponding query command for each sub-query task in the query task, and send the query command to the corresponding target shard. The query command is a query statement command that contains the target global transaction version number generated based on the sub-query task. Receive the query result returned by the target shard; the query result is the target data to be exported corresponding to the target global transaction version number determined by the target shard according to the query command; Process the query results and export the processing results; The step of obtaining the target global transaction version number includes: Send a request to the global transaction version management node, wherein the request is a request to obtain the target global transaction version number; The system receives the target global transaction version number returned by the global transaction version management node, wherein the target global transaction version number is determined by the global transaction version management node based on the current global transaction version number.

2. The method according to claim 1, characterized in that, The step of determining at least one target fragment according to the export instruction includes: Parse the export command to obtain the fragment containing the data to be exported; The shard containing the data to be exported is designated as the target shard, and the number of target shards is at least one.

3. The method according to claim 1, characterized in that, The query results include the target data to be exported corresponding to the target global transaction version number determined by the target shard according to the query command.

4. A data export device, characterized in that, include: The acquisition module is used to obtain the target global transaction version number when an export instruction is received; The determination module is used to determine at least one target fragment based on the export instructions; The target shard is the database shard where the data to be exported corresponds to the export command. The execution module is used to create a corresponding query task for the target shard based on the target global transaction version number, and to perform a data export operation based on the query task; The execution module includes: The task creation unit is used to create a query task containing the target global transaction version number for the target shard; A partitioning unit is used to divide the query task into at least one sub-query task based on the distribution information of the data to be exported on the target partition; the sub-query task is to divide the query task into multiple batch queries, with each batch query corresponding to a task. The subquery task includes the target global transaction version number, and each subquery task is executed in parallel. The execution module further includes: The command creation unit is used to create a corresponding query command for each sub-query task in the query task, and send the query command to the corresponding target shard. The query command is a query statement command containing the target global transaction version number generated based on the sub-query task. The result receiving unit is used to receive the query result returned by the target shard; the query result is the target data to be exported corresponding to the target global transaction version number determined by the target shard according to the query command. A processing unit is used to process the query results and export the processing results; The acquisition module includes: A request sending unit is used to send an acquisition request to the global transaction version management node, wherein the acquisition request is a request to acquire the target global transaction version number; The version number receiving unit is used to receive the target global transaction version number returned by the global transaction version management node, wherein the target global transaction version number is determined by the global transaction version management node based on the current global transaction version number.

5. An exported node, characterized in that, The exported nodes include: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data export method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the data export method of any one of claims 1-3.

7. A data export system, characterized in that, The system includes: the export node as described in claim 5, and at least one target shard.

8. The system according to claim 7, characterized in that, Also includes: A global transaction version management node is used to receive the retrieval request sent by the export node, determine the target global transaction version number based on the current global transaction version number, and return the target global transaction version number to the export node, wherein the retrieval request is a request to obtain the target global transaction version number.

Images