Data exporting method and device, storage medium and electronic equipment

Abstract

The invention discloses a data exporting method and device, a storage medium and electronic equipment. The method relates to the field of financial science and technology or other fields, and comprises the following steps: determining a first thread and a second thread from a plurality of threads under the condition that a data export request is received; reading a key and a data type corresponding to the key from the target database node through a first database connection instance in the first thread; obtaining a value corresponding to the key from the target database node through a second database connection instance in the first thread; and exporting the read key, the value corresponding to the key and the data type corresponding to the key as a target file through a second thread. Through the data exporting method and device, the problem that the exporting efficiency is low when the data in the database node is exported as the file in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of financial technology or other fields, and more specifically, to a data export method, apparatus, storage medium and electronic device. Background Technology

[0002] In modern information technology systems, data migration and backup are crucial for ensuring business continuity and data security. With the exponential growth of data volume and the increasing demand for real-time data processing, efficiently and accurately exporting data from database nodes to files has become a critical and complex task. Currently, data export in related technologies often relies on a single thread to perform a full scan of the database, reading key-value pairs one by one and writing the data to a file, resulting in low data export efficiency.

[0003] There is currently no effective solution to the aforementioned problems in the relevant technologies. Summary of the Invention

[0004] The main objective of this application is to provide a data export method, apparatus, storage medium, and electronic device to solve the problem of low export efficiency when exporting data from database nodes to files in related technologies.

[0005] To achieve the above objectives, according to one aspect of this application, a data export method is provided. The method includes: upon receiving a data export request, determining a first thread and a second thread from multiple threads; reading a key and its corresponding data type from a target database node through a first database connection instance in the first thread; obtaining the value corresponding to the key from the target database node through a second database connection instance in the first thread; and exporting the read key, its corresponding value, and its corresponding data type as a target file through the second thread.

[0006] Optionally, the data export method further includes: after reading N keys and their corresponding data types, constructing a message object based on the N keys and their corresponding data types, where N is a positive integer; and sending the message object to a first blocking queue, where the first blocking queue is used to provide data for the second database connection instance.

[0007] Optionally, the data export method further includes: retrieving a message object from a first blocking queue; searching for a value corresponding to a key in the message object in a target database node; constructing a first data object based on the message object and the value corresponding to the key in the message object during the process of retrieving the value corresponding to the key from the target database node through a second database connection instance in the first thread; and sending the first data object to a second blocking queue, wherein the second blocking queue is used to provide data for the second thread.

[0008] Optionally, the data export method further includes: encapsulating the M first data objects into a data set, where M is a positive integer greater than 1; and sending the data set to a second blocking queue.

[0009] Optionally, the data export method further includes: obtaining a first data object from a second blocking queue; converting the format of the first data object to obtain a second data object; and, if the transmission of the first data object stops in the second blocking queue, exporting all obtained second data objects as a target file.

[0010] Optionally, the data export method also includes: performing data verification processing on the first data object according to preset verification rules to obtain a verification result; if the verification result indicates that the verification is passed, performing format conversion on the first data object to obtain a second data object.

[0011] Optionally, the data export method further includes: before reading the key and the data type corresponding to the key from the target database node through the first database connection instance in the first thread, obtaining the data access mode of the database indicated by the data export request; if the data access mode is the first access mode, determining the hot standby node corresponding to the database as the target database node, wherein the first access mode is a mode that indicates accessing data across data shard nodes; if the data access mode is other than the first access mode, determining the first database node indicated in the data shard route corresponding to the database as the target database node, wherein the data shard route is used to record the storage relationship between the data in the database and the database nodes.

[0012] To achieve the above objectives, according to another aspect of this application, a data export apparatus is provided. The apparatus includes: a first determining module, configured to determine a first thread and a second thread from a plurality of threads upon receiving a data export request; a first acquiring module, configured to read keys and their corresponding data types from a target database node through a first database connection instance in the first thread; a second acquiring module, configured to acquire the value corresponding to the key from the target database node through a second database connection instance in the first thread; and an exporting module, configured to export the read keys, their corresponding values, and their corresponding data types as a target file through the second thread.

[0013] Optionally, the data export device further includes: a first construction module, used to construct a message object based on N keys and their corresponding data types after reading N keys and their corresponding data types, where N is a positive integer; and a first sending module, used to send the message object to a first blocking queue, wherein the first blocking queue is used to provide data for the second database connection instance.

[0014] Optionally, the first acquisition module further includes: an acquisition submodule for acquiring a message object from a first blocking queue; a search submodule for searching for a value corresponding to a key in the message object in a target database node; the data export device further includes: a second construction module for constructing a first data object based on the message object and the value corresponding to the key in the message object; and a second sending module for sending the first data object to a second blocking queue, wherein the second blocking queue is used to provide data for the second thread.

[0015] Optionally, the second sending module further includes: an encapsulation submodule for encapsulating M first data objects into a data set, where M is a positive integer greater than 1; and a sending submodule for sending the data set to a second blocking queue.

[0016] Optionally, the export module further includes: an acquisition submodule for acquiring a first data object from a second blocking queue; a processing submodule for converting the format of the first data object to obtain a second data object; and an export submodule for exporting all obtained second data objects as a target file when the transmission of the first data object stops in the second blocking queue.

[0017] Optionally, the processing submodule further includes: a processing unit, used to perform data verification processing on the first data object according to preset verification rules to obtain a verification result; and a conversion unit, used to perform format conversion on the first data object to obtain a second data object if the verification result indicates that the verification has passed.

[0018] Optionally, the data export device further includes: a second acquisition module, used to acquire the data access mode of the database indicated by the data export request; a second determination module, used to determine the hot standby node corresponding to the database as the target database node when the data access mode is a first access mode, wherein the first access mode is a mode that indicates accessing data across data shard nodes; and a third determination module, used to determine the first database node indicated in the data shard route corresponding to the database as the target database node when the data access mode is other than the first access mode, wherein the data shard route is used to record the storage relationship between data in the database and database nodes.

[0019] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the above-described data export method.

[0020] To achieve the above objectives, according to another aspect of this application, an electronic device is provided, the electronic device including a memory storing an executable program; and a processor for running the program, wherein the program executes the above-described data export method during runtime.

[0021] To achieve the above objectives, according to another aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the data export method described above.

[0022] In this embodiment, by utilizing a first thread to read keys, data types, and values ​​from the target database node, and a second thread to export the read keys, data types, and values ​​to a target file, different steps in the data export process are executed through different threads. This enables parallel execution of data reading and file export, improving data export efficiency. Furthermore, by employing different database connection instances within the first thread to read different data content, parallel data acquisition through independent connection instances is achieved, further enhancing data processing throughput and improving data export efficiency.

[0023] Therefore, the method provided in this application achieves the goal of executing different steps in the data export process through different threads, thereby improving the technical effect of data export efficiency and solving the technical problem of low export efficiency when exporting data from database nodes to files in related technologies. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 This is a hardware structure block diagram of a computer terminal provided according to an embodiment of this application;

[0026] Figure 2 This is a flowchart of a data export method provided according to an embodiment of this application;

[0027] Figure 3 This is a schematic diagram of a data export device provided according to an embodiment of this application;

[0028] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] The financial technology field or other fields disclosed herein can be used in the financial technology field, or in any field other than the financial technology field. The application fields of the financial technology field or other fields disclosed herein are not limited.

[0032] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0033] Example 1

[0034] According to an embodiment of this application, an embodiment of a data export method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a data export method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor (MCU) or a field-programmable gate array (FPGA), etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output (I / O) interface, a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0036] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0037] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data export method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned data export method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0038] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0039] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0040] Under the aforementioned operating environment, this application provides the following: Figure 2 The data export method shown. Figure 2 This is a flowchart of the data export method according to Embodiment 1 of this application.

[0041] Step S201: Upon receiving a data export request, determine the first thread and the second thread from multiple threads.

[0042] Optionally, electronic devices, application systems, servers, and other devices can be used as the execution subject of this application. In this embodiment, the target processing system is used as the execution subject to execute the above-described data export method.

[0043] The data export request described above is used to request the export of data from the database to a file. A data export request can request the export of all data from the database to a file, or it can request the export of specified data from the database to a file. The database includes multiple database nodes.

[0044] In an optional embodiment, during system initialization, a corresponding number of threads are created based on the number of threads configured by the staff. Different threads are assigned to perform specific database operations, so that one thread focuses on data reading while another thread focuses on data exporting. Upon receiving a data export request, based on the function of each thread, the thread used for data reading is designated as the first thread, and the thread used for data export is designated as the second thread.

[0045] In another alternative embodiment, upon receiving a data export request, an idle thread is determined from multiple threads. Then, one of the idle threads is designated as the first thread, and the other idle thread is designated as the second thread.

[0046] Step S202: Read the key and the corresponding data type from the target database node through the first database connection instance in the first thread.

[0047] In an optional embodiment, the database to which the data export request is directed includes multiple database nodes. The target database node refers to the database node among the multiple database nodes used to store the data to be exported. A database connection instance can also be referred to as a connection object, connection instance, database session, etc.

[0048] In an optional embodiment, in the first thread, a database connection instance named keyAndTypeNode is initialized. This instance, also known as the first database connection instance, is used to read keys and their corresponding data types from the target database node via batch queries. Optionally, the jedis.pipelined() method can be used to reduce network round trips and improve read efficiency.

[0049] Optionally, the data types mentioned above can include strings, integers, floating-point numbers, lists, sets, arrays, etc. Different data types have different storage structures.

[0050] Step S203: Obtain the value corresponding to the key from the target database node through the second database connection instance in the first thread.

[0051] Optionally, in the first thread, another database connection instance, `valueNode`, is also initialized. This instance, also known as the second database connection instance, is used to retrieve the values ​​corresponding to the keys in batches based on the keys and types obtained in step S202. Optionally, a pipeline mechanism can be used, sending multiple commands at once via the `jedis.pipelined()` method to reduce network latency and improve data reading speed. Pipeline is a batch processing mechanism for caching the database, as exemplified by the `jedis.pipelined()` method. The client can send the next command without waiting for a single command response; multiple commands are packaged into a single message and sent. All commands are executed sequentially on the server without being interrupted by other commands.

[0052] Step S204: The read keys, the corresponding values, and the corresponding data types of the keys are exported to a target file through the second thread.

[0053] In an optional embodiment, the second thread can directly write the read keys, the corresponding values, and the corresponding data types into the initial file according to a predefined format. If no more keys, values, and data types are read, the initial file is determined as the target file.

[0054] In another alternative embodiment, the second thread can directly write the read key, the value corresponding to the key, and the data type corresponding to the key into a predefined format into a buffer ByteBuffer. When the ByteBuffer reaches a certain capacity (e.g., 8K), the data in the buffer is written into a file channel. When no more keys, values, and data types are read, the data in the file channel is written into the initial file to obtain the target file.

[0055] It should be noted that the first thread can execute steps S202 and S203 in multiple batches, and the second thread can execute step S204 after the first thread completes each batch, thereby achieving asynchronous parallel operation and improving data export efficiency.

[0056] In this embodiment, by utilizing a first thread to read keys, data types, and values ​​from the target database node, and a second thread to export the read keys, data types, and values ​​to a target file, different steps in the data export process are executed through different threads. This enables parallel execution of data reading and file export, improving data export efficiency. Furthermore, by employing different database connection instances within the first thread to read different data content, parallel data acquisition through independent connection instances is achieved, further enhancing data processing throughput and improving data export efficiency.

[0057] Therefore, the method provided in this application achieves the goal of executing different steps in the data export process through different threads, thereby improving the technical effect of data export efficiency and solving the technical problem of low export efficiency when exporting data from database nodes to files in related technologies.

[0058] Optionally, in the data export method provided in this application embodiment, during the process of reading keys and corresponding data types from the target database node through the first database connection instance in the first thread, the method further includes: after reading N keys and corresponding data types, constructing a message object based on the N keys and corresponding data types, where N is a positive integer; and sending the message object to a first blocking queue, wherein the first blocking queue is used to provide data for the second database connection instance.

[0059] In an optional embodiment, a first database connection instance, keyAndTypeNode, is initialized in the first thread. This instance iteratively scans the keys in the target database node and obtains the corresponding data type for each key scanned in sequence via a Pipeline. Each piece of data in the target database node includes a key, a data type, and a value. The data type in the data containing the key can be determined as the data type corresponding to the key. After reading N keys and their corresponding data types, the first database connection instance constructs a message object List from the N keys and their corresponding data types. <nosmsg>nostMsgList. After obtaining the message object, the message object is placed into the first blocking queue KeyQueue for the consumer (i.e., the second database connection instance) to process.

[0060] As the scanning process progresses, the first database connection instance continuously sends message objects to the first blocking queue. Each message object includes N keys and their corresponding data types. The first database connection instance stops scanning when the scan times out, the maximum number of scans is reached, a stop flag is detected, or the cursor traversal is complete. After stopping the scan, the first database connection instance sends an end flag to the first blocking queue keyQueue to notify the second database connection instance.

[0061] In an optional embodiment, to reduce the consumption of system resources, N is a positive integer greater than 1.

[0062] It should be noted that by introducing message object construction and a first blocking queue, the above approach enhances the multi-threaded parallel processing capabilities and the continuity of the data export process. Specifically, setting the value of N allows for more efficient use of data in batch processing, reduces the number of data transfers between connected instances, optimizes memory usage, and thus accelerates the overall progress of data export. The existence of the first blocking queue improves the smoothness of data processing and the effective allocation of resources, avoiding resource waste caused by data waiting between two instances.

[0063] Optionally, in the data export method provided in this application embodiment, obtaining the value corresponding to the key from the target database node through the second database connection instance in the first thread includes: obtaining a message object from a first blocking queue; searching for the value corresponding to the key in the message object in the target database node; during the process of obtaining the value corresponding to the key from the target database node through the second database connection instance in the first thread, the method further includes: constructing a first data object based on the message object and the value corresponding to the key in the message object; sending the first data object to the second blocking queue, wherein the second blocking queue is used to provide data for the second thread.

[0064] Optionally, the system can initialize a second database connection instance, valueNode, and use this instance to cyclically retrieve a list of message objects from the first blocking queue, KeyQueue. <nosmsg>During message retrieval from the first blocking queue, KeyQueue, the retrieval process is interrupted if an end marker is received; if no message is received, the retrieval continues; if a message object is received, the Pipeline iterates through the target database nodes based on the type and key in the message object, thereby obtaining the values ​​corresponding to each key in the message object in batches. Each piece of data in the target database node includes a key, data type, and value, and the value in the data containing the key can be determined as the value corresponding to the key. Then, a first data object, NosMsg, is constructed based on the type, key, and corresponding value in the message object, meaning one message object corresponds to one data object. The first data object, NosMsg, is then sent to the second blocking queue, nosMsgDeque, and the counter keyValueCount is updated. This counter records the number of keys with determined data types and values. Before the first database connection instance scans the target database nodes, the system first initializes the counter keyValueCount, used to store the number of scanned data items in the current task, to 0.

[0065] It should be noted that distributing data type and value retrieval across different instances can significantly improve the efficiency of data reading and processing. By introducing a first blocking queue and a second blocking queue, asynchronous transmission of data processing is achieved, reducing the synchronization waiting time between threads and accelerating the data processing pipeline.

[0066] Optionally, in the data export method provided in this application embodiment, sending the first data object to the second blocking queue includes: encapsulating M first data objects into a data set, where M is a positive integer greater than 1; and sending the data set to the second blocking queue.

[0067] In an optional embodiment, after reading and processing each M first data objects, these M first data objects are encapsulated into a data collection NosMsgList. This data collection can be a list or a queue type to enable batch data transfer. Then, the encapsulated data collection NosMsgList is sent to a second blocking queue nosMsgDeque for use by the second thread responsible for data export. The second blocking queue acts as an intermediate container for data transfer, ensuring that data is delivered to the file export thread in an orderly manner.

[0068] It should be noted that by encapsulating M first data objects and sending them in batches to the second blocking queue, the data export method is further optimized. This mechanism not only significantly improves the efficiency of data processing and transmission, but also reduces the overhead of inter-thread communication, thereby improving the resource utilization and stability of the entire system.

[0069] Optionally, in the data export method provided in this application embodiment, the read key, the value corresponding to the key, and the data type corresponding to the key are exported to the target file through the second thread, including: obtaining a first data object from the second blocking queue; performing format conversion on the first data object to obtain a second data object; and exporting all obtained second data objects to the target file when the transmission of the first data object stops in the second blocking queue.

[0070] Optionally, a database connection instance NodePool, also known as a third database connection instance, can be initialized in the second thread. The NodePool instance loops through the second blocking queue nosMsgDeque, retrieving messages (e.g., the first data object or a data set) from the messages to obtain the first data object. Then, the first data object is format-converted to obtain the second data object, which is then stored in a ByteBuffer. Every 8KB of data in the ByteBuffer is filled, it is written to the file channel fileChannel. For example, the format conversion could be to convert the first data object to "type"... key The format is "value\n".

[0071] The database connection instance NodePool executes the aforementioned process in a loop until a stop request is received or the second blocking queue nosMsgDeque is empty, meaning the transmission of the first data object stops in the second blocking queue. Afterward, the contents of the last ByteBuffer are written to the fileChannel. Finally, the second data object from the fileChannel is written to the initial file, resulting in the target file.

[0072] In an optional embodiment, the system supports rate limiting. Before the first thread exports data, the user can select the processing speed as needed, thereby preventing excessive speed from impacting production in certain scenarios. If no selection is made, there is no limit by default; if selected, there is a minimum TPS (Transactions Per Second) limit: a minimum of 100 TPS. The database connection instance NodePool can monitor TPS and implement rate limiting accordingly, in order to obtain messages from the second blocking queue nosMsgDeque.

[0073] It should be noted that exporting the data to a target file when the termination condition is met can improve the reliability of data export.

[0074] Optionally, in the data export method provided in this application embodiment, converting the format of the first data object to obtain the second data object includes: performing data verification processing on the first data object according to preset verification rules to obtain a verification result; and if the verification result indicates that the verification is passed, converting the format of the first data object to obtain the second data object.

[0075] In an optional embodiment, before format conversion, the first data object obtained from the second blocking queue nosMsgDeque is first validated. The purpose of data validation is to check whether the data meets preset validation rules, such as those used for data integrity checks or data type correctness checks, to improve the quality of the exported data.

[0076] If data validation passes, the first data object is formatted to obtain the second data object. If data validation fails, the first data object is not exported to the target file.

[0077] It should be noted that performing data validation before exporting data can further improve the accuracy of the target file.

[0078] Optionally, in the data export method provided in this application embodiment, before reading the key and the data type corresponding to the key from the target database node through the first database connection instance in the first thread, the method further includes: obtaining the data access mode of the database indicated by the data export request; if the data access mode is a first access mode, determining the hot standby node corresponding to the database as the target database node, wherein the first access mode is a mode that indicates accessing data across data shard nodes; if the data access mode is other than the first access mode, determining the first database node indicated in the data shard route corresponding to the database as the target database node, wherein the data shard route is used to record the storage relationship between the data in the database and the database node.

[0079] Optionally, before starting data reading, the database access pattern is parsed from the data export request to determine whether it is a global tenant or another access request. The global tenant is the first access pattern. A global tenant refers to a tenant pattern that requires access across all data shard nodes.

[0080] When the data access mode is global tenant (i.e., the first access mode), it means that data needs to be read across multiple data shard nodes. In this case, the target processing system can traverse the data shard routing to find the lazy-loaded site that stores the data to be exported, and then find the hot standby node from the lazy-loaded site as the target database node. The data shard routing is used to record the storage relationship between data in the database and database nodes. The database can be divided into multiple sites (including lazy-loaded sites), and each site can be understood as a server cluster containing multiple database nodes.

[0081] When the data access mode is other than the first access mode, that is, when it is not a global tenant, it means that the data only needs to be read within a single database node. In this case, the first database node indicated in the data sharding route can be selected as the target database node.

[0082] It should be noted that the above method enables accurate determination of the target database node.

[0083] In another alternative embodiment, the target processing system may also disregard the data access mode and directly find the database node storing the data to be exported based on the data sharding route, thereby using the database node as the target database node.

[0084] Therefore, the method provided in this application achieves the goal of executing different steps in the data export process through different threads, thereby improving the technical effect of data export efficiency and solving the technical problem of low export efficiency when exporting data from database nodes to files in related technologies.

[0085] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0086] Example 2

[0087] This application also provides a data export apparatus. It should be noted that the data export apparatus of this application can be used to execute the data export method provided in this application. The data export apparatus provided in this application will be described below.

[0088] According to embodiments of this application, an apparatus for implementing the above-described data export method is also provided, such as... Figure 3 As shown, the device includes:

[0089] The first determining module 301 is used to determine the first thread and the second thread from multiple threads when a data export request is received;

[0090] The first acquisition module 302 is used to read the key and the corresponding data type from the target database node through the first database connection instance in the first thread;

[0091] The second acquisition module 303 is used to acquire the value corresponding to the key from the target database node through the second database connection instance in the first thread;

[0092] Export module 304 is used to export the read keys, the values ​​corresponding to the keys, and the data types corresponding to the keys to a target file through a second thread.

[0093] In this embodiment, by utilizing a first thread to read keys, data types, and values ​​from the target database node, and a second thread to export the read keys, data types, and values ​​to a target file, different steps in the data export process are executed through different threads. This enables parallel execution of data reading and file export, improving data export efficiency. Furthermore, by employing different database connection instances within the first thread to read different data content, parallel data acquisition through independent connection instances is achieved, further enhancing data processing throughput and improving data export efficiency.

[0094] Therefore, the method provided in this application achieves the goal of executing different steps in the data export process through different threads, thereby improving the technical effect of data export efficiency and solving the technical problem of low export efficiency when exporting data from database nodes to files in related technologies.

[0095] Optionally, in the data export device provided in the embodiments of this application, the data export device further includes: a first construction module, used to construct a message object based on N keys and the data types corresponding to the keys after reading N keys and the corresponding data types, where N is a positive integer; and a first sending module, used to send the message object to a first blocking queue, wherein the first blocking queue is used to provide data for the second database connection instance.

[0096] Optionally, in the data export device provided in this application embodiment, the first acquisition module further includes: an acquisition submodule, used to acquire a message object from a first blocking queue; a search submodule, used to search for a value corresponding to a key in the message object in a target database node; the data export device further includes: a second construction module, used to construct a first data object based on the message object and the value corresponding to the key in the message object; and a second sending module, used to send the first data object to a second blocking queue, wherein the second blocking queue is used to provide data for a second thread.

[0097] Optionally, in the data export device provided in the embodiments of this application, the second sending module further includes: an encapsulation submodule, used to encapsulate M first data objects into a data set, wherein M is a positive integer greater than 1; and a sending submodule, used to send the data set to a second blocking queue.

[0098] Optionally, in the data export apparatus provided in this application embodiment, the export module further includes: an acquisition submodule, used to acquire a first data object from a second blocking queue; a processing submodule, used to perform format conversion on the first data object to obtain a second data object; and an export submodule, used to export all obtained second data objects as a target file when the transmission of the first data object stops in the second blocking queue.

[0099] Optionally, in the data export device provided in this application embodiment, the processing submodule further includes: a processing unit, used to perform data verification processing on the first data object according to preset verification rules to obtain a verification result; and a conversion unit, used to perform format conversion on the first data object to obtain a second data object when the verification result indicates that the verification has passed.

[0100] Optionally, in the data export apparatus provided in this application embodiment, the data export apparatus further includes: a second acquisition module, used to acquire the data access mode of the database indicated by the data export request; a second determination module, used to determine the hot standby node corresponding to the database as the target database node when the data access mode is a first access mode, wherein the first access mode is a mode that indicates accessing data across data shard nodes; and a third determination module, used to determine the first database node indicated in the data shard route corresponding to the database as the target database node when the data access mode is other than the first access mode, wherein the data shard route is used to record the storage relationship between data in the database and database nodes.

[0101] It should be noted that the first determining module 301, the first acquiring module 302, the second acquiring module 303, and the exporting module 304 mentioned above correspond to steps S201 to S204 in Embodiment 1. The four modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0102] Example 3

[0103] Embodiments of this application may provide an electronic device. Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (Only one is shown) processor 1002, memory 1004, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0104] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0105] The processor can invoke information and applications stored in memory via a transmission device to perform the following steps: upon receiving a data export request, determine a first thread and a second thread from multiple threads; read the key and the corresponding data type from the target database node through a first database connection instance in the first thread; obtain the value corresponding to the key from the target database node through a second database connection instance in the first thread; and export the read key, the corresponding value, and the corresponding data type to a target file through the second thread.

[0106] The processor can also invoke information and applications stored in the memory via the transmission device to perform the following steps: after reading N keys and their corresponding data types, construct a message object based on the N keys and their corresponding data types, where N is a positive integer; send the message object to a first blocking queue, where the first blocking queue is used to provide data for the second database connection instance.

[0107] The processor can also invoke information and applications stored in memory via a transmission device to perform the following steps: retrieving a message object from a first blocking queue; searching for a value corresponding to a key in the message object in a target database node; constructing a first data object based on the message object and the value corresponding to the key in the message object during the process of retrieving the value corresponding to the key from the target database node through a second database connection instance in the first thread; and sending the first data object to a second blocking queue, wherein the second blocking queue is used to provide data for the second thread.

[0108] The processor can also invoke information and applications stored in memory via a transmission device to perform the following steps: encapsulate M first data objects into a data set, where M is a positive integer greater than 1; and send the data set to a second blocking queue.

[0109] The processor can also invoke information and applications stored in the memory via the transmission device to perform the following steps: obtain a first data object from the second blocking queue; convert the format of the first data object to obtain a second data object; and, if the transmission of the first data object stops in the second blocking queue, export all obtained second data objects as a target file.

[0110] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: perform data verification processing on the first data object according to the preset verification rules to obtain the verification result; if the verification result indicates that the verification is passed, perform format conversion on the first data object to obtain the second data object.

[0111] The processor can also invoke information and applications stored in memory via a transmission device to perform the following steps: before reading keys and their corresponding data types from the target database node through the first database connection instance in the first thread, obtain the data access mode of the database indicated by the data export request; if the data access mode is the first access mode, determine the hot standby node corresponding to the database as the target database node, wherein the first access mode is a mode that indicates accessing data across data shard nodes; if the data access mode is other than the first access mode, determine the first database node indicated in the data shard route corresponding to the database as the target database node, wherein the data shard route is used to record the storage relationship between data in the database and database nodes.

[0112] In this embodiment, by utilizing a first thread to read keys, data types, and values ​​from the target database node, and a second thread to export the read keys, data types, and values ​​to a target file, different steps in the data export process are executed through different threads. This enables parallel execution of data reading and file export, improving data export efficiency. Furthermore, by employing different database connection instances within the first thread to read different data content, parallel data acquisition through independent connection instances is achieved, further enhancing data processing throughput and improving data export efficiency.

[0113] Therefore, the method provided in this application achieves the goal of executing different steps in the data export process through different threads, thereby improving the technical effect of data export efficiency and solving the technical problem of low export efficiency when exporting data from database nodes to files in related technologies.

[0114] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0115] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0116] Example 4

[0117] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data export method provided in Embodiment 1.

[0118] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0119] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform data export method steps.

[0120] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0121] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0125] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0126] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / nosmsg> < / nosmsg>

Claims

1. A data export method, characterized in that, include: Upon receiving a data export request, determine the first thread and the second thread from among multiple threads; The first database connection instance in the first thread reads the key and the corresponding data type from the target database node; The value corresponding to the key is obtained from the target database node through the second database connection instance in the first thread; The second thread then exports the read keys, their corresponding values, and their corresponding data types to a target file.

2. The method according to claim 1, characterized in that, In the process of reading keys and their corresponding data types from the target database node through the first database connection instance in the first thread, the method further includes: After reading N keys and their corresponding data types, a message object is constructed based on the N keys and their corresponding data types, where N is a positive integer; The message object is sent to a first blocking queue, wherein the first blocking queue is used to provide data for the second database connection instance.

3. The method according to claim 2, characterized in that, Retrieving the value corresponding to the key from the target database node through the second database connection instance in the first thread includes: Retrieve the message object from the first blocking queue; Search the target database node for the value corresponding to the key in the message object; In the process of retrieving the value corresponding to the key from the target database node through the second database connection instance in the first thread, the method further includes: A first data object is constructed based on the message object and the values ​​corresponding to the keys in the message object; The first data object is sent to the second blocking queue, wherein the second blocking queue is used to provide data to the second thread.

4. The method according to claim 3, characterized in that, Sending the first data object to the second blocking queue includes: Encapsulate the M first data objects into a data set, where M is a positive integer greater than 1; The data set is sent to the second blocking queue.

5. The method according to claim 3, characterized in that, The second thread exports the read keys, their corresponding values, and their corresponding data types to a target file, including: Retrieve the first data object from the second blocking queue; The first data object is formatted to obtain the second data object; If the transmission of the first data object stops in the second blocking queue, all the second data objects obtained are exported as the target file.

6. The method according to claim 5, characterized in that, The first data object is format-converted to obtain a second data object, including: The first data object is subjected to data verification processing according to preset verification rules to obtain the verification result; If the verification result indicates that the verification is successful, the first data object is converted to a new format to obtain a second data object.

7. The method according to claim 1, characterized in that, Before reading the key and its corresponding data type from the target database node through the first database connection instance in the first thread, the method further includes: Obtain the data access mode of the database indicated by the data export request; When the data access mode is the first access mode, the hot standby node corresponding to the database is determined as the target database node, wherein the first access mode is a mode that represents accessing data across data shard nodes. When the data access mode is other than the first access mode, the first database node indicated in the data sharding route corresponding to the database is determined as the target database node, wherein the data sharding route is used to record the storage relationship between the data in the database and the database node.

8. A data export device, characterized in that, include: The first determining module is used to determine the first thread and the second thread from multiple threads when a data export request is received; The first acquisition module is used to read the key and the data type corresponding to the key from the target database node through the first database connection instance in the first thread; The second acquisition module is used to acquire the value corresponding to the key from the target database node through the second database connection instance in the first thread; The export module is used to export the read keys, the corresponding values ​​of the keys, and the data types of the keys to a target file through the second thread.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the data export method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the data export method according to any one of claims 1 to 7.

Images