Data batch import method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data batch import method and device, electronic equipment and storage medium, and relates to the technical field of computers. In response to a data batch import request for a target upstream system, each piece of current business data in the target upstream system, the size of a data file in a database associated with the target upstream system, and the data volume of each storage partition of the database are obtained. Based on the data volume of each storage partition and the data size of each piece of business data, multiple pieces of business data are segmented to obtain business data corresponding to each storage partition. Based on the size of the data file, the business data corresponding to each storage partition is divided to obtain each data file to which the business data of the storage partition belongs. Each data file of each storage partition is used to replace the current data file of the database. Thus, the impact of data batch import on the service performance of the database is effectively avoided, and the data batch import efficiency is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for batch data import. Background Technology

[0002] Currently, for key-value databases, batch data import can typically be performed using shell / Python / Java scripts. However, this method can consume significant database resources for extended periods, increasing pressure on the database cluster service, impacting database performance, and resulting in low batch import efficiency. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for batch data import, which aims to solve the problem of improving the efficiency of batch data import while reducing the impact on database service performance.

[0004] According to one aspect of this disclosure, a method for batch data import is provided, comprising:

[0005] In response to a batch data import request for a target upstream system, obtain each piece of current business data in the target upstream system, the size of the data file in the database associated with the target upstream system, and the amount of data in each storage partition of the database;

[0006] Based on the data volume of each storage partition and the data size of each piece of business data, multiple pieces of business data are segmented to obtain the business data corresponding to each storage partition;

[0007] Based on the data file size, the business data corresponding to each storage partition is divided to obtain the data files to which the business data of the storage partition belongs;

[0008] The current data files of the database are replaced using the data files of each of the storage partitions.

[0009] According to another aspect of this disclosure, a data batch import apparatus is provided, comprising:

[0010] The acquisition module is used to respond to a batch data import request for a target upstream system by acquiring each piece of current business data in the target upstream system, the size of the data file in the database associated with the target upstream system, and the amount of data in each storage partition of the database.

[0011] The segmentation module is used to segment multiple pieces of business data based on the data volume of each storage partition and the data size of each piece of business data, so as to obtain the business data corresponding to each storage partition.

[0012] The partitioning module is used to partition the business data corresponding to each storage partition based on the size of the data file, and obtain the data files to which the business data of the storage partition belongs;

[0013] The replacement module is used to replace the current data files of the database using the data files of each of the storage partitions.

[0014] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0015] At least one processor;

[0016] And, a memory that is communicatively connected to at least one processor;

[0017] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the methods of the above embodiments.

[0018] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the method according to the above embodiments.

[0019] This disclosure provides a method, apparatus, electronic device, and storage medium for batch data import. First, in response to a batch data import request for a target upstream system, it obtains each current business data item in the target upstream system, the size of the data files in the database associated with the target upstream system, and the data volume of each storage partition in the database. Then, based on the data volume of each storage partition and the size of each business data item, it segments the multiple business data items to obtain the business data corresponding to each storage partition. Next, based on the data file size, it divides the business data corresponding to each storage partition to obtain the data files belonging to the business data of that storage partition. Finally, it uses the data files of each storage partition to replace the current data file in the database. Thus, after obtaining the full amount of business data from the upstream system, the business data is distributed to each storage partition based on the data volume of each storage partition in the database. The business data in each storage partition is divided based on the data file size in the database to obtain the data files belonging to the business data of each storage partition. The data files of each storage partition are then used to replace the current data file in the database. This effectively avoids the impact on database service performance during batch data import, improving the efficiency of batch data import. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0021] Figure 1 A flowchart illustrating a method for batch data import provided in this embodiment of the disclosure;

[0022] Figure 2 A flowchart illustrating a method for batch data import provided in this embodiment of the disclosure;

[0023] Figure 3 A flowchart illustrating a method for batch data import provided in this embodiment of the disclosure;

[0024] Figure 4 A flowchart illustrating a method for batch data import provided in this embodiment of the disclosure;

[0025] Figure 5 A flowchart illustrating a method for batch data import provided in this embodiment of the disclosure;

[0026] Figure 6 This is a schematic diagram of the structure of a data batch import device provided in an embodiment of the present disclosure;

[0027] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0028] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0030] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0031] It should be noted that the acquisition, transmission, storage, use, and processing of data in this disclosed technical solution all comply with the relevant provisions of national laws and regulations.

[0032] The data batch import method of this disclosure will now be described in detail with reference to the accompanying drawings.

[0033] Figure 1 This is a flowchart illustrating a method for batch data import provided in an embodiment of this disclosure.

[0034] like Figure 1 As shown, the method includes:

[0035] Step 101: In response to the batch data import request for the target upstream system, obtain the current business data of each piece of data in the target upstream system, the size of the data file in the database associated with the target upstream system, and the amount of data in each storage partition of the database.

[0036] It should be noted that the target upstream system, also known as the "target business system," can be of any type, and the business data it generates can be stored in its associated database. For example, the target upstream system can be a distributed business system (such as HDFS), and this disclosure does not limit it in this regard.

[0037] HDFS is short for Hadoop Distributed File System, and Hadoop is a distributed system infrastructure.

[0038] It should be noted that the data batch import request for the target upstream system can be a data batch import request triggered when certain conditions are met (such as reaching the trigger period, etc.), or it can be a data batch import request triggered according to actual needs. This disclosure does not limit it in this way.

[0039] The business data in the target upstream system is a tag field, that is, a key-value pair field, which is not limited in this disclosure.

[0040] The database associated with the upstream system of the target is a distributed key-value database, which is not limited in this disclosure.

[0041] The size of the data file can be preset. For example, the size of the data file in the database can be 20 megabytes (MB) or 50 MB, etc., and this disclosure does not limit it.

[0042] It should be noted that the database storage partitions and the amount of data in each storage partition can be determined based on the amount of business data in the target upstream system and the database's storage and computing resources. The amount of data in each storage partition may be different or the same, and this disclosure does not limit this.

[0043] In this disclosure, in response to a batch data import request for a target upstream system, the current business data in the target upstream system, the size of the data file in the database associated with the target upstream system, and the amount of data in each storage partition of the database can be obtained first, so as to provide a data foundation for batch importing the business data in the target upstream system into the database using the batch data import method provided in this disclosure.

[0044] Step 102: Based on the data volume of each storage partition and the data size of each business data item, the multiple business data items are divided to obtain the business data corresponding to each storage partition.

[0045] In this disclosure, after obtaining the current business data of each item in the target upstream system and the data volume of each storage partition of the database, in order to improve the performance of database services (such as the efficiency of data query service performance, etc.), the business data of the target upstream system can be distributed and stored in each storage partition. At this time, the total data volume of the business data of the target upstream system can be determined first based on the data size of each item of business data. Then, based on the data volume of each storage partition and the total data volume of the business data of the target upstream system, multiple items of business data are divided to obtain the business data corresponding to each storage partition.

[0046] It should be noted that since the amount of data in the value field of each business data item may be different, and the data size of each business data item may be different, this disclosure does not impose any restrictions on this.

[0047] It should be noted that, provided that the storage resources of the database meet the requirements, the value field of the business data can also be serialized. In this case, when determining the data size of each piece of business data, the serialization overhead of that business data is also included. This disclosure does not limit this.

[0048] It should be noted that when splitting multiple business data based on the data volume of the storage partition and the size of each business data item, any preset splitting method can be used. For example, the business data can be split based on the hash value of the key field of the business data (such as the key field of key-value pairs), or it can be split based on the range of the identifier of the key field of the business data (such as the key identifier of key-value pairs), etc. This disclosure does not limit this.

[0049] Step 103: Based on the data file size, divide the business data corresponding to each storage partition to obtain the data files to which the business data of the storage partition belongs.

[0050] In this disclosure, after obtaining the business data corresponding to each storage partition and the size of the data file in the database, the business data corresponding to each storage partition can be divided based on the size of the data file. Then, each part of the divided business data is stored in a data file to obtain the data files to which the business data of each storage partition belongs. This allows for the offline generation of the underlying data files of the database associated with the target upstream system, providing conditions for improving the efficiency of batch data import.

[0051] It should be noted that the number of data files to which the business data belongs in each storage partition may be different or the same, and the amount of business data stored in each data file may be similar. This disclosure does not limit this.

[0052] Optionally, when dividing the business data of each storage partition based on the data file size to obtain the data files to which the business data of the storage partition belongs, the number of data files required for the business data of each storage partition can be determined first based on the data file size and the total data size of the business data of each storage partition, and the corresponding number of data files can be generated. Then, the business data of each storage partition can be divided based on the data file size, and then each part of the business data after division can be stored into data files to obtain the data files to which the business data of each storage partition belongs. This disclosure does not limit this.

[0053] Optionally, when dividing the business data of each storage partition based on the data file size, the number of data files required for the business data of the target upstream system can be determined first based on the data file size and the total data size of the business data of the target upstream system, and the corresponding number of data files can be generated. Then, based on the data file size, each part of the business data after division is stored in a data file. After that, based on the business data corresponding to each storage partition, the data files to which the business data of each storage partition belongs are determined. This disclosure does not limit this.

[0054] Step 104: Replace the current data file in the database using the data files in each storage partition.

[0055] In this disclosure, after obtaining the data files to which the business data of each storage partition belongs, the data files of each storage partition can be used to replace the current data files of the database associated with the target upstream system, thereby improving the efficiency of batch data import while realizing batch data import into the database.

[0056] In this embodiment, in response to a batch data import request for the target upstream system, the system first obtains each current business data item in the target upstream system, the size of the data file in the database associated with the target upstream system, and the data volume of each storage partition in the database. Then, based on the data volume of each storage partition and the data size of each business data item, the system segments the multiple business data items to obtain the business data corresponding to each storage partition. Next, based on the data file size, the system divides the business data corresponding to each storage partition to obtain the data files belonging to the business data of that storage partition. Finally, the system uses the data files of each storage partition to replace the current data file in the database. Thus, after obtaining the full amount of business data from the upstream system, the system distributes the business data to each storage partition based on the data volume of each storage partition in the database, and divides the business data of each storage partition based on the data file size in the database to obtain the data files belonging to the business data of each storage partition. The system then uses the data files of each storage partition to replace the current data file in the database. This effectively avoids the impact on the service performance of the database during batch data import, thereby improving the efficiency of batch data import.

[0057] Figure 2 This is a flowchart illustrating a method for batch data import provided in an embodiment of this disclosure.

[0058] like Figure 2 As shown, the method includes:

[0059] Step 201: In response to the batch data import request for the target upstream system, obtain the current business data of each piece of data in the target upstream system, the size of the data file in the database associated with the target upstream system, and the amount of data in each storage partition of the database.

[0060] The specific implementation of step 201 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0061] Step 202: Obtain the first field of each piece of business data.

[0062] The first field is the key field of the business data, that is, the key field of the business data.

[0063] In this disclosure, after obtaining each piece of business data in the target upstream system, before importing the business data into the database in batches, in order to improve the efficiency of batch data import, the business data in the target upstream system can be globally sorted. At this time, the first field of each piece of business data can be obtained first.

[0064] Step 203: Determine the first value corresponding to the first field of each piece of business data.

[0065] The first value can be a data identifier for the business data, and it can be an identifier of any form. For example, the first value corresponding to the first field of the business data can be the hash value of the first field, etc., and this disclosure does not limit it in this way.

[0066] It should be noted that the method for determining the first value corresponding to the first field of each piece of business data can be any pre-set method. For example, the hash value of the first field of the business data can be calculated using a pre-set hash function (such as MD5 or SHA-1) and determined as the first value of the business data. Alternatively, the hash value of the first field can be converted or encoded in a pre-set manner to obtain the first value, etc. This disclosure does not limit this method.

[0067] MD5 is short for Message-Digest Algorithm 5 (MD5).

[0068] SHA-1 is short for Secure Hash Algorithm 1 (SHA-1).

[0069] In this disclosure, after obtaining the first field of each piece of business data, in order to perform global sorting of the business data, the first value corresponding to the first field of each piece of business data can be determined first.

[0070] Step 204: Sort the multiple business data based on the size of the first value corresponding to each business data.

[0071] In this disclosure, after determining the first value corresponding to the first field of each piece of business data, multiple pieces of business data can be sorted based on the size of the first value corresponding to each piece of business data. For example, the business data can be sorted from largest to smallest based on the first value of each piece of business data, thereby achieving global ordering of business data and providing conditions for improving the efficiency of batch data import. This disclosure does not limit this.

[0072] Step 205: Based on the data volume of each storage partition and the data size of each business data item, the sorted business data items are divided to obtain the business data corresponding to each storage partition.

[0073] It should be noted that when dividing multiple sorted business data based on the data volume of each storage partition and the data size of each business data, the multiple sorted business data can be divided sequentially starting from the first business data, based on the data volume of each storage partition and the data size of each business data, so that the business data corresponding to each storage partition is ordered. This disclosure does not limit this.

[0074] Step 206: Based on the data file size, divide the business data corresponding to each storage partition to obtain the data files to which the business data of the storage partition belongs.

[0075] Step 207: Replace the current data file in the database using the data files in each storage partition.

[0076] The specific implementation of steps 206 to 207 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0077] In this embodiment of the disclosure, firstly, in response to a batch data import request for the target upstream system, the system obtains each current business data item in the target upstream system, the size of the data file in the database associated with the target upstream system, and the data volume of each storage partition in the database. Then, it obtains the first field of each business data item and determines the first value corresponding to the first field of each business data item. Based on the size of the first value corresponding to each business data item, the system sorts the multiple business data items. Then, based on the data volume of each storage partition and the data size of each business data item, the system divides the sorted multiple business data items to obtain the business data corresponding to each storage partition. Finally, based on the data file size, the system divides the business data corresponding to each storage partition to obtain the data files to which the business data of that storage partition belongs, and uses the data files of each storage partition to replace the current data file in the database. Therefore, after obtaining the business data from the upstream system, the first value corresponding to the first field of each piece of business data is determined, and the business data is sorted based on the size of the first value. Based on the data volume of each storage partition of the database and the size of the data file, the sorted business data is divided and partitioned to obtain the data files to which the business data of each storage partition belongs. The current data files of the database are then replaced using these data files, thereby reducing the amount of resources occupied during the batch data import process and improving the efficiency of batch data import.

[0078] Figure 3 This is a flowchart illustrating a method for batch data import provided in an embodiment of this disclosure.

[0079] like Figure 3 As shown, the method includes:

[0080] Step 301: In response to the batch data import request for the target upstream system, obtain the current business data of each piece of data in the target upstream system, the size of the data file in the database associated with the target upstream system, and the amount of data in each storage partition of the database.

[0081] Step 302: Obtain the first field of each piece of business data.

[0082] Step 303: Determine the first value corresponding to the first field of each piece of business data.

[0083] Step 304: Sort the multiple business data based on the size of the first value corresponding to each business data.

[0084] Step 305: Based on the data volume of each storage partition and the data size of each business data item, the sorted multiple business data items are divided to obtain the business data corresponding to each storage partition.

[0085] Step 306: Based on the data file size, divide the business data corresponding to each storage partition to obtain the data files to which the business data of the storage partition belongs.

[0086] The specific implementation of steps 301 to 306 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0087] Step 307: Determine the storage location information of each data file in the underlying storage space of the database for each storage partition.

[0088] It should be noted that the underlying storage space of a database can be the database's disk storage space.

[0089] In this disclosure, after obtaining the data files to which the business data of each storage partition belongs, when replacing the current data file of the database using the data files of each storage partition, in order to ensure that the database can provide accurate and reliable services normally after the data files are replaced, it is also necessary to replace the index information in the database. At this time, the storage location information of each data file of each storage partition in the underlying storage space of the database can be determined first.

[0090] Step 308: Obtain the range of the first values ​​of the business data stored in each data file of each storage partition, and the partition identifier of each storage partition.

[0091] The partition identifier can be an identifier used to represent a partition, and it can be any pre-set identifier. For example, the partition identifier can be the name of the storage partition, or it can be a partition sequence number identifier determined according to the sorting order of the business data corresponding to each storage partition, etc. This disclosure does not limit it in this way.

[0092] In this disclosure, after determining the storage location information of each data file of each storage partition in the underlying storage space of the database, the range of the first value of the first business data stored in each data file can be determined based on the first value of the first business data stored in each data file of each storage partition and the first value of the last business data stored, and the partition identifier of each storage partition can be obtained, thereby providing a data basis for determining the index information of each storage partition.

[0093] Step 309: Based on the storage location information of each data file in each storage partition, the range of the first value of the business data stored in each data file, and the partition identifier of each storage partition, determine the metadata index information corresponding to each storage partition.

[0094] The metadata index information can be the index information of business data in each storage partition. It may include partition identifier, data file identifier, data range of business data stored in the data file, etc. This disclosure does not limit it.

[0095] The data file identifier can be an identifier used to represent a data file, and it can be any pre-set identifier. For example, the data file identifier can be the name of the data file, or it can be determined by sorting the data files according to the first value of the business data stored in the data files and determining the corresponding sequence number of the data file, which is then used as the data file identifier. This disclosure does not limit this.

[0096] It should be noted that when the data structure in the underlying storage space of the database is a hierarchical data structure (such as LSM numbers, etc.), the metadata index information may also include the level number where the data file is located, etc. This disclosure does not limit this.

[0097] LSM is short for Log-Structured-Merge-Tree.

[0098] In this disclosure, after determining the storage location information of each data file of each storage partition in the underlying storage space of the database, the metadata index information corresponding to each storage partition can be determined based on the storage location information of each data file of each storage partition, the range of the first value of the business data stored in each data file, and the partition identifier of each storage partition.

[0099] Step 310: Replace the current metadata index information and current data files of the database based on the metadata index information corresponding to each storage partition and each data file.

[0100] In this disclosure, after determining the metadata index information corresponding to each storage partition, the current metadata index information and current data files of the database can be replaced based on the metadata index information corresponding to each storage partition and each data file. This ensures that the database can provide accurate and reliable services while realizing the batch import of business data from the target upstream system into the database.

[0101] In this embodiment of the disclosure, firstly, in response to a batch data import request for the target upstream system, the system obtains each current business data item in the target upstream system, the size of the data file in the database associated with the target upstream system, and the data volume of each storage partition in the database. It then obtains the first field of each business data item, determines the first value corresponding to the first field of each business data item, sorts the multiple business data items based on the size of the first value corresponding to each business data item, and then segments the sorted multiple business data items based on the data volume of each storage partition and the data size of each business data item to obtain the business data corresponding to each storage partition. Finally, based on the data file size, the system divides the business data corresponding to each storage partition into... The process involves partitioning the database to obtain the data files containing the business data for each partition. Then, the storage location of each data file within the database's underlying storage space is determined. The range of the first values ​​of the business data stored in each data file of each partition, as well as the partition identifier of each partition, are obtained. Based on the storage location information, the range of the first values ​​of the business data stored in each data file, and the partition identifier of each partition, the metadata index information corresponding to each storage partition is determined. Finally, based on the metadata index information of each storage partition and the data files, the current metadata index information and the current data files of the database are replaced. Thus, after obtaining the data files containing the business data for each storage partition, determining the storage location information of each data file in the database, and using the storage location information, the range of business data in each data file, and the partition identifier of each storage partition, the metadata index information of each storage partition is determined. Using the metadata index information of each storage partition and the data files, the current index information and the current data files of the database are replaced, thereby improving the efficiency of batch data import while ensuring the database provides reliable and accurate services.

[0102] Figure 4 This is a flowchart illustrating a method for batch data import provided in an embodiment of this disclosure.

[0103] like Figure 4 As shown, the method includes:

[0104] Step 401: In response to a batch data import request for the target upstream system, obtain each current business data in the target upstream system, the size of the data file in the database associated with the target upstream system, and the amount of data in each storage partition of the database.

[0105] Step 402: Obtain the first field of each business data item.

[0106] Step 403: Determine the first value corresponding to the first field of each piece of business data.

[0107] Step 404: Sort the multiple business data based on the size of the first value corresponding to each business data.

[0108] Step 405: Based on the data volume of each storage partition and the data size of each business data item, the sorted multiple business data items are divided to obtain the business data corresponding to each storage partition.

[0109] Step 406: Based on the data file size, divide the business data corresponding to each storage partition to obtain the data files to which the business data of the storage partition belongs.

[0110] The specific implementation of steps 401 to 406 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0111] Step 407: Sort each data file under multiple storage partitions based on the business data stored in each data file of each storage partition and the order of the sorted business data.

[0112] In this disclosure, after obtaining the data files to which the business data of the storage partition belongs, in order to ensure that the data files of each storage partition are stored in the underlying storage space of the database in an orderly manner, each data file under multiple storage partitions can be sorted based on the business data stored in each data file of each storage partition and the order of multiple sorted business data, so that the data files required by the business data of the target upstream system are globally ordered, thereby improving the efficiency of batch data import.

[0113] Step 408: Based on the order of each data file, determine the number of layers in the underlying storage space of the database where each data file is stored and its position in that layer.

[0114] The position of the data file in the layer can be which number the data file is in that layer, and this disclosure does not limit this.

[0115] In this disclosure, when the data structure of the underlying storage space of the database is a hierarchical structure, in order to determine the storage location information of each data file, after sorting the data files to which the business data belongs, the number of layers in the underlying storage space of the database and the position in that layer can be determined based on the order of each data file.

[0116] Optionally, when determining the level and location of each data file in the underlying storage space of the database based on the order of each data file, it can be based on the order of each data file and the storage level of the i-th level in the underlying storage space of the database. i-1 A number of data files are defined, and the number of layers in which each data file is stored and its position in that layer are determined sequentially, where i is a consecutive positive integer starting from 1, and this disclosure does not limit this.

[0117] Step 409: Determine the storage location information of each data file in the underlying storage space of the database by storing the number of layers where each data file is stored and its position in that layer.

[0118] In this disclosure, after determining the number of layers in the underlying storage space of the database where each data file is stored and its position in that layer, the number of layers in which each data file is stored and its position in that layer can be used to determine the storage location information of each data file, thereby improving the efficiency of batch data import.

[0119] Step 410: Obtain the range of the first values ​​of the business data stored in each data file of each storage partition, and the partition identifier of each storage partition.

[0120] Step 411: Based on the storage location information of each data file in each storage partition, the range of the first value of the business data stored in each data file, and the partition identifier of each storage partition, determine the metadata index information corresponding to each storage partition.

[0121] Step 412: Replace the current metadata index information and current data files of the database based on the metadata index information corresponding to each storage partition and each data file.

[0122] The specific implementation of steps 410 to 412 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0123] In this embodiment of the disclosure, firstly, in response to a batch data import request for a target upstream system, the system obtains each current business data item in the target upstream system, the size of the data file in the database associated with the target upstream system, and the data volume of each storage partition in the database. It then obtains the first field of each business data item, determines the first value corresponding to the first field, and sorts the multiple business data items based on the size of the first value. Next, based on the data volume of each storage partition and the size of each business data item, the sorted multiple business data items are segmented to obtain the business data corresponding to each storage partition. Then, based on the data file size, the business data corresponding to each storage partition is divided to obtain the data files to which the business data of that storage partition belongs. Finally, based on the business data stored in each data file of each storage partition and the sorted multiple business data items... The process involves sorting each data file across multiple storage partitions according to their order. Based on this order, the layer number and location of each data file within the database's underlying storage space are determined. This layer number and location define the storage location information for each data file within the database's underlying storage space. Next, the range of the first values ​​of the business data stored in each data file of each storage partition, along with the partition identifier of each storage partition, are obtained. Based on the storage location information of each data file in each storage partition, the range of the first values ​​of the business data stored in each data file, and the partition identifier of each storage partition, the metadata index information corresponding to each storage partition is determined. Finally, based on the metadata index information corresponding to each storage partition and the data files, the current metadata index information and the current data files of the database are replaced. Therefore, after obtaining the data files to which the business data of each storage partition of the database belongs, the data files to which the business data belongs are globally sorted based on the order of the sorted business data. Based on the order of each data file, the layer number and position of each data file in the underlying storage space of the database are determined. The layer number and position in the layer are determined as the underlying storage location information of each data file. Together with the business data range of each data file and the partition identifier of the storage partition to which it belongs, the index information of each storage partition is determined. Using the index information and the data file, the current index information and the current data file of the database are replaced. Thus, while ensuring the order of the business data imported into the database, the efficiency of batch data import is improved.

[0124] Figure 5 This is a flowchart illustrating a method for batch data import provided in an embodiment of this disclosure.

[0125] like Figure 5 As shown, the method includes:

[0126] Step 501: In response to the batch data import request for the target upstream system, obtain the current business data of each piece of data in the target upstream system, the size of the data file in the database associated with the target upstream system, and the amount of data in each storage partition of the database.

[0127] Step 502: Obtain the first field of each piece of business data.

[0128] Step 503: Determine the first value corresponding to the first field of each piece of business data.

[0129] Step 504: Sort the multiple business data based on the size of the first value corresponding to each business data.

[0130] Step 505: Based on the data volume of each storage partition and the data size of each business data item, the sorted multiple business data items are divided to obtain the business data corresponding to each storage partition.

[0131] Step 506: Based on the data file size, divide the business data corresponding to each storage partition to obtain the data files to which the business data of the storage partition belongs.

[0132] Step 507: Determine the storage location information of each data file in the underlying storage space of the database for each storage partition.

[0133] Step 508: Obtain the range of the first values ​​of the business data stored in each data file of each storage partition, and the partition identifier of each storage partition.

[0134] Step 509: Based on the storage location information of each data file in each storage partition, the range of the first value of the business data stored in each data file, and the partition identifier of each storage partition, determine the metadata index information corresponding to each storage partition.

[0135] The specific implementation of steps 501 to 509 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0136] Step 510: Based on the business data corresponding to each storage partition and the order of the sorted business data, merge the metadata index information of each storage partition in sequence to obtain metadata index files corresponding to multiple storage partitions.

[0137] In this disclosure, after determining the metadata index information corresponding to each storage partition, the merging order of the metadata index information of each storage partition can be determined based on the business data corresponding to each storage partition and the order among the sorted business data. Then, based on the merging order, the metadata index information of each storage partition is merged sequentially to obtain metadata index files corresponding to multiple storage partitions.

[0138] Step 511: Stop the database service.

[0139] In this disclosure, after obtaining the metadata index files corresponding to multiple storage partitions and the data files of each storage partition, the database service is stopped before replacing the current index file and the current data file of the database.

[0140] It should be noted that when the metadata index file and each data file are not obtained from the target upstream system (such as the HDFS system), the metadata index file and each data file can be downloaded from the system to the local machine or database server first, and then the database service can be stopped. This disclosure does not limit this.

[0141] Step 512: Replace the current metadata index file and current data file of the database using the metadata index file and the data files of each storage partition, and restart the database service.

[0142] In this disclosure, after stopping the database service, the current metadata index file and current data file of the database are replaced using the metadata index file and the data files of each storage partition, and then the database service is restarted, thereby realizing the batch import of business data from the target upstream system into the database and improving the efficiency of batch data import.

[0143] It should be noted that before replacing the current metadata index file and current data file of the database using the metadata index file and the data files of each storage partition, the current metadata index file and current data file of the database can also be backed up, and this disclosure does not limit this.

[0144] It should be noted that when replacing the current metadata index file and current data files of the database using the metadata index file and the data files of each storage partition, the metadata index file and the data files of each storage partition can be directly added to the specified directory of the database, and then the database service can be restarted to load the new data files, thereby completing the batch import of data. This disclosure does not limit this.

[0145] In this embodiment, firstly, in response to a batch data import request for the target upstream system, the system obtains each current business data item in the target upstream system, the size of the data file in the database associated with the target upstream system, and the data volume of each storage partition in the database. It then obtains the first field of each business data item, determines the first value corresponding to the first field, and sorts the multiple business data items based on the size of the first value. Next, based on the data volume of each storage partition and the data size of each business data item, the sorted multiple business data items are segmented to obtain the business data corresponding to each storage partition. Finally, based on the data file size, the business data corresponding to each storage partition is divided to obtain the data files to which the business data of that storage partition belongs, and the location of each data file in the database is determined. The system obtains the storage location information of the underlying storage space, the range of the first values ​​of the business data stored in each data file of each storage partition, and the partition identifier of each storage partition. Based on the storage location information of each data file of each storage partition, the range of the first values ​​of the business data stored in each data file, and the partition identifier of each storage partition, it determines the metadata index information corresponding to each storage partition. Then, based on the business data corresponding to each storage partition and the order of multiple sorted business data, it sequentially merges the metadata index information of each storage partition to obtain metadata index files corresponding to multiple storage partitions. The database service is then stopped. Finally, the current metadata index file and current data file of the database are replaced using the metadata index files and the data files of each storage partition, and the database service is restarted. Therefore, after obtaining the data files and metadata index information of each storage partition of the database, based on the order of multiple sorted business data and the business data corresponding to each storage partition, the metadata index information of each storage partition is merged to obtain the corresponding index file. The current index file and data file of the database are then replaced using the index file and the data files, thereby improving the efficiency of batch data processing.

[0146] To achieve the above embodiments, this disclosure also proposes a data batch import device.

[0147] Figure 6 This is a schematic diagram of the structure of a data batch import device provided in an embodiment of the present disclosure.

[0148] like Figure 6 As shown, the data batch import device 600 may include:

[0149] The acquisition module 601 is used to respond to a batch data import request for the target upstream system, and to acquire each piece of business data currently in the target upstream system, the size of the data file in the database associated with the target upstream system, and the amount of data in each storage partition of the database.

[0150] The segmentation module 602 is used to segment multiple business data based on the data volume of each storage partition and the data size of each business data to obtain the business data corresponding to each storage partition;

[0151] The partitioning module 603 is used to partition the business data corresponding to each storage partition based on the data file size, and obtain the data files to which the business data of the storage partition belongs.

[0152] Replacement module 604 is used to replace the current data file of the database using the data files of each storage partition.

[0153] Optionally, the segmentation module 602 described above is specifically used for:

[0154] Retrieve the first field of each piece of business data;

[0155] Determine the first value corresponding to the first field of each piece of business data;

[0156] Sort multiple business data based on the size of the first value corresponding to each business data;

[0157] Based on the data volume of each storage partition and the data size of each business data item, the sorted business data items are divided to obtain the business data corresponding to each storage partition.

[0158] Optionally, the aforementioned replacement module 604 is specifically used for:

[0159] Determine the storage location information of each data file in the underlying storage space of the database for each storage partition;

[0160] Obtain the range of the first values ​​of the business data stored in each data file of each storage partition, as well as the partition identifier of each storage partition;

[0161] Based on the storage location information of each data file in each storage partition, the range of the first value of the business data stored in each data file, and the partition identifier of each storage partition, determine the metadata index information corresponding to each storage partition.

[0162] Based on the metadata index information corresponding to each storage partition and each data file, the current metadata index information and the current data files of the database are replaced.

[0163] Optionally, the aforementioned replacement module 604 is also used for:

[0164] Based on the business data stored in each data file of each storage partition, and the order among the sorted business data, sort each data file under multiple storage partitions;

[0165] Based on the order of each data file, determine the number of layers in the underlying storage space of the database where each data file is stored and its position in that layer;

[0166] The number of layers where each data file is stored and its position within that layer determine the storage location information of each data file in the underlying storage space of the database.

[0167] Optionally, the aforementioned replacement module 604 is also used for:

[0168] Based on the order of each data file, and the i-th layer of storage in the underlying storage space of the database... i-1 Given a number of data files, determine the level in which each data file is stored and its position within that level, where i is a consecutive positive integer starting from 1.

[0169] Optionally, the aforementioned replacement module 604 is also used for:

[0170] Based on the business data corresponding to each storage partition and the order of multiple sorted business data, the metadata index information of each storage partition is merged in turn to obtain metadata index files corresponding to multiple storage partitions.

[0171] Stop the database service;

[0172] Using the metadata index file and the data files of each storage partition, replace the current metadata index file and the current data files of the database, and then restart the database service.

[0173] The functions and specific implementation principles of the modules described in this embodiment can be found in the above method embodiments, and will not be repeated here.

[0174] In this disclosure, in response to a batch data import request for the target upstream system, the system first obtains each piece of current business data in the target upstream system, the size of the data files in the database associated with the target upstream system, and the data volume of each storage partition in the database. Then, based on the data volume of each storage partition and the size of each piece of business data, the system segments multiple pieces of business data to obtain the business data corresponding to each storage partition. Next, based on the data file size, the system divides the business data corresponding to each storage partition to obtain the data files to which the business data of that storage partition belongs. Finally, the system uses the data files of each storage partition to replace the current data file in the database. Thus, after obtaining the full amount of business data from the upstream system, the system distributes the business data to each storage partition based on the data volume of each storage partition in the database, and divides the business data of each storage partition based on the data file size in the database to obtain the data files to which the business data of each storage partition belongs. The system uses the data files of each storage partition to replace the current data file in the database. This effectively avoids the impact on the service performance of the database during batch data import, thereby improving the efficiency of batch data import.

[0175] Figure 7 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown.

[0176] Figure 7 The electronic device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0177] like Figure 7As shown, electronic device 12 is represented in the form of a general-purpose computing device. Components of electronic device 12 may include, but are not limited to: one or more processors or processing units 16, memory 28, and a bus 18 connecting different system components (including memory 28 and processing unit 16). Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0178] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0179] Memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 7 Not shown; usually referred to as a "hard drive".

[0180] although Figure 7Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a Compact Disc Read-Only Memory (CD-ROM), a Digital Video Disc Read-Only Memory (DVD-ROM), or other optical media). In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0181] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0182] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable human interaction with electronic device 12, and / or with any device that enables electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays 35, RAID systems, tape drives, and data backup storage systems.

[0183] The processing unit 16 executes various functional applications and parameter information determination by running the program stored in the memory 28, such as implementing the data batch import method mentioned in the foregoing embodiments.

[0184] To implement the above embodiments, this disclosure also proposes a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the data batch import method proposed in the foregoing embodiments of this disclosure.

[0185] To implement the above embodiments, this disclosure also proposes a computer program product that, when the instruction processor in the computer program product is executed, performs the data batch import method as proposed in the foregoing embodiments of this disclosure.

[0186] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0187] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0188] It should be noted that in the description of this disclosure, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0189] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0190] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0191] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0192] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0193] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0194] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0195] Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A method for batch data import, characterized in that, include: In response to a batch data import request for a target upstream system, obtain each piece of current business data in the target upstream system, the size of the data file in the database associated with the target upstream system, and the amount of data in each storage partition of the database; Based on the data volume of each storage partition and the data size of each piece of business data, multiple pieces of business data are segmented to obtain the business data corresponding to each storage partition; Based on the data file size, the business data corresponding to each storage partition is divided to obtain the data files to which the business data of the storage partition belongs; The current data files of the database are replaced using the data files of each of the storage partitions.

2. The method as described in claim 1, wherein, The step of segmenting multiple pieces of business data based on the data volume of each storage partition and the data size of each piece of business data to obtain the business data corresponding to each storage partition includes: Obtain the first field of each piece of business data; Determine the first value corresponding to the first field of each piece of business data; Based on the magnitude of the first value corresponding to each piece of business data, sort the multiple pieces of business data; Based on the data volume of each storage partition and the data size of each piece of business data, the sorted multiple pieces of business data are divided to obtain the business data corresponding to each storage partition.

3. The method as described in claim 2, characterized in that, The step of replacing the current data file of the database using each data file of each of the storage partitions includes: Determine the storage location information of each data file in the underlying storage space of the database for each of the storage partitions; Obtain the range of first values ​​of business data stored in each data file of each storage partition, and the partition identifier of each storage partition; Based on the storage location information of each data file in each storage partition, the range of the first value of the business data stored in each data file, and the partition identifier of each storage partition, the metadata index information corresponding to each storage partition is determined. Based on the metadata index information corresponding to each storage partition and each data file, the current metadata index information and the current data files of the database are replaced.

4. The method as described in claim 3, characterized in that, Determining the storage location information of each data file in the underlying storage space of the database for each storage partition includes: Based on the business data stored in each data file of each storage partition, and the order among the sorted business data, each data file under the multiple storage partitions is sorted. Based on the order of each data file, determine the number of layers in the underlying storage space of the database where each data file is stored and its position in that layer; The number of layers in which each data file is stored and its position within that layer are determined to be the storage location information of each data file in the underlying storage space of the database.

5. The method of claim 4, wherein, Determining the level and position of each data file within the underlying storage space of the database based on the order of each data file includes: Based on the order of each of the data files, and the i-th layer of storage in the underlying storage space of the database... i-1 A number of data files are defined, and the layer number and position of each data file in that layer are determined sequentially, where i is a consecutive positive integer starting from 1.

6. The method as described in claim 3, characterized in that, The replacement of the current metadata index information and current data files of the database based on the metadata index information corresponding to each storage partition and each data file includes: Based on the business data corresponding to each storage partition and the order among the sorted business data, the metadata index information of each storage partition is merged sequentially to obtain the metadata index file corresponding to the multiple storage partitions; Stop the service of the database; Using the metadata index file and the data files of each storage partition, the current metadata index file and the current data files of the database are replaced, and the database service is restarted.

7. A data batch import device, characterized in that, include: The acquisition module is used to respond to a batch data import request for a target upstream system by acquiring each piece of current business data in the target upstream system, the size of the data file in the database associated with the target upstream system, and the amount of data in each storage partition of the database. The segmentation module is used to segment multiple pieces of business data based on the data volume of each storage partition and the data size of each piece of business data, so as to obtain the business data corresponding to each storage partition. The partitioning module is used to partition the business data corresponding to each storage partition based on the size of the data file, and obtain the data files to which the business data of the storage partition belongs; The replacement module is used to replace the current data files of the database using the data files of each of the storage partitions.

8. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that may be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images