Data batch importing method and device, electronic equipment and storage medium

Abstract

The invention provides a data batch importing method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: in response to a data batch import request for a target upstream system, obtaining each piece of current business data in the target upstream system, the size of a data file in a database associated with the target upstream system, and the data volume of each storage partition of the database; based on the data volume of each storage partition and the data size of each piece of business data, segmenting the plurality of pieces of business data to obtain business data corresponding to each storage partition; dividing the business data corresponding to each storage partition based on the size of the data file to obtain each data file to which the business data of the storage partition belongs; and replacing the current data file of the database by utilizing each data file of each storage partition. Therefore, the influence on the service performance of the database during data batch import is effectively avoided, and the data batch import efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for batch data import. Background Art

[0002] Currently, for KV databases, data can usually be imported in batches based on shell / python / java scripts. However, this method may occupy high database resources for a long time, causing increased pressure on database cluster services, affecting database service performance, and low batch import efficiency. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device and storage medium for batch data import, which are used to solve the problem of how to improve the efficiency of batch data import while reducing the impact on the service performance of the database.

[0004] According to one aspect of the present disclosure, a method for batch importing data is provided, comprising:

[0005] In response to a data batch import request for a target upstream system, obtaining each piece of current business data in the target upstream system, the size of a data file in a database associated with the target upstream system, and the amount of data in each storage partition of the database;

[0006] Based on the data volume of each storage partition and the data size of each piece of business data, multiple pieces of business data are segmented to obtain business data corresponding to each storage partition;

[0007] Based on the size of the data files, the business data corresponding to each of the storage partitions is divided to obtain the data files to which the business data of the storage partition belongs;

[0008] The current data files of the database are replaced by using the data files of each storage partition.

[0009] According to another aspect of the present disclosure, there is provided a device for batch importing data, comprising:

[0010] An acquisition module, configured to respond to a data batch import request for a target upstream system, acquire each piece of current business data in the target upstream system, the size of a data file in a database associated with the target upstream system, and the amount of data in each storage partition of the database;

[0011] A segmentation module, used to segment the multiple business data based on the data volume of each storage partition and the data size of each business data, to obtain the business data corresponding to each storage partition;

[0012] A partitioning module, used to partition the business data corresponding to each storage partition based on the size of the data file, and obtain the data files to which the business data of the storage partition belongs;

[0013] The replacement module is used to replace the current data files of the database using the data files of each storage partition.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor;

[0016] and, a memory communicatively coupled to the at least one processor;

[0017] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the method of the above embodiment.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method according to the above embodiment.

[0019] The present disclosure provides a method, device, electronic device and storage medium for batch data import. First, in response to a data batch import request for a target upstream system, each current business data in the target upstream system, the data file size in the database associated with the target upstream system, and the data volume of each storage partition of the database are obtained, and then based on the data volume of each storage partition and the data size of each business data, multiple business data are segmented to obtain the business data corresponding to each storage partition, and then based on the data file size, the business data corresponding to each storage partition is divided to obtain the data files to which the business data of the storage partition belongs, and finally, the data files of each storage partition are used to replace the current data files of the database. Thus, after obtaining the full amount of business data of the upstream system, the business data is dispersed to each storage partition based on the data volume of each storage partition of the database, and the business data of each storage partition is divided based on the data file size of the database to obtain the data files to which the business data of each storage partition belongs, and the data files of each storage partition are used to replace the current data files of the database, thereby effectively avoiding the impact on the service performance of the database during the batch data import on the basis of realizing the batch data import, and improving the efficiency of the batch data import. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] Figure 1 A flowchart of a method for batch importing data provided by an embodiment of the present disclosure;

[0022] Figure 2 A flowchart of a method for batch importing data provided by an embodiment of the present disclosure;

[0023] Figure 3 A flowchart of a method for batch importing data provided by an embodiment of the present disclosure;

[0024] Figure 4 A flowchart of a method for batch importing data provided by an embodiment of the present disclosure;

[0025] Figure 5 A flowchart of a method for batch importing data provided by an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of the structure of a device for batch data importing provided by an embodiment of the present disclosure;

[0027] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure.

[0028] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0030] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0031] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations.

[0032] The following describes in detail the method for batch importing data in an embodiment of the present disclosure with reference to the accompanying drawings.

[0033] Figure 1 A flowchart of a method for batch importing data provided by an embodiment of the present disclosure.

[0034] like Figure 1 As shown, the method includes:

[0035] Step 101, in response to a data batch import request for a target upstream system, obtain each piece of current business data in the target upstream system, the size of data files in a database associated with the target upstream system, and the data volume of each storage partition of the database.

[0036] It should be noted that the target upstream system may also be referred to as a "target business system", which may be any type of system, and the business data generated by it may be stored in its associated database. For example, the target upstream system may be a distributed business system (such as HDFS), which is not limited in this disclosure.

[0037] Among them, HDFS is the abbreviation of Hadoop Distributed File System (HDFS), and Hadoop is a distributed system infrastructure.

[0038] It should be noted that the data batch import request for the target upstream system can be a data batch import request triggered when certain conditions (such as reaching a trigger period, etc.) are met, or it can also be a data batch import request triggered according to actual needs. This disclosure does not limit this.

[0039] Among them, the business data in the target upstream system is a tag field, that is, a key-value pair field, which is not limited in the present disclosure.

[0040] The database associated with the target upstream system is a distributed key-value database, which is not limited in the present disclosure.

[0041] The data file size may be preset, for example, the data file size in the database may be 20 megabytes (MB) or 50 MB, etc., which is not limited in the present disclosure.

[0042] It should be noted that the storage partitions of the database and the data volume of each storage partition can be determined based on the data volume of the business data of the target upstream system and the storage resources and computing resources of the database. The data volume of each storage partition may be different, or may be the same, and the present disclosure does not limit this.

[0043] In the present disclosure, in response to a request for batch import of data to a target upstream system, each current business data in the target upstream system, the size of the data file in the database associated with the target upstream system, and the amount of data in each storage partition of the database can be first obtained to provide a data basis for batch importing the business data in the target upstream system into the database through the data batch import method provided by the present disclosure.

[0044] Step 102, based on the data volume of each storage partition and the data size of each business data, multiple business data are divided to obtain business data corresponding to each storage partition.

[0045] In the present disclosure, after obtaining each current business data in the target upstream system and the data volume of each storage partition of the database, in order to improve the database service performance (such as the efficiency of data query service performance, etc.), the business data of the target upstream system can be stored in each storage partition in a dispersed manner. At this time, the total data volume of the business data of the target upstream system can be first determined based on the data size of each business data, and then based on the data volume of each storage partition and the total data volume of the business data of the target upstream system, multiple business data can be divided to obtain the business data corresponding to each storage partition.

[0046] It should be noted that, since the amount of data in the value field of each piece of business data may be different, the data size of each piece of business data may be different, and the present disclosure does not limit this.

[0047] It should be noted that, when the storage resources of the database meet the conditions, the value field of the business data can also be serialized. At this time, when determining the data size of each business data, the serialization overhead of the business data is also included, and the present disclosure does not limit this.

[0048] It should be noted that when multiple business data are segmented based on the data volume of the storage partition and the data size of each business data, any preset segmentation method can be used. For example, the business data can be segmented based on the hash value of the key field of the business data (such as the key field of the key-value pair data, etc.), or the business data can be segmented based on the range of the identifier of the key field of the business data (such as the key identifier of the key-value pair data, etc.), and the present disclosure does not limit this.

[0049] Step 103: based on the size of the data files, divide the business data corresponding to each storage partition to obtain the data files to which the business data of the storage partition belongs.

[0050] In the present disclosure, after obtaining the business data corresponding to each storage partition and the size of the data files in the database, the business data corresponding to each storage partition can be divided based on the data file size, and then each part of the divided business data can be stored in a data file respectively, to obtain the data files belonging to the business data of each storage partition, thereby generating the underlying data files of the database associated with the target upstream system offline, providing conditions for improving the efficiency of batch data import.

[0051] It should be noted that the number of data files to which the business data of each storage partition belongs may be different, or may be the same, and the amount of business data stored in each data file is similar, which is not limited in the present disclosure.

[0052] Optionally, when the business data of each storage partition is divided based on the data file size to obtain the data files belonging to the business data of the storage partition, the number of data files required for the business data of each storage partition can be first determined based on the data file size and the total data size of the business data of each storage partition, and a corresponding number of data files can be generated. Then, based on the data file size, the business data of each storage partition can be divided, and then each part of the divided business data can be stored in a data file respectively to obtain the data files belonging to the business data of each storage partition. The present disclosure does not limit this.

[0053] Optionally, when dividing the business data of each storage partition based on the data file size, you can first determine the number of data files required for the business data of the target upstream system based on the data file size and the total data size of the business data of the target upstream system, and generate a corresponding number of data files. Then, based on the data file size, each part of the divided business data is stored in a data file respectively. Then, based on the business data corresponding to each storage partition, determine the data files to which the business data of each storage partition belongs. The present disclosure does not limit this.

[0054] Step 104: Use the data files in each storage partition to replace the current data files in the database.

[0055] In the present disclosure, after obtaining the data files belonging to the business data of each storage partition, the data files of each storage partition can be used to replace the current data files of the database associated with the target upstream system, thereby improving the efficiency of batch data importing into the database while realizing it.

[0056] In the disclosed embodiment, first, in response to a data batch import request for a target upstream system, each current business data in the target upstream system, the data file size in the database associated with the target upstream system, and the data volume of each storage partition of the database are obtained, and then based on the data volume of each storage partition and the data size of each business data, multiple business data are segmented to obtain the business data corresponding to each storage partition, and then based on the data file size, the business data corresponding to each storage partition is divided to obtain the data files to which the business data of the storage partition belongs, and finally, the data files of each storage partition are used to replace the current data files of the database. Thus, after obtaining the full amount of business data of the upstream system, the business data is dispersed to each storage partition based on the data volume of each storage partition of the database, and the business data of each storage partition is divided based on the data file size of the database to obtain the data files to which the business data of each storage partition belongs, and the data files of each storage partition are used to replace the current data files of the database, thereby effectively avoiding the impact on the service performance of the database during the batch import of data and improving the efficiency of the batch import of data.

[0057] Figure 2 A flowchart of a method for batch importing data provided by an embodiment of the present disclosure.

[0058] like Figure 2 As shown, the method includes:

[0059] Step 201, in response to a data batch import request for a target upstream system, obtain each piece of current business data in the target upstream system, the size of data files in a database associated with the target upstream system, and the data volume of each storage partition of the database.

[0060] The specific implementation form of step 201 can refer to the detailed description in other embodiments of the present disclosure, and will not be described in detail here.

[0061] Step 202: Obtain the first field of each piece of business data.

[0062] Among them, the first field is the key field of the business data, that is, the key field of the business data.

[0063] In the present disclosure, after obtaining each current business data in the target upstream system, before importing the business data into the database in batches, in order to improve the efficiency of batch data import, the business data of the target upstream system can be globally sorted. At this time, the first field of each business data can be obtained first.

[0064] Step 203: determine the first value corresponding to the first field of each piece of business data.

[0065] The first value may be a data identifier of the business data, which may be an identifier in any form. For example, the first value corresponding to the first field of the business data may be a hash value of the first field, etc., which is not limited in the present disclosure.

[0066] It should be noted that the method for determining the first value corresponding to the first field of each piece of business data may be any pre-set method. For example, a hash value of the first field of the business data may be calculated by a preset hash function (such as MD5, SHA-1), and determined as the first value of the business data, or the hash value of the first field may be converted or encoded by a preset method to obtain the first value, etc., and the present disclosure does not limit this.

[0067] Among them, MD5 is the abbreviation of Message-Digest Algorithm 5 (MD5).

[0068] Among them, SHA-1 is the abbreviation of Secure Hash Algorithm 1 (SHA-1).

[0069] In the present disclosure, after obtaining the first field of each piece of business data, in order to globally sort the business data, the first value corresponding to the first field of each piece of business data may be first determined.

[0070] Step 204: sort the multiple pieces of business data based on the size of the first value corresponding to each piece of business data.

[0071] In the present disclosure, after determining the first value corresponding to the first field of each piece of business data, multiple pieces of business data can be sorted based on the size of the first value corresponding to each piece of business data. For example, the business data can be sorted from large to small based on the first value of each piece of business data, thereby achieving global ordering of the business data and providing conditions for improving the efficiency of batch data import. The present disclosure does not limit this.

[0072] Step 205 , based on the data volume of each storage partition and the data size of each piece of business data, the sorted multiple pieces of business data are segmented to obtain business data corresponding to each storage partition.

[0073] It should be noted that when the sorted multiple business data are divided based on the data volume of each storage partition and the data size of each business data, the sorted multiple business data can be divided in order starting from the first business data based on the data volume of each storage partition and the data size of each business data, so that the business data corresponding to each storage partition are ordered. The present disclosure does not limit this.

[0074] Step 206: divide the business data corresponding to each storage partition based on the size of the data files, and obtain the data files to which the business data of the storage partition belongs.

[0075] Step 207: Use the data files in each storage partition to replace the current data files in the database.

[0076] The specific implementation forms of step 206 to step 207 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0077] In the disclosed embodiment, first, in response to a data batch import request for a target upstream system, each current business data in the target upstream system, the data file size in a database associated with the target upstream system, and the data volume of each storage partition of the database are obtained, then the first field of each business data is obtained, and the first value corresponding to the first field of each business data is determined, and based on the size of the first value corresponding to each business data, multiple business data are sorted, and then based on the data volume of each storage partition and the data size of each business data, the sorted multiple business data are divided to obtain the business data corresponding to each storage partition, and finally based on the data file size, the business data corresponding to each storage partition is divided to obtain the data files belonging to the business data of the storage partition, and the data files of each storage partition are used to replace the current data files of the database. Therefore, after obtaining the business data of the upstream system, the first value corresponding to the first field of each business data is determined, and the business data is sorted based on the size of the first value. Based on the data volume of each storage partition of the database and the size of the data file, the sorted business data is segmented and divided, and the data files belonging to the business data of each storage partition are obtained. Then, each data file is used to replace the current data file of the database, thereby reducing the amount of resources occupied during the batch data import process and improving the efficiency of the batch data import.

[0078] Figure 3 A flowchart of a method for batch importing data provided by an embodiment of the present disclosure.

[0079] like Figure 3 As shown, the method includes:

[0080] Step 301, in response to a data batch import request for a target upstream system, obtain each piece of current business data in the target upstream system, the size of data files in a database associated with the target upstream system, and the data volume of each storage partition of the database.

[0081] Step 302: Obtain the first field of each piece of business data.

[0082] Step 303: determine the first value corresponding to the first field of each piece of business data.

[0083] Step 304: sort the multiple pieces of business data based on the size of the first value corresponding to each piece of business data.

[0084] Step 305 , based on the data volume of each storage partition and the data size of each piece of business data, the sorted multiple pieces of business data are segmented to obtain business data corresponding to each storage partition.

[0085] Step 306: divide the business data corresponding to each storage partition based on the size of the data files, and obtain the data files to which the business data of the storage partition belongs.

[0086] The specific implementation forms of steps 301 to 306 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0087] Step 307: Determine the storage location information of each data file of each storage partition in the underlying storage space of the database.

[0088] It should be noted that the underlying storage space of the database may be the disk storage space of the database.

[0089] In the present disclosure, after obtaining the data files belonging to the business data of each storage partition, when using the data files of each storage partition to replace the current data files of the database, in order to ensure that the database can provide accurate and reliable services normally after the data files are replaced, it is also necessary to replace the index information in the database. At this time, the storage location information of each data file of each storage partition in the underlying storage space of the database can be determined first.

[0090] Step 308: Obtain the range of the first value of the business data stored in each data file of each storage partition, and the partition identifier of each storage partition.

[0091] The partition identifier may be an identifier used to characterize a partition, and may be an identifier in any pre-set form. For example, the partition identifier may be the name of a storage partition, or may be a partition serial number identifier determined according to the sorting order of the business data corresponding to each storage partition, etc., which is not limited in the present disclosure.

[0092] In the present disclosure, after determining the storage location information of each data file of each storage partition in the underlying storage space of the database, the range of the first value of the business data stored in each data file can be determined based on the first value of the first business data stored in each data file of each storage partition and the first value of the last business data stored, and the partition identifier of each storage partition can be obtained, thereby providing a data basis for determining the index information of each storage partition.

[0093] Step 309, based on the storage location information of each data file of each storage partition, the range of the first value of the business data stored in each data file, and the partition identifier of each storage partition, determine the metadata index information corresponding to each storage partition.

[0094] The metadata index information may be index information of business data in each storage partition, which may include a partition identifier, a data file identifier, a data range of business data stored in a data file, etc., which is not limited in the present disclosure.

[0095] The data file identifier may be an identifier used to characterize the data file, and may be an identifier in any pre-set form. For example, the data file identifier may be the name of the data file, or the data files may be sorted according to the first value of the business data stored in the data file, and the sequence number corresponding to the data file may be determined and determined as the data file identifier, which is not limited in the present disclosure.

[0096] It should be noted that when the data structure in the underlying storage space of the database is a hierarchical data structure (such as the number of LSMs, etc.), the metadata index information may also include the number of layers where the data file is located, etc., and the present disclosure does not limit this.

[0097] Among them, LSM is the abbreviation of Log-Structured-Merge-Tree (LSM).

[0098] In the present disclosure, after determining the storage location information of each data file of each storage partition in the underlying storage space of the database, the metadata index information corresponding to each storage partition can be determined based on the storage location information of each data file of each storage partition, the range of the first value of the business data stored in each data file, and the partition identifier of each storage partition.

[0099] Step 310: Based on the metadata index information and data files corresponding to each storage partition, the current metadata index information and data files of the database are replaced.

[0100] In the present disclosure, after determining the metadata index information corresponding to each storage partition, the current metadata index information and current data files of the database can be replaced based on the metadata index information and each data file corresponding to each storage partition, thereby ensuring that the database can provide accurate and reliable services while realizing batch import of business data of the target upstream system into the database.

[0101] In the disclosed embodiment, first, in response to a data batch import request for a target upstream system, each business data currently in the target upstream system, the data file size in a database associated with the target upstream system, and the data volume of each storage partition of the database are obtained, and the first field of each business data is obtained, and a first value corresponding to the first field of each business data is determined. Based on the size of the first value corresponding to each business data, multiple business data are sorted, and then based on the data volume of each storage partition and the data size of each business data, the sorted multiple business data are segmented to obtain the business data corresponding to each storage partition, and based on the data file size, the business data corresponding to each storage partition is segmented. The data files to which the business data of each storage partition belongs are divided, and then the storage location information of each data file of each storage partition in the underlying storage space of the database is determined, and the range of the first value of the business data stored in each data file of each storage partition and the partition identification of each storage partition are obtained. Based on the storage location information of each data file of each storage partition, the range of the first value of the business data stored in each data file, and the partition identification of each storage partition, the metadata index information corresponding to each storage partition is determined, and finally, based on the metadata index information corresponding to each storage partition and each data file, the current metadata index information of the database and the current data file are replaced. Thus, after obtaining the data files to which the business data of each storage partition of the database belongs, the storage location information of each data file in the database is determined, and the metadata index information of each storage partition is determined by the storage location information of each data file, the range of the business data of each data file and the partition identification of each storage partition, and the current index information of the database and the current data file are replaced by using the metadata index information of each storage partition and each data file, thereby improving the efficiency of batch import of data on the basis of ensuring that the database provides reliable and accurate services.

[0102] Figure 4 A flowchart of a method for batch importing data provided by an embodiment of the present disclosure.

[0103] like Figure 4 As shown, the method includes:

[0104] Step 401, in response to a data batch import request for a target upstream system, obtain each piece of current business data in the target upstream system, the size of data files in a database associated with the target upstream system, and the data volume of each storage partition of the database.

[0105] Step 402: Obtain the first field of each piece of business data.

[0106] Step 403: determine the first value corresponding to the first field of each piece of business data.

[0107] Step 404: sort the multiple pieces of business data based on the size of the first value corresponding to each piece of business data.

[0108] Step 405 , based on the data volume of each storage partition and the data size of each piece of business data, the sorted multiple pieces of business data are segmented to obtain business data corresponding to each storage partition.

[0109] Step 406: based on the size of the data files, divide the business data corresponding to each storage partition to obtain the data files to which the business data of the storage partition belongs.

[0110] The specific implementation forms of steps 401 to 406 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0111] Step 407 , sorting each data file under the multiple storage partitions based on the business data stored in each data file of each storage partition and the order of the sorted multiple business data.

[0112] In the present disclosure, after obtaining the data files belonging to the business data of the storage partition, in order to ensure that the data files of each storage partition are ordered when stored in the underlying storage space of the database, each data file under multiple storage partitions can be sorted based on the business data stored in each data file of each storage partition, and the order of the sorted multiple business data, so that the data files required for the business data of the target upstream system are globally ordered, thereby improving the efficiency of batch data import.

[0113] Step 408, based on the sequence of each data file, determine the layer number of each data file stored in the underlying storage space of the database and the position in the layer.

[0114] The position of the data file in the layer may be the nth file of the data file in the layer, which is not limited in the present disclosure.

[0115] In the present disclosure, when the data structure of the underlying storage space of the database is a hierarchical structure, in order to determine the storage location information of each data file, after sorting the data files belonging to the business data, the number of layers in which each data file is stored in the underlying storage space of the database and the position in the layer can be determined based on the order of each data file.

[0116] Optionally, when determining the number of layers in which each data file is stored in the underlying storage space of the database and the position in the layer based on the order of each data file, the number of layers in which each data file is stored in the underlying storage space of the database and the position in the layer may be determined based on the order of each data file and the number of layers in which the i-th storage 2 is stored in the underlying storage space of the database. i-1 A number of data files are stored, and the number of layers and the position of each data file in the layer are determined in turn, where i is a continuous positive integer starting from 1, which is not limited in the present disclosure.

[0117] Step 409: Determine the layer number and the position of each data file in the layer as the storage location information of each data file in the bottom storage space of the database.

[0118] In the present disclosure, after determining the layer number and position of each data file in the underlying storage space of the database, the layer number and position of each data file in the layer can be determined as the storage location information of each data file, thereby improving the efficiency of batch data import.

[0119] Step 410: Obtain the range of the first value of the business data stored in each data file of each storage partition, and the partition identifier of each storage partition.

[0120] Step 411, based on the storage location information of each data file of each storage partition, the range of the first value of the business data stored in each data file, and the partition identifier of each storage partition, determine the metadata index information corresponding to each storage partition.

[0121] Step 412: based on the metadata index information and the data files corresponding to each storage partition, the current metadata index information and the current data files of the database are replaced.

[0122] The specific implementation forms of steps 410 to 412 may refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0123] In the disclosed embodiment, first, in response to a data batch import request for a target upstream system, each business data currently in the target upstream system, the size of a data file in a database associated with the target upstream system, and the data volume of each storage partition of the database are obtained, and the first field of each business data is obtained, and a first value corresponding to the first field of each business data is determined. Based on the size of the first value corresponding to each business data, multiple business data are sorted, and then based on the data volume of each storage partition and the data size of each business data, the sorted multiple business data are divided to obtain the business data corresponding to each storage partition, and then based on the data file size, the business data corresponding to each storage partition is divided to obtain the data files to which the business data of the storage partition belongs, and based on the business data stored in each data file of each storage partition, and the sorted multiple business data According to the order between the data files, each data file under the multiple storage partitions is sorted, and based on the order of each data file, the number of layers in which each data file is stored in the underlying storage space of the database and the position in the layer are determined, and the number of layers in which each data file is stored and the position in the layer are determined as the storage location information of each data file in the underlying storage space of the database, and then the range of the first value of the business data stored in each data file of each storage partition and the partition identifier of each storage partition are obtained, and based on the storage location information of each data file of each storage partition, the range of the first value of the business data stored in each data file, and the partition identifier of each storage partition, the metadata index information corresponding to each storage partition is determined, and finally, based on the metadata index information corresponding to each storage partition and each data file, the current metadata index information and the current data file of the database are replaced. Therefore, after obtaining the data files belonging to the business data of each storage partition of the database, the data files belonging to the business data are globally sorted based on the order of the sorted business data, and the number of layers in which each data file is stored in the underlying storage space of the database and the position in the layer are determined based on the order of each data file. The number of layers and the position in the layer are determined as the underlying storage location information of each data file, and together with the business data range of each data file and the partition identifier of the storage partition to which it belongs, are determined as the index information of each storage partition. The index information and data files are used to replace the current index information and current data files of the database, thereby improving the efficiency of batch import of data while ensuring the order between the business data in the batch imported database.

[0124] Figure 5 A flowchart of a method for batch importing data provided by an embodiment of the present disclosure.

[0125] like Figure 5 As shown, the method includes:

[0126] Step 501, in response to a data batch import request for a target upstream system, obtain each piece of current business data in the target upstream system, the size of data files in a database associated with the target upstream system, and the data volume of each storage partition of the database.

[0127] Step 502: Obtain the first field of each piece of business data.

[0128] Step 503: determine the first value corresponding to the first field of each piece of business data.

[0129] Step 504: sort the multiple pieces of business data based on the size of the first value corresponding to each piece of business data.

[0130] Step 505 , based on the data volume of each storage partition and the data size of each piece of business data, the sorted multiple pieces of business data are segmented to obtain business data corresponding to each storage partition.

[0131] Step 506: divide the business data corresponding to each storage partition based on the size of the data files, and obtain the data files to which the business data of the storage partition belongs.

[0132] Step 507: Determine the storage location information of each data file of each storage partition in the underlying storage space of the database.

[0133] Step 508: Obtain the range of the first value of the business data stored in each data file of each storage partition, and the partition identifier of each storage partition.

[0134] Step 509, based on the storage location information of each data file of each storage partition, the range of the first value of the business data stored in each data file, and the partition identifier of each storage partition, determine the metadata index information corresponding to each storage partition.

[0135] The specific implementation forms of steps 501 to 509 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0136] Step 510, based on the business data corresponding to each storage partition and the order between the sorted multiple business data, the metadata index information of each storage partition is merged in sequence to obtain metadata index files corresponding to multiple storage partitions.

[0137] In the present disclosure, after determining the metadata index information corresponding to each storage partition, the merging order of the metadata index information of each storage partition can be determined based on the business data corresponding to each storage partition and the order between the sorted multiple business data, and then based on the merging order, the metadata index information of each storage partition is merged in turn to obtain metadata index files corresponding to multiple storage partitions.

[0138] Step 511, stop the database service.

[0139] In the present disclosure, after obtaining metadata index files corresponding to multiple storage partitions and data files of each storage partition, before replacing the current index files and current data files of the database, the database service is first stopped.

[0140] It should be noted that when the metadata index file and each data file are obtained in the target upstream system (such as the HDFS system), the metadata index file and each data file can be first downloaded from the system to the local or database server, and then the database service can be stopped. This disclosure does not limit this.

[0141] Step 512: Use the metadata index file and the data files of each storage partition to replace the current metadata index file and the current data files of the database, and restart the database service.

[0142] In the present disclosure, after stopping the database service, the metadata index file and the data files of each storage partition are used to replace the current metadata index file and the current data files of the database, and then the database service is restarted, thereby realizing batch import of the business data of the target upstream system into the database, and improving the efficiency of batch data import.

[0143] It should be noted that before using the metadata index file and the data files of each storage partition to replace the current metadata index file and the current data files of the database, the current metadata index file and the current data files of the database can also be backed up, and the present disclosure does not limit this.

[0144] It should be noted that when using the metadata index file and the data files of each storage partition to replace the current metadata index file and the current data files of the database, the metadata index file and the data files of each storage partition can be directly added to the specified directory of the database, and then the database service can be restarted to allow the database to load the new data files, thereby completing the batch import of data. The present disclosure does not limit this.

[0145] In the disclosed embodiment, first, in response to a data batch import request for a target upstream system, each business data currently in the target upstream system, the size of a data file in a database associated with the target upstream system, and the data volume of each storage partition of the database are obtained, and the first field of each business data is obtained, and a first value corresponding to the first field of each business data is determined. Then, based on the size of the first value corresponding to each business data, multiple business data are sorted, and then based on the data volume of each storage partition and the data size of each business data, the sorted multiple business data are segmented to obtain the business data corresponding to each storage partition, and based on the data file size, the business data corresponding to each storage partition is divided to obtain the data files belonging to the business data of the storage partition, and the data files of each storage partition are determined in the database. The storage location information in the underlying storage space of the storage partition is obtained, the range of the first value of the business data stored in each data file of each storage partition, and the partition identifier of each storage partition are obtained, and based on the storage location information of each data file of each storage partition, the range of the first value of the business data stored in each data file, and the partition identifier of each storage partition, the metadata index information corresponding to each storage partition is determined, and then based on the business data corresponding to each storage partition and the order between the sorted multiple business data, the metadata index information of each storage partition is merged in sequence to obtain the metadata index files corresponding to multiple storage partitions, and the service of the database is stopped, and finally, the metadata index file and the current data file of the database are replaced by using the metadata index file and the data files of each storage partition, and the service of the database is restarted. Thus, after obtaining the data files of each storage partition of the database and the metadata index information of each storage partition, based on the order of the sorted multiple business data and the business data corresponding to each storage partition, the metadata index information of each storage partition is merged to obtain the corresponding index file, and the index file and the data files are used to replace the current index file and data file of the database, thereby improving the efficiency of data batch processing.

[0146] In order to implement the above embodiment, the embodiment of the present disclosure also proposes a data batch import device.

[0147] Figure 6 A schematic diagram of the structure of a data batch import device provided in an embodiment of the present disclosure.

[0148] like Figure 6 As shown, the data batch import device 600 may include:

[0149] The acquisition module 601 is used to respond to the data batch import request for the target upstream system, obtain each business data currently in the target upstream system, the size of the data file in the database associated with the target upstream system, and the data volume of each storage partition of the database;

[0150] A segmentation module 602 is used to segment multiple business data based on the data volume of each storage partition and the data size of each business data to obtain business data corresponding to each storage partition;

[0151] A partitioning module 603 is used to partition the business data corresponding to each storage partition based on the size of the data file, and obtain the data files to which the business data of the storage partition belongs;

[0152] The replacement module 604 is used to replace the current data files of the database using the data files of each storage partition.

[0153] Optionally, the segmentation module 602 is specifically used to:

[0154] Get the first field of each business data;

[0155] Determine a first value corresponding to a first field of each piece of business data;

[0156] sorting the plurality of business data based on the size of the first value corresponding to each piece of business data;

[0157] Based on the data volume of each storage partition and the data size of each business data, the sorted multiple business data are segmented to obtain the business data corresponding to each storage partition.

[0158] Optionally, the replacement module 604 is specifically used to:

[0159] Determine the storage location information of each data file of each storage partition in the underlying storage space of the database;

[0160] Obtaining a range of first values ​​of business data stored in each data file of each storage partition and a partition identifier of each storage partition;

[0161] Determine metadata index information corresponding to each storage partition based on storage location information of each data file of each storage partition, a range of a first value of the business data stored in each data file, and a partition identifier of each storage partition;

[0162] Based on the metadata index information and data files corresponding to each storage partition, the current metadata index information and current data files of the database are replaced.

[0163] Optionally, the replacement module 604 is further used to:

[0164] Sort each data file under multiple storage partitions based on the business data stored in each data file of each storage partition and the order between the sorted multiple business data;

[0165] Based on the order of each data file, determine the layer number and position of each data file in the underlying storage space of the database;

[0166] The number of the layer where each data file is stored and the position in the layer are determined as the storage position information of each data file in the bottom storage space of the database.

[0167] Optionally, the replacement module 604 is further used to:

[0168] Based on the order of each data file and the i-th storage 2 in the underlying storage space of the database i-1 The number of data files is determined, and the number of layers and the position of each data file in the layer are determined in turn, where i is a consecutive positive integer starting from 1.

[0169] Optionally, the replacement module 604 is further used to:

[0170] Based on the business data corresponding to each storage partition and the order between the sorted multiple business data, the metadata index information of each storage partition is merged in sequence to obtain metadata index files corresponding to multiple storage partitions;

[0171] Stop the database service;

[0172] Use the metadata index file and the data files of each storage partition to replace the current metadata index file and the current data files of the database, and restart the database service.

[0173] The functions and specific implementation principles of the above modules in the embodiments of the present disclosure can be referred to the above method embodiments, and will not be repeated here.

[0174] In the present disclosure, first, in response to a data batch import request for a target upstream system, each current business data in the target upstream system, the data file size of the database associated with the target upstream system, and the data volume of each storage partition of the database are obtained, and then based on the data volume of each storage partition and the data size of each business data, multiple business data are segmented to obtain the business data corresponding to each storage partition, and then based on the data file size, the business data corresponding to each storage partition is divided to obtain the data files belonging to the business data of the storage partition, and finally, the data files of each storage partition are used to replace the current data files of the database. Thus, after obtaining the full amount of business data of the upstream system, the business data is dispersed to each storage partition based on the data volume of each storage partition of the database, and the business data of each storage partition is divided based on the data file size of the database to obtain the data files belonging to the business data of each storage partition, and the data files of each storage partition are used to replace the current data files of the database, thereby effectively avoiding the impact on the service performance of the database during the batch import of data and improving the efficiency of the batch import of data.

[0175] Figure 7 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown.

[0176] Figure 7 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0177] like Figure 7As shown, the electronic device 12 is in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, memory 28, and a bus 18 connecting different system components (including memory 28 and processing unit 16). The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any bus structure in a variety of bus structures. For example, these architectures include, but are not limited to, an Industry Standard Architecture (hereinafter referred to as: ISA) bus, a Micro Channel Architecture (hereinafter referred to as: MCA) bus, an Enhanced ISA bus, a Video Electronics Standards Association (hereinafter referred to as: VESA) local bus, and a Peripheral Component Interconnection (hereinafter referred to as: PCI) bus.

[0178] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0179] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 Not shown, often called a "hard drive").

[0180] although Figure 7Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (such as a compact disc read only memory (hereinafter referred to as: CD-ROM), a digital versatile disc read only 15 (Digital Video Disc Read Only Memory; hereinafter referred to as: DVD-ROM) or other optical media) may be provided. In these cases, each drive can be connected to the bus 18 through one or more data medium interfaces. The memory 28 may include at least one program product having a set (for example, at least one) of program modules, which are configured to perform the functions of the various embodiments of the present disclosure.

[0181] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0182] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), may also communicate with one or more devices that enable the human body to interact with the electronic device 12, and / or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. In addition, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the electronic device 12 through a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0183] The processing unit 16 executes various functional applications and parameter information determinations by running the programs stored in the memory 28, such as implementing the data batch import method mentioned in the above embodiment.

[0184] In order to implement the above embodiments, the present disclosure further proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data batch import method proposed in the above embodiments of the present disclosure is implemented.

[0185] In order to implement the above embodiments, the present disclosure further proposes a computer program product. When an instruction processor in the computer program product is executed, the data batch import method proposed in the above embodiments of the present disclosure is executed.

[0186] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0187] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0188] It should be noted that, in the description of the present disclosure, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "plurality" is two or more.

[0189] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0190] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0191] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0192] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0193] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0194] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0195] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. A method for batch importing data, characterized in that: include: In response to a data batch import request for a target upstream system, obtaining each piece of current business data in the target upstream system, the size of a data file in a database associated with the target upstream system, and the amount of data in each storage partition of the database; Based on the data volume of each storage partition and the data size of each piece of business data, multiple pieces of business data are segmented to obtain business data corresponding to each storage partition; Based on the size of the data files, the business data corresponding to each of the storage partitions is divided to obtain the data files to which the business data of the storage partition belongs; The current data files of the database are replaced by using the data files of each storage partition.

2. The method of claim 1, wherein: The step of segmenting the plurality of business data based on the data volume of each storage partition and the data size of each business data to obtain the business data corresponding to each storage partition includes: Obtaining the first field of each piece of business data; Determine a first value corresponding to the first field of each piece of the business data; sorting the plurality of pieces of business data based on the size of the first value corresponding to each piece of the business data; Based on the data volume of each storage partition and the data size of each piece of business data, the sorted multiple pieces of business data are segmented to obtain the business data corresponding to each storage partition.

3. The method according to claim 2, characterized in that The replacing the current data files of the database by using the data files of each storage partition includes: Determining storage location information of each data file of each storage partition in the underlying storage space of the database; Acquire a range of first values ​​of the business data stored in each data file of each storage partition, and a partition identifier of each storage partition; Determine metadata index information corresponding to each storage partition based on storage location information of each data file of each storage partition, a range of first values ​​of business data stored in each data file, and a partition identifier of each storage partition; Based on the metadata index information and the data files corresponding to each storage partition, the current metadata index information and the current data files of the database are replaced.

4. The method according to claim 3, characterized in that The determining of storage location information of each data file of each storage partition in the underlying storage space of the database includes: Based on the business data stored in each data file of each storage partition and the order of the sorted multiple business data, sort each data file under the multiple storage partitions; Based on the sequence of each of the data files, determining the number of layers in which each of the data files is stored in the underlying storage space of the database and the position in the layer; The number of the layer where each data file is stored and the position in the layer are determined as the storage position information of each data file in the bottom storage space of the database.

5. The method of claim 4, wherein: Determining the number of layers in which each data file is stored in the underlying storage space of the database and the position in the layer based on the order of each data file includes: Based on the order of each of the data files and the i-th layer storage 2 in the underlying storage space of the database i-1 A number of data files are stored, and the number of layers in which each data file is stored and the position in the layer are determined in sequence, wherein i is a continuous positive integer starting from 1.

6. The method according to claim 3, characterized in that The replacing the current metadata index information and the current data files of the database based on the metadata index information and the data files corresponding to each storage partition includes: Based on the business data corresponding to each of the storage partitions and the order of the sorted multiple business data, the metadata index information of each of the storage partitions is merged in sequence to obtain metadata index files corresponding to the multiple storage partitions; Stop the service of the database; The metadata index file and the data files of each storage partition are used to replace the current metadata index file and the current data files of the database, and the service of the database is restarted.

7. A device for batch importing data, characterized in that: include: An acquisition module, configured to respond to a data batch import request for a target upstream system, acquire each piece of current business data in the target upstream system, the size of a data file in a database associated with the target upstream system, and the amount of data in each storage partition of the database; A segmentation module, used to segment the multiple business data based on the data volume of each storage partition and the data size of each business data, to obtain the business data corresponding to each storage partition; A partitioning module, configured to partition the business data corresponding to each storage partition based on the size of the data file, and obtain data files to which the business data of the storage partition belongs; The replacement module is used to replace the current data files of the database using the data files of each storage partition.

8. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that may be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images