Data import method and device

Abstract

The embodiment of the present application provides a data import method and device, including: in response to a data import instruction, dividing the data to be imported into multiple data slices, and constructing an import relationship according to each data slice and each storage slice of the column-based database management system, wherein the import relationship is used to indicate the corresponding relationship between the data slice and the storage slice, and importing each data slice into the storage slice having a corresponding relationship with the data slice. By constructing the import relationship between the data slice and the storage slice, the data slice is imported into the corresponding storage slice based on the import relationship, thereby avoiding the technical problem of low import rate caused by different data to be imported competing for the same storage node when importing the data to be imported based on the random import strategy in the related technology. By performing the import operation based on the import relationship, targeted import can be achieved, thereby improving the technical effect of improving the import rate.

Description

Technical Field

[0001] The embodiments of the present application relate to the fields of computer technology and big data, and in particular to a data import method and device. Background Art

[0002] The Spark processor is a fast and versatile computing engine designed for large-scale data processing. The Spark processor can read data in a distributed manner, perform various conversions and processing on the data, and then import the processed data into a distributed storage target (such as a database management system, specifically ClickHouse, which is a column-based database management system for real-time analysis of big data).

[0003] In the prior art, a method for importing data includes: obtaining a data import request, the data import request including one or more data to be imported, randomly allocating storage slices to the one or more data to be imported in each storage slice of a database management system, and importing the one or more data to be imported into the corresponding storage slices.

[0004] In the process of implementing the present application, the inventors found that there are at least the following problems in the prior art: when importing the data to be imported into the database management system by the random import method, there may be a technical problem of a low import rate due to different data to be imported competing for the same storage node. Summary of the invention

[0005] The embodiments of the present application provide a data import method and device to solve the problem of low data import efficiency.

[0006] In a first aspect, an embodiment of the present application provides a data import method, comprising:

[0007] In response to a data import instruction, the data to be imported is divided into a plurality of data slices, and an import relationship is constructed according to each of the data slices and each storage slice of the column-based database management system, wherein the import relationship is used to indicate a corresponding relationship between the data slice and the storage slice;

[0008] Each of the data slices is imported into a storage slice that has a corresponding relationship with the data slice.

[0009] It is worth noting that, in this embodiment, by constructing an import relationship between data shards and storage shards, and the import relationship is used to indicate the corresponding relationship between data shards and storage shards, so that the data shards can be imported into the corresponding storage shards based on the import relationship, the technical problem of low import rate caused by different data to be imported competing for the same storage node when importing the data to be imported based on the random import strategy in the related technology is avoided. By performing the import operation based on the import relationship, targeted import can be achieved, thereby improving the technical effect of improving the import rate.

[0010] In some embodiments, constructing an import relationship according to each of the data shards and each storage shard of the column-based database management system includes:

[0011] Generate a list including address information of each storage shard, and broadcast the list in a plurality of import threads for performing import operations;

[0012] According to each of the import threads, the import relationship is constructed based on the modulo processing results between the list and each of the data slices.

[0013] It is worth noting that, in this embodiment, by determining the list and constructing the import relationship based on the modulo result between the list and the data shards, the data shards can be distributed relatively evenly among the storage shards, so that the load of each storage shard is relatively balanced, thereby improving the performance of data import and improving the technical effect of protecting and utilizing the resources of the column-based database management system.

[0014] In some embodiments, according to each of the import threads, the import relationship is constructed by taking a modulo processing result between the list and each of the data slices, including:

[0015] Assigning an index value to each of the data slices, and assigning a data slice including the index value to each of the import threads;

[0016] According to each of the import threads, the import relationship is constructed by taking the modulo results of the allocated data slices and the lists.

[0017] It is worth noting that, in this embodiment, by assigning index values ​​and assigning data shards including index values ​​to import threads, so as to construct an import relationship based on the modulo structure between the assigned data shards and lists by the import threads, the efficiency of constructing the import relationship can be improved, and the data shards can be evenly distributed among the storage shards, thereby achieving the technical effect of improving resource utilization.

[0018] In some embodiments, according to each of the import threads, the allocated data slices and the modulo results between the lists are used to construct the import relationship, including:

[0019] Through each of the import threads, the total number of the storage shards in the list is determined, and according to each of the import threads, the index value of the allocated data shard and the modulo result of the total number of the storage shards are taken to construct the import relationship.

[0020] It is worth mentioning that in this embodiment, each import thread performs modulo processing to obtain a modulo result. Since the import threads are independent of each other and execute in parallel, the efficiency of determining the modulo result can be improved, thereby improving the technical effect of improving the efficiency of constructing the import relationship.

[0021] In some embodiments, importing each of the data shards into a storage shard that has a corresponding relationship with the data shard includes:

[0022] Through each of the import threads, the data slices allocated by each of the import threads are imported into the storage slices with corresponding relationships in parallel.

[0023] It is worth noting that, in this embodiment, by executing the import operation in parallel based on multiple import threads, the technical effect of improving the efficiency of importing each data shard can be achieved.

[0024] In some embodiments, if any data shard fails to be imported, the method further includes:

[0025] According to a preset time interval, the arbitrary data slice is re-imported into a storage slice having a corresponding relationship.

[0026] It is worth noting that in the present embodiment, in the event of failure to import any data shard, the data import device can re-import after a time interval, and still import based on the import relationship, which can avoid the technical problem of waste of resources caused by complete failure of the previous operation of re-running (i.e., re-competing for storage shards) in the related technology, improve resource utilization, improve import efficiency, and avoid the problem of data loss caused by re-running, that is, it also improves the accuracy and reliability of data import.

[0027] In some embodiments, if any data shard fails to be imported, the method further includes:

[0028] Determine a storage shard that has a corresponding relationship with the arbitrary data shard, determine a copy of the storage shard that has a corresponding relationship with the arbitrary data shard, and import the arbitrary data shard into the copy.

[0029] It is worth noting that, in this embodiment, by importing any data shards that fail to be imported into the corresponding replica, a technical effect of high performance and high fault tolerance can be achieved, and a technical effect of improving the flexibility of data import can be achieved.

[0030] In some embodiments, the multiple storage shards are determined based on the number of acquired data import tasks and / or the load of the column-based database management system, wherein the data import tasks include the data to be imported.

[0031] It is worth noting that, in this embodiment, by determining the storage shards based on the number and / or load of data import tasks, it is possible to prevent over-exploitation of the resources of the column-based database management system, and to maximize the load balancing of the column-based database management system, thereby improving the technical effect of rational utilization of resources.

[0032] In some embodiments, the number of the data import tasks is less than a threshold of the data import tasks of the multiple storage shards; and / or the load of the multiple storage shards is less than a preset load threshold.

[0033] In a second aspect, an embodiment of the present application provides a data import device, comprising:

[0034] A segmentation unit, configured to segment the data to be imported into a plurality of data segments in response to a data import instruction;

[0035] A construction unit, configured to construct an import relationship according to each of the data shards and each storage shard of the column-based database management system, wherein the import relationship is used to indicate a corresponding relationship between the data shards and the storage shards;

[0036] The first importing unit imports each of the data slices into a storage slice that has a corresponding relationship with the data slice.

[0037] In some embodiments, the building block comprises:

[0038] A generating subunit, configured to generate a list including address information of each of the storage slices;

[0039] A broadcast subunit, used for broadcasting the list in a plurality of import threads for performing import operations;

[0040] A construction subunit is used to construct the import relationship based on the modulo processing results between the list and each of the data slices according to each of the import threads.

[0041] In some embodiments, the construction subunit includes:

[0042] An allocation module, configured to allocate an index value to each of the data slices, and allocate a data slice including the index value to each of the import threads;

[0043] A construction module is used to construct the import relationship based on each of the import threads, the allocated data slices, and the modulo results between the lists.

[0044] In some embodiments, the construction module is used to determine the total number of storage shards in the list through each of the import threads, and construct the import relationship based on the modulus result between the index value of the allocated data shard and the total number of storage shards of each of the import threads.

[0045] In some embodiments, the first import unit is used to import the data slices allocated by each import thread into the corresponding storage slices in parallel through each import thread.

[0046] In some embodiments, the apparatus further comprises:

[0047] The second importing unit is used to re-import any data slice into a storage slice having a corresponding relationship according to a preset time interval if the import of any data slice fails.

[0048] In some embodiments, the apparatus further comprises:

[0049] A determination unit, configured to determine a storage shard corresponding to any data shard if the import of any data shard fails, and to determine a copy of the storage shard corresponding to the any data shard;

[0050] The third importing unit is used to import the arbitrary data slice into the replica.

[0051] In some embodiments, the multiple storage shards are determined based on the number of acquired data import tasks and / or the load of the column-based database management system, wherein the data import tasks include the data to be imported.

[0052] In some embodiments, the number of the data import tasks is less than a threshold of the data import tasks of the multiple storage shards; and / or the load of the multiple storage shards is less than a preset load threshold.

[0053] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0054] Memory; Memory for storing instructions executable by the processor;

[0055] The processor is configured to execute the method as described in the first aspect.

[0056] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0057] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0058] In a sixth aspect, an embodiment of the present application provides a data import system, comprising: a column-based database management system, and a data import device as described in the second aspect.

[0059] The data import method and device provided by the embodiment of the present application include: in response to a data import instruction, dividing the data to be imported into multiple data slices, and constructing an import relationship according to each data slice and each storage slice of the column-based database management system, wherein the import relationship is used to indicate the corresponding relationship between the data slice and the storage slice, and importing each data slice into the storage slice having a corresponding relationship with the data slice. By constructing the import relationship between the data slice and the storage slice, the data slice is imported into the corresponding storage slice based on the import relationship, thereby avoiding the technical problem of low import rate caused by different data to be imported competing for the same storage node when importing the data to be imported based on the random import strategy in the related technology. By performing the import operation based on the import relationship, targeted import can be achieved, thereby improving the technical effect of import rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0061] Figure 1 A flowchart of a data import method according to an embodiment of the present application;

[0062] Figure 2 A flowchart of a data import method according to another embodiment of the present application;

[0063] Figure 3 A flowchart of a data import method according to another embodiment of the present application;

[0064] Figure 4 A data import device according to an embodiment of the present application;

[0065] Figure 5A data import device according to an embodiment of the present application;

[0066] Figure 6 A data import device according to an embodiment of the present application;

[0067] Figure 7 A schematic diagram of an electronic device according to an embodiment of the present application.

[0068] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0069] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0070] The nouns involved in the embodiments of the present application are explained as follows:

[0071] Data shard: refers to a component unit that constitutes data. For example, in this embodiment, a data shard is a component unit that constitutes the data to be imported. One data to be imported may include multiple data shards.

[0072] Storage shard: refers to a component unit used to constitute a storage device. For example, in this embodiment, a storage shard is a component unit that constitutes a column-based database management system. If the column-based database management system is a storage device including multiple machines, then one storage shard can be one machine, or one machine can also include multiple storage shards.

[0073] Import thread: refers to the smallest unit that the operating system can schedule operations.

[0074] Modulo processing: It can also be called remainder operation.

[0075] With the development of Internet technology, especially the rapid progress of e-commerce technology, the amount of data has increased significantly. How to improve the efficiency of data import has become a problem that needs to be solved urgently.

[0076] For example, the application scenario of the data import method can be: the data import device can import the data in the offline data warehouse into the data import column database management system; or, the data import device can import the data in the distributed file system (Distributed File System, HDFS) into the data import column database management system.

[0077] It is worth noting that the application scenarios of the above two data import methods are only used to illustrate the scenarios in which the data import methods may be applied, and cannot be understood as a limitation on the application scenarios.

[0078] In the related art, a computing engine (such as Spark) processor can be used to import data in an offline data warehouse into a columnar database management system. In the specific implementation, a random import strategy is usually adopted, such as randomly selecting a storage shard from the columnar database management system and writing the data to be imported to the randomly selected storage shard.

[0079] However, the random import strategy may result in a technical problem of low import rate due to different data to be imported competing for the same storage node.

[0080] In order to avoid the above technical problems, the inventors of the present application, through creative work, obtained the inventive concept of the present application: constructing an import relationship between a data shard and a storage shard, so as to import the data shard into a storage shard that has an import relationship with the data shard based on the import relationship.

[0081] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0082] See also Figure 1 , Figure 1 A flowchart of a data import method according to an embodiment of the present application is shown.

[0083] like Figure 1 As shown, the method includes:

[0084] S101: In response to a data import instruction, the data to be imported is divided into a plurality of data fragments.

[0085] Exemplarily, the executor of this embodiment is a data import device, and the data import device can be a server (such as a cloud server, or a local server), a terminal device, a processor (such as a Spark processor), or a chip, etc., which is not limited in this embodiment.

[0086] The data import instruction is an instruction for instructing the data import device to perform an import operation on the data to be imported.

[0087] In this embodiment, there is no limitation on the source of the data import instruction.

[0088] In one example, the data import instruction may be triggered by a user to the import device. For example, a staff member initiates the data import instruction to the data import device.

[0089] In another example, the data import instruction may be automatically triggered by the import device based on a preset trigger condition. For example, when the trigger condition is that the amount of data reaches a preset threshold, the import device automatically triggers the data import instruction and performs subsequent data import operations.

[0090] In another example, the data import instruction may be initiated by other devices to the data import device. For example, the scheduling device initiates the data import instruction to the data import device based on the load information of the column database management system.

[0091] It should be understood that the above examples are only used to illustrate possible sources of data import instructions and should not be construed as limitations on data import instructions.

[0092] In this embodiment, the segmentation strategy for the imported data is not limited. For example, segmentation parameters may be preset, and the imported data may be segmented based on the segmentation parameters to obtain multiple data fragments.

[0093] S102: Building an import relationship according to each data shard and each storage shard of the column-based database management system.

[0094] The import relationship is used to indicate the corresponding relationship between the data shards and the storage shards.

[0095] The column-based database management system includes multiple storage shards, and one storage shard can be understood as a storage node. Alternatively, the column-based database management system can be understood as a storage cluster including multiple machines, and one storage shard is one machine. Of course, multiple shards can also be set in one machine.

[0096] S103: Import each data shard into a storage shard that has an import relationship with the data shard.

[0097] In combination with the above analysis, in this embodiment, a correspondence between data shards and storage shards is constructed. Therefore, for any data shard, the data import device can import the arbitrary data shard into a storage node that has a corresponding relationship with the arbitrary data shard, thereby realizing targeted import.

[0098] Based on the above analysis, it can be seen that in this embodiment, the feature of constructing an import relationship between data shards and storage shards is introduced, and the import relationship is used to indicate the corresponding relationship between data shards and storage shards, so as to import data shards into corresponding storage shards based on the import relationship, thereby avoiding the technical problem of low import rate caused by different to-be-imported data competing for the same storage node when importing to-be-imported data based on a random import strategy in the related technology. By performing the import operation based on the import relationship, targeted import can be achieved, thereby improving the technical effect of import rate.

[0099] See also Figure 2 , Figure 2 This is a flowchart of a data import method according to another embodiment of the present application.

[0100] like Figure 2 As shown, the method includes:

[0101] S201: In response to a data import instruction, split the data to be imported into multiple data fragments.

[0102] For example, for the description of S201, reference may be made to S101, which will not be repeated here.

[0103] S202: Generate a list including address information of each storage segment, and broadcast the list in multiple import threads for performing import operations.

[0104] Exemplarily, the data import device can determine the shards included in the column-based database management system based on the connection properties between the shards in the column-based database management system (based on the above analysis, a machine can be understood as a node, or a shard, or a node including multiple shards. In this embodiment, shards are used as an example for illustrative explanation) (such as the connection relationship between the shards), and store the Internet Protocol (IP) address of each shard as a list (hostlist).

[0105] Among them, the data importing device can be a spark processing device, and when the data importing device is a spark processing device, the importing thread can be understood as an executor in the spark processing device.

[0106] The data import device includes multiple import threads, and the import threads are used to import data slices into storage slices that have a corresponding relationship with the data slices.

[0107] S203: According to each of the import threads, a modulo processing result is performed on the list and each data slice, and an import relationship is established according to the modulo processing result.

[0108] It is worth noting that, in this embodiment, by determining the list and constructing the import relationship based on the modulo result between the list and the data shards, the data shards can be distributed relatively evenly among the storage shards, so that the load of each storage shard is relatively balanced, thereby improving the performance of data import and improving the technical effect of protecting and utilizing the resources of the column-based database management system.

[0109] In some embodiments, S203 may include the following steps:

[0110] The first step: assign an index value to each data slice, and assign a data slice including the index value to each import thread.

[0111] In some embodiments, the data importing device may assign an index value to each data shard based on the total number of data shards.

[0112] For example, the data importing device determines the total number of data shards and assigns an index value to each data shard according to the total number of data shards.

[0113] Exemplarily, the data import device determines the total number of data shards and obtains that the total number of data shards is 18, that is, there are 18 data shards in total.

[0114] The data import device assigns an index value to each data shard according to the total number of each data shard. The index value can represent the sorting value of the data shard, such as the index value 1 of the first data shard, and so on, which are not listed one by one here.

[0115] Step 2: According to each import thread, the allocated data shards and the modulo results between lists are used to build an import relationship.

[0116] It is worth noting that, in this embodiment, by assigning index values ​​and assigning data shards including index values ​​to import threads, so as to construct an import relationship based on the modulo structure between the assigned data shards and lists by the import threads, the efficiency of constructing the import relationship can be improved, and the data shards can be evenly distributed among the storage shards, thereby achieving the technical effect of improving resource utilization.

[0117] In some embodiments, the second step may include the following sub-steps:

[0118] The first sub-step: determining the total number of storage shards in the list through each import thread.

[0119] For example, any thread determines the total number of storage shards and obtains that the total number of storage shards is 10, that is, there are 10 storage shards in total.

[0120] Second sub-step: According to each import thread, the index value of the allocated data shard and the total number of storage shards are modulo the result to construct an import relationship.

[0121] For example, for the data shard with an index value of 18, the modulo processing result of the data import device is 8. Accordingly, the data shard with an index value of 18 is assigned to the 8th storage shard, that is, there is a corresponding relationship between the data shard with an index value of 18 and the 8th storage shard.

[0122] It is worth mentioning that in this embodiment, each import thread performs modulo processing to obtain a modulo result. Since the import threads are independent of each other and execute in parallel, the efficiency of determining the modulo result can be improved, thereby improving the technical effect of improving the efficiency of constructing the import relationship.

[0123] It should be noted that, based on S202 and S203, in this embodiment, the import relationship is constructed by modulo processing. In other embodiments, the import relationship can also be constructed based on a bucketing strategy.

[0124] In other embodiments, the data import device may also directly determine the modulo result, construct an import relationship based on the modulo result, and allocate data slices to each import thread so that each import thread imports the allocated data slices into the storage slices with corresponding relationships.

[0125] Based on the above two different strategies for constructing import relationships, it can be seen that in some embodiments, the data import device can construct an import relationship based on the modulo processing results of multiple import threads, or directly determine the modulo processing results and construct an import relationship based on the modulo processing results.

[0126] Relatively speaking, for the scenario of independent import threads, it can be preferred to use the data import device to directly construct the import relationship based on the modulo processing results. For the scenario of multiple import threads, it can be preferred to construct the import relationship based on the modulo processing results of each import thread.

[0127] S204: Import each data shard into a storage shard that has a corresponding relationship with the data shard.

[0128] For example, the description of S204 may refer to S103, which will not be repeated here.

[0129] In combination with the above embodiments, it can be known that the data import device can assign each data shard to a corresponding import thread. Therefore, in some embodiments, S204 may include: through each import thread, importing the data shards assigned to each import thread in parallel into the storage shards with corresponding relationships.

[0130] It is worth noting that, in this embodiment, by executing the import operation in parallel based on multiple import threads, the technical effect of improving the efficiency of importing each data shard can be achieved.

[0131] In some embodiments, among each data shard, there may be one or more data shards that fail to be imported. In the case of any data shard import failure, in one example, the data import method of this embodiment may also include: re-importing any data shard into a corresponding storage shard according to a preset time interval.

[0132] That is to say, in the event of failure to import any data shard, the data import device can re-import after a time interval, and still import based on the import relationship, which can avoid the technical problem of resource waste caused by complete failure of the previous operation of re-running (i.e., re-competing for storage shards) in the related technology, thereby improving resource utilization, improving import efficiency, and avoiding the problem of data loss caused by re-running, that is, the technical effect of improving the accuracy and reliability of data import.

[0133] In another example, the data import method of the embodiment may further include: determining a storage shard corresponding to any data shard, determining a copy of the storage shard corresponding to any data shard, and importing any data shard into the copy.

[0134] In this embodiment, by importing any data shards that fail to be imported into the corresponding replica, a technical effect of high performance and high fault tolerance can be achieved, and a technical effect of improving the flexibility of data import can be achieved.

[0135] See also Figure 3 , Figure 3 This is a flowchart of a data import method according to another embodiment of the present application.

[0136] like Figure 3 As shown, the method includes:

[0137] S301: The scheduling processor receives the data import tasks sent by the user and allocates a task queue to each user.

[0138] The number of users may be one or more. Similarly, the number of data import tasks may be one or more, and the number of data to be imported in the data import task may be one or more.

[0139] In some embodiments, the scheduling processor may set a task threshold for the data import tasks in the task queue. For example, the scheduling processor sets the task threshold to 5, that is, the task queue may include at most 5 data import tasks.

[0140] S302: The scheduling processor determines a threshold value of the number of data import tasks for each storage slice of the column-based database management system; and / or determines a load of the column-based database management system.

[0141] The threshold value of the number of data import tasks for each storage shard may be understood as the maximum number of data import tasks that can be stored in each storage shard, that is, the maximum capacity of data that can be stored in each storage shard.

[0142] S303: If the scheduling processor determines the load of the column-based database management system based on the number threshold of data import tasks for each storage shard; and / or determines that the column-based database management system meets the import requirements of the data import task, then a data import instruction is initiated to the Spark processor.

[0143] Exemplarily, the scheduling processor can determine the load of the column-based database management system based on the processor CPU resources and memory resources of the column-based database management system, and when the load reaches a preset load threshold (determining that the column-based database management system does not meet the import requirements of the data import task), the scheduling task is not executed, that is, the data import instruction is not initiated to the Spark processor; on the contrary, if the load does not reach the load threshold (determining that the column-based database management system meets the import requirements of the data import task), a data import instruction is initiated to the Spark processor, and the scheduling processor selects a relatively low load from each storage shard (the scheduling processor can set the threshold based on demand, historical records, and experiments, and when the load is less than the threshold, it is determined that the load is relatively low), and carries the selected storage shard in the import instruction, so that the Spark processor imports the data to be imported into the storage shard selected by the scheduling processor.

[0144] In some embodiments, if the scheduling processor determines that the column-based database management system does not meet the import requirements of the data import task, the scheduling operation may be performed again after a preset time period.

[0145] If there are multiple users, the scheduling processor can perform corresponding scheduling operations according to the preset priority.

[0146] For example, when the resources of a column-based database management system are limited, the scheduling processor can give priority to executing data import tasks initiated by users with higher priorities.

[0147] It is worth noting that, in this embodiment, by dynamically deciding whether to execute the data import operation by the scheduling processor, it is possible to prevent excessive exploitation of the resources of the column-based database management system, and to maximize the load balancing of the column-based database management system, thereby improving the technical effect of rational utilization of resources.

[0148] S304: The Spark processor divides the data to be imported into multiple data slices, and builds an import relationship according to each data slice and each storage slice of the column-based database management system.

[0149] The import relationship is used to indicate the corresponding relationship between the data shards and the storage shards.

[0150] S305: The Spark processor imports each data shard into a storage shard that has a corresponding relationship with the data shard.

[0151] For example, regarding the principles of S304 to S305, reference may be made to the principles described in S101 to S103, or reference may be made to the principles described in S201 to S204, which will not be repeated here.

[0152] In some embodiments, after the data to be imported is imported into a column-based database management system, import operation information can be generated, including time information (including scheduling time information and waiting scheduling time information generated by a scheduling device, and may also include import time information generated by a spark processor, etc.) and data size information, etc., and a corresponding load curve chart can also be generated to improve the performance of subsequent data import and avoid waste of resources.

[0153] In some embodiments, the scheduling processor and the Spark processor may be deployed on the same server or on different servers, which is not limited in this embodiment.

[0154] See also Figure 4 , Figure 4 This is a data import device according to an embodiment of the present application.

[0155] like Figure 4 As shown, the data import device 400 includes:

[0156] The segmentation unit 401 is used to segment the data to be imported into multiple data slices in response to the data import instruction.

[0157] The construction unit 402 is used to construct an import relationship according to each of the data shards and each storage shard of the column-based database management system, wherein the import relationship is used to indicate the corresponding relationship between the data shards and the storage shards.

[0158] The first importing unit 403 imports each of the data slices into a storage slice that has a corresponding relationship with the data slice.

[0159] See also Figure 5 , Figure 5 This is a data import device according to an embodiment of the present application.

[0160] like Figure 5 As shown, the data import device 500 includes:

[0161] The segmentation unit 501 is used to segment the data to be imported into multiple data slices in response to the data import instruction.

[0162] In some embodiments, the multiple storage shards are determined based on the number of acquired data import tasks and / or the load of the column-based database management system, wherein the data import tasks include the data to be imported.

[0163] In some embodiments, the number of the data import tasks is less than a threshold of the data import tasks of the multiple storage shards; and / or the load of the multiple storage shards is less than a preset load threshold.

[0164] The construction unit 502 is used to construct an import relationship according to each of the data shards and each storage shard of the column-based database management system, wherein the import relationship is used to indicate the corresponding relationship between the data shards and the storage shards.

[0165] Combination Figure 5 It can be seen that, in some embodiments, the construction unit 502 includes:

[0166] The generating subunit 5021 is used to generate a list including the address information of each of the storage slices.

[0167] The broadcast subunit 5022 is used to broadcast the list in multiple import threads used to perform import operations.

[0168] The construction subunit 5023 is used to construct the import relationship according to each of the import threads and the modulo processing results between the list and each of the data slices.

[0169] In some embodiments, the construction subunit 5023 includes:

[0170] The allocation module is used to allocate an index value to each of the data slices, and allocate a data slice including the index value to each of the import threads.

[0171] A construction module is used to construct the import relationship based on each of the import threads, the allocated data slices, and the modulo results between the lists.

[0172] In some embodiments, the construction module is used to determine the total number of storage shards in the list through each of the import threads, and construct the import relationship based on the modulus result between the index value of the allocated data shard and the total number of storage shards of each of the import threads.

[0173] The first importing unit 503 imports each of the data slices into a storage slice that has a corresponding relationship with the data slice.

[0174] In some embodiments, the first import unit 503 is used to import the data slices allocated by each import thread into the corresponding storage slices in parallel through each import thread.

[0175] The second importing unit 504 is configured to re-import any data slice into a storage slice having a corresponding relationship according to a preset time interval if the import of any data slice fails.

[0176] See also Figure 6 , Figure 6 This is a data import device according to an embodiment of the present application.

[0177] like Figure 6 As shown, the data import device 600 includes:

[0178] The segmentation unit 601 is used to segment the data to be imported into multiple data slices in response to the data import instruction.

[0179] The construction unit 602 is used to construct an import relationship according to each of the data shards and each storage shard of the column-based database management system, wherein the import relationship is used to indicate the corresponding relationship between the data shards and the storage shards.

[0180] The first importing unit 603 imports each of the data slices into a storage slice that has a corresponding relationship with the data slice.

[0181] The determination unit 604 is configured to determine a storage shard corresponding to the arbitrary data shard if the import of the arbitrary data shard fails, and determine a copy of the storage shard corresponding to the arbitrary data shard.

[0182] The third importing unit 605 is used to import the arbitrary data slice into the replica.

[0183] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0184] According to an embodiment of the present application, the present application also provides a computer program product, the program product includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0185] like Figure 7 , is a block diagram of an electronic device according to a data import method of an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0186] like Figure 7 As shown, the electronic device includes: one or more processors 701, a memory 702, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 701 is taken as an example.

[0187] The memory 702 is a non-transient computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the data import method provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the data import method provided in the present application.

[0188] The memory 702 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the data import method in the embodiment of the present application. The processor 701 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 702, that is, the data import method in the above method embodiment is implemented.

[0189] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device of the data import method, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the electronic device of the data import method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0190] The electronic device of the data import method may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0191] The input device 703 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device of the data import method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 704 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0192] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0193] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0194] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0195] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0196] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0197] According to another aspect of the embodiment of the present application, the embodiment of the present application also provides a data import system, including: a column database management system, and a data import device as described in any of the above embodiments.

[0198] In some embodiments, the data importing device may be a Spark processor.

[0199] In some embodiments, the system may further include:

[0200] A scheduling processor is used to: the scheduling processor receives data import tasks sent by users and allocates a task queue to each user; it is also used to: determine the number threshold of data import tasks for each storage slice of the column-based database management system; and / or determine the load of the column-based database management system; it is also used to: if the scheduling processor determines that the column-based database management system meets the import requirements of the data import task based on the number threshold of data import tasks for each storage slice and / or determines the load of the column-based database management system, then initiate a data import instruction to the Spark processor.

[0201] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0202] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data import method, comprising: In response to a data import instruction, the data to be imported is divided into a plurality of data slices, and an import relationship is constructed according to each of the data slices and each storage slice of the column-based database management system, wherein the import relationship is used to indicate a corresponding relationship between the data slice and the storage slice; Through each import thread, each data slice allocated by the import thread is imported into a storage slice with a corresponding relationship in parallel; the import relationship is established according to each data slice and each storage slice of the column-based database management system, including: Generate a list including address information of each storage shard, and broadcast the list in a plurality of import threads for performing import operations; Assigning an index value to each of the data slices, and assigning a data slice including the index value to each of the import threads; Through each of the import threads, the total number of the storage shards in the list is determined, and according to each of the import threads, the index value of the allocated data shard and the modulo result of the total number of the storage shards are taken to construct the import relationship.

2. According to the method of claim 1, if any data shard fails to be imported, the method further comprises: According to a preset time interval, the arbitrary data slice is re-imported into a storage slice having a corresponding relationship.

3. According to the method of claim 1, if any data shard fails to be imported, the method further comprises: Determine a storage shard that has a corresponding relationship with the arbitrary data shard, determine a copy of the storage shard that has a corresponding relationship with the arbitrary data shard, and import the arbitrary data shard into the copy.

4. The method according to claim 1, wherein: Each of the storage shards is determined based on the number of acquired data import tasks and / or the load of the column-based database management system, wherein the data import tasks include the data to be imported.

5. The method according to claim 4, wherein: The number of the data import tasks is less than a threshold of the data import tasks of the plurality of the storage slices; and / or the load of the plurality of the storage slices is less than a preset load threshold.

6. A data import device, comprising: A segmentation unit, configured to segment the data to be imported into a plurality of data segments in response to a data import instruction; A construction unit, configured to construct an import relationship according to each of the data shards and each of the storage shards of the column-based database management system, wherein the import relationship is used to indicate a corresponding relationship between the data shards and the storage shards; An import unit, used to import, through each import thread, the data slices allocated by each import thread into the storage slices having a corresponding relationship in parallel; The construction unit is specifically used to generate a list including address information of each storage shard, and broadcast the list in multiple import threads used to perform import operations; assign an index value to each data shard, and assign a data shard including the index value to each import thread; determine the total number of each storage shard in the list through each import thread, and according to each import thread, take the modulus result between the index value of the assigned data shard and the total number of each storage shard, to construct the import relationship.

7. An electronic device comprising: Memory, processor; Memory; a memory for storing instructions executable by the processor; The processor is configured to execute the method as claimed in any one of claims 1 to 5.

8. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are used to implement the method according to any one of claims 1 to 5 when executed by a processor.

9. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

10. A data import system, comprising: A column-based database management system, and a data import device as claimed in claim 6.

Images