Data import method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention discloses a data importing method and device, electronic equipment and a computer readable medium. According to one specific embodiment, the method comprises the steps that current import data transaction information is obtained, and the current import data transaction information comprises a first to-be-imported data set; determining a virtual fragment group as a first virtual fragment group according to the virtual fragment corresponding to each piece of first to-be-imported data; for each first virtual fragment, screening out a target physical fragment from the at least one physical fragment corresponding to the first virtual fragment to obtain at least one target physical fragment; performing splitting processing on each target physical fragment to generate a target physical sub-fragment group, and obtaining at least one target physical sub-fragment group; and according to the at least one target physical sub-fragment group, generating a data import result for the first virtual fragment. The implementation mode is related to big data, it can be ensured that data distribution on all nodes and fragments is relatively balanced, and the data query efficiency is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technologies, and more particularly, to data import methods, devices, electronic devices, and computer-readable media. Background Art

[0002] Data import is a technology used to import data into each physical shard in a distributed scenario to support subsequent data analysis. Currently, when performing data import, the commonly adopted method is to directly import each piece of data to be imported into the corresponding physical shard according to sharding rules (such as hash sharding, random sharding), and return the data import result.

[0003] However, the inventors found that when using the above method for data import, the following technical problems often exist:

[0004] If the data import result is directly returned after importing the data into the corresponding physical shard according to the sharding rules, it is likely to cause uneven data distribution among the physical shards. Even if the random sharding method can be used to ensure that the data distribution in each shard is roughly balanced, when performing data queries, a large amount of computing power is usually required to query all shards, resulting in waste of computing resources and long query time.

[0005] The above information disclosed in this background art section is only used to enhance the understanding of the background of the inventive concept, and thus, it may include information that does not constitute the prior art known to those of ordinary skill in the art in this country. Summary of the Invention

[0006] The content part of the present disclosure is used to briefly introduce the concepts, which will be described in detail in the following detailed implementation part. The content part of the present disclosure is not intended to identify the key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] Some embodiments of the present disclosure propose data import methods, devices, electronic devices, and computer-readable media to solve the technical problems mentioned in the above background art section.

[0008] In a first aspect, some embodiments of the present disclosure provide a data import method, which includes: obtaining current import data transaction information, where the current import data transaction information includes a first dataset to be imported; determining a virtual shard group as a first virtual shard group according to the virtual shards corresponding to each first piece of data to be imported in the first dataset to be imported, where there is at least one initial physical shard corresponding to each virtual shard; for each first virtual shard in the first virtual shard group, performing the following data import steps: screening out target physical shards from the at least one physical shard corresponding to the first virtual shard to obtain at least one target physical shard, where the target physical shard is a physical shard whose corresponding shard data quantity meets a preset quantity condition, and the at least one physical shard is obtained by importing a first subset of data to be imported into the at least one initial physical shard, and the first subset of data to be imported corresponds to the first virtual shard; performing a splitting process on each of the at least one target physical shards to generate a target physical sub-shard group, obtaining at least one target physical sub-shard group; generating a data import result for the first virtual shard according to the at least one target physical sub-shard group.

[0009] Optionally, the method further includes: in response to receiving a data query request, determining a target query virtual shard corresponding to the data query request; in response to determining that there is a target query physical shard corresponding to the target query virtual shard, determining the difference set between at least one query physical shard and at least one target query physical shard as a non-target query physical shard group, where the target query physical shard is the target physical shard corresponding to the target query virtual shard, and the query physical shard is the physical shard corresponding to the target query virtual shard; performing a data query operation for the data query request according to the non-target query physical shard group to obtain a data query result.

[0010] Optionally, the performing a data query operation for the data query request according to the non-target query physical shard group to obtain a data query result includes: in response to determining that the splitting process of the at least one target query physical shard is not completed, selecting the target query physical shards that have been split completely from the at least one target query physical shards as the split physical shards to obtain a split physical shard group; performing a data query operation for the data query request according to the non-target query physical shard group and the split physical shard group to obtain a data query result.

[0011] Optionally, performing a data query operation on the data query request according to the non-target query physical shard group and the split physical shard group described above to obtain a data query result, including: selecting, from the at least one target query physical shards, the target query physical shards that have not completed splitting as the physical shards with incomplete splitting to obtain a physical shard group with incomplete splitting; performing a data query operation on the data query request according to the non-target query physical shard group, the split physical shard group, and the physical shard group with incomplete splitting to obtain a data query result.

[0012] Optionally, splitting each of the at least one target physical shards to generate a target physical sub-shard group, including: for each target physical shard, performing the following splitting steps: generating a shard split number according to the target physical shard; creating an initial target physical sub-shard group according to the shard split number; and importing the shard data corresponding to the target physical shard into the initial target physical sub-shard group to obtain a target physical sub-shard group.

[0013] Optionally, after importing the shard data corresponding to the target physical shard into the initial target physical sub-shard group to obtain a target physical sub-shard group, the method further includes: binding the shard information group corresponding to the target physical sub-shard group with the shard information corresponding to the first virtual shard, and deleting the target physical shard.

[0014] Optionally, before generating the data import result for the first virtual shard according to the at least one target physical sub-shard group, the method further includes: in response to receiving the next import data transaction information, determining the virtual shard group corresponding to the next import data transaction information as the second virtual shard group, where the next import data transaction information represents an import data transaction executed after the current import data transaction; in response to determining that there is a target initial physical shard in the second initial physical shard group, selecting the target initial physical shard from the second initial physical shard group to obtain at least one target initial physical shard, where each second initial physical shard is an initial physical shard corresponding to the second virtual shard group and for importing the second subset of data to be imported, and the second subset of data to be imported is a subset of the second data set to be imported corresponding to the next import data transaction information, and the target initial physical shard is a target physical shard corresponding to the first virtual shard group and not yet completed splitting; determining the difference set between the second initial physical shard group and the at least one target initial physical shard as the non-target initial physical shard group; for each target initial physical shard, importing the second subset of data to be imported corresponding to the target initial physical shard into the target initial physical shard and the corresponding initial target physical sub-shard group; for each non-target initial physical shard, importing the second subset of data to be imported corresponding to the non-target initial physical shard into the non-target initial physical shard.

[0015] Optionally, after screening out the target physical shards from the at least one physical shard corresponding to the first virtual shard to obtain at least one target physical shard, the method further includes: in response to detecting that there is no target physical shard in the at least one physical shard, determining the at least one physical shard and the corresponding at least one shard data as the data import result.

[0016] Optionally, generating the data import result for the first virtual shard according to the at least one target physical sub-shard group includes: in response to determining that there is a key unsplit physical shard in the at least one physical shard, selecting the key unsplit physical shard from the at least one physical shard to obtain at least one key unsplit physical shard; generating the data import result for the first virtual shard according to the at least one key unsplit physical shard and the at least one target physical sub-shard group.

[0017] Second aspect, some embodiments of the present disclosure provide a data import device, the device includes: an acquisition unit configured to acquire current import data transaction information, where the current import data transaction information includes a first dataset to be imported; a determination unit configured to determine a virtual shard group as a first virtual shard group according to virtual shards corresponding to each first piece of data to be imported in the first dataset to be imported, where there is at least one initial physical shard corresponding to each virtual shard; an execution unit configured to, for each first virtual shard in the first virtual shard group, perform the following data import steps: screen out target physical shards from at least one physical shard corresponding to the first virtual shard to obtain at least one target physical shard, where the target physical shard is a physical shard whose corresponding shard data quantity meets a preset quantity condition, and the at least one physical shard is obtained by importing a first subset of data to be imported into the at least one initial physical shard, and the first subset of data to be imported corresponds to the first virtual shard; perform splitting processing on each target physical shard in the at least one target physical shard to generate a target physical sub-shard group, obtaining at least one target physical sub-shard group; generate a data import result for the first virtual shard according to the at least one target physical sub-shard group.

[0018] Optionally, the data import device is further configured to: in response to receiving a data query request, determine a target query virtual shard corresponding to the data query request; in response to determining that there is a target query physical shard corresponding to the target query virtual shard, determine the difference set between at least one query physical shard and at least one target query physical shard as a non-target query physical shard group, where the target query physical shard is a target physical shard corresponding to the target query virtual shard, and the query physical shard is a physical shard corresponding to the target query virtual shard; perform a data query operation for the data query request according to the non-target query physical shard group, obtaining a data query result.

[0019] Optionally, the data import device is further configured to: in response to determining that the splitting processing of the at least one target query physical shard is not completed, select the target query physical shards that have been split and completed from the at least one target query physical shard as split physical shards, obtaining a split physical shard group; perform a data query operation for the data query request according to the non-target query physical shard group and the split physical shard group, obtaining a data query result.

[0020] Optionally, the above data import device is further configured to: select, from the above at least one target query physical shard, the target query physical shards that have not completed splitting as the physical shards with incomplete splitting, to obtain a group of physical shards with incomplete splitting; and perform a data query operation for the above data query request according to the above non-target query physical shard group, the above split physical shard group, and the above group of physical shards with incomplete splitting, to obtain a data query result.

[0021] Optionally, the execution unit is further configured to: for each target physical shard, perform the following splitting processing steps: generate a shard splitting number according to the above target physical shard; create an initial target physical sub-shard group according to the above shard splitting number; and import the shard data corresponding to the above target physical shard into the above initial target physical sub-shard group to obtain a target physical sub-shard group.

[0022] Optionally, the execution unit is further configured to: bind the shard information group corresponding to the above target physical sub-shard group with the shard information of the above first virtual shard, and delete the above target physical shard.

[0023] Optionally, the execution unit is further configured to: in response to receiving the next import data transaction information, determine the virtual shard group corresponding to the above next import data transaction information as the second virtual shard group, where the above next import data transaction information represents an import data transaction executed after the current import data transaction; in response to determining that there is a target initial physical shard in the second initial physical shard group, select the target initial physical shard from the above second initial physical shard group to obtain at least one target initial physical shard, where each second initial physical shard is the initial physical shard corresponding to the above second virtual shard group and for importing a second subset of data to be imported, the above second subset of data to be imported is a subset of the second dataset to be imported corresponding to the above next import data transaction information, and the target initial physical shard is the target physical shard corresponding to the above first virtual shard group and with incomplete splitting; determine the difference set between the above second initial physical shard group and the above at least one target initial physical shard as the non-target initial physical shard group; for each target initial physical shard, import the second subset of data to be imported corresponding to the above target initial physical shard into the above target initial physical shard and the corresponding initial target physical sub-shard group; and for each non-target initial physical shard, import the second subset of data to be imported corresponding to the above non-target initial physical shard into the above non-target initial physical shard.

[0024] Optionally, the execution unit is further configured to: in response to detecting that there is no target physical shard in the above at least one physical shard, determine the above at least one physical shard and the corresponding at least one shard data as the data import result.

[0025] Optionally, the execution unit is further configured to: in response to determining that there is a critical unsplit physical shard in the at least one physical shard, select the critical unsplit physical shard from the at least one physical shard to obtain at least one critical unsplit physical shard; and generate an import result for the first virtual shard according to the at least one critical unsplit physical shard and the at least one target physical sub-shard group.

[0026] In a third aspect, some embodiments of the present disclosure provide an electronic device, including: one or more processors; a storage device storing one or more programs thereon, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0027] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0028] In a fifth aspect, some embodiments of the present disclosure provide a computer program product including a computer program, and the computer program, when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0029] The above embodiments of the present disclosure have the following beneficial effects: Through the data import method of some embodiments of the present disclosure, the data distribution among physical shards can be made more balanced, and the waste of computing resources and the time consumption can be reduced during data query. Specifically, the reasons for the uneven data distribution among physical shards or the waste of computing resources and long query time during data query are as follows: If the data import result is directly returned after importing the data into the corresponding physical shards according to the sharding rules, it is easy to cause uneven data distribution among physical shards. Even if the random sharding method can be used to ensure that the data distribution of each shard is roughly balanced, when data query is performed, a large amount of computing power is usually required to query all shards, resulting in waste of computing resources and long query time. Based on this, in the data import method of some embodiments of the present disclosure, first, the current data import transaction information is obtained. Among them, the above current data import transaction information includes the first dataset to be imported. Thus, the data to be imported for subsequent sharding storage can be obtained. Then, according to the virtual shards corresponding to each first data to be imported in the above first dataset to be imported, a virtual shard group is determined as the first virtual shard group. Among them, there is at least one initial physical shard corresponding to each virtual shard. Thus, according to the sharding rules, the virtual shard corresponding to each data to be imported can be determined, which is convenient for subsequent importing the data to be imported into the physical shard corresponding to the virtual shard. Finally, for each first virtual shard in the above first virtual shard group, the following data import steps are executed: Filter out the target physical shards from the at least one physical shards corresponding to the above first virtual shard to obtain at least one target physical shard; where the target physical shard is a physical shard whose corresponding shard data quantity meets the preset quantity condition, and the above at least one physical shard is obtained by importing the first subset of data to be imported into the above at least one initial physical shard, and the above first subset of data to be imported corresponds to the above first virtual shard; Split each of the above at least one target physical shards to generate a target physical sub-shard group to obtain at least one target physical sub-shard group; According to the above at least one target physical sub-shard group, a data import result for the above first virtual shard is generated. Thus, for each virtual shard with corresponding data to be imported, after importing the data into the physical shard corresponding to the virtual shard, the target physical shards that meet the preset quantity condition can be split and disassembled, so that the large shards are split into shards with appropriate data volume. Therefore, in the data import method of some embodiments of the present disclosure, by splitting the original large shards for storing data into shards with appropriate data volume, the data distribution among physical shards can be made more balanced, and multiple storage disks on a single machine or multiple machine nodes can be effectively utilized. Also, because, through the sharding rules, the virtual shard corresponding to the data can be determined first, and then the data can be imported into the physical shard corresponding to the virtual shard.Thus, when performing data queries, it is only necessary to query data for each physical shard corresponding to the virtual shard, thereby reducing waste of computing resources and shortening query time. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the elements and elements are not necessarily drawn to scale.

[0031] Figure 1 is a schematic diagram of an application scenario of a data import method according to some embodiments of the present disclosure;

[0032] Figure 2 is a flowchart of some embodiments of the data import method according to the present disclosure;

[0033] Figure 3 is a schematic diagram of an application scenario of the data import step during shard splitting of the data import method according to the present disclosure;

[0034] Figure 4 is a flowchart of some other embodiments of the data import method according to the present disclosure;

[0035] Figure 5 is a schematic structural diagram of some embodiments of the data import apparatus according to the present disclosure;

[0036] Figure 6 is a schematic structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0037] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0038] In addition, it should be noted that for the sake of description, only parts related to the relevant invention are shown in the drawings. Without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other.

[0039] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules, or units, and are not used to limit the order of the functions performed by these devices, modules, or units or their interdependent relationships.

[0040] It should be noted that the modifications of "one" and "multiple" mentioned in this disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0041] The names of the messages or information exchanged between multiple devices in the embodiments of this disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0042] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0043] Figure 1 It is a schematic diagram of an application scenario of a data import method according to some embodiments of the present disclosure.

[0044] In Figure 1In the application scenario, first, the electronic device 101 obtains the current imported data transaction information 102. Among them, the current imported data transaction information 102 may include the first dataset to be imported 1021. For example, the first dataset to be imported 1021 may include {the first data to be imported 1, the first data to be imported 2, the first data to be imported 3}. Then, according to the virtual shards corresponding to each first data to be imported in the first dataset to be imported 1021, a virtual shard group is determined as the first virtual shard group 103. Among them, there is at least one initial physical shard corresponding to each virtual shard. For example, the first virtual shard group 103 may include the first virtual shard 1, the first virtual shard 2, and the first virtual shard 3. The first virtual shard 1 may correspond to the initial physical shard 11. The first virtual shard 2 may correspond to the initial physical shard 21. The initial physical shard 21 may include the imported data 1 and the imported data 2. The first virtual shard 3 may correspond to the initial physical shards 31 and 32. Finally, for each first virtual shard in the first virtual shard group 103, the following data import steps are executed: filtering out the target physical shards from the at least one physical shard 104 corresponding to the first virtual shard to obtain at least one target physical shard 105, where the target physical shard is a physical shard whose corresponding shard data number meets the preset number condition, and the at least one physical shard is obtained by importing the first subset of data to be imported into the at least one initial physical shard, and the first subset of data to be imported corresponds to the first virtual shard; performing a splitting process on each of the at least one target physical shards 105 to generate a target physical sub-shard group, obtaining at least one target physical sub-shard group 106; generating a data import result for the first virtual shard according to the at least one target physical sub-shard group. For example, the at least one physical shard 104 may include the physical shard 1, the physical shard 2, and the physical shard 3. Among them, the physical shard 1 may be the initial physical shard 21 after importing the first data to be imported 1. The physical shard 2 may be the initial physical shard 32 after importing the first data to be imported 2. The physical shard 3 may be the initial physical shard 11 after importing the first data to be imported 3. The at least one target physical shard 105 may include the target physical shard 1. The target physical shard 1 corresponds to the physical shard 1. The at least one target physical sub-shard group 106 may include the target physical sub-shard group 1. The target physical sub-shard group 1 may include the target physical sub-shard 1 and the target physical sub-shard 2. The target physical sub-shard 1 may include the imported data 2 and the first data to be imported 1. The target physical sub-shard 2 may include the imported data 1.

[0045] It should be noted that the above electronic device 101 can be hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is embodied as software, it can be installed in the above-listed hardware devices. It can be implemented as, for example, multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made here.

[0046] It should be understood that Figure 1 the number of terminal devices in

[0047] Continuing to refer to Figure 2 , a flow 200 of some embodiments of the data import method according to the present disclosure is shown. The data import method includes the following steps:

[0048] Step 201, obtain current import data transaction information.

[0049] In some embodiments, the execution subject of the data import method (such as Figure 1 the electronic device 101 shown) can obtain the current import data transaction information through a wired connection method or a wireless connection method. Among them, the above current import data transaction information can be information about the data import transaction for the target data table at the current moment. The above data import transaction can be a database transaction for importing data. The above target data table can be a distributed table for OLAP (Online Analytical Processing) services. The above target data table can be associated with a virtual shard set. The virtual shards in the virtual shard set can be sub-tables obtained by logically partitioning the above target data table. Each virtual shard does not occupy physical address space. Each virtual shard has an associated virtual shard identifier. The virtual shard identifier is the unique identifier of the virtual shard. The above current import data transaction information can include a first dataset to be imported. The first data to be imported in the first dataset to be imported can be data to be imported into the target data table. The first data to be imported in the first dataset to be imported can include a primary key value. The primary key value can be the primary key value corresponding to the primary key field.

[0050] It should be noted that the above wireless connection method can include, but is not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other currently known or future-developed wireless connection methods.

[0051] Step 202: Determine a virtual shard group as the first virtual shard group according to the virtual shards corresponding to each piece of first data to be imported in the first dataset to be imported.

[0052] In some embodiments, the above execution entity may determine a virtual shard group as the first virtual shard group according to the virtual shards corresponding to each piece of first data to be imported in the first dataset to be imported. Among them, there is at least one initial physical shard corresponding to each virtual shard. The initial physical shard may be a sub-table obtained by physically partitioning the above target data table and not importing the first data to be imported. Each initial physical shard occupies a physical address space.

[0053] As an example, first, for each piece of first data to be imported in the first dataset to be imported, the above execution entity may select a virtual shard from the above virtual shard set as the virtual shard corresponding to the first data to be imported through a preset sharding method. The sharding method may include, but is not limited to, at least one of the following: hash sharding, range-based sharding, and random sharding. Then, deduplicate the obtained virtual shards corresponding to each piece of first data to be imported to obtain a virtual shard group as the first virtual shard group.

[0054] Step 203: For each first virtual shard in the first virtual shard group, perform the following data import steps:

[0055] Step 2031: Select target physical shards from at least one physical shard corresponding to the first virtual shard to obtain at least one target physical shard.

[0056] In some embodiments, the above execution entity may select target physical shards from at least one physical shard corresponding to the first virtual shard to obtain at least one target physical shard. Among them, the target physical shard may be a physical shard whose corresponding shard data quantity meets a preset quantity condition. The shard data may be the data in the corresponding shard. The shard data may include original data and imported data. The original data may be the data stored in the corresponding physical shard before the first subset of data to be imported is not imported. The imported data may be the data in the first subset of data to be imported imported into the physical shard. The preset quantity condition may be that the quantity of shard data in the physical shard is greater than a preset quantity threshold. The preset quantity threshold may be an upper limit value of the data quantity set in advance. The at least one physical shard may be obtained by importing the first subset of data to be imported into the at least one initial physical shard. The first subset of data to be imported corresponds to the first virtual shard. The first subset of data to be imported may be a subset of the first dataset to be imported.

[0057] As an example, first, the above-mentioned execution entity can import the first data subset to be imported corresponding to the above-mentioned first virtual shard into at least one initial physical shard corresponding to the above-mentioned first virtual shard through a preset data import method, to obtain at least one physical shard. The above-mentioned preset data import method may include, but is not limited to, at least one of the following: a batch import method and a single import method. The above-mentioned batch import method may be to import data in batches and in different times into different initial physical shards. The above-mentioned single import method may be to import data one by one into any initial physical shard. Then, in response to determining that there is a target physical shard among the obtained at least one physical shards, filter out the target physical shard from the at least one physical shards corresponding to the above-mentioned first virtual shard, to obtain at least one target physical shard.

[0058] Optionally, after filtering out the target physical shard from the at least one physical shards corresponding to the above-mentioned first virtual shard to obtain at least one target physical shard, the above-mentioned execution entity may, in response to detecting that there is no target physical shard among the at least one physical shards, determine the at least one physical shard and the corresponding at least one shard data as the data import result. Wherein, the shard data in the at least one shard data may correspond one by one to the physical shards in the at least one physical shards.

[0059] As an example, first, the above-mentioned execution entity may, in response to detecting that there is no target physical shard among the at least one physical shards, for each physical shard in the at least one physical shards, determine the physical shard and the corresponding shard data as the target shard import result. Then, determine the obtained at least one target shard import result as the data import result.

[0060] Step 2032, perform a splitting process on each target physical shard in the at least one target physical shards, to generate a target physical sub-shard group, to obtain at least one target physical sub-shard group.

[0061] In some embodiments, the above-mentioned execution entity may perform a splitting process on each target physical shard in the at least one target physical shards, to generate a target physical sub-shard group, to obtain at least one target physical sub-shard group. Wherein, the target physical sub-shards in the target physical sub-shard group may be sub-shards corresponding to the target physical shards. The sub-shards may be sub-tables corresponding to sub-tables.

[0062] As an example, the above-mentioned execution entity may, for each target physical shard in the at least one target physical shards, perform the following steps:

[0063] The first step is to use the above-mentioned target physical shard as the physical shard to be split, and perform the following shard splitting process steps:

[0064] The first sub-step is to create a preset number of free physical sub-shards through database sharding operations as a group of free physical sub-shards. Among them, the above preset number can be a pre-set number greater than 1. The above free physical sub-shards can be sub-shards corresponding to the above target physical shard that have not imported any data.

[0065] The second sub-step is to import the shard data corresponding to the physical shard to be split into the above group of free physical sub-shards to obtain a group of non-free physical sub-shards. Among them, the non-free physical sub-shards in the above group of non-free physical sub-shards can be free physical sub-shards after importing data. The shard data corresponding to the physical shard to be split can be imported into each of the free physical sub-shards included in the above group of free physical sub-shards through the above preset data import method to obtain a group of non-free physical sub-shards.

[0066] The third sub-step is, for each non-free physical sub-shard, in response to determining that the non-free physical sub-shard meets the preset non-free data condition, to determine the non-free physical sub-shard as a target physical sub-shard. Among them, the above preset non-free data condition can be that the number of shard data corresponding to the non-free physical sub-shard does not exceed the above preset number threshold.

[0067] Optionally, the above execution entity can also, in response to determining that the non-free physical sub-shard does not meet the preset non-free data condition, use the non-free physical sub-shard as the physical shard to be split and execute the above shard splitting processing step again.

[0068] The second step is to determine each of the obtained target physical sub-shards as a group of target physical sub-shards.

[0069] In some optional implementation manners of some embodiments, the above execution entity can perform splitting processing on each target physical shard among the above at least one target physical shards through the following steps to generate a group of target physical sub-shards:

[0070] The first step is, for each target physical shard, to execute the following splitting processing steps:

[0071] The first sub-step is to generate a shard splitting number according to the above target physical shard. Among them, the above shard splitting number can be the number of sub-shards obtained after splitting the target physical shard.

[0072] As an example, first, the above execution entity can determine the ratio of the number of shard data corresponding to the above target physical shard to the above preset number threshold as a number ratio. Then, the value obtained by rounding up the above number ratio is determined as the shard splitting number.

[0073] Optionally, the above-mentioned execution entity may also determine the ratio of the shard data capacity corresponding to the above-mentioned target physical shard to a preset capacity threshold as the capacity ratio. The above-mentioned preset capacity threshold may be an upper limit value of the pre-set data capacity. Then, the value obtained by rounding up the above-mentioned capacity ratio is determined as the shard split number.

[0074] The second sub-step is to create an initial target physical sub-shard group according to the above-mentioned shard split number. Among them, the initial target physical sub-shards in the above-mentioned initial target physical sub-shard group may be sub-shards corresponding to the above-mentioned target physical shard and without any imported data. Each initial target physical sub-shard corresponds to a physical partition shard identifier. The above-mentioned physical partition shard identifier may be the unique identifier of the physical partition shard. The above-mentioned physical partition shard may be a shard obtained by physically partitioning a data table. It should be noted that the above-mentioned physical partition shard identifier is only used to identify the physical partition shard itself and is not used to distinguish whether the above-mentioned physical partition shard has imported data.

[0075] As an example, first, the above-mentioned execution entity may create the number of initial target physical sub-shards equal to the shard split number through database sharding operations. Then, each of the created initial target physical sub-shards is determined as the initial target physical sub-shard group.

[0076] The third sub-step is to import the shard data corresponding to the above-mentioned target physical shard into the above-mentioned initial target physical sub-shard group to obtain a target physical sub-shard group.

[0077] As an example, the above-mentioned execution entity may import the shard data corresponding to the above-mentioned target physical shard into each of the initial target physical sub-shards included in the above-mentioned initial target physical sub-shard group through the above-mentioned preset data import method to obtain a target physical sub-shard group.

[0078] Optionally, after importing the shard data corresponding to the above-mentioned target physical shard into the above-mentioned initial target physical sub-shard group to obtain a target physical sub-shard group, the above-mentioned execution entity may also bind the shard information group corresponding to the above-mentioned target physical sub-shard group with the shard information corresponding to the above-mentioned first virtual shard, and delete the above-mentioned target physical shard. Among them, the shard information in the shard information group corresponding to the above-mentioned target physical sub-shard group may include a physical partition shard identifier. The shard information corresponding to the above-mentioned first virtual shard may include a virtual shard identifier.

[0079] As an example, first, the above-mentioned execution entity may determine the physical partition shard identifiers corresponding to the above-mentioned target physical sub-shard group and the virtual shard identifier corresponding to the above-mentioned first virtual shard as the bound shard information. Then, delete the above-mentioned target physical shard.

[0080] Step 2033: Generate a data import result for the first virtual shard based on at least one target physical sub-shard group.

[0081] In some embodiments, the above-mentioned execution entity may generate a data import result for the first virtual shard based on the at least one target physical sub-shard group. Wherein, the data import result may be information on each physical partition shard corresponding to the first virtual shard and the corresponding shard data for the imported data.

[0082] As an example, first, in response to determining that there is no critical unsplit physical shard among the at least one physical shard, for each target physical sub-shard in the at least one target physical sub-shard group, the target physical sub-shard and the corresponding shard data are determined as the sub-shard data import result. Wherein, the critical unsplit physical shard may be a physical shard for which the number of corresponding shard data does not meet the above-mentioned preset number condition. Then, each obtained sub-shard data import result is determined as the data import result for the first virtual shard information.

[0083] In some optional implementation manners of some embodiments, the above-mentioned execution entity may generate a data import result for the first virtual shard based on the at least one target physical sub-shard group through the following steps:

[0084] First step: In response to determining that there is a critical unsplit physical shard among the at least one physical shard, select the critical unsplit physical shard from the at least one physical shard to obtain at least one critical unsplit physical shard.

[0085] Second step: Generate a data import result for the first virtual shard based on the at least one critical unsplit physical shard and the at least one target physical sub-shard group.

[0086] As an example, first, for each critical unsplit physical shard, the critical unsplit physical shard and the corresponding shard data are determined as the first sub-shard data import result. Then, for each target physical sub-shard in the at least one target physical sub-shard group, the target physical sub-shard and the corresponding shard data are determined as the second sub-shard data import result. Finally, at least one first sub-shard data import result and each second sub-shard data import result are determined as the data import result for the first virtual shard.

[0087] Optionally, before generating the data import result for the first virtual shard based on the at least one target physical sub-shard group, the above-mentioned execution entity may further perform the following steps:

[0088] First step, in response to receiving the information of the next import data transaction, determine the virtual shard group corresponding to the above-mentioned information of the next import data transaction as the second virtual shard group. Among them, the above-mentioned information of the next import data transaction can represent an import data transaction executed after the current import data transaction. The above-mentioned information of the next import data transaction can include a second dataset to be imported. The second data to be imported in the above-mentioned second dataset to be imported can be data of the target data table to be imported.

[0089] As an example, first, for each second data to be imported in the above-mentioned second dataset to be imported, the above-mentioned execution entity can select a virtual shard from the above-mentioned virtual shard set through a preset sharding method as the virtual shard corresponding to the second data to be imported. Then, de-duplicate the obtained virtual shards corresponding to each second data to be imported to obtain a virtual shard group as the second virtual shard group.

[0090] Second step, in response to determining that there is a target initial physical shard in the second initial physical shard group, select the target initial physical shard from the above-mentioned second initial physical shard group to obtain at least one target initial physical shard. Among them, each second initial physical shard can be an initial physical shard for importing a subset of the second data to be imported corresponding to the above-mentioned second virtual shard group. The above-mentioned subset of the second data to be imported can be a subset of the second dataset to be imported corresponding to the above-mentioned information of the next import data transaction. The target initial physical shard can be an unfinished split target physical shard corresponding to the above-mentioned first virtual shard group.

[0091] Third step, determine the difference set between the above-mentioned second initial physical shard group and the above-mentioned at least one target initial physical shard as the non-target initial physical shard group.

[0092] Fourth step, for each target initial physical shard, import the subset of the second data to be imported corresponding to the above-mentioned target initial physical shard into the above-mentioned target initial physical shard and the corresponding initial target physical sub-shard group.

[0093] As an example, the above-mentioned execution entity can perform the following steps for each target initial physical shard:

[0094] First sub-step, import the subset of the second data to be imported corresponding to the above-mentioned target initial physical shard into the above-mentioned target initial physical shard.

[0095] Second sub-step, through the above-mentioned preset data import method, import the subset of the second data to be imported corresponding to the above-mentioned target initial physical shard into each initial target physical sub-shard included in the corresponding initial target physical sub-shard group.

[0096] In the fifth step, for each non-target initial physical shard, import the second data subset to be imported corresponding to the non-target initial physical shard into the non-target initial physical shard.

[0097] As an example, the execution entity can, for each non-target initial physical shard, import the second data subset to be imported corresponding to the non-target initial physical shard into the non-target initial physical shard through the preset data import method.

[0098] As an example, Figure 3 shows a schematic diagram of an application scenario of the data import step during splitting according to the data import method of the present disclosure. First, during splitting, the execution entity receives the next import data transaction 301. Then, when any second data to be imported 302 corresponding to the next import data transaction 301 hits the second initial physical shard 303 according to the sharding method and the second initial physical shard is the target physical shard being split, import any second data to be imported 302 into the second initial physical shard 303 and the initial target physical sub-shard group 304 corresponding to the second initial physical shard 303 at the same time. For example, the initial target physical sub-shard group 304 may include an initial target physical sub-shard 1 and an initial target physical sub-shard 2. The second data to be imported 302 may finally be imported into the second initial physical shard 303 and the initial target physical sub-shard 1.

[0099] Optionally, the execution entity may also, in response to detecting that the import data transaction fails, perform an import data transaction rollback operation on the target physical shard and the corresponding initial target physical sub-shard group corresponding to the import data transaction.

[0100] Optionally, the execution entity may also, in response to detecting that the splitting process of any target physical shard fails, delete the initial target physical sub-shard group corresponding to the any target physical shard.

[0101] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: Through the data import method of some embodiments of the present disclosure, the data distribution among physical shards can be made more balanced, and the waste of computing resources and the time consumption can be reduced during data query. Specifically, the reasons for the uneven data distribution among physical shards or the waste of computing resources and long query time during data query are as follows: If the data import result is directly returned after importing the data into the corresponding physical shards according to the sharding rule, it is easy to cause uneven data distribution among physical shards. Even if the random sharding method can be used to ensure that the data distribution of each shard is roughly balanced, however, when data query is performed, a large amount of computing power is usually required to query all shards, resulting in waste of computing resources and long query time. Based on this, in the data import method of some embodiments of the present disclosure, first, the current data import transaction information is obtained. Among them, the above-mentioned current data import transaction information includes the first data set to be imported. Thus, the data to be imported for subsequent sharding storage can be obtained. Then, according to the virtual shards corresponding to each first data to be imported in the above-mentioned first data set to be imported, a virtual shard group is determined as the first virtual shard group. Among them, there is at least one initial physical shard corresponding to each virtual shard. Thus, according to the sharding rule, the virtual shard corresponding to each data to be imported can be determined, which is convenient for subsequently importing the data to be imported into the physical shard corresponding to the virtual shard. Finally, for each first virtual shard in the above-mentioned first virtual shard group, the following data import steps are executed: filtering out the target physical shards from at least one physical shard corresponding to the above-mentioned first virtual shard to obtain at least one target physical shard; where the target physical shard is a physical shard whose corresponding shard data quantity meets the preset quantity condition, and the above-mentioned at least one physical shard is obtained by importing the first data subset to be imported into the above-mentioned at least one initial physical shard, and the above-mentioned first data subset to be imported corresponds to the above-mentioned first virtual shard; splitting each target physical shard in the above-mentioned at least one target physical shard to generate a target physical sub-shard group to obtain at least one target physical sub-shard group; generating a data import result for the above-mentioned first virtual shard according to the above-mentioned at least one target physical sub-shard group. Thus, for each virtual shard with corresponding data to be imported, after importing the data into the physical shard corresponding to the virtual shard, the target physical shards that meet the preset quantity condition can be split, so that the large shards are split into shards with appropriate data volume. Therefore, in the data import method of some embodiments of the present disclosure, by splitting the original large shards for storing data into shards with appropriate data volume, the data distribution among physical shards can be made more balanced, and multiple storage disks on a single machine or multiple machine nodes can be effectively utilized. Also, because, through the sharding rule, the virtual shard corresponding to the data can be determined first, and then the data can be imported into the physical shard corresponding to the virtual shard.Therefore, when performing data query, it is only necessary to query data for each physical shard corresponding to the virtual shard, thereby reducing waste of computing resources and shortening query time.

[0102] Further refer to Figure 4 , which shows the flow 400 of some other embodiments of the data import method. The flow 400 of the data import method includes the following steps:

[0103] Step 401, obtain the current imported data transaction information.

[0104] Step 402, determine a virtual shard group as the first virtual shard group according to the virtual shards corresponding to each first data to be imported in the first data set to be imported.

[0105] Step 403, for each first virtual shard in the first virtual shard group, perform the following data import steps:

[0106] Step 4031, screen out target physical shards from at least one physical shard corresponding to the first virtual shard to obtain at least one target physical shard.

[0107] Step 4032, perform splitting processing on each target physical shard in the at least one target physical shard to generate a target physical sub-shard group, and obtain at least one target physical sub-shard group.

[0108] Step 4033, generate a data import result for the first virtual shard according to the at least one target physical sub-shard group.

[0109] In some embodiments, the specific implementation of steps 401-403 and the technical effects brought by them can refer to Figure 2 Steps 201-step 203 in the corresponding embodiments, which will not be elaborated here.

[0110] Step 404, in response to receiving a data query request, determine the target query virtual shard corresponding to the data query request.

[0111] In some embodiments, the above execution subject may, in response to receiving a data query request, determine the target query virtual shard corresponding to the data query request. Among them, the data query request may be a request for querying data for the above target data table. The data query request may include a query keyword. The query keyword may be a keyword for querying. The target query virtual shard may be a virtual shard corresponding to the data query request.

[0112] As an example, first, the above-mentioned execution entity can select a virtual shard from the above-mentioned virtual shard group as the target query virtual shard according to the query keyword included in the above-mentioned data query request by using the above-mentioned preset sharding method.

[0113] Step 405, in response to determining that there is a corresponding target query physical shard for the target query virtual shard, determine the difference set between at least one query physical shard and at least one target query physical shard as the non-target query physical shard group.

[0114] In some embodiments, the above-mentioned execution entity can, in response to determining that there is a corresponding target query physical shard for the above-mentioned target query virtual shard, determine the difference set between at least one query physical shard and at least one target query physical shard as the non-target query physical shard group. Among them, the target query physical shard can be the target physical shard corresponding to the above-mentioned target query virtual shard. The query physical shard is the physical shard corresponding to the above-mentioned target query virtual shard.

[0115] As an example, first, the above-mentioned execution entity can determine at least one physical shard corresponding to the above-mentioned target query virtual shard as at least one query physical shard. Then, in response to determining that there is a corresponding target physical shard for the above-mentioned target query virtual shard, select the target physical shard from at least one physical shard corresponding to the above-mentioned target query virtual shard as the target query physical shard to obtain at least one target query physical shard. Finally, determine the difference set between at least one query physical shard and at least one target query physical shard as the non-target query physical shard group.

[0116] Step 406, perform a data query operation for the data query request according to the non-target query physical shard group to obtain a data query result.

[0117] In some embodiments, the above-mentioned execution entity can perform a data query operation for the above-mentioned data query request according to the above-mentioned non-target query physical shard group to obtain a data query result.

[0118] As an example, first, for each non-target query physical shard in the above-mentioned non-target query physical shard group, the above-mentioned execution entity can perform a data query operation on the shard data corresponding to the above-mentioned non-target query physical shard according to the query keyword included in the above-mentioned data query request to obtain a shard data query result. Then, determine the obtained shard data query results as the data query result.

[0119] In some optional implementation manners of some embodiments, the above-mentioned execution entity can perform a data query operation for the above-mentioned data query request according to the above-mentioned non-target query physical shard group through the following steps to obtain a data query result:

[0120] First, in response to determining that the physical shard splitting process for the at least one target query has not been completed, select the target query physical shards that have completed splitting from the at least one target query physical shards as the split physical shards, and obtain a split physical shard group.

[0121] Second, according to the non-target query physical shard group and the split physical shard group, perform a data query operation for the data query request to obtain a data query result.

[0122] As an example, first, the execution entity may determine each non-target query physical shard in the non-target query physical shard group and each target physical sub-shard corresponding to the split physical shard group as the physical shards to be queried, and obtain a physical shard group to be queried. Then, for each physical shard to be queried in the physical shard group to be queried, according to the query keyword included in the data query request, perform a data query operation on the shard data corresponding to the physical shard to be queried to obtain a shard data query result. Then, determine the obtained shard data query results as the data query result.

[0123] In some optional implementation manners of some embodiments, the execution entity may perform the following steps to perform a data query operation for the data query request according to the non-target query physical shard group and the split physical shard group to obtain a data query result:

[0124] First, select the target query physical shards that have not completed splitting from the at least one target query physical shards as the physical shards with incomplete splitting, and obtain a physical shard group with incomplete splitting.

[0125] Second, according to the non-target query physical shard group, the split physical shard group, and the physical shard group with incomplete splitting, perform a data query operation for the data query request to obtain a data query result.

[0126] As an example, first, the execution entity may determine each non-target query physical shard in the non-target query physical shard group, each target physical sub-shard corresponding to the split physical shard group, and each physical shard with incomplete splitting corresponding to the physical shard group with incomplete splitting as the physical shards to be queried, and obtain a physical shard group to be queried. Then, for each physical shard to be queried in the physical shard group to be queried, according to the query keyword included in the data query request, perform a data query operation on the shard data corresponding to the physical shard to be queried to obtain a shard data query result. Then, determine the obtained shard data query results as the data query result.

[0127] It can be seen from Figure 4 that compared withFigure 2 Compared with the description of some corresponding embodiments, Figure 4 The process 400 of the data import method in some corresponding embodiments embodies the step of executing a data query request to generate a data query result. Thus, the solutions described in these embodiments first determine the virtual shards corresponding to the data query request, and then perform data query operations on each physical shard corresponding to the virtual shards to obtain the data query result. Among them, this step can be executed in the following three ways: simultaneously perform data queries on each non-target query physical shard that does not need to be split; simultaneously perform data queries on each non-target query physical shard that does not need to be split and each target physical sub-shard corresponding to each split physical shard that has been split; simultaneously perform data queries on each non-target query physical shard that does not need to be split, each target physical sub-shard corresponding to each split physical shard that has been split, and each physical shard that has not been split. Thus, data queries can be simultaneously performed on each physical shard related to the query under the corresponding virtual shard to shorten the query duration, and the data query process is not affected by the shard splitting process. Therefore, the efficiency of data query can be improved.

[0128] Further referring to Figure 5 , as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a data import device. These device embodiments correspond to Figure 2 the method embodiments shown, and the device can be specifically applied to various electronic devices.

[0129] As Figure 5As shown, a data import device 500 includes: an acquisition unit 501, a determination unit 502, and an execution unit 503. Among them, the acquisition unit 501 is configured to acquire current import data transaction information, where the current import data transaction information includes a first data set to be imported; the determination unit 502 is configured to determine a virtual shard group, as a first virtual shard group, according to the virtual shards corresponding to each first data to be imported in the first data set to be imported, where there is at least one initial physical shard corresponding to each virtual shard; the execution unit 503 is configured to, for each first virtual shard in the first virtual shard group, perform the following data import steps: screen out target physical shards from at least one physical shard corresponding to the first virtual shard to obtain at least one target physical shard, where the target physical shard is a physical shard whose corresponding shard data number meets a preset number condition, and the at least one physical shard is obtained by importing a first data subset to be imported into the at least one initial physical shard, and the first data subset to be imported corresponds to the first virtual shard; perform splitting processing on each target physical shard in the at least one target physical shard to generate a target physical sub-shard group to obtain at least one target physical sub-shard group; generate a data import result for the first virtual shard according to the at least one target physical sub-shard group.

[0130] In some alternative implementation manners of some embodiments, the data import device 500 may be further configured to: in response to receiving a data query request, determine a target query virtual shard corresponding to the data query request; in response to determining that there is a target query physical shard corresponding to the target query virtual shard, determine the difference set between at least one query physical shard and at least one target query physical shard as a non-target query physical shard group, where the target query physical shard is a target physical shard corresponding to the target query virtual shard, and the query physical shard is a physical shard corresponding to the target query virtual shard; perform a data query operation for the data query request according to the non-target query physical shard group to obtain a data query result.

[0131] In some alternative implementation manners of some embodiments, the data import device 500 may be further configured to: in response to determining that the splitting processing of the at least one target query physical shard is not completed, select the target query physical shards that have been split and completed from the at least one target query physical shard as split physical shards to obtain a split physical shard group; perform a data query operation for the data query request according to the non-target query physical shard group and the split physical shard group to obtain a data query result.

[0132] In some alternative implementations of some embodiments, the above data import device 500 may be further configured to: select, from the above at least one target query physical shard, the target query physical shards that have not completed splitting as the physical shards with incomplete splitting, to obtain a group of physical shards with incomplete splitting; and perform a data query operation for the above data query request according to the above non-target query physical shard group, the above split physical shard group, and the above group of physical shards with incomplete splitting, to obtain a data query result.

[0133] In some alternative implementations of some embodiments, the above execution unit 503 may be further configured to: for each target physical shard, perform the following splitting processing steps: generate a shard splitting number according to the above target physical shard; create an initial group of target physical sub-shards according to the above shard splitting number; and import the shard data corresponding to the above target physical shard into the above initial group of target physical sub-shards to obtain a group of target physical sub-shards.

[0134] In some alternative implementations of some embodiments, the above execution unit 503 may be further configured to: bind the shard information group corresponding to the above group of target physical sub-shards with the shard information of the above first virtual shard, and delete the above target physical shard.

[0135] In some alternative implementations of some embodiments, the above execution unit 503 may be further configured to: in response to receiving the information of the next import data transaction, determine the virtual shard group corresponding to the above information of the next import data transaction as the second virtual shard group, where the above information of the next import data transaction represents an import data transaction executed after the current import data transaction; in response to determining that there is a target initial physical shard in the second initial physical shard group, select the target initial physical shard from the above second initial physical shard group to obtain at least one target initial physical shard, where each second initial physical shard is the initial physical shard corresponding to the above second virtual shard group and for importing the second subset of data to be imported, and the second subset of data to be imported is a subset of the second dataset to be imported corresponding to the above information of the next import data transaction, and the target initial physical shard is the target physical shard corresponding to the above first virtual shard group and with incomplete splitting; determine the difference set between the above second initial physical shard group and the above at least one target initial physical shard as the non-target initial physical shard group; for each target initial physical shard, import the second subset of data to be imported corresponding to the above target initial physical shard into the above target initial physical shard and the corresponding initial group of target physical sub-shards; and for each non-target initial physical shard, import the second subset of data to be imported corresponding to the above non-target initial physical shard into the above non-target initial physical shard.

[0136] In some alternative implementations of some embodiments, the above-mentioned execution unit 503 may further be configured to: in response to detecting that the target physical shard does not exist in the at least one physical shard, determine the at least one physical shard and the corresponding at least one shard data as the data import result.

[0137] In some alternative implementations of some embodiments, the above-mentioned execution unit 503 may also be further configured to: in response to determining that there is a critical unsplit physical shard in the at least one physical shard, select the critical unsplit physical shard from the at least one physical shard to obtain at least one critical unsplit physical shard; generate a data import result for the first virtual shard according to the at least one critical unsplit physical shard and the at least one target physical sub-shard group.

[0138] It can be understood that the various units described in the data import device 500 correspond to the respective steps in the method described with reference to Figure 2 Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 500 and the units included therein, and will not be elaborated herein.

[0139] Next, with reference to Figure 6 , which shows a schematic structural diagram of an electronic device 600 suitable for implementing some embodiments of the present disclosure (such as Figure 1 the electronic device 101 therein). Figure 6 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0140] As Figure 6 shown, the electronic device 600 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 into the random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.

[0141] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0142] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0143] It should be noted that in some embodiments of the present disclosure, the above-mentioned computer-readable medium may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable signal medium may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0144] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed network.

[0145] The above computer-readable medium may be included in the above electronic device; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device is caused to: obtain current import data transaction information, where the current import data transaction information includes a first data set to be imported; determine a virtual shard group as a first virtual shard group according to the virtual shards corresponding to each first data to be imported in the first data set to be imported, where there is at least one initial physical shard corresponding to each virtual shard; for each first virtual shard in the first virtual shard group, perform the following data import steps: screen out target physical shards from at least one physical shard corresponding to the first virtual shard to obtain at least one target physical shard, where the target physical shard is a physical shard whose corresponding shard data quantity meets a preset quantity condition, and the at least one physical shard is obtained by importing a first subset of data to be imported into the at least one initial physical shard, and the first subset of data to be imported corresponds to the first virtual shard; perform splitting processing on each of the at least one target physical shards to generate a target physical sub-shard group, obtaining at least one target physical sub-shard group; generate a data import result for the first virtual shard according to the at least one target physical sub-shard group.

[0146] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0148] The units described in some embodiments of the present disclosure can be implemented in software or in hardware. The described units can also be provided in a processor. For example, it can be described as: a processor includes an acquisition unit, a determination unit, and an execution unit. Among them, the names of these units do not constitute a limitation on the unit itself in some cases. For example, the acquisition unit can also be described as "the unit for acquiring the current imported data transaction information".

[0149] The functions described above herein can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0150] Some embodiments of the present disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any one of the above data import methods.

[0151] The above description is only some preferred embodiments of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A data import method, comprising: obtaining current import data transaction information, where the current import data transaction information includes a first data set to be imported; determining a virtual shard group as a first virtual shard group according to the virtual shards corresponding to each first data to be imported in the first data set to be imported, where there is at least one initial physical shard corresponding to each virtual shard; for each first virtual shard in the first virtual shard group, performing the following data import steps: screening out target physical shards from at least one physical shard corresponding to the first virtual shard to obtain at least one target physical shard, where the target physical shard is a physical shard whose corresponding shard data quantity meets a preset quantity condition, and the at least one physical shard is obtained by importing a first subset of data to be imported into the at least one initial physical shard, and the first subset of data to be imported corresponds to the first virtual shard; performing a splitting process on each target physical shard in the at least one target physical shard to generate a target physical sub-shard group, obtaining at least one target physical sub-shard group; generating a data import result for the first virtual shard according to the at least one target physical sub-shard group.

2. The method according to claim 1, wherein, the method further comprises: responding to receiving a data query request, determining a target query virtual shard corresponding to the data query request; responding to determining that there is a target query physical shard corresponding to the target query virtual shard, determining a difference set between at least one query physical shard and at least one target query physical shard as a non-target query physical shard group, where the target query physical shard is a target physical shard corresponding to the target query virtual shard, and the query physical shard is a physical shard corresponding to the target query virtual shard; performing a data query operation for the data query request according to the non-target query physical shard group to obtain a data query result.

3. The method according to claim 2, wherein, the performing a data query operation for the data query request according to the non-target query physical shard group to obtain a data query result includes: responding to determining that the splitting process of the at least one target query physical shard is not completed, selecting the target query physical shards that have been split completely from the at least one target query physical shard as split physical shards, obtaining a split physical shard group; performing a data query operation for the data query request according to the non-target query physical shard group and the split physical shard group to obtain a data query result.

4. The method according to claim 3, wherein, the performing a data query operation for the data query request according to the non-target query physical shard group and the split physical shard group to obtain a data query result includes: selecting the target query physical shards that have not been split completely from the at least one target query physical shard as unsplit physical shards, obtaining an unsplit physical shard group; Perform a data query operation for the data query request according to the non-target query physical shard group, the split physical shard group, and the incomplete split physical shard group, to obtain a data query result.

5. The method according to claim 1, wherein, the splitting process for each target physical shard in the at least one target physical shard to generate a target physical sub-shard group includes: For each target physical shard, perform the following splitting steps: Generate a shard split number according to the target physical shard; Create an initial target physical sub-shard group according to the shard split number; Import the shard data corresponding to the target physical shard into the initial target physical sub-shard group to obtain a target physical sub-shard group.

6. The method according to claim 5, wherein, after importing the shard data corresponding to the target physical shard into the initial target physical sub-shard group to obtain a target physical sub-shard group, the method further includes: Bind the shard information group corresponding to the target physical sub-shard group with the shard information of the first virtual shard, and delete the target physical shard.

7. The method according to claim 1, wherein, before generating a data import result for the first virtual shard according to the at least one target physical sub-shard group, the method further includes: In response to receiving next import data transaction information, determine the virtual shard group corresponding to the next import data transaction information as a second virtual shard group, where the next import data transaction information represents an import data transaction executed after the current import data transaction; In response to determining that there is a target initial physical shard in the second initial physical shard group, select the target initial physical shard from the second initial physical shard group to obtain at least one target initial physical shard, where each second initial physical shard is an initial physical shard corresponding to the second virtual shard group for importing a second subset of data to be imported, the second subset of data to be imported is a subset of a second data set to be imported corresponding to the next import data transaction information, and the target initial physical shard is an incomplete split target physical shard corresponding to the first virtual shard group; Determine the difference set between the second initial physical shard group and the at least one target initial physical shard as a non-target initial physical shard group; For each target initial physical shard, import the second subset of data to be imported corresponding to the target initial physical shard into the target initial physical shard and the corresponding initial target physical sub-shard group; For each non-target initial physical shard, import the second subset of data to be imported corresponding to the non-target initial physical shard into the non-target initial physical shard.

8. The method according to claim 1, wherein, after screening out target physical shards from at least one physical shard corresponding to the first virtual shard to obtain at least one target physical shard, the method further includes: In response to detecting that the target physical shard does not exist in the at least one physical shard, determining the at least one physical shard and the corresponding at least one shard data as a data import result.

9. The method according to claim 1, wherein, the generating a data import result for the first virtual shard according to the at least one target physical sub-shard group includes: in response to determining that there is a critical unsplit physical shard in the at least one physical shard, selecting the critical unsplit physical shard from the at least one physical shard to obtain at least one critical unsplit physical shard; generating a data import result for the first virtual shard according to the at least one critical unsplit physical shard and the at least one target physical sub-shard group.

10. A data import device, comprising: an acquisition unit configured to acquire current import data transaction information, wherein the current import data transaction information includes a first data set to be imported; a determination unit configured to determine a virtual shard group as a first virtual shard group according to the virtual shards corresponding to each first data to be imported in the first data set to be imported, wherein there is at least one initial physical shard corresponding to each virtual shard; an execution unit configured to, for each first virtual shard in the first virtual shard group, perform the following data import steps: screening out target physical shards from the at least one physical shard corresponding to the first virtual shard to obtain at least one target physical shard, wherein the target physical shard is a physical shard whose corresponding shard data quantity meets a preset quantity condition, the at least one physical shard is obtained by importing a first data subset to be imported into the at least one initial physical shard, and the first data subset to be imported corresponds to the first virtual shard; performing a splitting process on each target physical shard in the at least one target physical shard to generate a target physical sub-shard group to obtain at least one target physical sub-shard group; generating a data import result for the first virtual shard according to the at least one target physical sub-shard group.

11. An electronic device, comprising: one or more processors; a storage device having stored thereon one or more programs, when the one or more programs are executed by the one or more processors, enabling the one or more processors to implement the method according to any one of claims 1-9.

12. A computer-readable medium having stored thereon a computer program, wherein, when the computer program is executed by a processor, implementing the method according to any one of claims 1-9.

13. A computer program product comprising a computer program, the computer program implementing the method according to any one of claims 1-9 when executed by a processor.

Images