Data loading method, device, equipment and medium

Abstract

Embodiments of the present disclosure relate to a data loading method, device, equipment and medium, wherein the method comprises: in response to a loading request of a plurality of candidate database tables, obtaining at least one target core in a central processing unit that meets a preset loading condition; judging whether the at least one target core meets a preset multi-channel loading construction condition; if the multi-channel loading construction condition is met, constructing a plurality of loading channels corresponding to the at least one target core; and loading the plurality of candidate database tables to a preset database according to the plurality of loading channels. Thus, the database table data is loaded in parallel in a multi-channel manner, the loading efficiency of the database table data is improved, the multi-channel is dynamically adjusted based on system load, resources can be reserved for other applications on the premise of maximizing hardware resources, and the data loading efficiency is further improved by loading in parallel in the granularity of the database table.

Description

Technical Field

[0001] This disclosure relates to the field of data synchronization technology, and in particular to a method, apparatus, device and medium for multi-channel parallel data initialization loading based on dynamic load. Background Technology

[0002] The initial loading of existing data is a crucial stage in real-time data synchronization using data synchronization tools. This initial loading primarily includes the initial loading of table structures and the initial loading of existing data. The key performance indicator for the table data initial loading stage is the maximum data throughput.

[0003] The key to maximizing data throughput lies in fully utilizing hardware resources to load existing data from the source database to the target database in the shortest possible time. Traditional methods primarily employ multi-threading or multi-processing, allocating tables to be loaded to a fixed number of threads or processes. The drawbacks of this approach are: 1. The initial number of threads or processes cannot be determined, potentially leading to system idleness or overload; 2. Based on a fixed allocation method, a particular thread or process may become the longest path, resulting in a backlog of tables waiting to be loaded while other threads or processes have already finished loading. Summary of the Invention

[0004] To address the aforementioned technical issues, this disclosure provides a method, apparatus, device, and medium for multi-channel parallel data initialization and loading based on dynamic load. It employs a multi-channel approach to load database table data in parallel, improving the loading efficiency of database table data. Furthermore, it dynamically adjusts the multi-channel configuration based on system load, maximizing hardware resource utilization while reserving resources for other applications. Additionally, parallel loading at the database table level further enhances loading efficiency.

[0005] This disclosure provides a data loading method, including the following steps: in response to loading requests from multiple candidate database tables, obtaining at least one target kernel in a central processing unit that meets preset loading conditions; determining whether the at least one target kernel meets preset multi-channel loading construction conditions; if the multi-channel loading construction conditions are met, constructing multiple loading channels corresponding to the at least one target kernel; and loading the multiple candidate database tables into a preset database according to the multiple loading channels.

[0006] In one embodiment of this disclosure, obtaining at least one target kernel in the central processing unit that meets preset loading conditions includes: obtaining a first free resource value for each candidate kernel in the central processing unit; and determining the candidate kernel whose first free resource value is greater than a preset resource threshold as the target kernel.

[0007] In one embodiment of this disclosure, constructing multiple loading channels corresponding to the at least one target kernel includes: obtaining a resource occupancy value for each loading channel and obtaining a second idle resource value for each target kernel; calculating the ratio of the resource occupancy value to the second idle resource value, and determining the number of channels based on the ratio; constructing multiple sets of loading threads corresponding to the number of channels in the at least one target kernel to realize the construction of the multiple loading channels, wherein each set of loading threads includes a read thread and a write thread.

[0008] In one embodiment of this disclosure, loading the plurality of candidate database tables into a preset database according to the plurality of loading channels includes: determining at least one target database table that has not been loaded in the plurality of candidate database tables during the current period; detecting the current state of the plurality of loading channels during the current period; if at least one target loading channel with the current state being idle is obtained, determining whether a first number of the at least one target loading channel is greater than or equal to a second number of the at least one target database table; if the first number is greater than or equal to the second number, determining at least one target loading channel corresponding to the at least one target database table, wherein the target loading channels corresponding to each target database table are different; and loading the corresponding target database table into the preset database according to the corresponding target loading channel.

[0009] In one embodiment of this disclosure, the method further includes: if the first quantity is less than the second quantity, determining the first quantity of target database tables in the at least one target database table; and storing the first quantity of target database tables in the preset database according to the at least one target loading channel, wherein the target database tables stored in each target loading channel are different.

[0010] In one embodiment of this disclosure, before loading the plurality of candidate database tables into a preset database according to the plurality of loading channels, the method includes: determining that the amount of data in each candidate database table is less than a preset threshold.

[0011] This disclosure also provides a data loading apparatus, comprising: an acquisition module, configured to acquire at least one target kernel in a central processing unit that meets preset loading conditions in response to loading requests for multiple candidate database tables; a judgment module, configured to determine whether the at least one target kernel meets preset multi-channel loading construction conditions; a construction module, configured to construct multiple loading channels corresponding to the at least one target kernel when the multi-channel loading construction conditions are met; and a storage module, configured to load the multiple candidate database tables into a preset database according to the multiple loading channels.

[0012] In one embodiment of this disclosure, the acquisition module is configured to: acquire a first free resource value for each candidate kernel in the central processing unit; and determine the candidate kernel whose first free resource value is greater than a preset resource threshold as the target kernel.

[0013] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the data loading method provided in this disclosure.

[0014] This disclosure also provides a computer-readable storage medium storing a computer program for executing the data loading method provided in this disclosure.

[0015] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0016] The data loading scheme provided in this disclosure responds to loading requests from multiple candidate database tables by acquiring at least one target kernel in the central processing unit that meets preset loading conditions. It then determines whether the at least one target kernel meets preset multi-channel loading construction conditions. If the multi-channel loading construction conditions are met, multiple loading channels corresponding to the at least one target kernel are constructed, and multiple candidate database tables are loaded into a preset database according to the multiple loading channels. Therefore, by using a multi-channel approach to load database table data in parallel, the loading efficiency of database table data is improved. Furthermore, the multi-channel approach dynamically adjusts based on system load, maximizing hardware resource utilization while reserving resources for other applications. Additionally, parallel loading at the database table level further enhances loading efficiency. Attached Figure Description

[0017] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0018] Figure 1 A schematic flowchart illustrating a data loading method provided in an embodiment of this disclosure;

[0019] Figure 2 A flowchart illustrating another data loading method provided in this embodiment of the disclosure;

[0020] Figure 3 A flowchart illustrating another data loading method provided in this embodiment of the disclosure;

[0021] Figure 4 This is a schematic diagram of a data loading scenario provided by an embodiment of the present disclosure;

[0022] Figure 5 This is a schematic diagram illustrating another data loading scenario provided by an embodiment of this disclosure;

[0023] Figure 6 This is a schematic diagram illustrating another data loading scenario provided by an embodiment of this disclosure;

[0024] Figure 7 This is a schematic diagram of the structure of a data loading device provided in an embodiment of the present disclosure. Detailed Implementation

[0025] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0026] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0027] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0028] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0029] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0031] To address the aforementioned problems, this disclosure provides a data loading method, which will be described below with reference to specific embodiments.

[0032] Figure 1 This is a flowchart illustrating a data loading method provided in an embodiment of the present disclosure. The method can be executed by a data loading device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method includes:

[0033] Step 101: In response to the loading requests of multiple candidate database tables, at least one target kernel in the central processing unit that meets the preset loading conditions is obtained.

[0034] The candidate database table can be any database table containing any data, and there are no restrictions here.

[0035] In this embodiment, in response to a loading request for multiple candidate database tables, the loading request can be triggered after the relevant database tables reach a preset synchronization condition.

[0036] In this embodiment, considering that in many data synchronization scenarios, the candidate database tables are all database tables with small data volumes, splitting at the single table level is meaningless. Sometimes, because splitting requires predicting the table's statistics in advance to decide on the splitting algorithm, the extra time consumed is even more than loading directly without splitting, which is equivalent to a negative optimization.

[0037] Therefore, in this embodiment, instead of judging whether the data volume of the candidate database table exceeds a preset data volume threshold, each candidate database table is stored as a single database table shard. Thus, there is no need to split individual tables; instead, the entire data loading task is split through a further dynamic load multi-channel approach, maximizing data loading performance in scenarios involving large-volume loading of small tables.

[0038] Further, acquire at least one target kernel in the central processing unit that meets the preset loading conditions.

[0039] In this embodiment, at least one target kernel in the central processing unit that meets preset loading conditions is obtained. It is understood that the target kernel meeting the preset loading conditions has sufficient remaining resources to avoid overloading during database table loading.

[0040] It should be noted that the preset loading conditions differ in different application scenarios, as shown in the following examples:

[0041] Example 1:

[0042] In this example, the first free resource value of each candidate kernel in the central processing unit is obtained. For example, the first free resource value of each candidate kernel can be obtained through relevant methods and functions. Then, the candidate kernel with the first free resource value greater than a preset resource threshold is determined as the target kernel. The preset resource threshold is determined according to experimental calibration.

[0043] Example 2:

[0044] In this embodiment, the task time level of the current processing task of each candidate core in the central processing unit is obtained. For example, a correspondence between different task time levels and corresponding task content is pre-built. The higher the task time level, the more likely the corresponding core will not be able to release the corresponding processing task in a short period of time. The task level of the current processing task of each candidate core is determined based on this correspondence.

[0045] If the task time level is less than the preset level, it means that the corresponding kernel may release the corresponding processing task in a short period of time, and the corresponding candidate kernel can be determined as the target kernel. Of course, in this embodiment, if the candidate kernel does not have any current processing tasks to be processed, the candidate kernel is directly determined as the target kernel.

[0046] Step 102: Determine whether at least one target kernel meets the preset multi-channel loading and building conditions.

[0047] In this embodiment, it is determined whether at least one target kernel meets the preset multi-channel loading and building conditions, such as the number of kernels required to build each loading channel in advance. If the number of at least one target kernel is less than the number of kernels, it indicates that the current target kernel cannot build multiple loading channels, thus failing to meet the multi-channel loading and building conditions.

[0048] Step 103: If the multi-channel loading construction conditions are met, then construct multiple loading channels corresponding to at least one target kernel.

[0049] In this embodiment, if the multi-channel loading construction conditions are met, then multiple loading channels corresponding to at least one target kernel are constructed.

[0050] In some possible embodiments, the resource occupancy value of each loading channel is obtained, and the second idle resource value of each target kernel is obtained. The ratio of the resource occupancy value and the second idle resource value is calculated. The number of channels is determined based on the ratio. If the ratio is not an integer, it is rounded down to the number of channels. Multiple sets of loading threads corresponding to the number of channels are constructed in at least one target kernel to realize the construction of multiple loading channels. Each set of loading threads includes a read thread and a write thread.

[0051] In other possible embodiments, such as this one, a load control thread can be constructed to monitor the current CPU load of the system. The number of read and write threads that can exist in the multi-channel system is determined based on the CPU's idle state. In this embodiment, the number of load channels that can be built for each target core is set, and multiple load channels corresponding to at least one target core are determined based on the number of target cores and the number of load channels that can be built for each target core.

[0052] For example, suppose the current system has 16 core processors, and the overall CPU utilization is 200%, meaning only 2 cores are working at full capacity. Then, 14 cores are available for allocation. Assuming one load channel (one read thread and one write thread) occupies 2 cores, then the remaining 14 cores can be allocated 7 load channels (14 / 2 = 7).

[0053] Furthermore, this load balancing thread allows setting a preset number of load channels that each target kernel can build. Setting it to 1 means one kernel can run one load channel; setting it to 2 means one kernel can run two load channels, and so on. When this parameter is set to 2, the remaining 14 kernels can run 14 read / write channels (14 * 2 / 2 = 14). The resource usage value can be determined based on the hardware system running different migration tasks, using empirical values ​​or the results of multiple tests.

[0054] In another embodiment of this disclosure, if the multi-channel loading condition is not met, multiple database table shards are randomly sorted to generate a queue, and the target kernel stores the database table shards in the queue one by one.

[0055] Step 104: Load multiple candidate database tables into the preset database according to multiple loading channels.

[0056] In this embodiment, multiple candidate database tables are loaded into a preset database using multiple loading channels. This preset database can be a target database corresponding to the source database in a data synchronization scenario. Specifically, when loading a candidate database table, the data storage link information of the currently stored candidate database table can be obtained, and the corresponding data can be retrieved from the corresponding source database and loaded based on this link information.

[0057] In one embodiment of this disclosure, one or more rows of data from each candidate database table can be encapsulated into an abstract object, and this abstract object can be placed into a write queue. The write thread of the loading channel continuously reads the abstract object from the queue, unpacks it into N data items, and then delivers them to a preset database. The data corresponding to each candidate database table can be linked to the source database to query the data in that table.

[0058] For example, if an abstract object corresponding to a candidate database table encapsulates multiple rows of data, then the corresponding encapsulation object could be:

[0059] {

[0060] Table: Table name

[0061] Meta: [

[0063] Column: {position: location, name: name, type: type}

[0064] Column: {position: position, name: name, type: type}

[0065] Column: {position: position, name: name, type: type} ]

[0067] Rows: [

[0069] {Row: {position: position, value: value}}

[0070] {Row: {position: position, value: value}},

[0071] {Row: {position: position, value: value}}

[0072] ], [

[0074] {Row: {position: position, value: value}},

[0075] {Row: {position: position, value: value}},

[0076] {Row: {position: position, value: value}} ]

[0078] }

[0079] Therefore, in this embodiment, when loading database table data through multiple channels, each loading channel can only retrieve the next database table for loading after processing the loading of one database table. This eliminates the task backlog within the multiple read threads of the multiple loading channels, thus achieving load balancing (i.e., ultimately only one set of read / write threads is left to process the last database table).

[0080] It should be noted that the methods for loading multiple candidate database tables into the preset database based on multiple loading channels vary in different application scenarios, as shown in the following example:

[0081] Example 1:

[0082] In this example, based on the actual number of loading channels, a read thread and a write thread are built for each channel. The read thread and the write thread are connected using a data output queue to store the relevant data.

[0083] like Figure 2 As shown, multiple candidate database tables are loaded into a preset database according to multiple loading channels, including:

[0084] Step 201: Identify at least one target database table that has not been loaded from among multiple candidate database tables in the current period.

[0085] In this embodiment, if multiple candidate database tables are being synchronized for the first time, then at least one candidate database table is considered as multiple target database tables. If multiple candidate database tables are not being synchronized for the first time, then the remaining unloaded database tables in at least one candidate database table are considered as target database tables.

[0086] Step 202: Detect the current status of multiple loading channels within the current cycle.

[0087] In this embodiment, the current status of multiple loading channels within the current period can be detected according to relevant methods and functions, that is, it can be determined whether each loading channel is loading data from database table shards.

[0088] Step 203: If at least one target loading channel is found to be currently in an idle state, determine whether the first number of at least one target loading channel is greater than or equal to the second number of at least one target database table.

[0089] In this embodiment, in order to achieve dynamic load, at least one target loading channel that is currently in an idle state is obtained, and it is determined whether the first number of at least one target loading channel is greater than or equal to the second number of at least one target database table.

[0090] Step 204: If the first quantity is greater than or equal to the second quantity, then determine at least one target loading channel corresponding to at least one target database table, wherein the target loading channels corresponding to each target database table are different.

[0091] Step 205: Load the corresponding target database table into the preset database according to the corresponding target loading channel.

[0092] In this embodiment, if the first quantity is greater than or equal to the second quantity, that is, each target database table can be individually assigned to the corresponding target loading channel, at least one target loading channel corresponding to at least one target database table is determined, wherein the target loading channels corresponding to each target database table are different.

[0093] For example, such as Figure 3 As shown, if the first quantity is 5 and the corresponding second quantity is 4, then a target loading channel is randomly selected for each target database table to store the target database table.

[0094] In this embodiment, if the first quantity is less than the second quantity, then the first quantity of target database tables is determined from at least one target database table. For example, at least one target database table is randomly sorted to generate a sorting queue, and the first quantity of target database tables is selected in order from front to back. Alternatively, the first quantity of target database tables is randomly determined from at least one target database table.

[0095] Then, based on at least one target loading channel, a first number of target database tables are loaded into a preset database, wherein each target loading channel stores a different target database table. The remaining target database tables are stored in subsequent cycles using the steps described above.

[0096] For example, such as Figure 4 As shown, if the first quantity is 5 and the corresponding second quantity is 6, then 5 target database tables are randomly selected from the 6 target database tables and assigned to 5 target loading channels for loading processing. Each target database table is assigned to a different target loading channel.

[0097] Example 2:

[0098] In this embodiment, multiple candidate database tables are directly split into multiple queues based on the number of loading channels. The number of candidate database tables corresponding to the multiple queues is no greater than a preset threshold. The preset threshold is related to the number of loading channels and the number of multiple database tables to ensure that the multiple candidate database tables are distributed as evenly as possible to the corresponding loading channels. Then, each loading channel loads the candidate database tables in the corresponding candidate database table queue.

[0099] To enable those skilled in the art to better understand the data loading method of the embodiments of this disclosure, the following description is provided in conjunction with a specific scenario. In this scenario, multiple candidate database tables are loaded from a source database to a target database. The construction of the loading channel and the processing of the corresponding candidate database tables can be handled by a load control thread.

[0100] In this embodiment, as Figure 5As shown, in response to loading requests from multiple candidate database tables, multiple loading channels are constructed based on the kernel resources of the central processing unit. Each loading channel contains one read thread and one write thread. When loading the corresponding database table shard, each loading channel can encapsulate one or more data rows from the corresponding candidate database table into an abstract object to obtain the output queue corresponding to each database table shard. Based on the output queue, each data row is written to the target database according to the write thread.

[0101] In practical applications, the load control thread has another parameter: maximum load. This parameter controls the maximum load the overall system can withstand. In this embodiment, the real-time load of the central processing unit is monitored during data storage. If the real-time load exceeds the maximum load, the difference between the real-time load and the maximum load is calculated. Based on this difference, a preset database is queried to determine the number of loading channels to stop loading. A corresponding number of loading channels are randomly selected and marked with a stop flag. After completing the storage of the abstract objects in the current queue, the loading channels marked with the stop flag stop loading data from the corresponding candidate database tables. Furthermore, the remaining data from the candidate database tables can be queued to other running loading channels for storage.

[0102] For example, if the maximum load is 0.8, it means that when the overall system load is greater than 80%, the load needs to be reduced. For instance, with a system of 16 cores, the CPU utilization it can handle is 1600% * 0.8 = 1280%. When the overall system load exceeds 1280%, at least one pair of read and write threads in a load channel needs to be dynamically stopped. This is done by continuously stopping read and write threads until the overall load is less than 80%. The specific stopping method is as follows: the load control thread randomly selects a read thread in the load channel, calls the read thread's cancel interface, and sets a cancel flag in the read thread. When the read thread detects this flag setting, it writes a stop flag to its write queue after completing the current table load task and then exits; the corresponding write thread reads the stop flag from its queue and also exits.

[0103] Of course, if the candidate database table is a particularly large granularity table when loading is performed at the database table level in this disclosure, the corresponding database table can be split and loaded. That is, in this embodiment, multiple database table shards corresponding to multiple candidate database tables are determined so that multi-channel storage can be performed on each database table shard in the future to improve loading efficiency.

[0104] It should be noted that the method for determining the sharding of multiple database tables corresponding to multiple candidate database tables differs in different application scenarios, as shown in the following example:

[0105] Example 1:

[0106] In this example, such as Figure 6 As shown, determining multiple database table shards corresponding to multiple candidate database tables includes:

[0107] Step 601: Determine whether the amount of data in each candidate database table is greater than the preset data amount threshold.

[0108] The preset data volume threshold can be determined based on the computing power of each CPU core. The larger the computing power of each core, the larger the corresponding preset data volume threshold.

[0109] Step 602: If the data volume is not greater than the preset data volume threshold, then the corresponding first candidate database table is determined as a database table shard.

[0110] In this embodiment, if the data volume of a candidate database table is not greater than a preset data volume threshold, it indicates that the corresponding database table may have a small data volume. Therefore, the corresponding first candidate database table is directly sharded as a database table, thereby avoiding splitting the database table and improving the loading efficiency of the database table while ensuring loading efficiency.

[0111] Furthermore, the Figure 6 Also includes:

[0112] Step 603: If the data volume exceeds the preset data volume threshold, the corresponding second candidate database table is split into multiple database table shards according to the preset splitting strategy, wherein the data volume of each database table shard does not exceed the preset data volume threshold.

[0113] In this embodiment, if the data volume is greater than the preset data volume threshold, it indicates that the corresponding candidate database table is a database table with a large data volume. Therefore, in order to improve the loading efficiency of the database table, the corresponding second candidate database table is split into multiple database table shards according to the preset splitting strategy, wherein the data volume of each database table shard is not greater than the preset data volume threshold.

[0114] For example, the preset splitting strategy can be to directly split the corresponding second candidate database table into shards of each database table with a data volume of the preset data volume threshold based on a preset data volume threshold. When splitting the corresponding second candidate database table according to the preset data volume threshold, if the remaining data volume of the second candidate database table is less than the preset data volume threshold, then the remaining data volume is used as a database table shard.

[0115] For example, the preset splitting strategy can be as follows: a splitting model is trained in advance based on a preset data volume threshold and the total data volume. The data volume of the corresponding second candidate database table and the preset data volume threshold are input into the splitting model to obtain multiple random data volumes whose summation value is the data volume of the second candidate database table. The second candidate database table is split according to these multiple random data volumes to obtain multiple database table shards.

[0116] In summary, the data loading method of this disclosure, in response to loading requests from multiple candidate database tables, obtains at least one target kernel in the central processing unit that meets preset loading conditions, determines whether the at least one target kernel meets preset multi-channel loading construction conditions, and then, if the multi-channel loading construction conditions are met, constructs multiple loading channels corresponding to the at least one target kernel, and loads multiple candidate database tables into a preset database according to the multiple loading channels. Therefore, by using a multi-channel approach to load database table data in parallel, the loading efficiency of database table data is improved. Furthermore, the multi-channel approach dynamically adjusts based on system load, maximizing the utilization of hardware resources while reserving resources for other applications. In addition, parallel loading at the database table level further improves loading efficiency.

[0117] To implement the above embodiments, this disclosure also proposes a data loading device.

[0118] Figure 7 This is a schematic diagram of a data loading device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 7 As shown, the device includes: an acquisition module 710, a judgment module 720, a construction module 730, and a storage module 740, wherein,

[0119] The acquisition module 710 is used to acquire at least one target kernel in the central processing unit that meets the preset loading conditions in response to loading requests from multiple candidate database tables.

[0120] The judgment module 720 is used to determine whether at least one target kernel meets the preset multi-channel loading and building conditions;

[0121] Module 730 is used to construct multiple loading channels corresponding to at least one target kernel when the multi-channel loading construction conditions are met.

[0122] Storage module 740 is used to load multiple candidate database tables into a preset database based on multiple loading channels.

[0123] In one embodiment of this disclosure, the acquisition module 720 is used for:

[0124] Obtain the first free resource value for each candidate core in the central processing unit;

[0125] Candidate kernels whose first idle resource value is greater than a preset resource threshold are identified as target kernels.

[0126] The data loading apparatus provided in this disclosure can execute the data loading method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0127] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implements the data loading method described above.

[0128] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium storing a computer program for performing any of the data loading methods provided in the present disclosure.

[0129] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0130] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0131] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0132] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0133] In response to loading requests from multiple candidate database tables, at least one target kernel in the central processing unit that meets preset loading conditions is obtained. It is then determined whether the at least one target kernel meets preset multi-channel loading construction conditions. If the multi-channel loading construction conditions are met, multiple loading channels corresponding to the at least one target kernel are constructed, and the multiple candidate database tables are loaded into the preset database according to the multiple loading channels. Therefore, by using a multi-channel approach to load database table data in parallel, the loading efficiency of database table data is improved. Furthermore, the multi-channel approach is dynamically adjusted based on system load to avoid system load overflow. In addition, parallel loading at the database table level further improves loading efficiency.

[0134] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0136] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0137] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0138] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0139] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0140] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0141] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A data loading method characterized by comprising: The method comprises the following steps: In response to a loading request of a plurality of candidate database tables, at least one target core in a central processor satisfying a preset loading condition is obtained; It is judged whether the at least one target core satisfies a preset multi-channel loading construction condition; If the multi-channel loading construction condition is satisfied, a resource occupation value of each loading channel is obtained, and a second idle resource value of each target core is obtained; The ratio of the resource occupation value and the second idle resource value is calculated, and the number of channels is determined according to the ratio; A plurality of groups of loading threads corresponding to the number of channels are constructed in the at least one target core to realize the construction of a plurality of loading channels, wherein each group of loading threads comprises a reading thread and a writing thread; The plurality of candidate database tables are loaded into a preset database according to the plurality of loading channels.

2. The method of claim 1, wherein, The at least one target core in the central processor satisfying the preset loading condition is obtained, comprising: A first idle resource value of each candidate core in the central processor is obtained; The candidate core whose first idle resource value is greater than a preset resource threshold value is determined as the target core.

3. The method of claim 1, wherein, The plurality of candidate database tables are loaded into the preset database according to the plurality of loading channels, comprising: At least one target database table in the plurality of candidate database tables that is not loaded in a current period is determined; The current state of the plurality of loading channels in the current period is detected; If at least one target loading channel in the current state is obtained, it is judged whether the first number of the at least one target loading channel is greater than or equal to the second number of the at least one target database table; If the first number is greater than or equal to the second number, at least one target loading channel corresponding to the at least one target database table is determined, wherein the target loading channel corresponding to each target database table is different; According to the corresponding target loading channel, the corresponding target database table is loaded into the preset database.

4. The method of claim 3, wherein, Further comprising: If the first number is less than the second number, the first number of target database tables in the at least one target database table is determined; According to the at least one target loading channel, the first number of target database tables is stored into the preset database, wherein the target database table stored by each target loading channel is different.

5. The method of claim 1, wherein, Before the plurality of candidate database tables are loaded into the preset database according to the plurality of loading channels, comprising: It is determined that the data amount of each candidate database table is less than a preset threshold value.

6. A data loading apparatus, characterized by comprising: Comprise: An acquisition module is configured to obtain at least one target core in a central processor satisfying a preset loading condition in response to a loading request of a plurality of candidate database tables; A judgment module is configured to judge whether the at least one target core satisfies a preset multi-channel loading construction condition; The construction module is configured to, when the multi-channel loading construction condition is met, acquire a resource occupation value of each loading channel, acquire a second idle resource value of each target core, calculate a ratio of the resource occupation value and the second idle resource value, determine a channel number according to the ratio, and construct a plurality of groups of loading threads corresponding to the channel number in the at least one target core to implement construction of a plurality of loading channels, wherein each group of the loading threads includes a read thread and a write-out thread. The storage module is configured to load the plurality of candidate database tables to a preset database according to the plurality of loading channels.

7. The apparatus of claim 6, wherein, The acquisition module is configured to: acquire a first idle resource value of each candidate core in the central processing unit; and determine a candidate core with a first idle resource value greater than a preset resource threshold as the target core.

8. An electronic device, comprising: The electronic device includes: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the data loading method of any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is used to execute the data loading method of any one of claims 1-5.

Images