Data loading method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a data loading method and device, electronic equipment, a storage medium and a program product, and particularly relates to the technical field of computers. According to the specific implementation scheme, the method comprises the steps that loading table information is obtained, wherein the loading table information indicates a data table needing to be loaded by data; a loading task corresponding to the loading table information is added to a target queue corresponding to the loading table information, and the loading task is a task for loading corresponding data into a corresponding data table; determining position information of the loading task in the target queue; and processing the loading task according to the position information. The loading tasks are added into the target queue, the loading tasks are processed according to the position information of the loading tasks in the target queue, sequential execution of the loading tasks is achieved by introducing the target queue, and the loading process processing capacity of the database is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data loading method, device, electronic equipment, storage medium and program product. Background Art

[0002] In database systems, there is often a need to import large quantities of data. These data often exist in different machines, and these data often need to be imported into the database system by starting different processes on different machines. Therefore, it is inevitable that multiple loading processes will load data into the same data table at the same time. The database system can only ensure that only one loading command is executed at the same time by locking it, avoiding conflicts and confusion caused by multiple loading commands operating at the same time.

[0003] However, there are two problems with the locking method: in a parallel state, if one process successfully locks the table, other processes must wait and compete for the lock, which can easily cause many problems such as deadlock and lock timeout, resulting in the failure of some loading processes and the inability to load; and the locking method cannot ensure that the data of different loading processes are loaded in order. Summary of the invention

[0004] The present invention provides a data loading method, device, electronic equipment, storage medium and program product to realize the sequential execution of loading tasks and improve the processing capability of the loading process.

[0005] According to one aspect of the present invention, there is provided a data loading method, comprising:

[0006] Obtaining loading table information, wherein the loading table information indicates a data table to be loaded with data;

[0007] Adding the loading task corresponding to the loading table information to the target queue corresponding to the loading table information, wherein the loading task is a task for loading the corresponding data into the corresponding data table;

[0008] Determine the position information of the loading task in the target queue;

[0009] The loading task is processed according to the position information.

[0010] According to another aspect of the present invention, there is provided a data loading device, comprising:

[0011] An acquisition module, used for acquiring loading table information, wherein the loading table information indicates a data table to be loaded with data;

[0012] An adding module, used for adding the loading task corresponding to the loading table information to the target queue corresponding to the loading table information, wherein the loading task is a task for loading the corresponding data into the corresponding data table;

[0013] A determination module, used to determine the position information of the loading task in the target queue;

[0014] A processing module is used to process the loading task according to the position information.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data loading method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data loading method described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the data loading method described in any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention obtains the loading table information, and adds the loading task corresponding to the loading table information to the target queue corresponding to the loading table information, and the loading task is a task for loading the corresponding data into the corresponding data table, then determines the position information of the loading task in the target queue, and finally processes the loading task according to the position information. The present invention adds the loading task to the target queue, and processes the loading task according to the position information of the loading task in the target queue, and realizes the sequential execution of the loading task by introducing the target queue, thereby improving the loading process processing capability of the database.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1A is a flow chart of a data loading method provided according to Embodiment 1 of the present invention;

[0025] Figure 1B is a flow chart of a method for rapid data loading provided according to Embodiment 1 of the present invention;

[0026] Figure 2 is a flowchart of a method for adding a loading task provided according to Embodiment 2 of the present invention;

[0027] Figure 3 is a structural schematic diagram of a data loading device provided according to Embodiment 3 of the present invention;

[0028] Figure 4 It is a block diagram of an electronic device provided according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1AA flowchart of a data loading method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of loading data into a database. The method can be executed by a data loading device. The data loading device can be implemented in the form of hardware and / or software. The data loading device can be configured in an electronic device. The electronic device can be a computer with data loading capabilities and a database system. Figure 1A As shown, the method includes:

[0033] S110: Obtain loading table information, where the loading table information indicates a data table into which data needs to be loaded.

[0034] In this embodiment, the database often encounters the need to import large quantities of data in scenarios such as data migration. In the database system, these data will exist in different machines, and these data often need to start different processes on different machines to import them into the database system. During data processing or data migration, the data information imported into the database is recorded and loaded into a data table. The data information is the data table information. The data table can be understood as the basic unit for storing data information in the database. Each data table has a unique table name, which is used to indicate the location of the data table in the database.

[0035] Specifically, in a database system, when a node of the database receives a loading request, it extracts the loading table related information in the loading request to obtain the loading table information. Then, the information related to data loading in the loading table information is repackaged and sent to another node related to data loading in the database, and the node parses the loading table information to finally obtain the available loading table information. At this time, the loading table information includes the data information required for data loading, such as the data table required to load the data.

[0036] For example, Figure 1B FIG. 1 is a flow chart of a method for rapid data loading according to the first embodiment of the present invention. Figure 1B When the database receives a fast loading request, that is, a request to load table T, it extracts the relevant information of table T and obtains Figure 1B The loading data packet in is sent to another node related to data loading, which unpacks the request data packet and extracts key information to finally obtain the loading table information of table T.

[0037] S120: Add the loading task corresponding to the loading table information to the target queue corresponding to the loading table information, wherein the loading task is a task for loading the corresponding data into the corresponding data table.

[0038] In this embodiment, in the database system, a loading task can be understood as a task of loading data into a target address. The target address can be a database or a data table, etc. When the database receives a loading request, it can process a loading request into a loading task, and load the data through the loading task. The target queue can be a hash linked list queue, and the target queue is used to store multiple loading tasks of the same loading table, so as to realize the sequential execution of the loading tasks.

[0039] Specifically, different loading tasks come from different loading requests. One loading table corresponds to multiple loading tasks, and multiple loading tasks are added to the target queue of the loading process management system in sequence. When the loading task corresponding to the loading table information enters the loading process management system for the first time, a corresponding target queue must be created for the loading table information. The target queue is used to store all loading tasks corresponding to the loading table information. The loading process management system can be understood as a system for managing multiple loading processes that load the same data table at the same time, to achieve the allocation and management of loading tasks. The loading process management system can use a hash linked list to implement the loading task queue, and can manage multiple loading tasks at the same time. Different loading task queues do not affect each other. By implementing such loading task queue management, the loading tasks can be executed in sequence.

[0040] For example, Figure 1B , the target queue can be a hash list queue. First, the loading process management system needs to be initialized, including initializing the memory and hash list required by the loading process management system. When a loading task enters the loading process management system, the hash value of the loading table information corresponding to the loading task can be compared with the hash value of the existing hash list queue in the loading process management system to determine whether there is a hash list queue corresponding to the loading task. Finally, the loading task is added to the end of the corresponding hash list queue, thereby completing the addition of the loading task to the hash list queue of the loading process management system.

[0041] S130: Determine the location information of the loading task in the target queue.

[0042] In this embodiment, multiple loading tasks corresponding to a loading table are added to the corresponding target queue. The position information can indicate the position of each loading task in the target queue. The multiple loading tasks are arranged according to the position information to ensure the order of the loading tasks in the target queue.

[0043] Specifically, for one of the loading tasks in the loading process management system, it is necessary to determine the position information of the loading task in the corresponding target queue according to the information of the loading task, that is, to determine whether the loading task is the first task in the target queue.

[0044] For example, Figure 1B In the loading process management system, the loading tasks are stored in the hash linked list queue in the order in which they enter the loading process management system. Each storage unit of the hash linked list queue can store the loading task, so the hash value of each storage unit of the hash linked list queue can be used as the hash value of the loading task to determine the location information of the loading task.

[0045] S140: Process the loading task according to the location information.

[0046] Specifically, for one of the loading tasks of the loading process management system, according to the location information of the loading task, it is determined whether the loading task is the first task in the target queue. If so, the loading task is directly processed; if not, the loading task is not processed first, and then the loading task is processed after it becomes the first task in the target queue. If all the loading tasks in the target queue corresponding to the loading table information are processed, it means that all the loading tasks of the data table are processed, and the loading of the data table is terminated.

[0047] For example, Figure 1B When the loading task is a newly created loading task, determine whether the newly created loading task is the first task in the hash list queue. If so, process the newly created loading task. If not, put the newly created loading task into a dormant state and process the first loading task in the current hash list queue. The newly created loading task waits to be awakened. The loading is terminated until all the loading tasks in the hash list queue corresponding to the loading table information are processed.

[0048] The technical solution of the embodiment of the present invention obtains the loading table information, and adds the loading task corresponding to the loading table information to the target queue corresponding to the loading table information, and the loading task is a task for loading the corresponding data into the corresponding data table, then determines the position information of the loading task in the target queue, and finally processes the loading task according to the position information. The present invention adds the loading task to the target queue, and processes the loading task according to the position information of the loading task in the target queue, and realizes the sequential execution of the loading task by introducing the target queue, thereby improving the loading process processing capability of the database.

[0049] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0050] In one embodiment, the obtaining of the loading table information includes:

[0051] Get the load request through the computing node;

[0052] Extracting the loading request through the computing node to obtain loading table information;

[0053] Packing the loading table information by the computing node to obtain a loading data packet;

[0054] transmitting the loading data packet to a metadata node via the computing node;

[0055] The loading data packet is parsed through the metadata node to obtain loading table information.

[0056] In this embodiment, in the database system, the computing node receives and processes various requests from the outside world, but does not store any data itself. The metadata node is located in the metadata storage module and provides metadata support for the entire database environment. Metadata is a kind of information that describes the attributes of data and is used to record the structured information of data, which only takes up a small amount of storage space. The loading data packet can be understood as a data set formed by packaging the information related to data loading in the loading table information.

[0057] Specifically, when the computing node of the database receives a loading request, the computing node extracts and parses the information in the loading request to obtain the loading table information. Then, the computing node packages the loading table information to obtain a loading data packet, which includes database information, loading request information, etc. The computing node transmits the loading data packet to the metadata node, at which time the computing node enters a dormant state and waits for the message from the metadata node. Since the amount of information contained in the loading data packet is large, the metadata node parses the loading data packet, extracts the key information in the loading data packet, and combines the key information to obtain the loading table information. The key information may include relevant information of the loading table and process information. The process here refers to the initiator of the loading command.

[0058] For example, Figure 1B , the computing node receives a request to load table T, extracts and parses the information in the loading request through the computing node, and obtains the loading table information of table T. Then the computing node packages the loading table information to obtain the loading data packet of table T, and sends it to the metadata node. The metadata node unpacks the loading data packet, extracts key information, and finally obtains the loading table information of table T.

[0059] In one embodiment, processing the loading task according to the position information includes:

[0060] When the position information indicates that the loading task is not the first task in the target queue, the task state of the loading task is set to a dormant state, and the first task in the target queue is taken out of the target queue to be awakened and executed;

[0061] In a case where the position information indicates that the loading task is the first task in the target queue, the loading task is retrieved from the target queue, awakened, and executed.

[0062] In this embodiment, the task state is the working state of the loading task, which may include a processing state and a dormant state. The processing state may be that a loading task is awakened for execution, while the dormant state is that a loading task enters a rest state and is not awakened.

[0063] Specifically, for a loading task in the target queue, when the position information of the loading task indicates that the loading task is not the first task in the target queue corresponding to the loading task, it is not yet the turn of the loading task to be executed, so the task state of the loading task is set to the dormant state, and the first loading task in the target queue corresponding to the loading task is taken out and awakened according to the position information, and the first task is processed and executed. At this time, the dormant loading task waits for the loading task before the loading task in the target queue to be taken out, awakened and executed, and then the loading task is processed. When the position information of the loading task indicates that the loading task is the first task in the target queue corresponding to the loading task, it means that the loading tasks before the loading task have been fully processed, and then the loading task is directly taken out, awakened and executed.

[0064] In one embodiment, the data loading method further includes: when the task executed in the target queue completes loading or fails to load, waking up and executing the next first task in the target queue.

[0065] Specifically, when a loading task in the target queue completes execution or fails to load, the loading task has entered the program to end loading. It is necessary to take out the next task in the target queue corresponding to the loading task according to the location information, and then take out the first loading task at this time, wake it up and execute it.

[0066] For example, after the loading task is awakened, the loading process of the entire environment is started. When the loading task is completed normally or an error occurs during the loading task, the loading task enters the end process. In this stage, other loading tasks in the same task hash list queue in the loading process management system need to be taken out, awakened and executed in sequence so that the loading tasks can be executed in sequence.

[0067] In one embodiment, the data loading method further includes: when the target task in the target queue is terminated, deleting the target task from the target queue, wherein the target task includes a task waiting to be awakened.

[0068] In this embodiment, the target task can be understood as a loading task. During the loading process, the loading task may be affected by external factors, such as power outage, network disconnection, etc., which may cause the loading task to fail to complete the loading. The target task may also be a loading task that has not yet been awakened.

[0069] Specifically, during the data loading process, if a loading task is terminated, the loading task is taken as the target task, the target task is deleted from the corresponding target queue, and the next task in the target queue corresponding to the target task is retrieved, woken up and executed according to the position information. The loading task can be terminated when the loading process has not yet been started.

[0070] For example, after the loading task enters the hash list queue, it enters the dormant state and waits to be awakened. However, before the next loading process is started, an unexpected error occurs, causing the loading task to be terminated. At this time, the loading task is used as the target task. Because the error occurs too early and the loading process has not actually started, the end process of loading will not be triggered. At this time, special processing is required to remove the target task in the loading task hash list queue (that is, the current error task). The target task should be at the end of the hash list queue at this time. Then, the first task in the hash list queue needs to be awakened, and the first task needs to be taken out, awakened, and executed.

[0071] Embodiment 2

[0072] Figure 2 This is a flow chart of a method for adding a loading task provided in the second embodiment of the present invention. This embodiment is based on the method for adding a loading task to a target queue in the above embodiment. Figure 2 As shown, the method includes:

[0073] S210: Determine whether a target queue corresponding to the loading table information exists in the loading process management system.

[0074] Specifically, the metadata node is used to determine whether there is a target queue corresponding to the loading table information in the loading process management system, and the address of the loading table information is compared with the address of the target queue already in the loading process management system. If the same address exists, there is a target queue corresponding to the loading table information.

[0075] Exemplarily, the loading process management system implements the queue by using a hash linked list. The implementation method is as follows: the hash value is calculated by using the address of the loading table information and the tasks are classified. Each hash value corresponds to a hash linked list, and the hash value records the information of the hash linked list. Through this information, all tasks corresponding to the hash value can be found at once, and each new task is added to the end of the hash linked list queue. That is, the metadata node uses the hash value to find the corresponding hash linked list, and then uses the hash linked list to find the loading task, and the hash linked list queue is used to ensure the execution order of the tasks.

[0076] S220: If yes, create a loading task corresponding to the loading table information, and add the loading task to the target queue corresponding to the loading table.

[0077] Specifically, if a target queue corresponding to the loading table information exists in the loading process management system, a loading task corresponding to the loading table information is directly created, and the loading task is added to the end of the target queue.

[0078] For example, Figure 1B If there is a hash linked list queue corresponding to the loading table information in the loading process management system, a loading task corresponding to the loading table information is created and the loading task is added to the end of the hash linked list queue.

[0079] S230: If not, create a target queue corresponding to the loading table information, create a loading task corresponding to the loading table information, and add the loading task to the target queue corresponding to the loading table.

[0080] Specifically, if the target queue corresponding to the loading table information does not exist in the loading process management system, the target queue corresponding to the loading table information is first created in the loading process management system, and then a loading task corresponding to the loading table information is created, and the loading task is added to the target queue. At this time, the loading task is the first task of the target queue.

[0081] For example, Figure 1B If the hash linked list queue corresponding to the loading table information does not exist in the loading process management system, a hash linked list queue corresponding to the loading table information is first created in the loading process management system. Then a loading task corresponding to the loading table information is created and the loading task is added to the hash linked list queue.

[0082] The technical solution of the embodiment of the present invention determines whether the target queue corresponding to the loading table information exists in the loading process management system. If so, a loading task corresponding to the loading table information is created, and the loading task is added to the target queue corresponding to the loading table. If not, the target queue corresponding to the loading table information is first created, and then a loading task corresponding to the loading table information is created, and the loading task is added to the target queue corresponding to the loading table. The method of adding the loading task to the target queue corresponding to the loading table in sequence is refined, and the purpose of sequentially executing the loading tasks is achieved.

[0083] Optionally, the target queue includes a hash linked list queue, the target queue corresponding to the loading table information includes a queue corresponding to a target hash value, and the target hash value is a hash value corresponding to the loading table information.

[0084] Specifically, the target queue is a queue for storing loading tasks in the loading process management system, which may include a hash list queue, and the metadata node puts the loading task into the target queue. The target queue corresponding to the loading table information includes a queue corresponding to the target hash value, and the hash value can be used to represent the address of a storage unit in the hash list queue. The target hash value is the address corresponding to the loading table information.

[0085] Embodiment 3

[0086] Figure 3 This is a schematic diagram of the structure of a data loading device provided by Embodiment 3 of the present invention. Figure 3 As shown, the device comprises:

[0087] An acquisition module 310 is used to acquire loading table information, where the loading table information indicates a data table into which data needs to be loaded;

[0088] An adding module 320, configured to add a loading task corresponding to the loading table information to a target queue corresponding to the loading table information, wherein the loading task is a task for loading the corresponding data into the corresponding data table;

[0089] A determination module 330, configured to determine location information of the loading task in the target queue;

[0090] The processing module 340 is used to process the loading task according to the position information.

[0091] The data loading device provided by the embodiment of the present invention first obtains the loading table information through the acquisition module, and adds the loading task corresponding to the loading table information to the target queue corresponding to the loading table information through the addition module, and the loading task is a task of loading the corresponding data into the corresponding data table, and then determines the position information of the loading task in the target queue through the determination module, and finally processes the loading task according to the position information through the processing module. Through the cooperation between the modules, the loading task is added to the target queue, and the loading task is processed according to the position information of the loading task in the target queue. By introducing the target queue, the sequential execution of the loading task is realized, and the loading process processing capacity of the database is improved.

[0092] In one embodiment, the acquisition module 310 is specifically configured to:

[0093] Get the load request through the computing node;

[0094] Extracting the loading request through the computing node to obtain loading table information;

[0095] Packing the loading table information by the computing node to obtain a loading data packet;

[0096] transmitting the loading data packet to a metadata node via the computing node;

[0097] The loading data packet is parsed through the metadata node to obtain loading table information.

[0098] In one embodiment, the adding module 320 is specifically configured to:

[0099] Determine whether a target queue corresponding to the loading table information exists in the loading process management system;

[0100] If so, create a loading task corresponding to the loading table information, and add the loading task to the target queue corresponding to the loading table;

[0101] If not, create a target queue corresponding to the loading table information, create a loading task corresponding to the loading table information, and add the loading task to the target queue corresponding to the loading table.

[0102] In one embodiment, the target queue includes a hash linked list queue, the target queue corresponding to the loading table information includes a queue corresponding to a target hash value, and the target hash value is a hash value corresponding to the loading table information.

[0103] In one embodiment, the processing module 340 is specifically configured to:

[0104] When the position information indicates that the loading task is not the first task in the target queue, the task state of the loading task is set to a dormant state, and the first task in the target queue is taken out of the target queue to be awakened and executed;

[0105] In a case where the position information indicates that the loading task is the first task in the target queue, the loading task is retrieved from the target queue, awakened, and executed.

[0106] In one embodiment, the data loading device further includes a wake-up module, which is specifically used to:

[0107] When the task executed in the target queue completes loading or fails to load, the next first task in the target queue is woken up and executed.

[0108] In one embodiment, the data loading device further includes a deletion module, which is specifically used to:

[0109] In the case where a target task in the target queue is terminated, the target task is deleted from the target queue, wherein the target task includes a task waiting to be awakened.

[0110] The data center submission device provided in the embodiment of the present invention can execute the data center submission method provided in any embodiment of the present invention, completes the submission of the data center through the mutual cooperation and collaboration between the modules, and has the functional modules and beneficial effects corresponding to the execution method.

[0111] Embodiment 4

[0112] According to an embodiment of the present invention, the present invention also provides an electronic device, a computer-readable storage medium and a computer program product.

[0113] Figure 4 It is a block diagram of an electronic device provided according to Embodiment 4 of the present invention, and the electronic device can implement the data loading method described in the embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0114] like Figure 4As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0115] A number of components in the electronic device are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0116] Processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. Processor 411 performs the various methods and processes described above, such as a data loading method.

[0117] In some embodiments, the data loading method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the data loading method described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured to perform the data loading method in any other appropriate manner (e.g., by means of firmware).

[0118] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0120] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0122] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0123] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0124] In some embodiments, the computer program product includes a computer program, and when the computer program is executed by a processor, the data loading method provided by the embodiment of the present invention is implemented.

[0125] The technical solution of the embodiment of the present invention is a data loading method, device, electronic device, storage medium and program product. By acquiring loading table information, the loading task corresponding to the loading table information is added to the target queue corresponding to the loading table information, and the loading task is a task for loading the corresponding data into the corresponding data table, and then the position information of the loading task in the target queue is determined, and finally the loading task is processed according to the position information. The present invention adds the loading task to the target queue and processes the loading task according to the position information of the loading task in the target queue. By introducing the target queue, the sequential execution of the loading task is realized, and the loading process processing capability of the database is improved.

[0126] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0127] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data loading method, characterized in that: include: Obtaining loading table information, wherein the loading table information indicates a data table to be loaded with data; Adding the loading task corresponding to the loading table information to the target queue corresponding to the loading table information, wherein the loading task is a task for loading the corresponding data into the corresponding data table; Determine the position information of the loading task in the target queue; The loading task is processed according to the position information.

2. The method according to claim 1, characterized in that The obtaining of the loading table information includes: Get the load request through the computing node; Extracting the loading request through the computing node to obtain loading table information; Packing the loading table information by the computing node to obtain a loading data packet; transmitting the loading data packet to a metadata node via the computing node; The loading data packet is parsed through the metadata node to obtain loading table information.

3. The method according to claim 1, characterized in that The adding the loading task corresponding to the loading table information to the target queue corresponding to the loading table information includes: Determine whether there is a target queue corresponding to the loading table information in the loading process management system; If yes, create a loading task corresponding to the loading table information, and add the loading task to the target queue corresponding to the loading table; If not, create a target queue corresponding to the loading table information, create a loading task corresponding to the loading table information, and add the loading task to the target queue corresponding to the loading table.

4. The method according to claim 3, characterized in that The target queue includes a hash linked list queue, the target queue corresponding to the loading table information includes a queue corresponding to a target hash value, and the target hash value is a hash value corresponding to the loading table information.

5. The method according to claim 1, characterized in that The processing of the loading task according to the position information includes: When the position information indicates that the loading task is not the first task in the target queue, the task state of the loading task is set to a dormant state, and the first task in the target queue is taken out of the target queue to be awakened and executed; In a case where the position information indicates that the loading task is the first task in the target queue, the loading task is retrieved from the target queue, awakened, and executed.

6. The method according to claim 1, characterized in that Also includes: When the task executed in the target queue completes loading or fails to load, the next first task in the target queue is woken up and executed.

7. The method according to claim 1, characterized in that Also includes: In the case where a target task in the target queue is terminated, the target task is deleted from the target queue, wherein the target task includes a task waiting to be awakened.

8. A data loading device, characterized in that: include: An acquisition module, used for acquiring loading table information, wherein the loading table information indicates a data table to be loaded with data; An adding module, used for adding the loading task corresponding to the loading table information to the target queue corresponding to the loading table information, wherein the loading task is a task for loading the corresponding data into the corresponding data table; A determination module, used to determine the position information of the loading task in the target queue; A processing module is used to process the loading task according to the location information.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data loading method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data loading method according to any one of claims 1 to 7 when executed.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the data loading method according to any one of claims 1 to 7.

Images