Data loading method and apparatus thereof

Abstract

This disclosure proposes a data loading method and apparatus, relating to the field of big data access technology. The method includes: in response to detecting a target data file to be loaded, obtaining configuration information of the target data file; determining M candidate databases to be loaded and the data table structure information corresponding to any candidate database based on the configuration information, and determining the loading order of the target data file, where M is a positive integer; and loading the target data file into the M candidate databases according to the data table structure information in the loading order. This disclosure can load various data file types according to configuration information, improving data loading efficiency, ensuring compatibility with various database types, enhancing data loading flexibility, avoiding redundant code caused by repeated development, avoiding wasted resources, and still achieving automatic loading when the upstream data file structure changes, exhibiting high compatibility and reliability.

Description

Technical Field

[0001] This disclosure relates to the field of big data access technology, and in particular to a data loading method and apparatus. Background Technology

[0002] With the rapid development of computer technology, databases are being used more and more widely, and data loading efficiency is a very important indicator of the overall performance of a database.

[0003] In related technologies, sharing or synchronizing data between systems via data files is a common method. Generally, when there is a need to load data files, the loading is customized according to the data file format of the other party. Data file loading is a common requirement for various systems. However, there are various differences in file types, file formats, and file arrival times between different systems, which may lead to repetitive and meaningless development and low efficiency. Therefore, how to improve the efficiency and flexibility of data loading and avoid redundant code caused by repeated development has become one of the important research directions. Summary of the Invention

[0004] This disclosure aims to at least partially address one of the technical problems in the related art. To this end, one objective of this disclosure is to propose a data loading method.

[0005] The second objective of this disclosure is to provide a data loading device.

[0006] The third objective of this disclosure is to propose an electronic device.

[0007] The fourth objective of this disclosure is to provide a non-transitory computer-readable storage medium.

[0008] The fifth objective of this disclosure is to provide a computer program product.

[0009] To achieve the above objectives, a data loading method is proposed in the first aspect of this disclosure, comprising:

[0010] In response to the detection of the target data file to be loaded, the configuration information of the target data file is obtained;

[0011] Based on the configuration information, determine the M candidate databases to be loaded and the data table structure information corresponding to any candidate database, and determine the loading order of the target data files, where M is a positive integer;

[0012] Based on the data table structure information, the target data file is loaded into M candidate databases in the order of loading.

[0013] In some implementations, there are multiple target data files. Determining the loading order of the target data files also includes:

[0014] Based on the configuration information, determine N data loading processes, and obtain the waiting time of the target data file and the concurrency of the data loading processes, where N is a positive integer;

[0015] The loading order of the target data files is determined based on the waiting time and the concurrency of the data loading process.

[0016] In some implementations, the target data file is loaded into M candidate databases in the order of loading, based on the data table structure information, including:

[0017] Based on the data table structure information, obtain the target table structure and the format of the target data file to be loaded.

[0018] Invoke N data loading processes to load the target data file into M candidate databases in the order of loading, based on the target data file's format and target table structure.

[0019] In some implementations, the target data file is loaded into M candidate databases in loading order according to the format of the target data file and the structure of the target table, including:

[0020] In response to the target data file being in data stream format, the target data file is directly loaded into the M candidate databases according to the target table structure and loading order; or

[0021] In response to the target data file being in compressed format, the target data file is decompressed to obtain the decompressed data file. Based on the target table structure, the decompressed data file is loaded into the M candidate databases in the loading order.

[0022] In some implementations, the data table structure information includes one or more database identifiers. Based on the data table structure information, the target data file is loaded into M candidate databases in loading order, including:

[0023] Based on one or more database identifiers, determine the target database corresponding to the target data file from M candidate databases;

[0024] Based on the data table structure information, the target data files are loaded into the target database corresponding to the target data files in the loading order.

[0025] In some implementations, the method also includes:

[0026] Obtain error information from the target data file during the data loading process;

[0027] Determine the loading result of the target data file based on the error information in the target data file;

[0028] If the loading of the target data file fails, the target data file is cleaned up.

[0029] In some implementations, it also includes:

[0030] Retrieve the job log based on the file identifier of the target data file and the loading result.

[0031] In some implementations, it also includes:

[0032] In response to the completion of loading of the target data file, obtain the first number of target data files that failed to load and the second number of target data files that were successfully loaded from the job log;

[0033] The total number of target data files is determined based on the sum of the first and second quantities.

[0034] If the ratio of the first quantity to the total quantity is greater than a preset ratio threshold, the loading is determined to have failed and an alarm is issued.

[0035] This disclosure allows for loading of various data file types based on configuration information, improving data loading efficiency, ensuring compatibility with various database types, enhancing data loading flexibility, avoiding redundant code caused by repeated development, preventing resource waste, and enabling automatic loading even when the upstream data file structure changes, resulting in high compatibility and reliability.

[0036] To achieve the above objectives, a second aspect of this disclosure provides a data loading apparatus, comprising:

[0037] The acquisition module is configured to acquire the configuration information of the target data file in response to the detection of the target data file to be loaded.

[0038] The determination module is configured to determine the M candidate databases to be loaded and the data table structure information corresponding to any candidate database based on the configuration information, and to determine the loading order of the target data files, where M is a positive integer;

[0039] The loading module is configured to load the target data file into M candidate databases in the order of loading, based on the data table structure information.

[0040] In some implementations, there are multiple target data files, and the loading module is also configured as follows:

[0041] Based on the configuration information, determine N data loading processes, and obtain the waiting time of the target data file and the concurrency of any data loading process, where N is a positive integer;

[0042] The loading order of the target data files is determined based on the waiting time and the concurrency of the data loading process.

[0043] In some implementations, loading modules is also configured as follows:

[0044] Based on the data table structure information, obtain the target table structure and the format of the target data file to be loaded.

[0045] Invoke N data loading processes to load the target data file into M candidate databases in the order of loading, based on the target data file's format and target table structure.

[0046] In some implementations, loading modules is also configured as follows:

[0047] In response to the target data file being in data stream format, the target data file is directly loaded into the M candidate databases according to the target table structure and loading order; or

[0048] In response to the target data file being in compressed format, the target data file is decompressed to obtain the decompressed data file. Based on the target table structure, the decompressed data file is loaded into the M candidate databases in the loading order.

[0049] In some implementations, the data table structure information includes one or more database identifiers, and the loading module is also configured as follows:

[0050] Based on one or more database identifiers, determine the target database corresponding to the target data file from M candidate databases;

[0051] Based on the data table structure information, the target data files are loaded into the target database corresponding to the target data files in the loading order.

[0052] In some implementations, the device also includes a cleanup module, configured to:

[0053] Obtain error information from the target data file during the data loading process;

[0054] Determine the loading result of the target data file based on the error information in the target data file;

[0055] If the loading of the target data file fails, the target data file is cleaned up.

[0056] In some implementations, the device also includes a log acquisition module, configured as follows:

[0057] Retrieve the job log based on the file identifier of the target data file and the loading result.

[0058] In some implementations, the device also includes an alarm module, configured as follows:

[0059] Get the first number of target data files that failed to load and the second number of target data files that successfully loaded from the job log;

[0060] The total number of target data files is determined based on the sum of the first and second quantities.

[0061] If the ratio of the first quantity to the total quantity is greater than a preset ratio threshold, the loading is determined to have failed and an alarm is issued.

[0062] To achieve the above objectives, a third aspect of this disclosure provides an electronic device comprising:

[0063] At least one processor; and

[0064] A memory that is communicatively connected to at least one processor; wherein,

[0065] The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform the data loading method provided in the first aspect of this disclosure.

[0066] To achieve the above objectives, a fourth aspect of this disclosure provides a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to cause a computer to perform a data loading method provided in the first aspect of this disclosure.

[0067] To achieve the above objectives, a fifth aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the data loading method provided in the first aspect of this disclosure. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating the data loading method according to the first embodiment of this disclosure;

[0069] Figure 2 This is a schematic flowchart of the data loading method according to the second embodiment of this disclosure;

[0070] Figure 3 This is a flowchart illustrating the data loading method according to the third embodiment of this disclosure;

[0071] Figure 4 This is a flowchart illustrating the data loading method according to the fourth embodiment of this disclosure;

[0072] Figure 5 This is a flowchart illustrating the data loading method according to the fifth embodiment of this disclosure;

[0073] Figure 6 This is a structural block diagram of the data loading device according to the first embodiment of this disclosure;

[0074] Figure 7 This is a schematic diagram of the structure of an electronic device according to the first embodiment of this disclosure. Detailed Implementation

[0075] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0076] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0077] The data loading method and apparatus of the present disclosure are described below with reference to the accompanying drawings.

[0078] Figure 1 This is a flowchart illustrating the data loading method of the first embodiment of this disclosure, as shown below. Figure 1 As shown, the method includes the following steps:

[0079] In step S101, in response to the detection of the target data file to be loaded, the configuration information of the target data file is obtained.

[0080] In this embodiment of the disclosure, the data file of the cache database is monitored. After the data file arrives at the cache database, it is determined that the target data file to be loaded has been detected, and the target data file and its configuration information are obtained.

[0081] Optionally, the configuration information includes data file names, downstream dependency channels, database type, loading method, and external table names. Optionally, the configuration information is pre-configured and does not require consideration of underlying resources / logic; it can be used directly after deployment.

[0082] In step S102, the M candidate databases to be loaded and the data table structure information corresponding to any candidate database are determined according to the configuration information, and the loading order of the target data files is determined, where M is a positive integer.

[0083] Optionally, the loading method of the configuration information is the type of candidate database. In this embodiment of the disclosure, the loading of data can be compatible with various data loading requirements. That is to say, the candidate database can be a distributed database, such as a distributed database (Creenplum, GP), a distributed database ORACLE, or a relational database (MySQL), a database Hive, etc.

[0084] In some implementations, there are multiple target data files. Taking M as an example, there is only one candidate database to be loaded. In other words, all target data files need to be loaded into the same type of database.

[0085] In some implementations, we take an integer M greater than 1 as an example. For any given target data file, the target data file can correspond to multiple types of databases. In other words, the target data file needs to be loaded into multiple candidate databases.

[0086] In some implementations, we take an integer M greater than 1 as an example. Different data files correspond to different types of databases. In other words, different target data files need to be loaded into different candidate databases.

[0087] Optionally, in this embodiment of the disclosure, the data table structure information includes, but is not limited to, the content of the configuration information. For example, the data table structure information may include the target data file's task identifier, file identifier, file name, compression mode, loading script, command file flag (COLFILE_FLAG), data table name, external table name, data identifier, etc. The data identifier can be the data format, such as a data stream (.dat) format, a compressed (.gz) format, etc. The compression mode can be 0 or 1 to distinguish whether the target file data is compressed. The external table is a table directly imported from the configuration information, that is, a custom table.

[0088] In some implementations, to improve data loading efficiency, multiple data loading processes can be invoked when loading multiple target data files. That is, multiple data loading processes are determined based on configuration information, and then the loading order of the target data files is determined based on the concurrency of the data loading processes. In some implementations, the loading order can be determined based on the waiting time of the target data files, with target data files with longer waiting times being loaded first. In some implementations, the loading order can be determined by combining the concurrency of the data loading processes and the waiting time of the target data files.

[0089] In step S103, the target data file is loaded into the M candidate databases according to the data table structure information in the loading order.

[0090] In some implementations, the target data file is converted into a data table structure. For any target data file, the data loading process is invoked, and the target data file is loaded into the candidate database corresponding to that target data file in the loading order according to the data table structure.

[0091] In this embodiment, the M candidate databases to be loaded and the data table structure information corresponding to any candidate database are determined according to the configuration information. The target data file is loaded into the M candidate databases according to the data table structure information. This disclosure can load various data file types according to the configuration information, improve the efficiency of data loading, be compatible with various types of databases, enhance the flexibility of data loading, avoid redundant code caused by repeated development, avoid wasting resources, and can still achieve automatic loading when the upstream data file structure changes, with high compatibility and reliability.

[0092] Figure 2 This is a flowchart illustrating the data loading method of the second embodiment of this disclosure, as shown below. Figure 2 As shown, the method includes the following steps:

[0093] In step S201, N data loading processes are determined according to the configuration information, and the waiting time of the target data file and the concurrency of the data loading processes are obtained, where N is a positive integer.

[0094] In some implementations, to improve data loading efficiency, in this embodiment of the disclosure, the arrival of the data file automatically triggers a data file loading program, and N data loading processes are determined according to configuration information. Optionally, the number of CPUs occupied by a single data loading process can be determined according to the size of the central processing unit (CPU) in the computer; that is, the concurrency of the data loading processes is determined according to the configuration information.

[0095] In some implementations, the number of target data files exceeds the concurrency of the data loading process. In such cases, the waiting time of the target data files can be obtained so that the loading order of the target data files can be determined based on the waiting time and the concurrency of the data loading process.

[0096] In step S202, the loading order of the target data files is determined based on the waiting time and the concurrency of the data loading process.

[0097] Concurrency refers to the number of data files that a loading process can load simultaneously. Higher concurrency requires higher instantaneous bandwidth. If other tasks are loading, the target data file needs to wait in a loop. Once other tasks are finished, the target data files with longer waiting times are released first. In other words, target data files with longer waiting times are loaded first, while concurrency is limited to avoid CPU overload. Each data loading process can load multiple target data files.

[0098] In step S203, N data loading processes are invoked to load the target data file into M candidate databases according to the data table structure information in the loading order.

[0099] For details regarding step S203, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0100] It should be noted that in this embodiment, the loading order is determined based on the waiting time and the concurrency of the data loading process. A streaming processing approach is used to load the target data file into M candidate databases in the specified loading order, separating downstream data computation from current data storage, and enabling real-time interaction with the candidate databases.

[0101] In this embodiment, N data loading processes are determined based on configuration information, and the waiting time of the target data file and the concurrency of the data loading processes are obtained. Based on the waiting time and the concurrency of the data loading processes, the loading order of the target data file is determined, and N data loading processes are called. The target data file is loaded into M candidate databases in the loading order according to the data table structure information. This does not occupy the downstream job batch processing time, improves the data loading efficiency and timeliness, and enhances the flexibility of data loading.

[0102] Figure 3 This is a flowchart illustrating the data loading method according to the third embodiment of this disclosure, as shown below. Figure 3 As shown, the method includes the following steps:

[0103] In step S301, in response to the detection of the target data file to be loaded, the configuration information of the target data file is obtained.

[0104] In step S302, the M candidate databases to be loaded and the data table structure information corresponding to any candidate database are determined according to the configuration information, and the loading order of the target data files is determined, where M is a positive integer.

[0105] For details regarding steps S301 to S302, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0106] In step S303, the target table structure and the format of the target data file are obtained based on the data table structure information.

[0107] Different types of candidate databases have different table structures. In some implementations, the configuration information includes custom table structures, which can be imported from the configuration information. In other implementations, if the configuration information does not include external table structures, they can be loaded by default as specified in the dynamic link library file.

[0108] Optionally, the target data file can be in various formats. In some implementations, the target data file is in a data stream format, while in others, it is in a compressed format.

[0109] In step S304, N data loading processes are invoked to load the target data file into M candidate databases in the order of loading, according to the format of the target data file and the target table structure.

[0110] In some implementations, in response to the target data file being in data stream format, the target data file is directly loaded into the M candidate databases in the loading order according to the target table structure.

[0111] In some implementations, in response to the target data file being in a compressed format, the target data file is decompressed, the decompressed data file is obtained, and the decompressed data file is loaded into M candidate databases according to the target table structure in the loading order.

[0112] In this embodiment, the target table structure and format of the target data file are obtained based on the data table structure information. The target data file is then loaded into M candidate databases according to its format and table structure. This disclosure allows for loading of various data file types based on configuration information, improving data loading efficiency, ensuring compatibility with different database types, enhancing data loading flexibility, avoiding redundant code from repeated development, preventing resource waste, and enabling automatic loading even when the upstream data file structure changes. It also demonstrates high compatibility and reliability.

[0113] Figure 4 This is a flowchart illustrating the data loading method of the fourth embodiment of this disclosure, as shown below. Figure 4 As shown, the method includes the following steps:

[0114] In step S401, in response to the detection of the target data file to be loaded, the configuration information of the target data file is obtained.

[0115] In step S402, the M candidate databases to be loaded and the data table structure information corresponding to any candidate database are determined according to the configuration information, and the loading order of the target data files is determined, where M is a positive integer.

[0116] In step S403, the target data file is loaded into the M candidate databases according to the data table structure information in the loading order.

[0117] For details regarding steps S401 to S403, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0118] In step S404, error information of the target data file is obtained during the data loading process.

[0119] In some implementations, the target data file may encounter loading errors. In such cases, other data files must wait while the target data file is being used / read. In this embodiment of the present disclosure, in order to avoid wasting time and improve data loading efficiency, error information of the target data file is obtained during the data loading process.

[0120] In step S405, the loading result of the target data file is determined based on the error information of the target data file.

[0121] In some implementations, the loading result of a target data file that fails to load is determined as a loading failure, while the loading result of a target data file that loads successfully is determined as a loading success.

[0122] In step S406, in response to the loading result of the target data file being a loading failure, the target data file is cleaned up.

[0123] To improve data loading efficiency, target data files that fail to load are cleaned up, allowing other data files to be loaded without waiting for them to finish loading or being retrieved.

[0124] To improve the accuracy of data loading, some implementations of this method further include obtaining a job log based on the file identifier of the target data file and the loading result, updating the job log according to the loading process of the target data files in the same task, and, in response to the completion of target data file loading, obtaining a first number of target data files that failed to load and a second number of target data files that were successfully loaded in the job log. After the target data files in the current task have been loaded, the total number of target data files is determined based on the sum of the first and second numbers. If the ratio of the first number to the total number is greater than a preset ratio threshold, it is determined as a loading failure, and an alarm is issued. Target data files with the same task identifier can be identified as target data files of the same task.

[0125] In this embodiment of the disclosure, the loading result of the target data file is determined based on the error information of the target data file. In response to the loading result of the target data file being a loading failure, the target data file is cleaned up, which can improve the data loading accuracy and enhance the flexibility of data loading.

[0126] Figure 5 This is a flowchart illustrating the data loading method according to the fifth embodiment of this disclosure, as shown below. Figure 5 As shown in this embodiment, in response to detecting a target data file to be loaded, the configuration information of the target data file is obtained. The concurrency of the data loading process is limited according to the configuration information. The loading order of the target data file is determined according to the waiting time of the target data file. The target database and target table structure corresponding to the target data file are determined according to the configuration information. The target table structure can be a custom table structure or a DLL table structure. The target data file is format-converted. If the target data file is in data stream format, it is directly loaded into the target database according to the target table structure. If the target data file is in compressed format, it is decompressed to obtain the decompressed data file, which is then loaded into the target database according to the target table structure. If the target data file loading is complete, a loading success message is sent to the downstream job.

[0127] This disclosure allows for loading of various data file types based on configuration information, improving data loading efficiency, ensuring compatibility with various database types, enhancing data loading flexibility, avoiding redundant code caused by repeated development, preventing resource waste, and enabling automatic loading even when the upstream data file structure changes, resulting in high compatibility and reliability.

[0128] Figure 6 This is a block diagram of a data loading apparatus according to a first embodiment of the present disclosure.

[0129] like Figure 6 As shown, the data loading device 600 of this embodiment includes:

[0130] The acquisition module 610 is configured to acquire the configuration information of the target data file in response to the detection of the target data file to be loaded.

[0131] The determination module 620 is configured to determine the M candidate databases to be loaded and the data table structure information corresponding to any candidate database based on the configuration information, and to determine the loading order of the target data files, where M is a positive integer;

[0132] Loading module 630 is configured to load the target data file into M candidate databases in loading order according to the data table structure information.

[0133] In some implementations, there are multiple target data files, and module 630 is also configured as follows:

[0134] Based on the configuration information, determine N data loading processes, and obtain the waiting time of the target data file and the concurrency of any data loading process, where N is a positive integer;

[0135] The loading order of the target data files is determined based on the waiting time and the concurrency of the data loading process.

[0136] In some implementations, loading module 630 is also configured as follows:

[0137] Based on the data table structure information, obtain the target table structure and the format of the target data file to be loaded.

[0138] Invoke N data loading processes to load the target data file into M candidate databases in the order of loading, based on the target data file's format and target table structure.

[0139] In some implementations, loading module 630 is also configured as follows:

[0140] In response to the target data file being in data stream format, the target data file is directly loaded into the M candidate databases according to the target table structure and loading order; or

[0141] In response to the target data file being in compressed format, the target data file is decompressed to obtain the decompressed data file. Based on the target table structure, the decompressed data file is loaded into the M candidate databases in the loading order.

[0142] In some implementations, the data table structure information includes one or more database identifiers, and the loading module 630 is also configured as follows:

[0143] Based on one or more database identifiers, determine the target database corresponding to the target data file from M candidate databases;

[0144] Based on the data table structure information, the target data files are loaded into the target database corresponding to the target data files in the loading order.

[0145] In some implementations, the data loading device 600 also includes a cleanup module 640, configured to:

[0146] Obtain error information from the target data file during the data loading process;

[0147] Determine the loading result of the target data file based on the error information in the target data file;

[0148] If the loading of the target data file fails, the target data file is cleaned up.

[0149] In some implementations, the data loading device 600 also includes a log acquisition module 650, configured as follows:

[0150] Retrieve the job log based on the file identifier of the target data file and the loading result.

[0151] In some implementations, the data loading device 600 also includes an alarm module 660, configured as follows:

[0152] Get the first number of target data files that failed to load and the second number of target data files that successfully loaded from the job log;

[0153] The total number of target data files is determined based on the sum of the first and second quantities.

[0154] If the ratio of the first quantity to the total quantity is greater than a preset ratio threshold, the loading is determined to have failed and an alarm is issued.

[0155] This disclosure allows for loading of various data file types based on configuration information, improving data loading efficiency, ensuring compatibility with various database types, enhancing data loading flexibility, avoiding redundant code caused by repeated development, preventing resource waste, and enabling automatic loading even when the upstream data file structure changes, resulting in high compatibility and reliability.

[0156] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. It should be noted that the electronic device can be any type of electronic device, including client devices, banking systems, and card organization systems.

[0157] like Figure 7 As shown, the above-mentioned electronic device 700 includes:

[0158] The present invention includes a memory 710 and a processor 720, and a bus 730 connecting different components (including the memory 710 and the processor 720). The memory 710 stores a computer program, and when the processor 720 executes the program, it implements the data loading method described in the embodiments of the present disclosure.

[0159] Bus 730 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0160] Electronic device 700 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 700, including volatile and non-volatile media, removable and non-removable media.

[0161] The memory 710 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 740 and / or cache memory 750. The electronic device 700 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 760 can be used to read and write non-removable, non-volatile magnetic media (…). Figure 7 Not shown; usually referred to as a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 730 via one or more data media interfaces. Memory 710 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0162] A program / utility 780 having a set (at least one) of program modules 770 may be stored in, for example, memory 710. Such program modules 770 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 770 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0163] Electronic device 700 can also communicate with one or more external devices 790 (e.g., keyboard, pointing device, display 791, etc.), and with one or more devices that enable a user to interact with the electronic device 700, and / or with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 792. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 793. Figure 7 As shown, network adapter 793 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0164] The processor 720 executes various functional applications and data processing by running programs stored in the memory 710.

[0165] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the data loading method of the present disclosure embodiment, and will not be repeated here.

[0166] In summary, in this embodiment of the present disclosure, in response to the detection of a target data file to be loaded, the configuration information of the target data file is obtained; based on the configuration information, the M candidate databases to be loaded and the data table structure information corresponding to any candidate database are determined; and based on the data table structure information, the target data file is loaded into the M candidate databases. This disclosure can load various data file types according to the configuration information, improve the efficiency of data loading, be compatible with various types of databases, enhance the flexibility of data loading, avoid redundant code caused by repeated development, avoid wasting resources, and can still achieve automatic loading when the upstream data file structure changes, with high compatibility and reliability.

[0167] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium.

[0168] When the instructions in the computer-readable storage medium are executed by the processor of the business server, the business server is able to perform the data loading method as described above. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0169] To implement the above embodiments, this disclosure also provides a computer program product, including a computer program, characterized in that the computer program, when executed by a processor, implements the data loading method as described above.

[0170] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0171] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data loading method, characterized by, include: In response to the detection of a target data file to be loaded, the configuration information of the target data file is obtained. There are multiple target data files, and the configuration information includes the data file name, downstream dependent channels, loading database type, loading method, and external table name. Based on the configuration information, determine the M candidate databases to be loaded and the data table structure information corresponding to any candidate database, where M is a positive integer; Based on the configuration information, N data loading processes are determined, and the waiting time of the target data file and the concurrency of the data loading processes are obtained, where N is a positive integer; The loading order of the target data files is determined based on the waiting time and the concurrency of the data loading process. Based on the data table structure information, obtain the target table structure and the format of the target data file for loading the target data file; The N data loading processes are invoked to load the target data file into the M candidate databases according to the loading order based on the format of the target data file and the target table structure. Alternatively, the data table structure information includes one or more database identifiers, and the target database corresponding to the target data file is determined from the M candidate databases based on the one or more database identifiers; The target data file is loaded into the target database corresponding to the target data file according to the loading order based on the data table structure information.

2. The method of claim 1, wherein, The step of loading the target data file into the M candidate databases according to the loading order based on the format of the target data file and the target table structure includes: In response to the target data file being in data stream format, the target data file is directly loaded into the M candidate databases according to the target table structure and the loading order; or In response to the fact that the target data file is in compressed format, the target data file is decompressed to obtain a decompressed data file, and the decompressed data file is loaded into the M candidate databases according to the loading order based on the target table structure.

3. The method according to claim 1, characterized in that, The method further includes: During the data loading process, obtain error information of the target data file; The loading result of the target data file is determined based on the error information of the target data file. If the loading result of the target data file is a loading failure, the target data file is cleaned up.

4. The method according to claim 3, characterized in that, Also includes: The job log is obtained based on the file identifier of the target data file and the loading result.

5. The method according to claim 4, characterized in that, Also includes: In response to the completion of loading the target data file, obtain a first number of target data files that failed to load and a second number of target data files that were successfully loaded from the job log; The total number of the target data files is determined based on the sum of the first quantity and the second quantity; If the ratio of the first quantity to the total quantity is greater than a preset ratio threshold, it is determined that the loading has failed and an alarm is issued.

6. A data loading device, characterized in that, For clients, including: The acquisition module is configured to acquire configuration information of the target data file in response to the detection of the target data file to be loaded. There are multiple target data files, and the configuration information includes the data file name, downstream dependent channels, loading database type, loading method, and external table name. The determination module is configured to determine, based on the configuration information, M candidate databases to be loaded and the data table structure information corresponding to any candidate database, where M is a positive integer; determine N data loading processes based on the configuration information, and obtain the waiting time of the target data file and the concurrency of any data loading process, where N is a positive integer; and determine the loading order of the target data file based on the waiting time and the concurrency of the data loading processes. The loading module is configured to obtain the target table structure and the format of the target data file based on the data table structure information; and to call the N data loading processes to load the target data file into the M candidate databases in the loading order according to the format and target table structure of the target data file. Alternatively, the data table structure information may include one or more database identifiers, and the loading module may be further configured to: Based on the one or more database identifiers, determine the target database corresponding to the target data file from the M candidate databases; The target data file is loaded into the target database corresponding to the target data file according to the loading order based on the data table structure information.

7. The apparatus according to claim 6, characterized in that, The loading module is also configured to: Since the target data file is in data stream format, the target data file is directly loaded into the M candidate databases according to the target table structure and the loading order. or In response to the fact that the target data file is in compressed format, the target data file is decompressed to obtain a decompressed data file, and the decompressed data file is loaded into the M candidate databases according to the loading order based on the target table structure.

8. The apparatus according to claim 6, characterized in that, The device also includes a cleaning module configured to: During the data loading process, obtain error information of the target data file; The loading result of the target data file is determined based on the error information of the target data file. If the loading result of the target data file is a loading failure, the target data file is cleaned up.

9. The apparatus according to claim 8, characterized in that, The device further includes a log acquisition module, configured to: The job log is obtained based on the file identifier of the target data file and the loading result.

10. The apparatus according to claim 9, characterized in that, The device also includes an alarm module, configured to: Obtain a first number of target data files that failed to load and a second number of target data files that successfully loaded from the job log; The total number of the target data files is determined based on the sum of the first quantity and the second quantity; If the ratio of the first quantity to the total quantity is greater than a preset ratio threshold, it is determined that the loading has failed and an alarm is issued.

11. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-5.

Images