Data loading method, device and computer equipment

Abstract

The application relates to a data loading method, device and computer equipment. The method comprises the following steps: obtaining loading configuration information of to-be-processed file data; pre-processing the to-be-processed file data according to the loading configuration information, obtaining to-be-loaded file data meeting a preset loading condition; and loading the to-be-loaded file data to a target database. The method can avoid problems such as human resource waste, low efficiency, untimeliness and inaccuracy caused by manual scheduling.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data loading method, apparatus, and computer device. Background Technology

[0002] The rapid development of big data technology and the advancement of informatization have led to a gradual increase in the amount of data accumulated by human society. Downloading this data into databases for analysis can consume excessive time and money.

[0003] Currently, loading file data from the database server into the corresponding table in the database typically involves manually scheduling a loading script. The main process consists of three steps: First, within a reasonable timeframe, developers check whether the file data has arrived at the database server; second, after the file data arrives at the database server, developers manually execute the loading script to load the file data into the corresponding table in the database; finally, after loading is complete, developers check whether the number of records loaded into the database table matches the number of records in the file data on the server to determine whether the file data has been loaded correctly.

[0004] However, current file data loading technology is relatively simple and straightforward to implement, and it can be well applied when the amount of file data is small and the time when the file data arrives at the data server is relatively fixed. However, in practical applications, file data loading methods lack timeliness and accuracy, and are costly. Summary of the Invention

[0005] Therefore, it is necessary to provide a data loading method, apparatus, computer device, computer-readable storage medium, and computer program product that can automatically load file data to address the aforementioned technical problems.

[0006] Firstly, this application provides a data loading method. The method includes:

[0007] Retrieve the loading configuration information for the file data to be processed;

[0008] The file data to be processed is preprocessed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions;

[0009] Load the file data to be loaded into the target database.

[0010] In one embodiment, the loading configuration information includes loading path information; the step of preprocessing the file data to be processed according to the loading configuration information to obtain file data to be loaded that meets preset loading conditions includes:

[0011] The file data to be processed is obtained and scanned according to the loading path information;

[0012] If a signal file containing the data to be processed is detected, the data to be processed is then verified.

[0013] The data to be processed after verification is converted to a preset data format to obtain the data to be loaded.

[0014] In one embodiment, the signal file includes information about all individual file data in the file data to be processed; the verification of the file data to be processed includes:

[0015] Obtain the first file data parameter information of the file data to be processed;

[0016] The first file data parameter information and the second file data parameter information recorded in the signal file are compared to verify the file data to be processed.

[0017] In one embodiment, the loading configuration information further includes loading table structure information; the step of preprocessing the file data to be processed according to the loading configuration information to obtain file data to be loaded that meets preset loading conditions includes:

[0018] Based on the first loaded table structure information and the second table structure information of the target database, obtain table structure change information; wherein, the table structure change information includes at least one of field type change, field name change, field order change, and field length change;

[0019] Modify at least one of the field structures of the second table structure information and the loading script corresponding to the target database based on the table structure change information.

[0020] In one embodiment, modifying at least one of the field structures of the second table structure information and the loading script corresponding to the target database based on the table structure change information includes:

[0021] If the structural change information includes changes in field length or an increase in field length, then modify the structural information of the second table.

[0022] If the structural change information includes changes in the field order, then modify the field structure of the loading script corresponding to the target database;

[0023] If the structural change information includes at least one of the field type change and field name change, then modify the field structure of the second table structure information and the field structure of the loading script corresponding to the target database.

[0024] In one embodiment, loading the file data to be loaded into the target database includes:

[0025] The signal file corresponding to the data to be loaded is renamed for the first time;

[0026] The data of the file to be loaded is loaded into the target database according to the signal file after the first renaming and the loading script of the target database;

[0027] The signal file is renamed a second time to complete the loading process.

[0028] In one embodiment, the number of files to be loaded is multiple, and loading the files to be loaded into the target database further includes:

[0029] Obtain the preloading order of the data from the multiple files to be loaded;

[0030] According to the preloading order, each file data to be loaded is loaded in the following manner:

[0031] The signal file corresponding to the data to be loaded is renamed for the first time;

[0032] The data of the file to be loaded is loaded into the target database according to the signal file after the first renaming and the loading script of the target database;

[0033] The signal file is renamed a second time to complete the loading process.

[0034] In one embodiment, after loading the file data to be loaded into the target database, the method further includes:

[0035] An end signal is output based on the signal indicating that the loading of the file data to be loaded is complete. The end signal includes at least one of a warning signal, a notification signal, and a cleanup signal.

[0036] Secondly, this application also provides a data loading apparatus, the apparatus comprising:

[0037] The configuration module is used to obtain the loading configuration information for the file data to be processed;

[0038] The loading preparation module preprocesses the file data to be processed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions.

[0039] The data loading module is used to load the data of the file to be loaded into the target database.

[0040] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0041] Retrieve the loading configuration information for the file data to be processed;

[0042] The file data to be processed is preprocessed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions;

[0043] Load the file data to be loaded into the target database.

[0044] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0045] Retrieve the loading configuration information for the file data to be processed;

[0046] The file data to be processed is preprocessed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions;

[0047] Load the file data to be loaded into the target database.

[0048] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0049] Retrieve the loading configuration information for the file data to be processed;

[0050] The file data to be processed is preprocessed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions;

[0051] Load the file data to be loaded into the target database.

[0052] The aforementioned data loading method, apparatus, computer equipment, storage medium, and computer program product preprocess the file data to be processed according to the loading configuration information to obtain file data that meets preset loading conditions. This avoids loading chaos or reloading caused by large amounts of file data with different formats, uncertain arrival times at the database server, and a large number of data files, thus significantly improving loading efficiency. Automatic updates to the loading script avoid the tedious work and script errors associated with manual loading. Upon arrival of the data file data, the file data to be loaded can be immediately loaded into the target database, meeting the application's timeliness requirements for loading data file data. Furthermore, after loading is complete, the consistency between the data entered into the database and the data file data can be automatically checked, freeing developers from much tedious work. Attached Figure Description

[0053] Figure 1This is an application environment diagram of the data loading method provided in one embodiment of this application;

[0054] Figure 2 This is a flowchart illustrating a data loading method provided in one embodiment of this application;

[0055] Figure 3 This is a flowchart illustrating the preprocessing steps for file data to be processed according to one embodiment of this application;

[0056] Figure 4 This is a flowchart illustrating the steps for verifying file data to be processed according to one embodiment of this application;

[0057] Figure 5 This is a flowchart illustrating a data loading method provided in another embodiment of this application;

[0058] Figure 6 This is a flowchart illustrating the steps for modifying target database information based on table structure change information in one embodiment of this application.

[0059] Figure 7 This is a flowchart illustrating the step of loading file data to be loaded into a target database according to one embodiment of this application;

[0060] Figure 8 This is a flowchart illustrating the step of loading file data to be loaded into a target database, provided in another embodiment of this application.

[0061] Figure 9 This is a flowchart illustrating a data loading method provided in yet another embodiment of this application;

[0062] Figure 10 This is a flowchart illustrating a data loading method provided in another embodiment of this application;

[0063] Figure 11 This is a structural block diagram of a loading scheduling system provided in one embodiment of this application;

[0064] Figure 12 This is a structural block diagram of a data loading device provided in one embodiment of this application;

[0065] Figure 13 This is an internal structural diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0066] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0068] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” or “having,” etc., specify the presence of the stated feature, whole, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof.

[0069] As described in the background section, current manual script loading techniques are relatively simple and straightforward, and are well-suited for applications with small file sizes and relatively fixed arrival times on the data server. However, in real-world applications, the arrival time of file data on the database server is uncertain, and the number of file data is large, failing to meet the application's need for timely data loading. Furthermore, developers must manually schedule the loading scripts upon arrival at the server, a tedious process that cannot guarantee accuracy. After the data is loaded into the database, developers need to verify the consistency between the loaded data and the database to ensure accurate data entry. Therefore, with the increasing volume of loaded file data and the system's real-time loading requirements, existing loading techniques can no longer fully meet application needs, necessitating the development of new systems and methods to address the current situation.

[0070] The data loading method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown includes upstream systems, data exchange platforms, and automatic data file loading and scheduling systems.

[0071] In this system, upstream systems communicate with the data exchange platform via a network, and the data exchange platform communicates with the data file autoloading scheduling system via a network. Upstream systems export data files and signal files and transmit them to the data exchange platform. Upon receiving the upstream signal files, the data exchange platform transmits the data files and signal files to a database server (not shown in the diagram). The scheduling system then loads the data files from the database server into the target database, thus transforming text data files into structured data.

[0072] In one embodiment, such as Figure 2 As shown, a data loading method is provided, the method comprising:

[0073] Step S200: Obtain the loading configuration information for the file data to be processed.

[0074] The file data to be processed includes the file data transmitted from the upstream system to the database server. The loading configuration information includes all information that can uniquely identify the file data and the information required to load it into the target database. The loading configuration information may include, but is not limited to, at least one of the following: loading path information, database connection information, required loading parameters, and database scheduling parameter table; the database connection information may include the database username, server password, and other connection-required information; the required loading parameters may specifically include the batch date of the file data, the file data load parameters, database encryption information, and server directory.

[0075] Step S400: Preprocess the file data to be processed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions.

[0076] Specifically, preprocessing the file data to be processed according to the loading configuration information includes checking whether various loading conditions are met, whether all the file data to be loaded is in place, checking whether the parameters required during the loading process are met, and converting the file data to be processed according to a preset format—all prerequisites that ensure the file data to be processed is successfully loaded into the target database. The file data to be loaded includes file data that meets the preset loading conditions and is transferred from the database server to the database server.

[0077] Step S600: Load the file data to be loaded into the target database.

[0078] Specifically, loading the file data to be loaded into the target database includes the process of loading the file data from the database server into the corresponding table of the database. The target database includes the database to be loaded, which can be a target repository that organizes, stores, and manages data according to a preset data structure.

[0079] The above data loading method preprocesses the file data to be processed according to the loading configuration information to obtain file data that meets the preset loading conditions. This avoids loading chaos or reloading caused by a large number of file data with different formats, uncertain arrival times at the database server, and a large number of data files, thus significantly improving loading efficiency. Automatic updates to the loading script avoid the tedious work and script errors caused by manual loading. Upon arrival of the data file data, it can be immediately loaded into the target database, meeting the application's timeliness requirements for loading data files. Furthermore, after loading, the consistency between the data entered into the database and the data file data can be automatically checked, freeing developers from much tedious work.

[0080] In one embodiment, such as Figure 3 As shown, step S400, which involves preprocessing the file data to be processed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions, includes steps S410 to S430.

[0081] Step S410: Obtain and scan the file data to be processed according to the loading path information.

[0082] Specifically, the loading configuration information includes loading path information; the loading path information can be various paths of the required server, such as the server connection path, file data storage path, database connection path, and the storage path of logs generated during the loading process. Obtaining and scanning the file data to be processed according to the loading path information includes sending the file data to be processed from the upstream system to the server according to the server connection path, and periodically scanning all data files in this batch of file data to be processed during the sending process. Before obtaining and scanning the file data to be processed according to the loading path information, the process also includes: determining whether the file data storage path, database connection information, and server directory parameters have been established; if not, then creating the file data storage path, database connection information, or server directory parameters.

[0083] Step S420: If a signal file containing the data to be processed is detected, then the data to be processed is verified.

[0084] Specifically, the signal file is a marker file indicating that all file data in this batch has been sent from the upstream system to the database server. That is, if the database server receives the signal file, it proves that all file data in this batch has been sent. After the file data in the same batch is sent to the server, the signal file is also sent to the server, marking the completion of the batch's data transmission. The naming and content of the signal file are agreed upon in advance according to the requirements of the upstream systems to facilitate identification during subsequent system use. If the naming and content of the signal file are inconsistent with the agreement, it will affect the identification of subsequent programs. Therefore, it is necessary to verify the content recorded in the signal file, that is, to verify the content recorded in the signal file with the content of the individual file data in the file data to be processed to ensure that the signal file content is consistent and that the system functions normally afterward.

[0085] Step S430: Perform a preset data format conversion on the verified file data to obtain the file data to be loaded.

[0086] Specifically, the file data to be processed can be text data, which may include, but is not limited to, at least one of TXT file data, CSV file data, and Excel file data. The preset data format conversion includes converting the different file data formats and contents into standard file data in a unified format. The standard file data may specifically be TXT file data, and the format and content may include the number of fields, field types, field value separators, line breaks, TXT file data character set type, etc.

[0087] In the above data loading method, scanning the signal file ensures that all single file data to be loaded in this batch has been sent. By converting the data to be processed into a preset data format, the various types of text data files received can be unified into one format, so that they can be loaded into the same database normally, avoiding the situation where file data import is rejected due to different file data formats.

[0088] In one embodiment, such as Figure 4 As shown, the signal file includes information about all single file data in the file data to be processed; step S420, the verification of the file data to be processed includes steps S421 to S422.

[0089] Step S421: Obtain the first file data parameter information of the file data to be processed.

[0090] Specifically, the first file data parameter information includes the file data parameter information of all single file data in the file data to be processed. The file data parameter information may specifically include the file data size, the number of file data records, etc.

[0091] Step S422: Compare the first file data parameter information with the second file data parameter information recorded in the signal file to verify the file data to be processed.

[0092] Specifically, the second file data parameter information includes the file data parameter information of all individual files in this batch of files to be processed. This file data parameter information may include file size, number of file records, etc. By determining whether the first file data parameter information is equal to the second file data parameter information, a signal indicating whether the signal file is normal can be obtained. If the first and second file data parameter information are completely identical, the verification passes, and the file data to be loaded is obtained. If any parameter in the first and second file data parameter information is different, an alarm signal is generated to warn the user to investigate the cause, until the second and first file data parameter information are completely identical.

[0093] Optionally, during the loading process of the file data to be loaded, comparison information can also be recorded in the database file data information table of the target database. The comparison information includes file data name, file data size, number of file data records, batch file data arrival time, and whether the file data information is consistent.

[0094] In the above data loading method, the content of the signal file is verified by comparing the data parameters of the first file with those of the second file, so as to avoid affecting the recognition of subsequent programs due to inconsistencies between the naming and content of the signal file and the convention.

[0095] In one embodiment, such as Figure 5 As shown, the data loading methods include:

[0096] Step S200: Obtain the loading configuration information for the file data to be processed;

[0097] Specifically, the loading configuration information includes loading table structure information. Each file data is sent along with a planned entry table structure information that corresponds one-to-one with the file data. The loading table structure information includes table structure information that can characterize the uniqueness of file data, such as field name, field type, field length, and field order.

[0098] Step S400: Preprocess the file data to be processed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions;

[0099] Step S510: Obtain table structure change information based on the first loaded table structure information and the second table structure information of the target database. The table structure change information includes at least one of the following: field type change, field name change, field order change, and field length change.

[0100] Specifically, the first table structure information that has been issued needs to be consistent with the second table structure information of the target database in order to ensure that the data is imported normally. Therefore, before the file data is officially loaded into the database, it is necessary to compare the first table structure information with the second table structure information to see if the two table structure information are consistent.

[0101] Step S520: Modify at least one of the field structures of the second table structure information and the loading script corresponding to the target database according to the table structure change information.

[0102] Specifically, if the structure information of the first table to be loaded is inconsistent with the structure information of the second table, then at least one of the field structures of the second table structure information and the loading script corresponding to the target database is modified, and the corresponding data loading script is updated according to the latest second table structure information of the target database.

[0103] Step S600: Load the file data to be loaded into the target database.

[0104] In the above data loading method, the planned table structure to be loaded into the database (i.e., the first table structure information) must be consistent with the target database (i.e., the second table structure information) to ensure that the data is imported normally. Therefore, before the data file data is loaded into the database, it is necessary to compare the first table structure information and the second table structure information to see if the two table structures are consistent. If they are inconsistent, the second table structure information needs to be modified, and the corresponding data loading script needs to be updated according to the latest database table structure.

[0105] In one embodiment, such as Figure 6 As shown, step S520, which involves modifying at least one of the field structures of the second table structure information and the loading script corresponding to the target database according to the table structure change information, includes steps S521 to S523.

[0106] Step S521: If the structure change information includes changes in field length or an increase in field length, then modify the structure information of the second table.

[0107] Specifically, if only the fields of the first loaded table structure information of the upstream system are extended, for example, the length of a field is increased from varchar(10) to varchar(100), in this case, the system will generate a database field extension statement according to the latest first loaded table structure information, and only extend the corresponding fields of the second table structure information of the target database. If the first loaded table structure information of the upstream system adds fields and the original field order remains unchanged, the system will generate a database field addition statement according to the latest first loaded table structure information, and only add fields to the corresponding positions of the second table structure information of the target database. However, if there are other structural changes besides the change in field length, then step S523 is executed.

[0108] Step S522: If the structural change information includes changes in the field order, then modify the field structure of the loading script corresponding to the target database.

[0109] Specifically, if only the field order of the first loaded table structure information in the upstream system changes—for example, the order of field A is changed from first to second—then the field order of the second table structure information in the target database is not adjusted. Instead, the order of field B in the data loading script corresponding to field A is changed from first to second to accommodate the change in the first loaded table structure information. However, if there are other structural changes besides the change in field order, then step S523 is executed.

[0110] Step S523: If the structure change information includes at least one of the field type change and field name change, then modify the field structure of the second table structure information and the loading script corresponding to the target database.

[0111] Specifically, if the field type changes, for example, field A changes from number to varchar, then the original second table structure information of the target database is backed up, and a table creation statement is generated based on the first loaded table structure information to recreate the latest second table structure information. The field type of field B in the original data loading script, corresponding to field A, changes from number to varchar. If the field name changes, for example, field A is changed from name1 to name2, then the original second table structure information of the target database is backed up, and a table creation statement is generated based on the first loaded table structure information to recreate the latest second table structure information. The field name of field B in the original data loading script, corresponding to field A, changes from name3 to name4. If both the field order and field length change, i.e., field A is added to the first loaded table structure information and the original field order changes, an execution statement for adding the new field is generated based on the field name and field type of the newly added field in the first table structure information. Field A is added to the second table structure information of the target database first, but the field order of the second table structure information of the target database is not adjusted. Instead, the order of field B in the corresponding data loading script of the target database is adjusted. If the field order and field type / field name both change, the original second table structure information of the target database is backed up, and a table creation statement is generated based on the first loaded table structure information to recreate the table and generate the latest second table structure information. The field order and field type / field name in the original data loading script will change accordingly. The structure change information includes, but is not limited to, changes in field type, field name, field order, and field length. If the first loaded table structure information undergoes changes other than changes in field type, field name, field order, or field length, the original second table structure information of the target database is backed up, and a table creation statement is generated based on the first loaded table structure information to recreate the table and generate the latest second table structure information. The fields in the original data loading script will change accordingly.

[0112] In one embodiment, such as Figure 7 As shown, step S600, loading the file data to be loaded into the target database includes steps S611 to S613.

[0113] Step S611: Rename the signal file corresponding to the file data to be loaded for the first time.

[0114] Specifically, after preprocessing the file data to be processed, the data loading stage begins, where the file data to be loaded is loaded from the database server to the target database. First, the signal files corresponding to the file data to be loaded are renamed to mark the corresponding batch of file data being loaded, preventing the same batch of file data and signal files from being re-sent during the loading process, which could lead to loading confusion or duplicate loading.

[0115] Step S612: Load the file data to be loaded into the target database according to the signal file after the first renaming and the loading script of the target database.

[0116] Specifically, a second verification is performed on the signal files after the first renaming to identify abnormal file data. This abnormal file data refers to file data that was deleted due to misoperation during the loading or preprocessing process. The signal files after the first renaming are read line by line. Each line read represents the basic information of a single file in this batch. Based on the basic information of the read single file, the corresponding single file is searched in the directory of the batch's file data to determine if it exists. If it exists, the corresponding data loading script for the target database is invoked to load the data into the database, and the loading information is recorded in the database log table. If it does not exist, the loading process for that single file is skipped, and the loading of the next single file in the signal file is performed, with the loading information also recorded in the database log table. The loading information includes, but is not limited to, loading status, number of records, loading start time, loading end time, and other information generated during this loading process.

[0117] Step S613: Rename the signal file a second time to complete the loading.

[0118] Specifically, after loading the single-file data corresponding to the last line of each signal file, the signal file, which has been renamed once, is renamed a second time to indicate that all file data in this batch has been loaded and the loading process is complete. If file data and corresponding signal files from the same batch are sent repeatedly during the loading process, the previously sent file data from the same batch will be loaded according to the above process, and then the repeated batch data will be loaded. During this process, the original name, the first renaming, and the second renaming of the signal files will all be different.

[0119] In one embodiment, such as Figure 8 As shown, the number of files to be loaded is multiple; step S600, loading the files to be loaded into the target database includes steps S621 to S625.

[0120] Step S621: Obtain the preloading order of the multiple files to be loaded.

[0121] Specifically, the batch can be a single file or multiple files, originating from a single upstream system or multiple upstream systems. The preloading order follows the order in which preprocessing is completed; that is, the batches of files to be loaded that have completed preprocessing are loaded first. There are no dependencies or constraints between the batches.

[0122] Step S622: Load each file data to be loaded according to the preloading order in the following manner:

[0123] Step S623: Rename the signal file corresponding to the file data to be loaded for the first time.

[0124] Specifically, the signal file names corresponding to the batch of files to be loaded are renamed to distinguish the corresponding batch of files to be loaded from other unloaded signal files, and to avoid loading chaos or reloading caused by the re-delivery of data files and signal files of the same batch during the loading process.

[0125] Step S624: Load the file data to be loaded into the target database according to the signal file after the first renaming and the loading script of the target database.

[0126] Specifically, a second verification is performed on the signal files after the first renaming to identify abnormal file data. This abnormal file data refers to file data that was deleted due to misoperation during the loading or preprocessing process. The signal files after the first renaming are read line by line. Each line read represents the basic information of a single file in this batch. Based on the basic information of the read single file, the corresponding single file is searched in the directory of the batch's file data to determine if it exists. If it exists, the corresponding data loading script for the target database is invoked to load the data into the database, and the loading information is recorded in the database log table. If it does not exist, the loading process for that single file is skipped, and the loading of the next single file in the signal file is performed, with the loading information also recorded in the database log table. The loading information includes, but is not limited to, loading status, number of records, loading start time, loading end time, and other information generated during this loading process.

[0127] Step S625: Rename the signal file a second time to complete the loading.

[0128] Specifically, after loading the single-file data corresponding to the last line of each signal file, the signal file, after its first renaming, is renamed a second time to indicate that the entire batch of file data has been loaded. Loading the next preprocessed file data then begins, continuing until all signal files have been renamed, signifying that all batches of file data for all signal files have been loaded into the target database. If file data and corresponding signal files from the same batch are repeatedly sent during the loading process, the previously received batch of file data is loaded according to the above process, and then the duplicate batch of file data is loaded. During this process, the original name, first renaming, and second renaming of each batch of signal files are all different.

[0129] Specifically, during the loading process in step S600, the data in the same table can be loaded in various ways, such as appending, overwriting with the same data, or loading after clearing the table data. This can be adaptively configured. Appending means that the original data in the second table structure information of the target database remains unchanged, and the data of the newly sent file data is loaded into the second table structure information. That is, if the original data in the second table structure information of the target database is 'a', and the data in the sent file data is 'b', after appending to the second table structure information of the target database, the data in the second table structure information of the target database will be 'a+b'. The same data refers to the same file data, that is, data with the same position, size, and structure. When the upstream sends file data 'a' for the first time, 'a' is loaded into the second table structure information of the target database. At this time, the data in the second table structure information of the target database is 'a'. If, for some reason, the upstream resends the file data 'a', the new file data 'a' will be reloaded into the second table structure information of the target database. At this time, the data in the second table structure information of the target database will be 'a'. "Clearing table data and then loading" means that the original data in the second table structure of the target database is 'b', and the upstream sends file data 'a'. The original data 'b' in the second table structure of the target database is cleared, and 'a' is loaded into the second table structure of the target database. At this time, the data in the second table structure of the target database is 'a'.

[0130] In one embodiment, such as Figure 9 As shown, the data loading methods include:

[0131] Step S200: Obtain the loading configuration information for the file data to be processed;

[0132] Step S400: Preprocess the file data to be processed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions;

[0133] Step S600: Load the file data to be loaded into the target database;

[0134] Step S700: Output an end signal based on the signal indicating that the loading of the file data to be loaded is complete. The end signal includes at least one of a warning signal, a notification signal, and a cleanup signal.

[0135] Specifically, if the end signal includes a warning signal, then a preset alarm action is executed according to the warning signal, and data quality warning issues detected by the data loading and storage according to the inspection rules are recorded in the corresponding log table to remind staff of potential risks during the loading process. These data quality warning issues include data corruption, data misalignment, or data loss. The preset actions include at least one of the following: indicator light alarm, buzzer alarm, voice alarm, sending SMS, and sending email. The indicator light alarm includes a constantly lit indicator light and a flashing indicator light; the buzzer alarm includes a long beep and a short beep; the voice alarm includes "Data abnormal, please perform maintenance"; and the SMS or email notification includes alerting staff to existing data quality warning issues. This allows staff to promptly and intuitively receive alerts about data anomalies based on sound, light, SMS, or email. If the end signal includes a notification signal, then staff are notified of the completion of loading to check the storage status and record the relevant loading result log. If the end signal includes a cleanup signal, then temporary file data, temporary log information, historical data file data, and other temporary data generated during the loading process are cleaned up according to the cleanup signal.

[0136] In one embodiment, the specific steps of the data loading method are as follows: Figure 10 As shown, it can be applied to Figure 11 The loading scheduling system may include an application system layer, a data storage layer, a data file layer, and a system scheduling layer. The data storage layer mainly stores business data and inspection result data processed by ETL (which describes the process of extracting, transforming, and loading data from the source to the destination). The system scheduling layer mainly includes data loading scheduling management, inspection rule scheduling management, data supplementation scheduling management, system resource management, and inspection early warning management.

[0137] Data loading scheduling management is used to automatically monitor whether file data has arrived, realize automatic and adaptive loading of file data into the database table structure in multiple batches, and monitor the loading results. The specific data loading steps include: the loading task is started by the server through a pre-set periodic task, entering the loading preparation stage, initializing the database scheduling parameter table information, and checking whether the file data path, database connection information, and server directory exist; if not, creating the file data path, database connection information, or server directory. The system begins to send the file data to be processed and the first loading table structure information corresponding one-to-one with the individual file data to the database server until the server scans the signal file of this batch, indicating that all the file data to be processed in this batch has been sent to the database server. Preprocessing is then performed on the files to be processed, i.e., verifying the content of the signal file corresponding to the file to be processed and converting the file data to a preset format to obtain the file data to be loaded. The system determines whether the first loading table structure information of the file to be processed is consistent with the second table structure information of the target database. If they are inconsistent, the second table structure information and / or the corresponding data loading script of the target database are updated. If the structure change information includes changes in field length or an increase in field length, the second table structure information is modified. If the structure change information includes changes in field order, the field structure of the loading script corresponding to the target database is modified. If the structure change information includes at least one of changes in field type and field name, the field structure of the second table structure information and the loading script corresponding to the target database is modified. The data loading stage begins. The system reads the signal files corresponding to the pre-processed file data to be loaded and renames the signal files corresponding to the batch of file data currently being loaded to distinguish them from other unloaded signal files. For example, the signal files are renamed to "signal file.tmp" for the first time. A second verification is performed on the renamed signal files to identify abnormal file data, which is file data that was deleted due to misoperation during the loading or pre-processing process. The signal file after the first renaming is read line by line. Each line read contains the basic information of a single file in this batch. Based on the basic information of the read single file, the corresponding single file is searched in the directory of the file data in this batch. It is determined whether the corresponding single file exists. If it exists, the data loading script corresponding to the target database is called to load the data into the database and the loading information is recorded in the database log table. If it does not exist, the loading process of the single file is skipped and the loading process of the single file corresponding to the next line in the signal file is started, and the loading information is recorded in the database log table.After loading the single-file data corresponding to the last line of each signal file, the signal file, which has been renamed once, is renamed a second time. For example, "signal file.tmp" is renamed to "signal file.finish" to indicate that all file data in this batch has been loaded. Then, the loading of the next preprocessed file begins, continuing until all signal files have been renamed. This signifies that all file data for all batches of signal files has been loaded into the target database, and the data loading process is complete.

[0138] The inspection and early warning management system is used to monitor and issue warnings regarding inspection results. After all data has been loaded, it can trigger functions such as email notifications, inspection rule scheduling, early warning notifications, and system resource management. The inspection rule scheduling management system checks the quality of the loaded data according to inspection rules, such as checking for garbled characters or misalignments. Specific inspection rules can be flexibly configured in the system based on the user's inspection needs. Any data quality warnings are recorded in the corresponding log tables, and users are notified via email to confirm the warnings. Temporary files, temporary log information, and historical data files generated during the loading process are cleaned up using specific strategies configured in the system according to actual needs. Data supplementation scheduling management mainly manages the scheduling of supplementary data and upstream backflow operations; system resource management mainly monitors database resource information.

[0139] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0140] Based on the same inventive concept, this application also provides a data loading apparatus for implementing the data loading method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more data loading apparatus embodiments provided below can be found in the limitations of the data loading method described above, and will not be repeated here.

[0141] In one embodiment, such as Figure 12As shown, a data loading device is provided, including a configuration module 10, a loading preparation module 20, and a data loading module 30; the configuration module 10 is used to obtain loading configuration information of the file data to be processed; the loading preparation module 20 is used to preprocess the file data to be processed according to the loading configuration information to obtain the file data to be loaded that meets the preset loading conditions; the data loading module 30 is used to load the file data to be loaded into the target database.

[0142] In one embodiment, the loading preparation module 20 is further configured to acquire and scan the file data to be processed according to the loading path information; if a signal file of the file data to be processed is scanned, the file data to be processed is verified; and the verified file data to be processed is converted to a preset data format to obtain the file data to be loaded.

[0143] In one embodiment, the loading preparation module 20 is further configured to obtain first file data parameter information of the file data to be processed; compare the first file data parameter information with the second file data parameter information recorded in the signal file to verify the file data to be processed.

[0144] In one embodiment, the loading preparation module 20 is further configured to obtain table structure change information based on the first loading table structure information and the second table structure information of the target database; wherein, the table structure change information includes at least one of field type change, field name change, field order change, and field length change; and modify at least one of the field structures of the second table structure information and the loading script corresponding to the target database based on the table structure change information.

[0145] In one embodiment, the loading preparation module 20 is further configured to modify the second table structure information if the structure change information includes changes in field length or an increase in field length; modify the field structure of the loading script corresponding to the target database if the structure change information includes changes in field order; and modify the second table structure information and the field structure of the loading script corresponding to the target database if the structure change information includes at least one of changes in field type and field name.

[0146] In one embodiment, the data loading module 30 is further configured to: rename the signal file corresponding to the file data to be loaded for the first time; load the file data to be loaded into the target database according to the first renamed signal file and the loading script of the target database; and rename the signal file for the second time to complete the loading.

[0147] In one embodiment, the data loading module 30 is further configured to obtain the pre-loading order of multiple files to be loaded; and to load each file according to the pre-loading order in the following manner: renaming the signal file corresponding to the file to be loaded for the first time; loading the file to be loaded into the target database according to the renamed signal file and the loading script of the target database; and renaming the signal file for the second time to complete the loading.

[0148] In one embodiment, the data loading device further includes a loading end module 40, which is used to output an end signal according to the signal that the loading of the file data to be loaded is complete. The end signal includes at least one of a warning signal, a notification signal, and a cleanup signal.

[0149] Each module in the aforementioned data loading device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0150] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 13 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data loading method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad located on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0151] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0152] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0153] Retrieve the loading configuration information for the file data to be processed;

[0154] The file data to be processed is preprocessed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions;

[0155] Load the file data to be loaded into the target database.

[0156] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0157] Retrieve the loading configuration information for the file data to be processed;

[0158] The file data to be processed is preprocessed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions;

[0159] Load the file data to be loaded into the target database.

[0160] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data loading method, characterized in that, The method includes: Retrieve the loading configuration information for the file data to be processed; The file data to be processed is preprocessed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions; Load the file data to be loaded into the target database; The loading configuration information includes loading path information; the preprocessing of the file data to be processed according to the loading configuration information to obtain file data to be loaded that meets preset loading conditions includes: The file data to be processed is obtained and scanned according to the loading path information; If a signal file for the file data to be processed is detected during scanning, the file data to be processed is then verified; the signal file is a flag file that indicates that all the file data to be processed has been sent to the database server. The data to be processed after verification is converted to a preset data format to obtain the data to be loaded. The number of files to be loaded is multiple, and loading the files to be loaded into the target database includes: Obtain the preloading order of the data from the multiple files to be loaded; According to the preloading order, the loading of each file data to be loaded is completed in the following manner: the signal file corresponding to the file data to be loaded is renamed for the first time; the file data to be loaded is loaded into the target database according to the renamed signal file and the loading script of the target database; the signal file is renamed for the second time to complete the loading. The signal file includes information about all individual files in the file data to be processed; the verification of the file data to be processed includes: Obtain the first file data parameter information of the file data to be processed; The first file data parameter information and the second file data parameter information recorded in the signal file are compared to verify the file data to be processed.

2. The method according to claim 1, characterized in that, The loading configuration information also includes loading table structure information; the step of preprocessing the file data to be processed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions includes: Table structure change information is obtained based on the first loaded table structure information and the second table structure information of the target database; wherein, the table structure change information includes at least one of the following: field type change, field name change, field order change, and field length change; Modify at least one of the field structures of the second table structure information and the loading script corresponding to the target database based on the table structure change information.

3. The method according to claim 2, characterized in that, Modifying at least one of the field structures of the second table structure information and the loading script corresponding to the target database based on the table structure change information includes: If the structural change information includes changes in field length or an increase in field length, then modify the structural information of the second table. If the structural change information includes changes in the field order, then modify the field structure of the loading script corresponding to the target database; If the structural change information includes at least one of the field type change and field name change, then modify the field structure of the second table structure information and the field structure of the loading script corresponding to the target database.

4. The method according to claim 1, characterized in that, After loading the file data to be loaded into the target database, the method further includes: An end signal is output based on the signal indicating that the loading of the file data to be loaded is complete. The end signal includes at least one of a warning signal, a notification signal, and a cleanup signal.

5. A data loading device, characterized in that, The device includes: The configuration module is used to obtain the loading configuration information for the file data to be processed; The loading preparation module preprocesses the file data to be processed according to the loading configuration information to obtain file data to be loaded that meets the preset loading conditions. The data loading module is used to load the file data to be loaded into the target database; The loading configuration information includes loading path information; the loading preparation module is also used to obtain and scan the file data to be processed according to the loading path information; if a signal file of the file data to be processed is scanned, the file data to be processed is verified; the signal file is a flag file that indicates that all the file data to be processed has been sent to the database server; the verified file data to be processed is converted to a preset data format to obtain the file data to be loaded. The number of files to be loaded is multiple, and the data loading module is further used to obtain the pre-loading order of the multiple files to be loaded; according to the pre-loading order, the loading of each file to be loaded is completed in the following manner: the signal file corresponding to the file to be loaded is renamed for the first time; the file to be loaded is loaded into the target database according to the renamed signal file and the loading script of the target database; the signal file is renamed for the second time to complete the loading; The signal file includes information about all individual files in the file data to be processed; the loading preparation module is also used to obtain the first file data parameter information of the file data to be processed; and compare the first file data parameter information with the second file data parameter information recorded in the signal file to verify the file data to be processed.

6. The apparatus according to claim 5, characterized in that, The loading preparation module is further configured to obtain table structure change information based on the first loading table structure information and the second table structure information of the target database; wherein, the table structure change information includes at least one of field type change, field name change, field order change, and field length change; and modify at least one of the field structures of the second table structure information and the loading script corresponding to the target database based on the table structure change information.

7. The apparatus according to claim 6, characterized in that, The loading preparation module is further configured to modify the second table structure information if the structure change information includes changes in field length or an increase in field length; modify the field structure of the loading script corresponding to the target database if the structure change information includes changes in field order; and modify the second table structure information and the field structure of the loading script corresponding to the target database if the structure change information includes at least one of changes in field type and field name.

8. The apparatus according to claim 5, characterized in that, The data loading device further includes a loading end module, which is used to output an end signal according to the signal that the data of the file to be loaded has been loaded. The end signal includes at least one of a warning signal, a notification signal, and a cleanup signal.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

Images