Data file loading method and apparatus

Abstract

The present specification relates to the technical field of data file loading, and discloses a data file loading method and device, wherein the method comprises: obtaining a target data file to be processed, a corresponding target format file and a target data quantity file from a data server; the target format file is a target format type generated according to a preset format specification for fields in the target data file; the target data quantity file is used to represent a data quantity contained in the target data file; the target data file and the corresponding target format file and target data quantity file are checked; and in the case that the target data file and the corresponding target format file and target data quantity file are all checked, the target data file is loaded into a target database. The above method can load data files of different format types into a database through the same interface.

Description

Technical Field

[0001] This specification relates to the field of data file loading technology, and in particular to a data file loading method and apparatus. Background Technology

[0002] Currently, different interface templates are typically configured for loading different data files. If the source data format changes due to system reasons, the data transmitted externally will change, leading to problems for data users who haven't adjusted their interfaces in time. Different external systems using different data sources and resulting in different data formats will also affect downstream data processing. Furthermore, as external data dependencies increase, the number of jobs also increases. This necessitates not only developing different jobs for different external data but also maintaining more jobs in the scheduling tool to meet business needs.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This specification provides a data file loading method and apparatus to solve the problem in the prior art where data loading errors or failures occur due to different source data formats and failure to adjust the interface in a timely manner.

[0005] This specification provides a data file loading method, applied to an application server, the method comprising:

[0006] The target data file to be processed, along with the corresponding target format file and target data volume file, are obtained from the data server. The target format file is a target format type generated from the fields in the target data file according to a preset format specification. The target data volume file is used to represent the amount of data contained in the target data file.

[0007] Verify the target data file, as well as the corresponding target format file and target data volume file;

[0008] If the target data file, the corresponding target format file, and the target data volume file all pass verification, the target data file will be loaded into the target database.

[0009] In one embodiment, obtaining the target data file to be processed, along with the corresponding target format file and target data volume file, from a data server includes:

[0010] The system receives a job allocation table sent by a scheduling server; the job allocation table includes identifiers of data files to be loaded; the scheduling server is used to obtain a file list from a data server, the file list includes the processing status of multiple data files, and the scheduling server is also used to perform job scheduling based on the processing status of the multiple data files and generate a job allocation table.

[0011] Based on the data file identifier in the job allocation table, the data file to be processed, along with the corresponding format file and data volume file, is obtained from the data server.

[0012] In one embodiment, the data server is used to receive data files, corresponding format files, and data volume files transmitted from the source server; the data server is also used to check the transmission progress of the data files, corresponding format files, and data volume files, generate a message based on the transmission progress, and send the message to the distributed publish-subscribe messaging system.

[0013] Accordingly, based on the data file identifier in the job allocation table, the data file to be processed, along with its corresponding format file and data volume file, is obtained from the data server, including:

[0014] Based on the data file identifier in the job allocation table, subscribe to the message corresponding to the data file identifier from the distributed publish-subscribe messaging system;

[0015] If the message corresponding to the data file identifier indicates successful transmission, the data file corresponding to the data file identifier, as well as the corresponding format file and data volume file, are retrieved from the data server.

[0016] In one embodiment, the verification of the target data file, the corresponding target format file, and the target data volume file includes:

[0017] The validity of the target data file is verified;

[0018] If the validity of the target data file passes the verification, determine whether the amount of data in the received target data file is consistent with the amount of data in the target data file.

[0019] If the amount of data in the received target data file is consistent with the amount of data in the target data file, determine whether the target format type in the target format file and the field format type involved in the target database meet the preset conditions;

[0020] If the target format type in the target format file and the field format type in the target database meet the preset conditions, the target data file and the corresponding target format file and target data volume file are determined to have been successfully verified.

[0021] In one embodiment, determining whether a preset condition is met between the target format type in the target format file and the field format types involved in the target database includes:

[0022] Determine whether the target format type in the target format file is the same as the field format type involved in the target database;

[0023] If it is determined that the target format type in the target format file is the same as the field format type involved in the target database, then it is determined that the target format type in the target format file and the field format type involved in the target database meet a preset condition.

[0024] In one embodiment, after determining whether the target format type in the target format file is the same as the field format type involved in the target database, the method further includes:

[0025] If it is determined that the target format type in the target format file is different from the field format type involved in the target database, it is determined whether the target format type in the target format file is compatible with the field format type involved in the target database;

[0026] If it is determined that the target format type in the target format file is compatible with the field format types involved in the target database, then it is determined that the target format type in the target format file and the field format types involved in the target database meet a preset condition.

[0027] In one embodiment, loading the target data file into the target database includes:

[0028] Read the configuration file; the configuration file includes the job identifier, the amount of data to be read, the start processing time, and the processing interval;

[0029] Based on the configuration file, the target data file is loaded into the target database.

[0030] In one embodiment, loading the target data file into the target database includes:

[0031] The target data file is loaded into a temporary table in the target database according to the format type in the target format file;

[0032] Read the table-level field details corresponding to the target data file;

[0033] Based on the table-level field details and the preset mapping function, a statement to be executed is generated; the preset mapping function is used to map fields in the temporary table to fields in the main table of the target database.

[0034] Execute the statement to be executed to update the data in the temporary table to the main table.

[0035] In one embodiment, after executing the statement to be executed to update the data in the temporary table to the main table, the method further includes:

[0036] Clear the temporary table; and / or

[0037] Collect statistical information from the positive table.

[0038] In one embodiment, after loading the target data file into the target database, the method further includes:

[0039] Determine whether the target data file has been successfully loaded into the target database;

[0040] If the target data file fails to be loaded into the target database, the process of loading the target data file into the target database is repeated until it is successfully loaded.

[0041] This specification also provides a data file loading device, applied to an application server, the device comprising:

[0042] The acquisition module is used to acquire the target data file to be processed, as well as the corresponding target format file and target data volume file from the data server; the target format file is a target format type generated according to a preset format specification for the fields in the target data file; the target data volume file is used to characterize the amount of data contained in the target data file.

[0043] The verification module is used to verify the target data file, as well as the corresponding target format file and target data volume file;

[0044] The loading module is used to load the target data file into the target database after the target data file, the corresponding target format file, and the target data volume file have all passed verification.

[0045] In one embodiment, the loading module specifically includes:

[0046] A loading unit is used to load the target data file into a temporary table in the target database according to the format type in the target format file;

[0047] The reading unit is used to read the table-level field detail information corresponding to the target data file;

[0048] The generation unit is used to generate a statement to be executed based on the table-level field details and a preset mapping function; the preset mapping function is used to map fields in the temporary table to fields in the main table of the target database.

[0049] An execution unit is used to execute the statement to be executed, so as to update the data in the temporary table to the main table accordingly.

[0050] This specification also provides a computer device, including a processor and a memory for storing processor-executable instructions, wherein the processor executes the instructions to implement the steps of the data file loading method described in any of the above embodiments.

[0051] This specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the data file loading method described in any of the above embodiments.

[0052] This specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the data file loading method described in any of the above embodiments.

[0053] This specification provides a data file loading method. An application server can obtain a target data file to be processed, along with a corresponding target format file and a target data volume file, from a data server. The target format file is a target format type generated according to a preset format specification for the fields in the target data file. The target data volume file represents the amount of data contained in the target data file. The target data file, its corresponding target format file, and target data volume file can be validated. If all validations pass, the target data file is loaded into the target database. In this scheme, the target format type generated according to a unified preset format specification for the field formats in the data file eliminates the need to configure different interface modules for loading different types of data files. This allows the data loading operation to support more diverse data sources, moving beyond a fixed interface. Furthermore, in the above scheme, when validating the data file, the corresponding data volume file and format file are simultaneously validated. Validating the data volume file ensures the integrity and accuracy of the received data file, while validating the format file ensures that the data file's format type matches the type in the target database. It also allows for adjustments to the field types in the target database based on the format type in the format file, going beyond fixed database table information and better responding to changes in upstream data and data sources. The data file is only loaded into the target database after all three (data file, format file, and data volume file) have been successfully validated, improving the success rate and accuracy of file loading. This data loading method supports different data sources, allows for on-the-spot field modification, enables dynamic loading configuration, and completes the data loading function. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of this specification and form part of it, do not constitute a limitation thereof. In the drawings:

[0055] Figure 1 A schematic diagram illustrating an application scenario of a data file loading method according to one embodiment of this specification is shown;

[0056] Figure 2 A flowchart of a data file loading method according to one embodiment of this specification is shown;

[0057] Figure 3 A flowchart of a data file loading method according to one embodiment of this specification is shown;

[0058] Figure 4 A schematic diagram of a data file loading device according to one embodiment of this specification is shown;

[0059] Figure 5 A schematic diagram of a computer device according to one embodiment of this specification is shown. Detailed Implementation

[0060] The principles and spirit of this specification will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this specification, and are not intended to limit the scope of this specification in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0061] Those skilled in the art will recognize that the embodiments described in this specification can be implemented as a system, apparatus, method, or computer program product. Therefore, the disclosure of this specification can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0062] This specification provides a data file loading method through its embodiments. Figure 1 A schematic diagram illustrating an application scenario of a data file loading method according to one embodiment of this specification is shown. For example... Figure 1 As shown, each of the multiple source servers can send a data file, a format file, and a data volume file to the data server. The format file is a target format type generated according to a preset format specification for the fields in the data file. The data volume file is used to represent the amount of data contained in the data file.

[0063] The data server can monitor the data file transfer progress from the source server in real time and generate corresponding messages based on the transfer progress, which are then sent to the distributed publish-subscribe messaging system. In one embodiment, the distributed publish-subscribe messaging system can be a Kafka messaging system.

[0064] In one embodiment, the scheduling server can obtain a file list from the data server. The file list may include the processing status of each data file across multiple data files. The processing status may include unprocessed, processing, and processed. The scheduling server can schedule application servers to execute data file loading jobs based on the processing status in the file list. In one embodiment, the scheduling server can generate a job allocation table. The job allocation table records the job information allocated to each application server.

[0065] In one embodiment, the application server can obtain a job allocation table from the scheduling server and subscribe to Kafka messages based on the job information in the job allocation table to obtain the transmission progress of the data file to be loaded. After the data file is transmitted to the data server, the application server can obtain the data file to be loaded, along with its corresponding format file and data volume file, from the data server. The application server can verify the obtained data file, format file, and data volume file. Then, the application server can read the configuration file, which may contain the job identifier, the amount of data to be read at one time, the start processing time, and the processing interval. The application server can then load the verified data file into a temporary table in the target database based on the information in the configuration file.

[0066] After all the data in the data file has been written to the temporary table, the application server can generate the corresponding statement to be executed using a shared preset mapping function, information about the source and target tables, and the execution date. Executing the statement will update the data in the temporary table to the main table of the target database.

[0067] In some embodiments of this specification, after successfully updating the main table, the application server can clear the temporary table and collect statistics about the main table. These statistics may include information such as the table's data volume, indexes, and sequence.

[0068] In some embodiments of this specification, if the data in the data file is not successfully loaded into the temporary table or not successfully updated into the main table, the application server may reload or update until the data in the data file is successfully updated into the main table.

[0069] In some embodiments of this specification, in the event of a verification failure, the application server can generate a reason for the failure and send that reason back to the corresponding source server, scheduling server, or data server. In the event of a format file verification failure, the format specification can be updated according to the new field format, and a new format file can be generated based on the updated format specification and reloaded.

[0070] Figure 2A flowchart of a data file loading method according to one embodiment of this specification is shown. While this specification provides method operation steps or apparatus structures as illustrated in the following embodiments or figures, more or fewer operation steps or module units may be included in the method or apparatus based on conventional or non-inventive effort. In steps or structures where there is no logically necessary causal relationship, the execution order of these steps or the module structure of the apparatus is not limited to the execution order or module structure described in the embodiments and figures of this specification. When the method or module structure is applied in a practical device or terminal product, it can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed processing environment) according to the method or module structure shown in the embodiments or figures.

[0071] Specifically, such as Figure 2 As shown, a data file loading method provided in one embodiment of this specification may include the following steps:

[0072] Step S201: Obtain the target data file to be processed, the corresponding target format file, and the target data volume file from the data server; the target format file is a target format type generated according to a preset format specification for the fields in the target data file; the target data volume file is used to characterize the amount of data contained in the target data file.

[0073] The embodiments in this specification can be applied to an application server. The application server can obtain the target data file to be processed, along with the corresponding target format file and target data volume file, from the data server. The data server can receive the data file, data volume file, and format file sent by the source server.

[0074] The target format file is a target format type generated for the fields in the target data file according to a preset format specification. In one embodiment, the target format file can be a unified format file generated by abstracting the formats corresponding to multiple field types in the target data file through a common configuration. Different data source formats can be maintained through parameters using the target format file. For example, regardless of the data source, the character type can be set to c..n according to a convention, where n represents length and c represents character type; other types are similar.

[0075] A target data volume file can be used to characterize the amount of data contained in the target data file. In one embodiment, the data volume may include information such as the data file's generation time, file size, and number of lines.

[0076] Step S202: Verify the target data file, the corresponding target format file, and the target data volume file.

[0077] Step S203: If the target data file, the corresponding target format file, and the target data volume file have all passed verification, the target data file is loaded into the target database.

[0078] After obtaining the target data file, its corresponding target format file, and its target data volume file, the application server can verify these files. If all verifications of the target data file, its corresponding target format file, and its target data volume file pass, the target data file is loaded into the target database.

[0079] In the above embodiments, the target format type can be generated according to a unified preset format specification for the field format in the data file. This eliminates the need to configure different interface modules for loading different types of data files, allowing the data loading job to support more diverse data sources and no longer be limited to a fixed interface. Furthermore, when validating the data file, the corresponding data volume file and format file are simultaneously validated. Validating the data volume file ensures the integrity and accuracy of the received data file, while validating the format file ensures that the format type of the data file matches the type in the target database. It also allows for adjustments to the field types in the target database based on the format type in the format file, going beyond fixed database table information and better responding to changes in upstream data and data sources. The data file is only loaded into the target database after all three (data file, format file, and data volume file) have been successfully validated, improving the success rate and accuracy of file loading. This data loading method supports different data sources, allows for on-the-spot modification of fields, enables dynamic loading configuration, and completes the data loading function.

[0080] In some embodiments of this specification, obtaining the target data file to be processed, along with the corresponding target format file and target data volume file, from the data server may include: receiving a job allocation table sent by a scheduling server; the job allocation table includes identifiers of data files to be loaded; the scheduling server is used to obtain a file list from the data server, the file list including the processing status of multiple data files, and the scheduling server is also used to perform job scheduling based on the processing status of the multiple data files to generate a job allocation table; and based on the data file identifiers in the job allocation table, obtaining the data file to be processed, along with the corresponding format file and data volume file, from the data server.

[0081] Specifically, the scheduling server can obtain a file list from the data server. The file list may include the processing status of multiple data files. The processing status can include unprocessed, processing, and processed. The scheduling server can schedule jobs for application servers based on the file processing status in the file list, generating a job allocation table. In one embodiment, the scheduling server can allocate jobs based on the job information to be processed and parameters such as the load and performance of each application server, generating a job allocation table. The application server can receive the job allocation table from the scheduling server and obtain the data files to be processed, along with their corresponding format files and data volume files, from the data server according to the data file identifiers recorded in the job allocation table. Through this method, job scheduling can be performed by the scheduling server, achieving load balancing among multiple application servers and improving data loading efficiency.

[0082] In some embodiments of this specification, the data server can be used to receive data files, corresponding format files, and data volume files transmitted from the source server; the data server is also used to check the transmission progress of the data files, corresponding format files, and data volume files, generate a message based on the transmission progress, and send the message to the distributed publish-subscribe messaging system; correspondingly, obtaining the data file to be processed, the corresponding format file, and data volume file from the data server based on the data file identifier in the job allocation table may include: subscribing to the message corresponding to the data file identifier from the distributed publish-subscribe messaging system based on the data file identifier in the job allocation table; and, if the message corresponding to the data file identifier indicates successful transmission, obtaining the data file, the corresponding format file, and data volume file from the data server.

[0083] Specifically, the data server can receive data files, along with corresponding format files and data volume files, transmitted from at least one source server. The data server can also check the transmission progress of the data files, format files, and data volume files transmitted from the source server. In some embodiments, when there are many dependent data files, parameter switches can be used to control whether to check the arrival status of certain files. For example, the data server can store a file list, which may include data file identifiers, dates, statuses, and switches. For data files with the switch on, the transmission progress of that file needs to be checked; for data files with the switch off, the transmission progress of that file does not need to be checked. The parameter switches in the file list allow selection of which data files to check, facilitating subsequent processing.

[0084] The data server can generate a corresponding message based on the transmission progress of the data file, its corresponding format file, and data volume file. The data server can also send this message to a distributed publish-subscribe messaging system. In some embodiments, the distributed publish-subscribe messaging system can be a Kafka messaging system. The application server can subscribe to messages from the distributed publish-subscribe messaging system to obtain real-time information about the transmission progress of the data file to be processed. When a message indicates that the data file and its corresponding format file and data volume file have been successfully transmitted, the application server can retrieve the corresponding data file, its corresponding format file, and its data volume file from the data server. In the above embodiments, the distributed publish-subscribe messaging system enables buffering between the data server and the application server, achieves asynchronous message processing, improves data loading efficiency, and can handle high-concurrency data loading requests.

[0085] In some embodiments of this specification, verifying the target data file and its corresponding target format file and target data volume file may include: verifying the legality of the target data file; if the legality verification of the target data file passes, determining whether the received data volume of the target data file is consistent with the data volume in the target data file; if the received data volume of the target data file is consistent with the data volume in the target data file, determining whether the target format type in the target format file and the field format type involved in the target database meet a preset condition; if the target format type in the target format file and the field format type involved in the target database meet the preset condition, determining that the target data file and its corresponding target format file and target data volume file have been successfully verified.

[0086] Specifically, the application server can perform a validity check on the target data file. If the validity check passes, it can then verify the target data volume file. It can determine whether the received data volume of the target data file matches the data volume in the target data file. If the received data volume matches the data volume in the target data file, it can then verify the target format file. The application server can determine whether the target format type in the target format file meets a preset condition with the field format types in the target database. Before determining whether the target format type in the target format file meets the preset condition, it can also determine whether the target format type in the target format file matches the format type in the received data file. If they match, it can then determine whether the target format type in the target format file meets the preset condition with the field format types in the target database. If the target format type in the target format file meets the preset condition with the field format types in the target database, the application server can determine that the target data file, the corresponding target format file, and the target data volume file have been successfully verified.

[0087] In this embodiment, the verification is performed in the order of target data file, target data volume file, and target format file. Those skilled in the art will understand that the verification order can be changed, as long as all files are successfully verified.

[0088] In some embodiments of this specification, determining whether a preset condition is met between the target format type in the target format file and the field format type involved in the target database includes: determining whether the target format type in the target format file is the same as the field format type involved in the target database; if it is determined that the target format type in the target format file is the same as the field format type involved in the target database, then determining that the preset condition is met between the target format type in the target format file and the field format type involved in the target database. The target format file can be validated using the above method.

[0089] In some embodiments of this specification, after determining whether the target format type in the target format file is the same as the field format type involved in the target database, the method further includes: if it is determined that the target format type in the target format file is different from the field format type involved in the target database, determining whether the target format type in the target format file is compatible with the field format type involved in the target database; if it is determined that the target format type in the target format file is compatible with the field format type involved in the target database, determining that the target format type in the target format file and the field format type involved in the target database satisfy a preset condition. Through the above method, the target format file can be verified.

[0090] If it is determined that the target format type in the target format file is incompatible with the field format types involved in the target database, an alarm message can be generated. Based on the alarm message, a new preset format specification can be generated, thereby generating a new format file.

[0091] If it is determined that the target format type in the target format file is compatible with the field format type designed in the target database, the corresponding field format type in the target database can be updated.

[0092] In some embodiments of this specification, loading the target data file into the target database includes: reading a configuration file; the configuration file includes a job identifier, the amount of data to be read, the start processing time, and the processing interval; and loading the target data file into the target database based on the configuration file.

[0093] Specifically, job configuration can be performed on the application server to generate a configuration file. The configuration file can include the job identifier, the amount of data to be read at one time, the start processing time, and the processing interval. The application server can then read the information in the configuration file and load the target data file into the target database based on that information.

[0094] The configuration file can be in the following format:

[0095] <jobid>=<Amount of data read at one time>_<Start processing time>_<Processing interval per record>_

[0096] in, <jobid>The job ID (I) is used to uniquely identify each job (1 <= JobId <= K), and can be obtained when each job starts. <Data read at one time> refers to the amount of data that the automatically called processing machine reads into the scheduling task table at one time for start time arrangement. <Start processing time> refers to the start time of the daily scheduled task processing; this can be adjusted as needed in case of changes, anomalies, or other situations. <Processing interval per record> refers to the processing interval for each record in the scheduling task table; a shorter interval can be configured if the system load is high and the processing time is fast, and vice versa.

[0097] The above embodiments provide a unified and flexibly configurable solution, allowing jobs on multiple application servers to effectively cooperate with the template database after startup by reading configuration files.

[0098] In some embodiments of this specification, loading the target data file into the target database includes: loading the target data file into a temporary table in the target database according to the format type in the target format file; reading the table-level field details corresponding to the target data file; generating an execution statement based on the table-level field details and a preset mapping function; the preset mapping function is used to map the fields in the temporary table to the fields in the main table of the target database; and executing the execution statement to update the data in the temporary table to the main table accordingly.

[0099] Specifically, the application server can load the target data file into a temporary table in the target database according to the format type in the target format file. After all the data is loaded into the temporary table, the application server can generate an execution statement based on a preset mapping function and table-level field details. The table-level field details can include the source table name, the target table name, and the execution date. The application server can also obtain table-level field details by acquiring system views corresponding to different data sources. An execution statement can be generated based on the parameters in the table-level field details and the preset mapping function. The preset mapping function can be a common function used to map fields in the temporary table to fields in the main table of the target database. The application server can execute the execution statement to update the data in the temporary table to the main table accordingly. In one embodiment, the execution statement can be an SQL statement. Through this method, the preset mapping function obtains system views or table-level field details corresponding to different data sources, and dynamically generates the execution statement by concatenating the field details. This eliminates the need for static configuration by enumerating all field information for each table individually, thus improving operational efficiency.

[0100] In some embodiments of this specification, after executing the statement to be executed to update the data in the temporary table to the main table, the process may further include: clearing the temporary table; and / or collecting statistical information from the main table.

[0101] After updating the data in the temporary table to the corresponding information in the main table, the application server can clear the temporary table to save storage space in the target database. The application server can also collect statistical information from the main table. This statistical information may include table data volume, indexes, sequence information, etc.

[0102] In some embodiments of this specification, after loading the target data file into the target database, the method may further include: determining whether the target data file has been successfully loaded into the target database; and if the target data file has not been successfully loaded into the target database, re-executing the loading of the target data file into the target database until the loading is successful.

[0103] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. For details, please refer to the foregoing descriptions of the relevant processing embodiments; they will not be repeated here.

[0104] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0105] The above method will be described below with reference to a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustration of this specification and does not constitute an improper limitation of this specification.

[0106] This specific embodiment provides a data loading method. This method can abstract the formats corresponding to all types through a general configuration, generate a unified format file, maintain different data sources through parameters, convert the unified format into the data format corresponding to each data source, and adjust the field information of the database table according to the data format file each time, not limited to fixed database table information. This can better cope with changes in upstream data and data sources. This solution also provides a general approach to address the problem of increased workloads caused by increased dependencies.

[0107] Please refer to Figure 3 The flowchart of the data loading method in this specific embodiment is shown. Figure 3 As shown, the method in this embodiment may include the following processing flow.

[0108] Step 1: File arrival check.

[0109] Before loading or processing a file, it's necessary to determine if the data files from external systems have already been transferred to the local machine. When there are many dependent external files, parameter switches are needed to control file arrival status for subsequent processing. Subscribe to Kafka messages to obtain file transfer information and distribute the data across different machines for processing according to different jobs.

[0110] Step 2: Document inspection.

[0111] The file check has been changed from checking only data files to checking three files: format files, data volume files, and finally, data files.

[0112] The format file specifies the format type for all fields. For example, the character type is set to c..n regardless of the data source, where n represents the length and c represents the character type. Other types are similar.

[0113] The data file records the file's creation time, file size, and number of lines.

[0114] The data file contains actual data, with each information element separated by |@|.

[0115] Upon confirmation of file arrival, the process changes from directly loading the data file as agreed to first performing a verification. The data volume file is retrieved and checked. It is then determined whether the size and number of lines of the received file match the size in the data volume file; if they do not match, a failure message is returned.

[0116] Perform a second format file check. Verify that the field types in the tables of the target database are consistent with the transmitted format file.

[0117] If there is a discrepancy, determine if the new format file is compatible with the old one. If it is compatible, first update the field types of the temporary table in the target database. After a normal update, update the field types in the main table.

[0118] Next, the data files are checked.

[0119] Step 3: Load the file.

[0120] After the preliminary steps are completed, the data is entered into the database according to the format file.

[0121] Step 4: Update the data.

[0122] Once all the data has been stored in the temporary table, a common function and the passed parameters are used to generate SQL statements corresponding to different data sources, which can meet various business needs without adding new jobs.

[0123] The common functions retrieve system views or table-level field details from different data sources. By concatenating the field details, the SQL is dynamically generated, eliminating the need for static configuration by enumerating all field information for each table individually.

[0124] Step 5: Clear the temporary table.

[0125] Step 6: Collect statistical information from the positive table.

[0126] In this embodiment, before the machine processing is automatically invoked, a configuration file can be read, which has the following format:

[0127] <jobid>=<Amount of data read at one time>_<Start processing time>_<Processing interval per record>_

[0128] in, <jobid>The job ID (I) is used to uniquely identify each job (1 <= JobId <= K), and can be obtained when each job starts. <Data read at one time> refers to the amount of data that the automatically called processing machine reads into the scheduling task table at one time for start time arrangement. <Start processing time> refers to the start time of the daily scheduled task processing; this can be adjusted as needed in case of changes, anomalies, or other situations. <Processing interval per record> refers to the processing interval for each record in the scheduling task table; a shorter interval can be configured if the system load is high and the processing time is fast, and vice versa.

[0129] Jobs on multiple servers process data from a single database, establishing a many-to-one relationship between jobs and the database. A unified standard is used to generate format files for different data sources and formats, allowing conversion to the target database's type without being restricted by data source or database type. For different data requirements, a new set of jobs is not added for unified checking, loading, and database entry; instead, the approved standard is used for loading, and a unified function is used for data updates. As described in the workflow, this solution supports repeated execution; if a job is abnormally interrupted at any step of the process, it can be manually re-executed without worrying about dirty data or data loss caused by improper operation.

[0130] The methods described in the above embodiments provide a unified and flexibly configurable solution, allowing jobs on multiple servers to effectively cooperate with the database after startup by reading configuration files. These embodiments also enable jobs to support more diverse data sources, moving beyond fixed interfaces. Furthermore, they support re-execution schemes; if a job is abnormally interrupted, the re-execution scheme provided by this invention supports manual, multiple reruns of the job without concerns about data loss or corrupted data. It can support different source systems and platform formats through configuration, maximizing development time savings.

[0131] Based on the same inventive concept, this specification also provides a data file loading device applied to an application server, as described in the following embodiments. Since the principle by which the data file loading device solves the problem is similar to that of the data file loading method, the implementation of the data file loading device can refer to the implementation of the data file loading method, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated. Figure 4 This is a structural block diagram of a data file loading device according to an embodiment of this specification, such as... Figure 4 As shown, it includes: acquisition module 401, verification module 402 and loading module 403. The structure is described below.

[0132] The acquisition module 401 is used to acquire the target data file to be processed, as well as the corresponding target format file and target data volume file from the data server; the target format file is a target format type generated according to a preset format specification for the fields in the target data file; the target data volume file is used to characterize the amount of data contained in the target data file.

[0133] The verification module 402 is used to verify the target data file and the corresponding target format file and target data volume file.

[0134] The loading module 403 is used to load the target data file into the target database if the target data file, the corresponding target format file, and the target data volume file have all passed verification.

[0135] In some embodiments of this specification, the loading module specifically includes:

[0136] A loading unit is used to load the target data file into a temporary table in the target database according to the format type in the target format file;

[0137] The reading unit is used to read the table-level field detail information corresponding to the target data file;

[0138] The generation unit is used to generate a statement to be executed based on the table-level field details and a preset mapping function; the preset mapping function is used to map fields in the temporary table to fields in the main table of the target database.

[0139] An execution unit is used to execute the statement to be executed, so as to update the data in the temporary table to the main table accordingly.

[0140] In some embodiments of this specification, the acquisition module may specifically include:

[0141] The receiving unit receives a job allocation table sent by the scheduling server; the job allocation table includes identifiers of data files to be loaded; the scheduling server is used to obtain a file list from the data server, the file list includes the processing status of multiple data files, and the scheduling server is also used to perform job scheduling based on the processing status of the multiple data files and generate a job allocation table.

[0142] The acquisition unit is used to acquire the data file to be processed, as well as the corresponding format file and data volume file, from the data server based on the data file identifier in the job allocation table.

[0143] In some embodiments of this specification, the data server is used to receive data files, corresponding format files, and data volume files transmitted from the source server; the data server is also used to check the transmission progress of the data files, corresponding format files, and data volume files, generate a message based on the transmission progress, and send the message to the distributed publish-subscribe messaging system;

[0144] Accordingly, the acquisition unit is specifically used for:

[0145] Based on the data file identifier in the job allocation table, subscribe to the message corresponding to the data file identifier from the distributed publish-subscribe messaging system;

[0146] If the message corresponding to the data file identifier indicates successful transmission, the data file corresponding to the data file identifier, as well as the corresponding format file and data volume file, are retrieved from the data server.

[0147] In some embodiments of this specification, the verification module specifically includes:

[0148] A legality verification unit is used to verify the legality of the target data file;

[0149] The first judgment unit is used to determine whether the amount of data in the received target data file is consistent with the amount of data in the target data file if the legality verification of the target data file passes.

[0150] The second judgment unit is used to determine whether the target format type in the target format file and the field format type involved in the target database meet preset conditions when the data volume of the received target data file is consistent with the data volume in the target data file.

[0151] The determining unit determines that the target data file and the corresponding target format file and target data volume file have been successfully verified if the target format type in the target format file and the field format type involved in the target database meet the preset conditions.

[0152] In some embodiments of this specification, the second determining unit is specifically used for:

[0153] Determine whether the target format type in the target format file is the same as the field format type involved in the target database;

[0154] If it is determined that the target format type in the target format file is the same as the field format type involved in the target database, then it is determined that the target format type in the target format file and the field format type involved in the target database meet a preset condition.

[0155] In some embodiments of this specification, the second determining unit is further specifically used for:

[0156] If it is determined that the target format type in the target format file is different from the field format type involved in the target database, it is determined whether the target format type in the target format file is compatible with the field format type involved in the target database;

[0157] If it is determined that the target format type in the target format file is compatible with the field format types involved in the target database, then it is determined that the target format type in the target format file and the field format types involved in the target database meet a preset condition.

[0158] In some embodiments of this specification, the loading module includes:

[0159] The reading unit is used to read the configuration file; the configuration file includes the job identifier, the amount of data to be read, the start processing time, and the processing interval.

[0160] The loading unit is used to load the target data file into the target database based on the configuration file.

[0161] In some embodiments of this specification, the loading module includes:

[0162] A loading unit is used to load the target data file into a temporary table in the target database according to the format type in the target format file;

[0163] The reading unit is used to read the table-level field detail information corresponding to the target data file;

[0164] The generation unit is used to generate a statement to be executed based on the table-level field details and a preset mapping function; the preset mapping function is used to map fields in the temporary table to fields in the main table of the target database.

[0165] An execution unit is used to execute the statement to be executed, so as to update the data in the temporary table to the main table accordingly.

[0166] In some embodiments of this specification, the loading module further includes:

[0167] A clear unit is used to clear the temporary table; and / or

[0168] A collection unit is used to collect statistical information from the positive table.

[0169] In some embodiments of this specification, the loading module is further configured to: determine whether the target data file has been successfully loaded into the target database; and, if the target data file has not been successfully loaded into the target database, re-execute the loading of the target data file into the target database until the loading is successful.

[0170] From the above description, it can be seen that the embodiments of this specification achieve the following technical effects: For the target format type generated according to a unified preset format specification for the field format in the data file, it is not necessary to configure different interface modules for file loading for different types of data files. Through the above scheme, the data loading operation can support more diverse data sources, no longer limited to a fixed interface. Furthermore, in the above scheme, when validating the data file, the corresponding data volume file and format file are simultaneously verified. Verifying the data volume file ensures the integrity and accuracy of the received data file, while verifying the format file ensures that the format type of the data file matches the type in the target database. It also allows adjustment of the field types in the target database based on the format type in the format file, not being limited to fixed database table information, and better responding to changes in upstream data and data sources. The data file is only loaded into the target database after all three data files (data file, format file, and data volume file) have been successfully verified, improving the success rate and accuracy of file loading. The above data loading method can support different data sources, modify fields at any time, realize dynamic loading configuration, and complete the data loading function.

[0171] This specification also provides a computer device, which can be found in the following description. Figure 5 The diagram shown illustrates the structural composition of a computer device based on the data file loading method provided in the embodiments of this specification. Specifically, the computer device may include an input device 51, a processor 52, and a memory 53. The memory 53 stores processor-executable instructions. When the processor 52 executes the instructions, it implements the steps of the data file loading method described in any of the above embodiments.

[0172] In this embodiment, the input device can specifically be one of the main devices for information exchange between the user and the computer system. The input device may include a keyboard, mouse, camera, scanner, light pen, handwriting input tablet, voice input device, etc.; the input device is used to input raw data and programs for processing these data into the computer. The input device can also receive data transmitted from other modules, units, and devices. The processor can be implemented in any suitable manner. For example, the processor can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc. The memory can specifically be a memory device used to store information in modern information technology. The memory can include multiple layers; in digital systems, anything that can store binary data can be considered memory; in integrated circuits, a circuit without physical form but with storage function is also called memory, such as RAM, FIFO, etc.; in a system, a storage device with physical form is also called memory, such as a memory stick, TF card, etc.

[0173] In this embodiment, the specific functions and effects implemented by the computer device can be explained in comparison with other embodiments, and will not be repeated here.

[0174] This specification also provides a computer storage medium based on a data file loading method, wherein the computer storage medium stores computer program instructions that, when executed by a processor, implement the steps of the data file loading method described in any of the above embodiments.

[0175] In this embodiment, the storage medium includes, but is not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions. The network communication unit can be an interface configured according to standards specified in the communication protocol for network connection communication.

[0176] In this embodiment, the specific functions and effects implemented by the program instructions stored in the computer storage medium can be explained by comparison with other embodiments, and will not be repeated here.

[0177] This specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the data file loading method described in any of the above embodiments.

[0178] Obviously, those skilled in the art will understand that the modules or steps of the embodiments described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this specification are not limited to any particular combination of hardware and software.

[0179] It should be understood that the above description is for illustrative purposes and not for limitation. Many embodiments and applications beyond the provided examples will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of this specification should not be determined by reference to the above description, but rather by reference to the foregoing claims and the full scope of their equivalents.

[0180] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to the embodiments described herein by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.< / jobid> < / jobid> < / jobid> < / jobid>

Claims

1. A method for loading data files, characterized in that, Applied to an application server, the method includes: The system retrieves the target data file to be processed, along with the corresponding target format file and target data volume file, from the data server. The target format file is a target format type generated based on the fields in the target data file according to a preset format specification. Through general configuration, the formats corresponding to multiple field types in the target data file are abstracted to generate a unified target format file. Different data source formats can be maintained through parameters using the target format file. The target data volume file is used to represent the amount of data contained in the target data file. The target data file, the corresponding target format file, and the target data volume file are validated. Validation of the target format file includes: verifying whether the field types involved in the tables in the target database are consistent with those in the target format file. If they are consistent, the target format file passes validation. If they are inconsistent, it is determined whether the target format type in the target format file is compatible with the field types involved in the tables in the target database. If they are compatible, the target format file passes validation, and the field types of the tables in the target database are updated according to the target format type. If the target data file, the corresponding target format file, and the target data volume file all pass verification, the target data file is loaded into the target database according to the target format type in the target format file.

2. The data file loading method of claim 1, wherein, Retrieve the target data file to be processed, along with the corresponding target format file and target data volume file, from the data server, including: The system receives a job allocation table sent by a scheduling server; the job allocation table includes identifiers of data files to be loaded; the scheduling server is used to obtain a file list from a data server, the file list includes the processing status of multiple data files, and the scheduling server is also used to perform job scheduling based on the processing status of the multiple data files and generate a job allocation table. Based on the data file identifier in the job allocation table, the data file to be processed, along with the corresponding format file and data volume file, is obtained from the data server.

3. The data file loading method of claim 2, wherein, The data server is used to receive data files, corresponding format files, and data volume files transmitted from the source server; the data server is also used to check the transmission progress of the data files, corresponding format files, and data volume files, generate a message based on the transmission progress, and send the message to the distributed publish-subscribe messaging system. Accordingly, based on the data file identifier in the job allocation table, the data file to be processed, along with its corresponding format file and data volume file, is obtained from the data server, including: Based on the data file identifier in the job allocation table, subscribe to the message corresponding to the data file identifier from the distributed publish-subscribe messaging system; If the message corresponding to the data file identifier indicates successful transmission, the data file corresponding to the data file identifier, as well as the corresponding format file and data volume file, are retrieved from the data server.

4. The data file loading method of claim 1, wherein, The target data file, its corresponding target format file, and target data volume file are validated, including: The validity of the target data file is verified; If the validity verification of the target data file passes, determine whether the amount of data in the received target data file is consistent with the amount of data in the target data file. If the amount of data in the received target data file is consistent with the amount of data in the target data file, determine whether the target format type in the target format file and the field format type involved in the target database meet the preset conditions; If the target format type in the target format file and the field format type in the target database meet the preset conditions, the target data file and the corresponding target format file and target data volume file are determined to have been successfully verified.

5. The data file loading method of claim 4, wherein, Determining whether the target format type in the target format file and the field format types involved in the target database meet preset conditions includes: Determine whether the target format type in the target format file is the same as the field format type involved in the target database; If it is determined that the target format type in the target format file is the same as the field format type involved in the target database, then it is determined that the target format type in the target format file and the field format type involved in the target database meet a preset condition.

6. The data file loading method of claim 5, wherein, After determining whether the target format type in the target format file is the same as the field format type involved in the target database, the process further includes: If it is determined that the target format type in the target format file is different from the field format type involved in the target database, it is determined whether the target format type in the target format file is compatible with the field format type involved in the target database; If it is determined that the target format type in the target format file is compatible with the field format types involved in the target database, then it is determined that the target format type in the target format file and the field format types involved in the target database meet a preset condition.

7. The data file loading method of claim 1, wherein, Loading the target data file into the target database includes: Read the configuration file; the configuration file includes the job identifier, the amount of data to be read, the start processing time, and the processing interval; Based on the configuration file, the target data file is loaded into the target database.

8. The data file loading method according to claim 1, characterized in that, Loading the target data file into the target database includes: The target data file is loaded into a temporary table in the target database according to the format type in the target format file; Read the table-level field details corresponding to the target data file; Based on the table-level field details and the preset mapping function, a statement to be executed is generated; the preset mapping function is used to map fields in the temporary table to fields in the main table of the target database. Execute the statement to be executed to update the data in the temporary table to the main table.

9. The data file loading method of claim 8, wherein, After executing the statement to be executed to update the data in the temporary table to the main table, the process further includes: Clear the temporary table; and / or Collect statistical information from the positive table.

10. The data file loading method of claim 1, wherein, After loading the target data file into the target database, the process also includes: Determine whether the target data file has been successfully loaded into the target database; If the target data file fails to be loaded into the target database, the process of loading the target data file into the target database is repeated until it is successfully loaded.

11. A data file loading apparatus characterized by comprising: The device, used in an application server, includes: The acquisition module is used to acquire the target data file to be processed, as well as the corresponding target format file and target data volume file, from the data server. The target format file is a target format type generated according to the fields in the target data file based on a preset format specification. Through general configuration, the formats corresponding to multiple field types in the target data file are abstracted to generate a target format file with a unified format. Different data source formats can be maintained through parameters using the target format file. The target data volume file is used to represent the amount of data contained in the target data file. The verification module is used to verify the target data file, the corresponding target format file, and the target data volume file. Verification of the target format file includes: verifying whether the field types involved in the tables in the target database are consistent with those in the target format file; if they are consistent, the verification passes; if they are inconsistent, it is determined whether the target format type in the target format file is compatible with the field types involved in the tables in the target database; if they are compatible, the verification passes and the field types of the tables in the target database are updated according to the target format type. The loading module is used to load the target data file into the target database according to the target format type in the target format file, provided that the target data file, the corresponding target format file, and the target data volume file have all passed verification.

12. The data file loading apparatus of claim 11, wherein The loading module specifically includes: A loading unit is used to load the target data file into a temporary table in the target database according to the format type in the target format file; The reading unit is used to read the table-level field detail information corresponding to the target data file; The generation unit is used to generate a statement to be executed based on the table-level field details and a preset mapping function; the preset mapping function is used to map fields in the temporary table to fields in the main table of the target database. An execution unit is used to execute the statement to be executed, so as to update the data in the temporary table to the main table accordingly.

13. A computer device, comprising: It includes a processor and a memory for storing processor-executable instructions, wherein the processor, when executing the instructions, implements the steps of the method according to any one of claims 1 to 10.

14. A computer readable storage medium having stored thereon computer instructions, wherein, When the instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 10.

15. A computer program product comprising computer programs / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 10.

Images