Task loading method and platform

Abstract

The invention discloses a task loading method and platform and relates to the technical field of data processing, and the task loading method comprises the steps that a to-be-loaded task is obtained through a visual interface operation interface; performing locking verification on the current database based on the to-be-loaded task to obtain a verification result; when the verification result is successful verification, downloading the target file based on the to-be-loaded task according to preset configuration information, and decomposing the target file into a plurality of sub-files; obtaining a plurality of file batches according to the plurality of sub-files, and loading the sub-tasks of each file batch in sequence to complete the to-be-loaded task; according to the method, locking verification is carried out on a current database in advance before task loading is carried out, when verification succeeds, it is determined that the current database meets the requirement for task loading of a to-be-loaded task, the to-be-loaded task is downloaded and decomposed into multiple batches of sub-files, loading is carried out on the basis of the sub-files in the same thread, and the task loading efficiency is improved. And data loading can be quickly completed while the stability of the database is ensured.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a task loading method and platform. Background Art

[0002] OLTP (Online Transaction Processing) is a category of software systems that supports transaction processing in daily operations. It is usually used for real-time data processing and can quickly execute a large number of simple, standardized transactions, such as banking transactions, retail sales, stock trading, etc. OLTP online data loading tool is a data service platform for online databases, providing functions such as data loading, loading task monitoring, and data quality verification.

[0003] In the process of online transaction, data usually needs to be loaded according to the uploaded data and data configuration of the user to improve the data reading efficiency and thus reduce the transaction response time; however, currently the database loads files slowly in order to ensure the stability of the database.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a task loading method and platform, which aims to solve the technical problem of slow data loading speed in order to ensure database stability when loading files in the current database.

[0006] To achieve the above purpose, the present application proposes a task loading method, wherein the task loading platform includes a visual interface operation interface and a current database, and the task loading method includes:

[0007] Obtaining the task to be loaded through the visual interface operation interface;

[0008] Performing a lock check on the current database based on the task to be loaded, and obtaining a check result;

[0009] When the verification result is successful, downloading the target file based on the task to be loaded according to the preset configuration information and decomposing the target file into multiple sub-files;

[0010] A plurality of file batches are obtained according to the plurality of sub-files, and the sub-tasks of each file batch are loaded in sequence to complete the tasks to be loaded.

[0011] In one embodiment, the lock verification of the current database based on the task to be loaded to obtain the verification result includes:

[0012] Obtaining the number of tasks being loaded in the current database, wherein the current database and the reference database are databases storing the tasks to be loaded, and the database executing the tasks to be loaded is the current database;

[0013] When the number of tasks being loaded does not exceed a threshold value of the amount of tasks that can be processed, identifying a reference database state of the reference database;

[0014] When the reference database is in a state of not being loaded, a verification result is obtained according to the database configuration of the current database.

[0015] In one embodiment, before obtaining the number of tasks being loaded in the current database, the method further includes:

[0016] Obtaining the task status of the task to be loaded, and when the task status is unprocessed, identifying the number of times the task to be loaded has been run;

[0017] When the number of weight runs does not exceed the weight run number threshold, obtaining the instance date of the task to be loaded;

[0018] A date status is obtained according to the instance date and the task instance list, and when the date status is the earliest date, a step of obtaining the number of tasks being loaded in the current database is performed.

[0019] In one embodiment, the database configuration of the current database includes cluster configuration, application access routing configuration, and information configuration;

[0020] When the reference database is in a state of not being loaded, obtaining a verification result according to the database configuration of the current database includes:

[0021] When the reference database is in a state of not being loaded, obtaining a cluster configuration state according to the cluster configuration;

[0022] When the cluster configuration state is that the cluster configuration is correct, obtaining the application access route configuration state according to the application access route configuration;

[0023] When the application access route configuration state is correctly configured, obtaining the information configuration state according to the information configuration;

[0024] When the information configuration status is that the information configuration is correct, verification success is taken as the verification result.

[0025] In one embodiment, when the verification result is successful, downloading the target file based on the task to be loaded according to preset configuration information and decomposing the target file into multiple sub-files includes:

[0026] When the verification result is successful, obtaining initial storage space;

[0027] Obtaining a target file according to the task to be loaded, wherein the target file includes a control information file and a data file;

[0028] The data file is split according to the control information file to obtain multiple sub-files.

[0029] In one embodiment, obtaining multiple file batches according to the multiple sub-files and sequentially loading subtasks of each file batch to complete the task to be loaded includes:

[0030] Obtain the number of processor cores of the current database, and obtain core parameters of the database thread pool according to the number of processor cores;

[0031] Obtaining data processing capability of the current database according to the core parameters, and dividing the sub-files into a plurality of file batches in sequence according to the data processing capability, wherein the file batches include a plurality of files;

[0032] The subtasks of each file batch are loaded in sequence until all the subfiles of each file batch are loaded, so as to complete the tasks to be loaded.

[0033] In one embodiment, after obtaining multiple file batches according to the multiple sub-files and sequentially loading the subtasks of each file batch, the method further includes:

[0034] Get the amount of loaded data and the type of data loaded;

[0035] Obtaining a data loading status for the day based on the amount of loaded data;

[0036] When the data loading status of the day is successfully loaded, data cleaning is performed on the current database according to the data loading type.

[0037] In one embodiment, the data loading type includes full loading and incremental loading;

[0038] When the data loading status of the day is successfully loaded, data cleaning is performed on the current database according to the data loading type, including:

[0039] When the data loading status of the day is successfully loaded and the data loading type is incremental loading, deleting the historical data of the current database according to the amount of loaded data;

[0040] When the data loading status of the day is successfully loaded and the data loading type is full load, a temporary storage space is obtained, and the loaded data volume is stored in the temporary storage space.

[0041] In one embodiment, the task loading platform further includes a real-time monitoring module;

[0042] The task loading method further includes:

[0043] Acquire the platform file generation status of the task to be loaded through the real-time monitoring module;

[0044] The file generation flag of the task to be loaded is adjusted according to the platform file generation status.

[0045] In one embodiment, the task loading platform further includes an abnormal alarm push module;

[0046] The task loading method further includes:

[0047] The abnormal alarm push module generates a database abnormality warning when the number of tasks being loaded in the current database does not exceed the threshold of the processable task volume or the number of re-runs exceeds the threshold of the number of re-runs or the cluster configuration status is a cluster configuration error or the application access routing configuration status is a configuration error or the information configuration status is an information configuration error.

[0048] One or more technical solutions proposed in this application have at least the following technical effects:

[0049] By locking and checking the current database in advance before loading the task, when the verification is successful, it is determined that the current database meets the requirements for loading the task to be loaded. The task to be loaded is downloaded and decomposed into multiple batches of sub-files, and loaded based on the same thread of the sub-files. This can quickly complete data loading while ensuring the stability of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0052] Figure 1 A flowchart of the first embodiment of the task loading method of this application is provided;

[0053] Figure 2 A task loading flow chart of the current database provided in Example 1 of the task loading method of this application;

[0054] Figure 3 A schematic diagram of splitting and loading a task to be loaded provided in the first embodiment of the task loading method of the present application;

[0055] Figure 4 A flowchart of the second embodiment of the task loading method of this application is provided; DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0057] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0058] The main solution of the embodiment of the present application is: obtain the task to be loaded through a visual interface operation interface; lock and verify the current database based on the task to be loaded to obtain a verification result; when the verification result is successful, download the target file based on the task to be loaded according to the preset configuration information and decompose the target file into multiple sub-files; obtain multiple file batches based on the multiple sub-files, and load the sub-tasks of each file batch in turn to complete the task to be loaded.

[0059] In the existing technology, OLTP is usually used for real-time data processing and can quickly execute a large number of simple and standardized transactions, such as banking transactions, retail sales, stock transactions, etc. The OLTP online data loading tool is a data service platform for online databases, providing functions such as data loading, loading task monitoring, and data quality verification.

[0060] In the process of online transaction, data usually needs to be loaded according to the uploaded data and data configuration of the user to improve the data reading efficiency and thus reduce the transaction response time; however, currently the database loads files slowly in order to ensure the stability of the database.

[0061] The present application provides a solution by pre-locking and checking the current database before loading the task. When the check is successful, it is determined that the current database meets the requirements for loading the task to be loaded. The task to be loaded is downloaded and decomposed into multiple batches of sub-files, which are loaded based on the same thread of the sub-files. This ensures that data loading can be completed quickly while ensuring the stability of the database.

[0062] It can be seen from the above embodiments that the present application discloses a task loading method and platform, which relates to the field of data processing technology, and discloses a task loading method, including: obtaining the task to be loaded through a visual interface operation interface; performing a lock check on the current database based on the task to be loaded to obtain a check result; when the check result is a successful check, downloading the target file based on the task to be loaded according to preset configuration information and decomposing the target file into multiple sub-files; obtaining multiple file batches based on the multiple sub-files, and loading the sub-tasks of each file batch in sequence to complete the task to be loaded; the method performs a lock check on the current database in advance before loading the task, and when the check is successful, it is determined that the current database meets the requirements for loading the task to be loaded, downloads the task to be loaded and decomposes it into multiple batches of sub-files, and loads them based on the same thread of the sub-files, so as to quickly complete the data loading while ensuring the stability of the database.

[0063] It should be noted that the execution subject of this embodiment can be a task loading platform with data processing, network communication and program running functions. The following takes the task loading platform including a visual interface operation interface as an example to illustrate this embodiment and the following embodiments.

[0064] Based on this, the present application embodiment provides a task loading method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the task loading method of the present application.

[0065] In this embodiment, the task loading method includes steps S10 to S40:

[0066] Step S10, obtaining the task to be loaded through the visual interface operation interface.

[0067] It is understandable that the task loading method in this embodiment is applied to a task loading platform, which includes a visual interface operation interface and a current database. Through the visual operation interface, data loading tasks can be quickly configured, take effect immediately, and quickly respond to business needs.

[0068] It should be further explained that the task loading platform also includes real-time monitoring of data loading tasks, abnormal alarm push, and data quality verification, to achieve full-process monitoring and management of data loading.

[0069] It is understandable that the task to be loaded can be a loading task instance generated according to the configuration information in the loading configuration table. In a specific implementation, the instance division can be performed at a fixed time every day to obtain the instance task of the day, that is, the task to be loaded. For example, multiple loading task instances are generated according to the configuration information filled in by the user in the loading configuration table, and each task instance is a task to be loaded.

[0070] Step S20, performing a lock check on the current database based on the task to be loaded to obtain a check result.

[0071] It can be understood that this embodiment uses the "AB library + AB table" core loading solution, and batch data updates do not affect external access, that is, the current database and the reference database, each database corresponds to a loading configuration table and other tables. In simple terms, each database has a corresponding table, for example, the current database has a loading configuration table corresponding to the current database, and the reference database has a loading configuration table corresponding to the reference database.

[0072] It should be noted that the lock check can be a check of the task to be loaded and the current database, to check whether the task to be loaded has been processed, the number of reruns, etc., and at the same time, to check the cluster configuration, routing configuration, and information configuration of the current database.

[0073] Step S30, when the verification result is successful, download the target file based on the task to be loaded according to preset configuration information and decompose the target file into multiple sub-files.

[0074] It is understandable that the verification result may include verification success and verification failure. If the verification fails, an alarm is issued; if the verification succeeds, the target file is downloaded based on the task to be loaded according to the preset configuration information and the target file is decomposed into multiple sub-files.

[0075] It should be noted that the preset configuration information may be the relevant information of the task to be loaded determined by the user when loading data through the task loading platform. The preset configuration information may be stored in a preset configuration table and directly queried after being filled in by the user.

[0076] It is understandable that the target file can be the file that the user wants to load. Before loading the file, the user needs to upload the file to the platform. At the same time, the target file uploaded by the user needs to be downloaded and then loaded according to the configuration table filled out by the user.

[0077] It should be noted that when a file needs to be loaded, the loading time may be too long because the file is too large, affecting the stability of the platform database, so the target file is divided into multiple small files.

[0078] In a feasible implementation, step S30 may include steps A31 to A33:

[0079] Step A31, when the verification result is successful, obtain initial storage space.

[0080] It is understandable that the initial storage space may be a storage space used to store a target file to be loaded by a user.

[0081] Step A32, obtaining a target file according to the task to be loaded, wherein the target file includes a control information file and a data file.

[0082] It should be noted that the temporary file storage directory is cleared, and the CTL file (control information file) and DAT file (data file) are downloaded from the data exchange platform to the temporary directory, that is, the storage directory corresponding to the initial storage space.

[0083] Step A33, splitting the data file according to the control information file to obtain multiple sub-files.

[0084] It is understandable that, according to the control information file provided, the original data file can be split in detail to obtain multiple independent sub-files. These sub-files will contain specific parts of the original data file, which are split according to the rules and parameters specified in the control information file. Each sub-file will contain a portion of data, which may be divided according to a specific format, time period or other criteria, and a large number of data files can be effectively managed and processed, making them easier to manage and analyze.

[0085] It should be noted that the appropriate number of file splitting lines is calculated according to the configuration information, and the database file is split into multiple small files according to the number of splitting lines.

[0086] In this implementation, by combining intelligent splitting of data files with loading concurrency control strategies, database resources are fully utilized to improve data loading efficiency while ensuring database stability. A distributed high-concurrency loading solution is used to complete loading of tens of millions of large tables in minutes, ensuring data timeliness.

[0087] The above are only two feasible implementations of step S30 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S30.

[0088] Step S40, obtaining multiple file batches according to the multiple sub-files, and sequentially loading the subtasks of each file batch to complete the task to be loaded.

[0089] It should be noted that the multiple sub-files are divided into multiple batches in sequence, and the sub-files of one batch are loaded each time until all the sub-files are loaded.

[0090] In the specific implementation, multiple threads can be used to load multiple subtasks of the same batch. The task loading flowchart of the current database can be found in Figure 2 , Figure 2The current database is used as an example for explanation, including loading configuration tables, clusters, loading task flow tables, application instances, multiple threads corresponding to each instance, and target tables (the target table can be understood as a task instance table).

[0091] It should be emphasized that before loading sub-files, the pre-load check should be repeated and cyclically performed on the task to be loaded (non-split tasks start loading directly, and split tasks need to ensure that only one instance can execute the loading to prevent excessive database pressure).

[0092] It should be noted that the task loading platform further includes a real-time monitoring module, through which the platform file generation status of the task to be loaded is obtained; and the file generation flag of the task to be loaded is adjusted according to the platform file generation status.

[0093] Among them, it is understandable that the generation status can include generated and not generated, and the upload progress of the target file corresponding to the task to be loaded on the platform is adjusted in real time based on the file generation status, whether the upload is completed, and whether the task can be loaded.

[0094] In the specific implementation, the real-time monitoring module is responsible for task initialization and file scanning: generating a loading task instance according to the configuration information in the loading configuration table, periodically scanning whether the exchange platform file corresponding to the instance is generated, and if the file has been generated, modifying the file generation flag of the corresponding loading task instance to have been generated.

[0095] It should be further explained that the task loading platform also includes an abnormal alarm push module, through which a database abnormality warning is generated when the number of tasks being loaded in the current database does not exceed the threshold of the processable task volume or the number of re-runs exceeds the threshold of the number of re-runs or the cluster configuration status is a cluster configuration error or the application access routing configuration status is a configuration error or the information configuration status is an information configuration error.

[0096] Step S40 also includes steps S41 to S43:

[0097] Step S41, obtaining the number of processor cores of the current database, and obtaining core parameters of the database thread pool according to the number of processor cores.

[0098] It is understandable that the number of processor cores of the current server can be queried through system commands or tools. For example, in the Linux system, we can use the 'lscpu' command to obtain the number of processor cores. By parsing the output of this command, we can get the number of processor cores.

[0099] It should be noted that after obtaining the number of processor cores, the core parameters of the database thread pool can be set according to this number of cores. The core parameters of the database thread pool refer to the maximum number of threads that can be used during the operation of the database. This parameter is usually closely related to the number of processor cores, because the number of processor cores determines the maximum number of tasks that the database can process in parallel.

[0100] It should be emphasized that, generally speaking, it is a common practice to set the core parameters of the thread pool to about twice the number of processor cores. This ensures that the database can fully utilize the processor's computing resources when processing concurrent requests, thereby improving the performance of the database.

[0101] In the actual implementation, the specific thread pool core parameters need to be adjusted according to the actual hardware configuration and business needs. For example, if the database mainly processes high-concurrency read operations, the core parameters of the thread pool can be appropriately increased to improve the parallelism of the read operation. On the contrary, if the database mainly processes write operations, the core parameters of the thread pool may need to be appropriately reduced to avoid performance degradation caused by excessive thread competition.

[0102] It is understandable that obtaining the number of processor cores of the current database is the basis for setting the core parameters of the database thread pool, which can fully utilize the computing resources of the processor and improve the performance and response speed of the database.

[0103] It should be noted that the sub-files are packaged in batches and submitted to the thread pool for loading sub-tasks (a library only allows one task to execute the loading sub-task). The submission and execution are looped until all the sub-files are loaded, and the loading progress is printed synchronously. The thread pool core parameters of this step are determined by the number of CPU cores of the online loading system.

[0104] Step S42, obtaining the data processing capability of the current database according to the core parameters, and dividing the sub-files into a plurality of file batches in sequence according to the data processing capability, wherein the file batches include a plurality of files.

[0105] It should be noted that the core parameters can accurately determine the data processing capabilities of the current database. Based on the evaluation results of this data processing capability, we will group the sub-files that need to be processed in turn to form multiple file batches. Each file batch contains several files, which will be processed according to the established order and rules to ensure the efficiency and orderliness of the entire data processing process.

[0106] Step S43, loading the subtasks of each file batch in sequence until all the subfiles of each file batch are loaded, so as to complete the tasks to be loaded.

[0107] In the specific implementation, the process of splitting and loading the task to be loaded can refer to Figure 3 , Figure 3 The target file downloaded from the exchange platform is processed by the instance and split into multiple tasks according to the user-configured strategy, and loaded through multiple threads in the same thread pool.

[0108] It should be noted that step S43 also includes obtaining the amount of loaded data and the data loading type, obtaining the data loading status of the day based on the loaded data amount, and when the data loading status of the day is successfully loaded, cleaning the current database according to the data loading type.

[0109] It should be noted that the post-load verification step verifies the data volume during loading and verifies whether the data for the day has been successfully loaded into the target table. Only the existence is verified, and no accuracy verification is performed.

[0110] It should be noted that the data loading types include full loading and incremental loading.

[0111] Among them, it should be emphasized that when the data loading status of the day is successfully loaded, the current database is cleaned up according to the data loading type, including: when the data loading status of the day is successfully loaded and the data loading type is incremental loading, deleting the historical data of the current database according to the loaded data volume; when the data loading status of the day is successfully loaded and the data loading type is full loading, obtaining temporary storage space and storing the loaded data volume in the temporary storage space.

[0112] Among them, it should be noted that whether it is incremental loading, incremental loading requires conditional deletion of historical data to prevent the target table data volume from expanding infinitely; full loading (overwriting) table switching verification, full loading is to load the data into a temporary table first, and then exchange the target table and the temporary table by renaming after the loading is completed to prevent accidental clearing of historical data under abnormal circumstances.

[0113] The present embodiment provides a task loading method, which performs a lock check on the current database before loading the task. When the check succeeds, it is determined that the current database meets the requirements for loading the task to be loaded. The task to be loaded is downloaded and decomposed into multiple batches of sub-files, which are loaded based on the same thread of the sub-files. This ensures that the data loading can be completed quickly while ensuring the stability of the database.

[0114] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , step S20 also includes steps S21 to S23:

[0115] Step S21, obtaining the number of tasks being loaded in the current database, the current database and the reference database are databases storing the tasks to be loaded, and the database executing the tasks to be loaded is the current database.

[0116] It is understandable that the data processing capacity of the database is limited. In order to avoid the database being overloaded due to excessive data loading and affecting the stability of the database, when loading tasks through the current database, you can check in advance the amount of loading tasks being processed by the current database to determine whether the current database can further process the tasks to be loaded.

[0117] It is understandable that the current database and the reference database have corresponding routing configurations. In order to ensure that the database access is not affected during the task loading process, the AB library + AB table method is adopted in this embodiment. When the current database is loading the task, the user can access the reference database. After the current database is loaded, the various tables corresponding to the current database are updated, and the data of the loaded task is synchronized to the reference database.

[0118] It should be noted that the current database and the reference database are not fixed. The database for loading tasks is defined as the current database, and another database for being accessed by users can be defined as the reference database.

[0119] In a feasible implementation manner, step S21 may include steps A21.1 to A2103 before step S21:

[0120] Step A2101, obtain the task status of the task to be loaded, and when the task status is unprocessed, identify the number of times the task to be loaded has been run.

[0121] It is understandable that the task status may include unprocessed and being processed. If the task status is being processed, the processing of the task to be loaded is stopped.

[0122] It should be understood that when a database loads a task to be loaded at the same time, if the task to be loaded is being processed, it proves that another database is being loaded. In order to ensure the stability of the database, the database route that processes the task to be loaded is closed and is no longer accessed. In order to ensure that user access is not affected, it is necessary to ensure that the current database does not process the task to be loaded.

[0123] It should be noted that it is determined whether the loading task has been processed or the maximum number of re-runs has been reached. If it has been processed by other application instances (processors) or the maximum number of re-runs has been reached, the task is skipped.

[0124] It is understandable that the number of reruns can be understood as the number of times the task to be loaded is processed but not successfully processed. It may be due to various problems such as errors in the task to be loaded and the inability to download the file, which causes the task to be selected for processing by the database multiple times, but the processing fails.

[0125] It should be understood that each weight-carrying operation will be recorded, and the number of weight-carrying operations can be obtained based on the record of the task to be loaded.

[0126] Step A2102, when the number of heavy runs does not exceed the heavy run number threshold, obtain the instance date of the task to be loaded.

[0127] It should be noted that when the number of weight-running times exceeds the weight-running times threshold, the task to be loaded can be skipped and other tasks to be loaded can be processed first.

[0128] It is understandable that each task to be loaded has a processing time, which can also be understood as the generation time of the task to be loaded, and can be directly obtained through the table recording the tasks to be loaded.

[0129] It should be noted that it is checked whether there is a task with an earlier task instance date (instantiation) being executed for the loading task. If so, the current task is skipped and the previous task is waited for to be completed.

[0130] Step A2103, obtaining a date status according to the instance date and the task instance list, and when the date status is the earliest date, executing the step of obtaining the number of tasks being loaded in the current database.

[0131] It is understandable that the date status includes the earliest date and the non-earliest date.

[0132] It should be noted that the tasks to be loaded are generally the tasks to be loaded on the same day, and the instance date is the current day. It is also possible that the tasks to be loaded cannot be successfully loaded due to some reasons on the previous day, and are delayed until the next day.

[0133] Furthermore, when processing the tasks to be loaded on the second day, the instance dates of the tasks to be loaded are compared and judged, and the processing starts from the task to be loaded with the earliest instance date.

[0134] In this implementation, the tasks to be loaded are pre-verified in terms of the number of reruns, task status, and instance date of the tasks to be loaded, so as to ensure that the tasks to be loaded that can be processed are processed first, thereby improving data loading efficiency.

[0135] The above are only two feasible implementations of step S10 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S10.

[0136] Step S22, when the number of the tasks being loaded does not exceed the threshold value of the processable task amount, identifying the reference database status of the reference database.

[0137] It should be noted that identifying the reference database status of the reference database can also be understood as determining whether there are tasks being loaded or failed to load in other libraries of the cluster. If so, the current loading task is skipped. If so, it proves that other databases are starting routing and are processing tasks and cannot be accessed.

[0138] It is understandable that the reference database status includes not loaded and loading.

[0139] It should be noted that when the reference database is in the loading state, if the reference database routing is closed, the reference database is inaccessible.

[0140] It should be noted that the determination may be made by determining whether the number of tasks being loaded by the database to be loaded reaches a set threshold. The threshold may be manually set in advance and may be adjusted according to actual conditions, and this embodiment does not limit this.

[0141] It should be understood that the processable task volume threshold may be the maximum task volume that the current database can process.

[0142] Step S23, when the reference database is in the unloaded state, obtaining a verification result according to the database configuration of the current database.

[0143] It is understandable that obtaining the verification result according to the database configuration of the current database can be used to further determine whether the current database configuration is correct. When the database configuration is correct, a verification success result is obtained.

[0144] It should be noted that finding out the tasks that need to be processed from the database and performing database verification in advance can screen out the loading tasks, and can schedule the tasks that can be loaded immediately as much as possible, reducing unnecessary verification time and distributed lock contention.

[0145] It should be emphasized that the data of the task to be loaded is not necessarily accurate, and a pre-task verification step with a lock is performed in advance (this step is used to quickly determine whether the task can be executed after waiting for a period of time). This step is implemented using the responsibility chain model.

[0146] In a feasible implementation, step S23 may include steps A231 to A234:

[0147] Step A231, when the reference database status is unloaded, obtain the cluster configuration status according to the cluster configuration.

[0148] It is understandable that each database has multiple tables, such as the task instance table, the loading control table, the application routing access table and the database information table. The database information table contains the corresponding cluster-library name-address-user name-password information; the application routing access table includes the routes configured for each application instance; the loading control table includes the routes, clusters and other information corresponding to the database cluster.

[0149] It should be noted that when loading the task to be loaded, the application instance and cluster loading the task to be loaded are checked to determine whether they are consistent with the information in the table. If they are consistent, the configuration is correct; if not, the configuration is wrong.

[0150] It is understandable that the cluster configuration status includes a correct cluster configuration and an incorrect cluster configuration.

[0151] It should be noted that it is necessary to determine whether the relationship between the database and the cluster to be loaded is correctly configured. If not configured or incorrectly configured, the current loading task is skipped.

[0152] Step A232, when the cluster configuration status is that the cluster configuration is correct, obtain the application access route configuration status according to the application access route configuration.

[0153] It is understandable that the application access route configuration status includes the application access route configuration being correct and the application access route configuration being incorrect.

[0154] It should be noted that obtaining the application access route configuration status according to the application access route configuration can be to query the application route access table through the route corresponding to the cluster in the current database to determine whether the application access route is correctly configured. If the query results are consistent, the application access route configuration status is that the application access route configuration is correct. If they are inconsistent, an alarm is issued and an application access route configuration error is fed back.

[0155] Step A233, when the application access route configuration status is correctly configured, obtain the information configuration status according to the information configuration.

[0156] It is understandable that the information configuration status includes correct information configuration and incorrect information configuration.

[0157] It should be noted that the information configuration status can be determined through the database information table, and login verification based on the cluster, library name, address, user name, and password in the database information table can determine whether the configuration is correct.

[0158] It is understandable that if the information configuration is wrong, an alarm is issued and the information configuration status is fed back as information configuration error.

[0159] It should be noted that it is necessary to determine whether the database information is configured and correct (the correct judgment condition is: the database can be connected and the required permissions are available. The verification results are cached and this step will not be repeated for a period of time. Manual refresh is supported after the database information is updated.)

[0160] Step A234, when the information configuration status is that the information configuration is correct, verification success is taken as the verification result.

[0161] It should be emphasized that the relevant loading status is updated, including routing status, task instance status, loading configuration status and other loading control information.

[0162] In this implementation, by verifying the application access routing configuration, database information configuration, and database cluster configuration of the database, it is ensured that the instance of the database processing the task to be loaded is correct, and the task to be loaded can be completed efficiently and accurately, avoiding database instability caused by database data confusion.

[0163] The above are only two feasible implementations of step S23 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S23.

[0164] This embodiment provides a task loading method, which implements a full-process monitoring and management mechanism for data loading through real-time monitoring of data loading tasks, pushing abnormal alarms, and pre-verification of loading tasks. On the premise of ensuring database stability, it fully utilizes database resources to improve data loading efficiency.

[0165] The present application provides a task loading platform, which adopts the task loading method in the above embodiment, and can solve the technical problem that the current database loading file is slow in order to ensure the stability of the database. Compared with the prior art, the beneficial effects of the task loading platform provided by the present application are the same as the beneficial effects of the task loading method provided by the above embodiment, and the other technical features in the task loading platform are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0166] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the task loading method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0167] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0168] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0169] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0170] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0171] The computer program product provided by the present application can solve the technical problem that the data loading speed is slow in order to ensure the stability of the database when loading files in the current database. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the task loading method provided by the above embodiment, which will not be repeated here.

[0172] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A task loading method, characterized in that: The task loading method is applied to a task loading platform, which includes a visual interface operation interface and a current database. The task loading method includes: Obtaining the task to be loaded through the visual interface operation interface; Performing a lock check on the current database based on the task to be loaded, and obtaining a check result; When the verification result is successful, downloading the target file based on the task to be loaded according to the preset configuration information and decomposing the target file into multiple sub-files; A plurality of file batches are obtained according to the plurality of sub-files, and the sub-tasks of each file batch are loaded in sequence to complete the tasks to be loaded.

2. The task loading method according to claim 1, characterized in that: The locking check of the current database based on the task to be loaded to obtain the check result includes: Obtaining the number of tasks being loaded in the current database, wherein the current database and the reference database are databases storing the tasks to be loaded, and the database executing the tasks to be loaded is the current database; When the number of tasks being loaded does not exceed a threshold value of the amount of tasks that can be processed, identifying a reference database state of the reference database; When the reference database is in a state of not being loaded, a verification result is obtained according to the database configuration of the current database.

3. The task loading method according to claim 2, characterized in that: Before obtaining the number of tasks being loaded in the current database, the method further includes: Obtaining the task status of the task to be loaded, and when the task status is unprocessed, identifying the number of times the task to be loaded has been run; When the number of weight runs does not exceed the weight run number threshold, obtaining the instance date of the task to be loaded; A date status is obtained according to the instance date and the task instance list, and when the date status is the earliest date, a step of obtaining the number of tasks being loaded in the current database is performed.

4. The task loading method according to claim 2, characterized in that: The database configuration of the current database includes cluster configuration, application access routing configuration and information configuration; When the reference database is in a state of not being loaded, obtaining a verification result according to the database configuration of the current database includes: When the reference database is in a state of not being loaded, obtaining a cluster configuration state according to the cluster configuration; When the cluster configuration state is that the cluster configuration is correct, obtaining the application access route configuration state according to the application access route configuration; When the application access route configuration state is correctly configured, obtaining the information configuration state according to the information configuration; When the information configuration status is that the information configuration is correct, verification success is taken as the verification result.

5. The task loading method according to claim 1, characterized in that: When the verification result is successful, downloading the target file based on the task to be loaded according to the preset configuration information and decomposing the target file into multiple sub-files includes: When the verification result is successful, obtaining initial storage space; Obtaining a target file according to the task to be loaded, wherein the target file includes a control information file and a data file; The data file is split according to the control information file to obtain multiple sub-files.

6. The task loading method according to claim 1, characterized in that: The step of obtaining multiple file batches according to the multiple sub-files and sequentially loading sub-tasks of each file batch to complete the task to be loaded includes: Obtain the number of processor cores of the current database, and obtain core parameters of the database thread pool according to the number of processor cores; Obtaining data processing capability of the current database according to the core parameters, and dividing the sub-files into a plurality of file batches in sequence according to the data processing capability, wherein the file batches include a plurality of files; The subtasks of each file batch are loaded in sequence until all the subfiles of each file batch are loaded, so as to complete the tasks to be loaded.

7. The task loading method according to claim 1, characterized in that: After obtaining multiple file batches according to the multiple sub-files and sequentially loading the subtasks of each file batch, the method further includes: Get the amount of loaded data and the type of data loaded; Obtaining a data loading status for the day based on the amount of loaded data; When the data loading status of the day is successfully loaded, data cleaning is performed on the current database according to the data loading type.

8. The task loading method as claimed in claim 7, characterized in that: The data loading types include full loading and incremental loading; When the data loading status of the day is successfully loaded, data cleaning is performed on the current database according to the data loading type, including: When the data loading status of the day is successfully loaded and the data loading type is incremental loading, deleting the historical data of the current database according to the amount of loaded data; When the data loading status of the day is successfully loaded and the data loading type is full load, a temporary storage space is obtained, and the loaded data volume is stored in the temporary storage space.

9. The task loading method according to claim 1, characterized in that: The task loading platform also includes a real-time monitoring module and an abnormal alarm push module; The task loading method further includes: Acquire the platform file generation status of the task to be loaded through the real-time monitoring module; Adjust the file generation flag of the task to be loaded according to the platform file generation status; The abnormal alarm push module generates a database abnormality warning when the number of tasks being loaded in the current database does not exceed the threshold of the processable task volume or the number of re-runs exceeds the threshold of the number of re-runs or the cluster configuration status is a cluster configuration error or the application access routing configuration status is a configuration error or the information configuration status is an information configuration error.

10. The task loading platform according to claim 1, characterized in that: The task loading platform includes a visual interface operation interface, a current database, a real-time monitoring module and an abnormal alarm push module; Obtaining the task to be loaded through the visual interface operation interface; Performing a lock check on the current database based on the task to be loaded, and obtaining a check result; When the verification result is successful, download the target file based on the task to be loaded according to the preset configuration information and decompose the target file into multiple sub-files; obtain multiple file batches according to the multiple sub-files, and load the sub-tasks of each file batch in sequence to complete the task to be loaded; Acquire the platform file generation status of the task to be loaded through the real-time monitoring module; Adjust the file generation flag of the task to be loaded according to the platform file generation status; The abnormal alarm push module generates a database abnormality warning when the number of tasks being loaded in the current database does not exceed the threshold of the processable task volume or the number of re-runs exceeds the threshold of the number of re-runs or the cluster configuration status is a cluster configuration error or the application access routing configuration status is a configuration error or the information configuration status is an information configuration error.

Images