Batch data exporting method and device

Abstract

The invention relates to the field of databases, can be applied to the fields of big data, intelligent operation and maintenance and financial science and technology, and provides a batch data export method and device. The method comprises the steps that file export information configured by a user according to a configuration interface is received, and the file export information is stored in a batch export instance table, the configuration interface comprises a configuration control related to a user configuration field in the batch export instance table; reading file export information in the batch export instance table one by one after a preset time point arrives, generating a file export task when reading one piece of information, and storing the file export task into a corresponding export time queue according to an export time point identifier; and for each export time queue, when the export time point arrives, executing each export file task in the export time queue, and storing execution information of each export file task into the batch export record table. According to the method, the requirement for the ability of configuration personnel can be lowered, unified management of batch export tasks is achieved, and abnormal export tasks can be checked.

Description

Technical Field

[0001] This article relates to the database field and can be used in the fields of big data, intelligent operation and maintenance, and financial technology, and in particular, to a method and device for batch exporting data. Background Art

[0002] With the gradual implementation of technologies such as product group decoupling and off-hosting, the demand for exporting various database data at the end of the day and during the day in batches for various system applications has increased. In the existing technology, batch task configuration is achieved by technicians logging into the server and configuring or modifying the batch deployment program in the server. This method requires high technical capabilities of technicians and has problems such as low configuration efficiency and poor operability. In addition, the formats of the batch deployment programs configured by various technicians are inconsistent, which is not conducive to the management and monitoring of the batch deployment program.

[0003] Therefore, considering the above problems comprehensively, there is an urgent need for a universal method for deploying batch export of multiple export time nodes to achieve unified management and monitoring of batch export of multiple export time nodes (such as end of day, daytime). Summary of the invention

[0004] This article is used to solve the problems in the existing technology that the batch task configuration of each system is complex, requires high technical personnel skills, has low accuracy and poor operability, and the formats of the batch deployment programs configured by each technician are inconsistent, which is not conducive to the management and monitoring of the batch deployment program.

[0005] In order to solve the above technical problems, this article provides a batch data export method, including:

[0006] Receive file export information configured by the user according to a configuration interface, and store the file export information in a batch export instance table, wherein the configuration interface includes configuration controls related to user configuration fields in the batch export instance table, and the batch export instance table includes the following user configuration fields: export job name, export time point identifier, export logic, export file name, and export directory;

[0007] After the preset time point arrives, the file export information in the batch export instance table is read one by one, an export file task is generated for each piece of information read, and the export file task is stored in the corresponding export time queue according to the export time point identifier;

[0008] For each export time queue, when the export time point arrives, each export file task in the export time queue is executed, and the execution information of each export file task is stored in the batch export record table.

[0009] As a further embodiment of the present invention, the batch export instance table further includes the following automatically configured fields: export batch identifier, export time;

[0010] The batch export record table includes the following fields: record identifier, export batch identifier, export batch date, export start time, export end time, export execution result, and error message;

[0011] The export file task includes file export information.

[0012] As a further embodiment of this invention, when the export time point arrives, each export file task in the export time queue is executed, and the execution information of each export file task is stored in the batch export record table, including:

[0013] When the export time point arrives, for each export file task in the export time queue, the following process is executed:

[0014] Insert the record data of the export file task into the batch export record table;

[0015] Update the record identifier, export batch identifier, export batch date, and export start time in the record data;

[0016] Create a target file in the export directory in the export file task;

[0017] Determine the export script according to the export logic, export file name and target file in the export file task;

[0018] Execute the export script to query the data satisfying the export logic from the export file name and export it to the target file;

[0019] During the execution of the export script, a pre-configured monitoring script is executed, so that the monitoring script records error information when the export script is executed abnormally;

[0020] The pre-configured batch complement program is called to update the export end time, export execution result and error information in the recorded data according to the export execution result and the error information detected by the monitoring script.

[0021] As a further embodiment of this invention, the method for exporting data in batches further includes:

[0022] Scan the export execution result of each data in the batch export record table;

[0023] Each time a piece of data is scanned, it is determined whether the export execution result in the data is a failure. If so, the data or the export execution result is sent to different persons in charge.

[0024] As a further embodiment of this invention, after each piece of data is scanned, the following is further included:

[0025] Get the export start time and export end time of the data;

[0026] According to the export start time and export end time, the export duration of the exported file data is counted;

[0027] Determine whether there is an abnormality based on the export duration of the exported file data;

[0028] If abnormal, a prompt message will be issued.

[0029] As a further embodiment of the present invention, the batch export record table further includes: an export status field, which is used to indicate whether the batch task is executed for the first time;

[0030] The monitoring script is also used to monitor the execution result of the export script;

[0031] Before updating the export end time, export execution result, and error information in the record data, the method further includes:

[0032] The monitoring script monitors whether the execution result of the export script is a failure. If the execution result is a failure, the export status field in the relevant record data in the batch export record table is updated to be not the first time, the export script is re-executed, and the number of consecutive failures is counted;

[0033] If the execution result is successful or the number of consecutive failures reaches a predetermined threshold, a pre-configured batch complement program is called to update the export end time, export execution result, and error information in the recorded data according to the export execution result and the error information detected by the monitoring script.

[0034] As a further embodiment of this invention, an export script is determined according to the export logic, the export file name and the target file in the export file task;

[0035] Find the script associated with the export logic in the export file task from the pre-configured script database;

[0036] Add the export file name and target file to the export file name variable in the found script to obtain the export script.

[0037] As a further embodiment of this invention, the batch export instance table further includes: a server address field;

[0038] According to the export logic, export file name and target file in the export file task, determine the export script and include:

[0039] According to the server address field, an export script is sent to a relevant server, and the relevant server executes the export script to query data satisfying the export logic from the export file name and export the data to the target file;

[0040] Receive the export execution result and error information sent by the monitoring script in the relevant server.

[0041] As a further embodiment of this invention, the method for exporting data in batches further includes:

[0042] Receive operation information configured by the user according to the configuration interface;

[0043] The related file export information in the batch export instance table is modified according to the operation information.

[0044] A second aspect of the present invention provides a device for exporting data in batches, comprising:

[0045] A configuration unit, configured to receive file export information configured by a user according to a configuration interface, and store the file export information in a batch export instance table, wherein the configuration interface includes configuration controls related to user configuration fields in the batch export instance table, and the batch export instance table includes the following user configuration fields: export job name, export time point identifier, export logic, export file name, and export directory;

[0046] A task generating unit, configured to read the file export information in the batch export instance table one by one after a preset time point, generate an export file task for each piece of information read, and store the export file task into a corresponding export time queue according to an export time point identifier;

[0047] The task execution unit is used for executing each export file task in each export time queue when the export time point arrives, and storing the execution information of each export file task in the batch export record table.

[0048] A third aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any of the foregoing embodiments when executing the computer program.

[0049] A fourth aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor of a computer device, implements the method described in the aforementioned embodiment.

[0050] The batch data export method and device provided in this article can be easily used to configure file export information, reduce the technical ability requirements of configuration personnel, and enable novices to quickly get started, greatly saving the communication cost of batch deployment. At the same time, it can achieve unified management of batch export tasks in various environments and improve the stability and convenience of export task management at each export time point. The file export information configured by the user is stored in the batch export task instance table, and the export task execution results are stored in the batch export record table, which can achieve unified management and monitoring of each batch export task, and facilitate the troubleshooting of abnormal export tasks.

[0051] For newly added export batches, the user only needs to configure new file export information to realize the automatic export function. For existing export batches, the user only needs to obtain the corresponding file export information and change the relevant information to automatically realize the export function. The batch data export method and device provided in this article can also improve the processing efficiency of export tasks.

[0052] In order to make the above and other purposes, features and advantages of this article more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of this article or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of this article. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 The structure diagram of the batch data export system in the embodiment of this article is shown;

[0055] Figure 2 A flow chart of a method for exporting data in batches according to an embodiment of this invention is shown;

[0056] Figure 3 A flow chart showing the process of executing the export file task in the embodiment of this article is shown;

[0057] Figure 4 A flow chart showing a process of batch exporting data system monitoring in an embodiment of this invention is shown;

[0058] Figure 5 Another flow chart of the process of monitoring the batch export data system in the embodiment of this article is shown;

[0059] Figure 6 Another flow chart showing the process of executing the export file task in the embodiment of this article;

[0060] Figure 7A schematic diagram of the configuration interface of the embodiment of this article is shown;

[0061] FIG. 8A to FIG. 8C Another schematic diagram showing the configuration interface of the embodiment of this article;

[0062] Fig. 9 A structural diagram of a device for exporting data in batches according to an embodiment of this invention is shown;

[0063] Fig.10 A structural diagram of a computer device according to an embodiment of this invention is shown.

[0064] Description of the accompanying symbols:

[0065] 101. Client;

[0066] 102. Server side;

[0067] 701. Added export controls;

[0068] 702. Modify the export control;

[0069] 703. Delete export control;

[0070] 901, configuration unit;

[0071] 902, task generation unit;

[0072] 903, task execution unit;

[0073] 1002. Computer equipment;

[0074] 1004. Processor;

[0075] 1006. Memory;

[0076] 1008. Driving mechanism;

[0077] 1010, input / output module;

[0078] 1012. Input device;

[0079] 1014. Output device;

[0080] 1016. Presentation equipment;

[0081] 1018. Graphical user interface;

[0082] 1020. Network interface;

[0083] 1022. Communication link;

[0084] 1024. Communication bus. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of this article to clearly and completely describe the technical solutions in the embodiments of this article. Obviously, the described embodiments are only part of the embodiments of this article, not all of the embodiments. Based on the embodiments of this article, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this article.

[0086] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0087] This specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel.

[0088] It should be noted that the batch data export method and device in this article can be used in the financial field, and can also be used in any field other than the financial field. The application field of the batch data export method and device in this article is not limited. As long as there is a field with batch data export application, it falls within the protection scope of this article.

[0089] In the prior art, batch task configuration is implemented by technicians logging into the server and configuring or modifying the batch deployment program in the server. This method requires high technical capabilities of the technicians and has problems such as low configuration efficiency and poor operability. In addition, the batch deployment program formats configured by each technician are inconsistent, which is not conducive to the management and monitoring of the batch deployment program. In order to solve this technical problem, this paper provides a batch export data system, such as Figure 1 As shown, it includes: a client 101 and a server 102.

[0090] The client 101 has a configuration interface for the user to input operation instructions according to the configuration controls in the configuration interface. In some embodiments of the present specification, the client may be a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a smart wearable device, etc. Among them, the smart wearable device may include a smart bracelet, a smart watch, smart glasses, a smart helmet, etc. Of course, the client is not limited to the above-mentioned electronic device with a certain entity, and it can also be software running in the above-mentioned electronic device.

[0091] Configuration interface such as Figure 7 , Fig. 8A , Figure 8B and Figure 8C As shown, Figure 7 The configuration interface shown includes: operation instruction controls, specifically including a new export control 701, a modified export control 702, and a deleted export control 703. The configuration interface after the new export control 701 is triggered is as follows Fig. 8A As shown, it is used for the user to configure the file to export information. Modify the configuration interface after the export control 702 is triggered as follows Figure 8B As shown, it is used to display or allow users to retrieve the configured file export information. Users can modify the user configuration fields in the displayed or retrieved file export information. Figure 8C As shown, it is used to display or allow users to retrieve the configured file export information, and users can delete the displayed or retrieved file export information. Figure 8B and Figure 8C The shown interface can also be displayed in an interface.

[0092] Specifically, Fig. 8A As shown, the configuration interface of the newly added export information includes configuration controls related to the user configuration fields in the batch export instance table, and the batch export instance table includes the following user configuration fields: export job name, export time point identifier, export logic, export file name, and export directory. Correspondingly, the batch export instance table includes at least the export job name, export time point identifier, export logic, export file name, and export directory, and also includes automatic configuration fields: export batch identifier and export time. Among them, the export job name is used to uniquely identify the export job. The export logic is used to limit the logic that the batch export data satisfies. The export file name is used to limit the file name of the exported data storage. The export time point identifier is used to limit the export start time, such as the end of the day and the daytime, with 0 representing the end of the day and 1 representing the daytime. The export directory is used to limit the target location for export file storage. The export batch identifier is automatically generated by the server side and is used to limit the serial number of the batch task (the primary key starts to increase from 1, that is, the primary key increases by 1 for each additional data item). The export time is used to limit the export start time.

[0093] The server end 102 has a built-in batch deployment program, which is used to receive file export information configured by the user according to the configuration interface, and store the file export information in the batch export instance table; after the preset time point arrives, the file export information in the batch export instance table is read one by one, and an export file task is generated for each information read (the export file task includes the file export information), and the export file task is stored in the corresponding export time queue according to the export time point identifier; for each export time queue, when the export time point arrives, each export file task in the export time queue is executed, and the execution information of each export file task is stored in the batch export record table.

[0094] The server 102 also has a built-in monitoring program for scanning the export execution result of each data in the batch export record table; each time a data is scanned, it is determined whether the export execution result in the data is a failure. If so, the data and the export execution result are sent to different persons in charge respectively.

[0095] The server end 102 is further configured to receive the deleted file export information input by the user according to the configuration interface, and delete the data corresponding to the deleted file information in the batch export instance table.

[0096] The server end 102 is further configured to receive modified file export information input by the user according to the configuration interface, and update the modified file export information to data at a corresponding position in the batch export instance table.

[0097] The server end 102 described in this article is a server with batch export requirements, and the number of servers may specifically include multiple ones.

[0098] The batch export record table includes the following fields: record identifier (record ID), export batch identifier, export batch date, export start time, export end time, export execution result, and error information. Among them, the record identifier is used to limit the export record sequence number (the primary key starts to increase from 1, that is, the primary key increases by 1 for each additional data entry). The export batch identifier is used to correspond to the data in the batch export instance table. The export batch date is used to limit the year, month, and day of the export batch. The export start time is used to record the start time of the batch export in a day. The export end time is used to record the end time of the batch export in a day. The export execution result is used to record whether the export task is successful, for example, succ indicates success and fail indicates failure. The error information is used to record any error execution step information and error information during the execution of the export task.

[0099] This embodiment can be used to configure file export information, reduce the technical ability requirements of configuration personnel, realize unified management of batch export tasks in various environments, and improve the stability and convenience of export task management at each export time point. The file export information configured by the user is stored in the batch export task instance table, and the export task execution results are stored in the batch export record table, which can realize unified management and monitoring of each batch export task and facilitate the investigation of abnormal export tasks.

[0100] For newly added export batches, the user only needs to configure new file export information to realize the automatic export function. For existing export batches, the user only needs to obtain the corresponding file export information and change the relevant information to automatically realize the export function. The batch data export method and device provided in this article can also improve the processing efficiency of export tasks.

[0101] In one embodiment of the present invention, a method for batch exporting data applied to a server is provided. In specific implementation, each step in the method is implemented by a batch deployment program, such as Figure 2 As shown, including:

[0102] Step 201: receiving file export information configured by the user according to the configuration interface, and storing the file export information in a batch export instance table.

[0103] Among them, the configuration interface includes configuration controls related to the user configuration fields in the batch export instance table. The batch export instance table includes the following user configuration fields: export job name (export batch ID), export time point identifier, export logic, export file name, and export directory.

[0104] When this step is implemented, after detecting the export job name entered by the user, the export job name in the historical file export information is retrieved, and the export job name entered by the user and the retrieved export job name are compared. If the export job name entered by the user is the same as one of the retrieved export job names, the user is prompted to re-enter the export job name or the user is prompted to modify the file export information of the export job name.

[0105] This article ensures the uniqueness of the file export information configuration by configuring the export job name.

[0106] In some implementations, in order to reduce user requirements, the export logic in this step can be input by the user in natural language, and the export script template is determined by identifying the natural language, and the executable export script is obtained after the relevant parameters are filled in. In other implementations, the export logic can also be an executable export script.

[0107] Step 202, after the preset time point arrives, the file export information in the batch export instance table is read one by one, an export file task is generated for each piece of information read, and the export file task is stored in the corresponding export time queue according to the export time point identifier.

[0108] In some implementations, the export time point identifier includes the end of the day and the daytime, which is used to limit the start time of the batch export. The preset time point is, for example, midnight every day, which is used to limit the start time of the batch deployment program. After each start, each data in the batch export instance table is read only once, thereby avoiding repeated operations.

[0109] Step 203: for each export time queue, when the export time point arrives, execute each export file task in the export time queue, and store the execution information of each export file task in the batch export record table.

[0110] When implementing, Figure 3 As shown, when the export time point arrives, for each export file task in the export time queue, the following process is executed:

[0111] Step 301: inserting the record data of the export file task into the batch export record table.

[0112] In this step, the record data inserted in the export file record is empty, that is, the specific values ​​of all fields are null.

[0113] Step 302, updating the record identifier, export batch identifier, export batch date, and export start time in the record data.

[0114] When the step is implemented, the record ID, export batch ID, export batch date and export start time in the record data are filled in with data. For example, the updated record data is:

[0115] Export batch identifier (e.g. instance ID): 1. Record identifier (e.g. export batch ID): 1. Export batch date: 20230708. Export start time: 21:00. Export end time: null. Export execution result: null.

[0116] Step 303: Create a target file in the export directory in the export file task.

[0117] In this step, in order to improve the readability of the target file, the export file name and the export batch date can be combined to obtain the file name. For example, if the export file name is BATCHTMP1 and the export batch date is 20230708, the target file name is BATCHTMP1_20230708.bin.

[0118] Step 304, determining the export script according to the export logic, export file name and target file in the export file task.

[0119] When this step is implemented, the script template associated with the export logic in the export file task is first found from the pre-configured script database, and then the export file name and the target file are added to the export file name variable in the found script template to obtain the export script. The script database contains the association between the export logic and the export script. In this step, the export logic can be a natural language description data. This step can simplify the difficulty of setting the export logic for the user. According to the export logic described in the natural language configured by the user, the export script that can be executed by the computer can be automatically determined, thereby reducing the user's operating cost.

[0120] Step 305, execute the export script to query the data that meets the export logic from the export file name and export it to the target file.

[0121] Step 306: Execute a pre-configured monitoring script during the execution of the export script, so that the monitoring script records error information when the export script is executed abnormally.

[0122] Step 307, calling the pre-configured batch complement program to update the export end time, export execution result, and error information in the record data according to the export execution result and the error information detected by the monitoring script.

[0123] Specifically, exporting the execution result includes determining the export execution result time and the export execution result, filling the export execution result time into the export end time field, filling the export execution result into the export execution result field, and resetting the detected error information into the error information field in the record data.

[0124] In one embodiment of the present invention, the batch export data method further includes: receiving operation information configured by the user according to the configuration interface; and modifying the related file export information in the batch export instance table according to the operation information, wherein the operation information includes deletion information and update information.

[0125] In one embodiment of this invention, Figure 4 As shown, the batch export data method also includes:

[0126] Step 401 , scanning the export execution result of each data in the batch export record table.

[0127] Step 402: Every time a piece of data is scanned, it is determined whether the export execution result in the piece of data is a failure. If so, the piece of data and the export execution result are sent to different persons in charge respectively.

[0128] When this step is implemented, the entire record data or only the export execution result can be sent to the person in charge according to their different responsibilities. The information sending method is, for example, email, which can be pre-configured during specific implementation, and this document does not make specific limitations on this.

[0129] This embodiment can perform all-round monitoring from both a global and individual batch perspective.

[0130] In one embodiment of this invention, Figure 5 As shown, the batch export data method also includes:

[0131] Step 501, obtaining the export start time and export end time in the data.

[0132] Step 502: Count the export duration of the exported file data according to the export start time and the export end time.

[0133] In this step, the export start time and the export end time refer to the export start time and the export end time. The export duration of the exported file data is obtained by subtracting the export start time from the export end time.

[0134] Step 503: Determine whether there is an abnormality based on the export time of the exported file data.

[0135] Specifically, when the export time of the exported file data is greater than a preset threshold, it is determined that there is an export anomaly, otherwise, it is normal. The preset threshold can be determined according to the amount of exported file data, and can also be determined by analyzing the average time taken for exporting historical exported files. This article does not limit its specific value.

[0136] Step 504: If it is abnormal, a prompt message is issued; if it is normal, no processing is required.

[0137] This embodiment can monitor the export duration, and can promptly detect and resolve export anomalies.

[0138] In one embodiment of the present invention, the batch export record table includes: record identifier, export batch identifier, export batch date, export start time, export end time, export execution result, error message and export status field. The export status field is used to indicate whether the batch task is executed for the first time, for example, 0 indicates the first time execution, and 1 indicates non-first time execution. Specifically, Figure 6 As shown, when the export time point arrives, for each export file task in the export time queue, the following process is executed:

[0139] Step 601: inserting record data of the export file task into the batch export record table.

[0140] Step 602, updating the record identifier, export batch identifier, export batch date, and export start time in the record data.

[0141] Step 603, setting the export status field in the record data to first time, for example, to 0, indicating that the export file task is executed for the first time.

[0142] Step 604: Create a target file in the export directory in the export file task.

[0143] Step 605, determining the export script according to the export logic, export file name and target file in the export file task.

[0144] Step 606, execute the export script to query the data that meets the export logic from the export file name and export it to the target file.

[0145] Step 607: Execute a pre-configured monitoring script during the execution of the export script, so that the monitoring script records error information when the export script is executed abnormally.

[0146] Step 608, the monitoring script monitors whether the execution result of the export script is a failure. If the execution result is a failure, the export status field in the relevant record data in the batch export record table is updated to non-first time, for example, 1, and returns to step 606 to continue executing the export script and count the number of consecutive failures.

[0147] Step 609, if the execution result is successful or the number of consecutive failures reaches a predetermined threshold, call the pre-configured batch complement program to update the export end time, export execution result, and error information in the record data according to the export execution result and the error information detected by the monitoring script.

[0148] This embodiment sets the export status and repeats the export task when the export script execution result is a failure. This has high fault tolerance and ensures that the batch export is correct.

[0149] In one embodiment of the present invention, the batch export instance table further includes: a server address field, which is used to indicate the server to be batch exported.

[0150] The above steps 304 and 605 further include:

[0151] (1) According to the server address field, the export script is sent to the relevant server, so that the relevant server executes the export script, searches for data that meets the export logic from the export file name and exports it to the target file. Among them, each server is configured with a monitoring script to record error information when the export script is executed abnormally.

[0152] (2) Receive the export execution results and error information sent by the monitoring script in the relevant server.

[0153] After obtaining the export execution result and error information, continue to execute the above steps 307 and 608.

[0154] This embodiment can achieve unified management of batch export data from multiple servers.

[0155] Based on the same inventive concept, this document also provides a batch data export device, as described in the following embodiments. Since the principle of solving the problem by the batch data export device is similar to that of the batch data export method, the implementation of the batch data export device can refer to the batch data export method, and the repeated parts will not be repeated. Specifically, Fig. 9 As shown, the batch data export device includes:

[0156] Configuration unit 901 is used to receive file export information configured by the user according to the configuration interface, and store the file export information in a batch export instance table. The configuration interface includes configuration controls related to user configuration fields in the batch export instance table. The batch export instance table includes the following user configuration fields: export time point identifier, export logic, export file name, and export directory.

[0157] The task generation unit 902 is used to read the file export information in the batch export instance table one by one after the preset time point, generate an export file task for each piece of information read, and store the export file task into the corresponding export time queue according to the export time point identifier.

[0158] The task execution unit 903 is used for executing each export file task in each export time queue when the export time point arrives, and storing the execution information of each export file task in the batch export record table.

[0159] Furthermore, it also includes: a monitoring unit, which is used to globally or individually monitor the data in the batch export record table. Specifically, it scans the export execution result of each data in the batch export record table; each time a data is scanned, it is determined whether the export execution result in the data is a failure. If so, the data and the export execution result are sent to different persons in charge respectively.

[0160] This embodiment can be conveniently used to configure file export information, reduce the technical ability requirements of configuration personnel, realize unified management of batch export tasks for various environments, and improve the stability and convenience of export task management at each export time point. The file export information configured by the user is stored in the batch export task instance table, and the export task execution results are stored in the batch export record table. It is possible to achieve unified management and monitoring of each batch export task, and it is convenient to troubleshoot abnormal export tasks. For newly added export batches, the user only needs to configure new file export information to realize the automatic export function. For existing export batches, the user only needs to obtain the corresponding file export information and change the relevant information to automatically realize the export function. The batch export data method and device provided in this article can also improve the processing efficiency of batch export.

[0161] The technical solution of this article is described below with a specific embodiment. It is assumed that the server is a WINDOWS server, and the data in the batch export instance table stored in the server is:

[0162] 1) Export batch ID: 1. Export file name: BATCHTMP1.bin, Export job name: Batch export job 1, Export time: 21:00, Export logic: select * from batchtmp1 where condition 1, Export directory: E:\batchreport\batchtmp1\, End of day flag: 1;

[0163] 2) Export batch ID: 2. Export file name: BATCHTMP2.bin. Export job name: Batch export job 2. Export time: 00:10. Export logic: select * from batchtmp2 where condition 2. Export directory: E:\batchreport\batchtmp2\. End-of-day flag: 0.

[0164] The batch deployment procedure on July 8, 2023 to execute the batch export data method process includes:

[0165] Step 0, automatically start the whole process at midnight every day.

[0166] Step 1, read two data records in the batch export instance table (read only once a day without repeated reading operations), and generate two export file tasks task1 to Step 2 and task2 to Step 3.

[0167] Step 2, wait until 21:00, insert data into the batch export record table: record ID: 1, export batch ID: 1, export batch date: 20230708 (the daytime batch is the data that meets the conditions on the evening of 20230708), export status: null, export start time: 21:00, export end time: null, export execution result: null. Go to Step 4.

[0168] Step 3, wait until 00:10, insert data into the batch export record table: record ID: 2, export batch ID: 2, export batch date: 20230707 (the end-of-day batch is the data that meets the conditions on 20230707, the day before 20230708, exported at midnight), export status: null, export start time: 00:10, export end time: null, export execution result: null. Go to Step 5.

[0169] Step 4, set the export status field of the data inserted in Step 2 to 0.

[0170] Step 4.1, create the target file BATCHTMP1_20230708.bin in the server directory E:\batchreport\batchtmp1\ (the file name is generated by the combination of the export file name and the export batch date field), and save the result returned by select * from batchtmp1 where condition 1 into the target file BATCHTMP1_20230708.bin. If the file is generated successfully, go to Step 8, if the file fails to be generated, go to Step 6.

[0171] Step 5, set the export status field of the data inserted in Step 3 to 0.

[0172] Step 5.1, create the target file BATCHTMP2_20230707.bin in the server directory E:\batchreport\batchtmp2\, and save the result returned by select*from batchtmp2 where condition 2 into the target file BATCHTMP2_20230707.bin. If the file is generated successfully or the export status field is 1, go to Step 8. If the file fails to be generated, go to Step 7.

[0173] Step 6, set the export status field of the data inserted in Step 2 to 1, and go to Step 4.1.

[0174] Step 7, set the export status field of the data inserted in Step 2 to 1, and go to Step 5.1.

[0175] Step 8, the complement program monitors the target file generation success, sets the export execution result to succ, and sets the export end time to the current time. The complement program monitors the export status field to 1, sets the export execution result to fail, sets the export end time to the current time, and records the error information in the error information record field, and then goes to Step 9.

[0176] Step 9, end.

[0177] The monitoring program performs full access monitoring from global and individual batch data, including:

[0178] Step 1, scan the entire table to batch export the export execution results in the record table, and then go to Step 2.

[0179] Step 2: If the export execution result recorded in the batch export record table is fail, go to Step 3.

[0180] Step 3, send a push email to the relevant person in charge, who will confirm the reason for the export failure and solve it before going to Step 4.

[0181] Step 4, scan the global records in the batch export record table and go to Step 5.

[0182] Step 5, send a reminder email with the start time, end time, error information and other fields of a single batch to the relevant technical person in charge for inspection, and then go to Step 6.

[0183] Step 6, end.

[0184] In one embodiment of the present invention, a computer device is also provided, such as Fig.10As shown, the computer device 1002 may include one or more processors 1004, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 1002 may also include any memory 1006, which is used to store any kind of information such as code, settings, data, etc. Non-limitingly, for example, the memory 1006 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 1002. In one case, when the processor 1004 executes an associated instruction stored in any memory or a combination of memories, the computer device 1002 may perform any operation of the associated instruction. The computer device 1002 also includes one or more drive mechanisms 1008 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0185] The computer device 1002 may also include an input / output module 1010 (I / O) for receiving various inputs (via input devices 1012) and for providing various outputs (via output devices 1014). A specific output mechanism may include a presentation device 1016 and an associated graphical user interface (GUI) 1018. In other embodiments, the input / output module 1010 (I / O), the input device 1012, and the output device 1014 may not be included, and the computer device 1002 may be used as a computer device in a network. The computer device 1002 may also include one or more network interfaces 1020 for exchanging data with other devices via one or more communication links 1022. One or more communication buses 1024 couple the components described above together.

[0186] The communication link 1022 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1022 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0187] The embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are executed.

[0188] The embodiments of this document also provide a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the method of any of the aforementioned embodiments.

[0189] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0190] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0191] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this article.

[0192] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0193] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0194] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.

[0195] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0196] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0197] Specific embodiments are used in this article to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for general technicians in this field, according to the ideas of this article, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this article.

Claims

1. A method for exporting data in batches, characterized in that: include: Receive file export information configured by the user according to a configuration interface, and store the file export information in a batch export instance table, wherein the configuration interface includes configuration controls related to user configuration fields in the batch export instance table, and the batch export instance table includes the following user configuration fields: export job name, export time point identifier, export logic, export file name, and export directory; After the preset time point arrives, the file export information in the batch export instance table is read one by one, an export file task is generated for each piece of information read, and the export file task is stored in the corresponding export time queue according to the export time point identifier; For each export time queue, when the export time point arrives, each export file task in the export time queue is executed, and the execution information of each export file task is stored in the batch export record table.

2. The method according to claim 1, characterized in that The batch export instance table also includes the following automatically configured fields: export batch identifier, export time; The batch export record table includes the following fields: record identifier, export batch identifier, export batch date, export start time, export end time, export execution result, and error information; The export file task includes file export information.

3. The method according to claim 2, characterized in that When the export time point arrives, each export file task in the export time queue is executed, and the execution information of each export file task is stored in the batch export record table, including: When the export time point arrives, for each export file task in the export time queue, the following process is executed: Insert the record data of the export file task into the batch export record table; Update the record identifier, export batch identifier, export batch date, and export start time in the record data; Create a target file in the export directory in the export file task; Determine the export script according to the export logic, export file name and target file in the export file task; Execute the export script to query the data satisfying the export logic from the export file name and export it to the target file; During the execution of the export script, a pre-configured monitoring script is executed, so that the monitoring script records error information when the export script is executed abnormally; The pre-configured batch complement program is called to update the export end time, export execution result and error information in the recorded data according to the export execution result and the error information detected by the monitoring script.

4. The method according to claim 3, characterized in that Also includes: Scan the export execution result of each data in the batch export record table; Each time a piece of data is scanned, it is determined whether the export execution result in the data is a failure. If so, the data and the export execution result are sent to different persons in charge respectively.

5. The method according to claim 4, characterized in that After each scan of data, it also includes: Get the export start time and export end time of the data; According to the export start time and export end time, the export duration of the exported file data is counted; Determine whether there is an abnormality based on the export duration of the exported file data; If abnormal, a prompt message will be issued.

6. The method according to claim 4, characterized in that The batch export record table also includes: an export status field, which is used to indicate whether the batch task is executed for the first time; The monitoring script is also used to monitor the execution result of the export script; Before updating the export end time, export execution result, and error information in the record data, the method further includes: The monitoring script monitors whether the execution result of the export script is a failure. If the execution result is a failure, the export status field in the relevant record data in the batch export record table is updated to be not the first time, the export script is re-executed, and the number of consecutive failures is counted; If the execution result is successful or the number of consecutive failures reaches a predetermined threshold, a pre-configured batch complement program is called to update the export end time, export execution result, and error information in the recorded data according to the export execution result and the error information detected by the monitoring script.

7. The method according to claim 3, characterized in that Determine the export script according to the export logic, export file name and target file in the export file task; Find out the script template associated with the export logic in the export file task from the pre-configured script database; Add the export file name and target file to the export file name variable in the found script template to obtain the export script.

8. The method according to claim 3, characterized in that The batch export instance table also includes: a server address field; According to the export logic, export file name and target file in the export file task, determine the export script and include: According to the server address field, an export script is sent to a relevant server, and the relevant server executes the export script to query data satisfying the export logic from the export file name and export the data to the target file; Receive the export execution result and error information sent by the monitoring script in the relevant server.

9. The method according to claim 1, characterized in that Also includes: Receive operation information configured by the user according to the configuration interface; The related file export information in the batch export instance table is modified according to the operation information.

10. A device for exporting data in batches, characterized in that: include: A configuration unit, configured to receive file export information configured by a user according to a configuration interface, and store the file export information in a batch export instance table, wherein the configuration interface includes configuration controls related to user configuration fields in the batch export instance table, and the batch export instance table includes the following user configuration fields: export job name, export time point identifier, export logic, export file name, and export directory; A task generating unit, configured to read the file export information in the batch export instance table one by one after a preset time point, generate an export file task for each piece of information read, and store the export file task into a corresponding export time queue according to an export time point identifier; The task execution unit is used for executing each export file task in each export time queue when the export time point arrives, and storing the execution information of each export file task in the batch export record table.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

12. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of a computer device, the method according to any one of claims 1 to 9 is implemented.

Images