Batch file processing method, apparatus and system

Abstract

The application discloses a batch file processing method, device and system, which can be used in the field of artificial intelligence technology, and the method comprises the following steps: receiving a first batch file transmitted by an upstream application server; determining the number of to-be-processed records in the first batch file, and determining the number of historical to-be-processed records in a historical first batch file; determining whether the change rate of the number of to-be-processed records in the first batch file relative to the number of historical to-be-processed records exceeds a preset change rate threshold value; if yes, performing data verification on the first batch file based on a preset rule; and if the verification is passed, forming a second batch file according to the first batch file and transmitting the second batch file to a downstream application server. The application can identify and verify the to-be-processed first batch file which may be abnormal, and improve the processing efficiency of the first batch file.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of batch file processing, in particular to the field of artificial intelligence, and especially to a batch file processing method, device and system. BACKGROUND

[0002] This section is intended to provide background or context to the embodiments of the application recited in the claims. The description herein is not admitted to be prior art merely by inclusion in this section.

[0003] At present, the processing of batch files of banks and other financial institutions is generally performed at night. After a current application receives a batch file transmitted by an upstream application, if the batch file is abnormal, the correctness of the data of the batch file needs to be confirmed by a human being before the batch file is processed again. In this way, on the one hand, the processing of batch files is generally performed at night, and the abnormal batch files need to be confirmed by a human being, which may cause business errors due to untimely processing. On the other hand, the confirmation of batch files by a human being is low in efficiency and prone to errors, which may continue to pass the error data to downstream applications. SUMMARY

[0004] An object of the present application is to provide a batch file processing method for identifying and verifying a to-be-processed batch file that may be abnormal, thereby improving the processing efficiency of batch files. Another object of the present application is to provide a batch file processing device. Still another object of the present application is to provide a batch file processing system. Yet another object of the present application is to provide a computer device. Still another object of the present application is to provide a readable medium.

[0005] To achieve the above objects, in one aspect, the present application discloses a batch file processing method, comprising:

[0006] receiving a first batch file transmitted by an upstream application server;

[0007] determining a to-be-processed record quantity of to-be-processed records in the first batch file, and determining a historical to-be-processed record quantity of historical first batch files;

[0008] determining whether a change rate of the to-be-processed record quantity of the first batch file relative to the historical to-be-processed record quantity exceeds a preset change rate threshold, if yes, performing data verification on the first batch file based on a preset rule, and if the verification is passed, forming a second batch file according to the first batch file and transmitting the second batch file to a downstream application server.

[0009] Preferably, the determination of the historical to-be-processed record quantity of the historical first batch files comprises:

[0010] extracting a file name of the first batch file;

[0011] Based on the filenames, obtain the first batch of historical files within a preset time period before the current time;

[0012] The number of pending records in the first batch of historical files is determined to obtain the total number of pending historical records.

[0013] Preferably, before determining whether the rate of change of the number of records to be processed in the first batch of files relative to the historical number of records to be processed exceeds a preset rate of change threshold:

[0014] The configuration information is obtained by determining the correspondence between the filenames of the first batch of files and the preset change rate threshold.

[0015] Preferably, it further includes:

[0016] Extract the filenames of the first batch of files;

[0017] The change rate threshold corresponding to the filename of the first batch of files is determined from the preset configuration information based on the filename.

[0018] Preferred,

[0019] If the rate of change of the number of pending records in the first batch of files relative to the historical number of pending records does not exceed a preset rate of change threshold, the first batch of files is stored.

[0020] Once the preset conditions are met, a second batch of files is generated based on the stored first batch of files, and the second batch of files is transmitted to the downstream application server for processing.

[0021] Preferably, the step of generating a second batch of files based on the stored first batch of files after the preset conditions are met specifically includes:

[0022] Once the preset conditions are met, feature values ​​corresponding to the downstream file features in the first batch of files are extracted according to the preset downstream file rules.

[0023] Based on the downstream file characteristics and corresponding feature values, a first batch of files is generated according to the downstream file rules, and the second batch of files is transmitted to the downstream application server for processing.

[0024] Preferably, the preset condition is that the file size of all the first batch of stored files reaches a preset value at each preset time interval.

[0025] Preferably, the data verification of the first batch of files based on preset rules specifically includes:

[0026] Based on preset rules, extract the fields to be verified and field attributes of each record to be processed in the first batch file;

[0027] The query results are obtained by querying the validation database based on the field to be validated and the field attributes.

[0028] If a query result exists, the corresponding record to be processed passes the verification; otherwise, the corresponding record to be processed fails the verification.

[0029] Preferably, it further includes:

[0030] For pending records that fail verification, a first batch of files is generated based on the pending records that pass verification. A second batch of files is then generated based on the first batch of files and transmitted to the downstream application server.

[0031] This application also discloses a batch file processing apparatus, comprising:

[0032] The file receiving and processing unit is used to receive the first batch of files transmitted by the upstream application server.

[0033] The data loading and processing unit is used to determine the number of pending records in the first batch file and the historical number of pending records in the first batch file.

[0034] The file generation and processing unit is used to determine whether the rate of change of the number of records to be processed in the first batch of files relative to the number of historical records to be processed exceeds a preset rate of change threshold. If so, the first batch of files is validated based on preset rules. If the validation passes, a second batch of files is generated based on the first batch of files and transmitted to the downstream application server.

[0035] This application also discloses a batch file processing system, including an upstream application server, a batch file processing device, and a downstream application server;

[0036] The batch file processing device is used to receive a first batch of files transmitted from an upstream application server; determine the number of pending records in the first batch of files, and determine the historical number of pending records in the first batch of files; determine whether the rate of change of the number of pending records in the first batch of files relative to the historical number of pending records exceeds a preset rate of change threshold; if so, perform data verification on the first batch of files based on preset rules; if the verification passes, form a second batch of files based on the first batch of files and transmit them to the downstream application server.

[0037] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0038] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0039] This application's batch file processing method receives a first batch of files transmitted from an upstream application server; determines the number of unprocessed records in the first batch of files, and determines the historical number of unprocessed records in the first batch of files; determines whether the rate of change of the number of unprocessed records in the first batch of files relative to the historical number of unprocessed records exceeds a preset rate of change threshold; if so, performs data verification on the first batch of files based on preset rules; if the verification passes, forms a second batch of files based on the first batch of files and transmits them to a downstream application server. Therefore, this application determines whether the rate of change of the number of unprocessed records in the first batch of files exceeds a preset rate of change threshold. If it exceeds the preset rate of change threshold, it automatically performs data verification on the first batch of files; if the verification passes, it then forms a second batch of files based on the first batch of files and transmits them to a downstream application server for processing. Therefore, this application identifies potentially problematic first batch files by monitoring the rate of change in the number of records to be processed in the first batch of files. The first batch of potentially problematic files is then validated. If the validation passes, a second batch of files is generated and sent to the downstream application server for normal processing. This avoids the business delays or errors caused by the need for manual intervention in the validation of abnormal files. At the same time, automated data validation improves the efficiency and accuracy of checking the first batch of files. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0041] Figure 1 This is a schematic diagram of a specific embodiment of the batch file processing system of this application;

[0042] Figure 2 This is a flowchart of a specific embodiment of the batch file processing method of this application;

[0043] Figure 3 This is a flowchart of a specific embodiment S200 of the batch file processing method of this application;

[0044] Figure 4 This is a flowchart of a specific embodiment S000 of the batch file processing method of this application;

[0045] Figure 5 This is a flowchart of a specific embodiment S000 of the batch file processing method of this application, including S020 and S030;

[0046] Figure 6 This is a flowchart of a specific embodiment S400 of the batch file processing method of this application;

[0047] Figure 7 This is a flowchart of a specific embodiment S420 of the batch file processing method of this application;

[0048] Figure 8 This is a flowchart of a specific embodiment S300 of the batch file processing method of this application;

[0049] Figure 9 This is a flowchart of a specific embodiment S500 of the batch file processing method of this application;

[0050] Figure 10 This is a structural diagram of a specific embodiment of the batch file processing device of this application;

[0051] Figure 11 A schematic diagram of a computer device suitable for implementing embodiments of the present invention is shown. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit this application.

[0053] It should be noted that the batch file processing method, apparatus and system disclosed in this application can be used in the field of artificial intelligence technology, or in any field other than artificial intelligence technology. The application field of the batch file processing method, apparatus and system disclosed in this application is not limited.

[0054] It should be noted that, in one or more embodiments of this application, the first batch of files refers to files transmitted by the upstream application server to the batch file processing device or the first batch of files generated by the device itself in a batch manner. The batch file processing device can generate a second batch of files based on the first batch of files and transmit the second batch of files to downstream users.

[0055] To facilitate understanding of the technical solution provided in this application, the relevant content of the technical solution is explained below. The batch file processing method provided in this application can receive a first batch of files transmitted from an upstream application server; determine the number of pending records in the first batch of files, and determine the historical number of pending records in the first batch of files; determine whether the rate of change of the number of pending records in the first batch of files relative to the historical number of pending records exceeds a preset rate of change threshold; if so, perform data verification on the first batch of files based on preset rules; if the verification passes, form a second batch of files based on the first batch of files and transmit them to a downstream application server. This application identifies potentially problematic first batch of files by monitoring the rate of change of the number of pending records in the first batch of files, performs data verification on potentially problematic first batch of files, and if the verification passes, forms a second batch of files and sends them to the downstream application server for normal processing. This avoids the business delays or errors caused by manual intervention in the verification of abnormal files, and improves the efficiency and accuracy of checking the first batch of files through automated data verification.

[0056] Figure 1 This is a schematic diagram of the structure of the batch file processing system provided in the embodiments of this application, as shown below. Figure 1 As shown, the batch file processing system provided in this application embodiment includes an upstream application server 1, a batch file processing device 2, and a downstream application server 3.

[0057] The batch file processing device 2 is used to receive a first batch of files transmitted by the upstream application server 1, determine the number of pending records in the first batch of files, determine the historical number of pending records in the first batch of files, determine whether the rate of change of the number of pending records in the first batch of files relative to the historical number of pending records exceeds a preset rate of change threshold, if so, perform data verification on the first batch of files based on preset rules, and if the verification passes, transmit the second batch of files to the downstream application server 3 for processing.

[0058] The following uses batch file processing device 2 as an example to illustrate the implementation process of the batch file processing method provided in this application embodiment. It is understood that the execution subject of the batch file processing method provided in this application embodiment includes, but is not limited to, the batch file processing device 2.

[0059] According to one aspect of this application, this embodiment discloses a method for batch file processing. For example... Figure 2 As shown, in this embodiment, the method includes:

[0060] S100: Receives the first batch of files transmitted from the upstream application server.

[0061] S200: Determine the number of pending records in the first batch file and the historical number of pending records in the first batch file.

[0062] S300: Determine whether the rate of change of the number of records to be processed in the first batch of files relative to the number of historical records to be processed exceeds a preset rate of change threshold. If so, perform data verification on the first batch of files based on preset rules. If the verification passes, form a second batch of files based on the first batch of files and transmit them to the downstream application server.

[0063] This application determines whether the rate of change of the number of records to be processed in the first batch of files exceeds a preset rate of change threshold. If it does, the first batch of files is automatically validated. If the validation passes, the first batch of files is then transmitted to the downstream application server for processing. Therefore, this application identifies potentially problematic first batch of files by monitoring the rate of change of the number of records to be processed in the first batch of files, validates these potentially problematic first batch of files, and if the validation passes, forms a second batch of files and sends them to the downstream application server for normal processing. This avoids the business delays or errors caused by the need for manual intervention in the validation of abnormal files, and improves the efficiency and accuracy of checking the first batch of files through automated data validation.

[0064] In alternative implementations, such as Figure 3 As shown, S200 determines the number of historical pending records for the first batch of historical files, including:

[0065] S210: Extract the filenames of the first batch of files.

[0066] S220: Obtain the first batch of historical files within a preset time period before the current time based on the filename.

[0067] S230: Determine the number of pending records in the first batch of historical files to obtain the total number of pending historical records.

[0068] Specifically, it is understood that the batch file processing device of this application can agree on the filename of the first batch file with the upstream application server. Then, based on the filename of the current first batch file, it can obtain a historical first batch file with the same filename, and thus determine the number of historical unprocessed records in the historical first batch file. For example, for a first batch file with the filename APPNAME-FILENAME.BIN, the first batch file with the filename APPNAME-FILENAME.BIN can be found in the processing data of historical first batch files.

[0069] Optionally, the first batch of historical files can be the first batch of files processed the day before the current time. In practical applications, the first batch of historical files can also be the first batch of files processed at other times. Those skilled in the art can set it according to the actual situation, and this application does not limit it in this regard.

[0070] The methods for determining the number of pending records in the first batch of files and the number of pending records in the historical first batch of files are conventional techniques in this field and will not be elaborated here.

[0071] The batch file processing device can receive a first batch of files sent by an upstream application server, perform data verification and processing on the first batch of files, and then save the processed data in the batch file processing device. That is, the batch file processing device can receive and process one or more first batch files from one or more upstream application servers, reprocess the data in the first batch files and save it to the database, and then generate a second batch of files according to a pre-agreed file format with the downstream application server before sending it to the downstream application server. The downstream application server can use the second batch of files as needed. In a specific example, the file format of the first batch of files is shown in Table 1. The file format of the second batch of files is similar to that of the first batch of files, and may include fields and corresponding field lengths, etc. The file formats of the first and second batch of files can be pre-agreed with the downstream application server.

[0072] Table 1

[0073] Format Field 1 Length 1 Field 2 Length 2 Field 3 Length 3 Field 4 Length 4 Field N Length N Example 50345 5 5 2 bbbb 4 010300 6 ... ...

[0074] Optionally, the data in the first batch of files and the second batch of files can be encrypted using an encryption algorithm to improve security. The first batch of files and the second batch of files can be compressed during transmission to improve the efficiency of batch file transmission.

[0075] In an optional implementation, the method further includes determining whether the rate of change of the number of records to be processed in the first batch of files relative to the historical number of records to be processed exceeds a preset rate of change threshold, as in step S000. Figure 4 As shown, the S000 includes:

[0076] S010: Determine the correspondence between the filenames of the first batch of files and the preset change rate threshold to obtain configuration information.

[0077] Specifically, it can be understood that the correspondence between configuration file names and preset change rate thresholds can be pre-configured to form configuration information. When the batch file processing device receives the first batch of files and determines the number of records to be processed in the first batch of files, it can read the preset change rate threshold corresponding to the currently received first batch of files from the configuration information according to the file name of the first batch of files, and detect whether the first batch of files is abnormal through the change rate threshold.

[0078] It should be noted that the preset change rate threshold can be a growth rate, which refers to the ratio of the increase in the number of pending records in the current first batch of files relative to the historical number of pending records to the historical number of pending records; the preset change rate threshold can also be a reduction rate, which refers to the ratio of the decrease in the number of pending records in the current first batch of files relative to the historical number of pending records to the historical number of pending records. The preset change rate threshold is preferably set to 5%. Of course, in practical applications, the change rate threshold can also be set according to the actual situation, and this application does not impose any limitations on this.

[0079] Optionally, the configuration information may also include information such as the application name corresponding to the filename, processing status, and processing batch, as shown in Table 2. If the verification fails, the processing status can be set to 0; if the verification passes, the processing status can be set to 1. The processing batch can be set according to the batch number corresponding to the processing procedure. At least some of the information in the configuration information can be obtained through pre-configuration, or it can be updated in the configuration information based on the actual processing status and the information of the first batch of files after receiving the first batch of files transmitted from the upstream application server.

[0080] Table 2

[0081] File Name Application Name Growth Rate Threshold (Percent) Processing Status Processing Batch AAAA-BBBB-CCCC.txt APP1 5 1 001 EEEE-FFFF-KKKK.txt APP2 10 1 002

[0082] In alternative implementations, such as Figure 5 As shown, the method further includes:

[0083] S020: Extract the filenames of the first batch of files.

[0084] S030: Determine the change rate threshold corresponding to the filename of the first batch of files from the preset configuration information based on the filename.

[0085] Specifically, it can be understood that after receiving the first batch of files transmitted from the upstream application server after the preset configuration information, the batch file processing device can parse the file name of the first batch of files and find the change rate threshold corresponding to the file name from the preset configuration information.

[0086] In alternative implementations, such as Figure 6As shown, the method further includes S400:

[0087] S410: If the rate of change of the number of records to be processed in the first batch of files relative to the number of historical records to be processed does not exceed a preset rate of change threshold, store the first batch of files.

[0088] S420: After the preset conditions are met, a second batch of files is generated based on the stored first batch of files, and the second batch of files is transmitted to the downstream application server for processing.

[0089] Specifically, it can be understood that if the rate of change of the number of pending records in the first batch of files relative to the historical number of pending records does not exceed the preset rate of change threshold, it means that the first batch of files has not changed much compared with the historical first batch of files, and is usually a normal first batch of files. The first batch of files passes the verification, and the second batch of files can be generated based on the stored first batch of files, and the second batch of files is transmitted to the downstream application server for processing.

[0090] In alternative implementations, such as Figure 7 As shown, the process of generating a second batch of files based on the stored first batch of files after the preset conditions are met specifically includes:

[0091] S421: After the preset conditions are met, extract the feature values ​​corresponding to the downstream file features in the first batch of files according to the preset downstream file rules.

[0092] S422: Generate a first batch of files according to the downstream file characteristics and corresponding feature values ​​and the downstream file rules, and transmit the second batch of files to the downstream application server for processing.

[0093] Specifically, for the first batch of files that pass verification, the first batch of files can be stored first. Then, the fields and corresponding data of the fields in the first batch of files can be extracted according to the pre-agreed file format. The extracted data can be stored in the form of a data table according to the preset format in the batch file processing device. The data in the stored data table can be periodically used to generate a second batch of files according to the file format of the downstream application server so that the second batch of files can be sent to the downstream application server for processing.

[0094] Optionally, the batch file processing device can periodically scan for the existence of a first batch of files that need to be processed, and the timing of the periodic scan can be determined based on information such as the processing time of the first batch of files. Furthermore, the batch file processing device can also support parallel processing of batch files; for example, it can simultaneously process a first batch of files named AAAA-BBBB-CCCC.txt and EEEE-FFFF-KKKK.txt.

[0095] In an optional implementation, the preset condition is that the file size of all the first batch of stored files reaches a preset value at preset time intervals. Of course, in practical applications, preset conditions can also be set according to actual needs, and this application does not limit this.

[0096] In alternative implementations, such as Figure 8 As shown, step S300, which performs data verification on the first batch of files based on preset rules, specifically includes:

[0097] S310: Extract the fields to be verified and field attributes of each record to be processed in the first batch file based on preset rules.

[0098] S320: Query the verification database based on the field to be verified and the field attribute to obtain the query result.

[0099] S330: If a query result exists, the corresponding record to be processed passes the verification; if no query result exists, the corresponding record to be processed fails the verification.

[0100] Specifically, it is understandable that for the first batch of files that exceeds the rate of change threshold, it is necessary to further examine each record in the first batch of files to determine whether there are any abnormal records in the first batch of files that cause the number of records to be processed in the first batch of files to exceed the preset rate of change threshold.

[0101] The batch file processing device can directly verify the first batch of files, extracting the fields to be verified from the records to be processed in the first batch of files according to preset rules, and extracting the corresponding field attributes. The preset rules can be determined based on normal records to be processed in the previous first batch of files. For example, Table 3 records multiple data records, including the field to be verified (Field 1) and its corresponding field attributes, which can be obtained from normal historical first batch files. The field attributes include scene, field 2, field 3, field 4…field n. Due to the different scenes attribute, the data of the field to be verified (Field 1) has different ranges or values, and the corresponding fields 2, 3, 4…field n also differ depending on the range or value of the field 1. Therefore, the fields to be verified and the field attributes of the records to be processed in the first batch of files can be extracted according to the preset rules corresponding to Table 3. The extracted values ​​of the field to be verified (Field 1) and the attribute values ​​of the fields to be processed are then queried in the database, specifically in Table 3 of the database, to obtain the query results. If a query result exists, indicating that a matching data record exists, it means that the record to be processed has appeared historically and is normal, and can be directly used to form the second batch of files. If no query results are found, meaning no matching data record exists, it indicates that the record to be processed has not appeared in the historical processing and is likely an abnormal record. Therefore, the verification of the record to be processed will fail.

[0102] Table 3

[0103]

[0104] In other optional implementations, a separate data verification server can be set up to verify the first batch of files. This data verification server can receive the fields to be verified and field attributes of the records to be processed in the first batch of files transmitted by the batch file processing device. The extracted values ​​of the fields to be verified and the attribute values ​​of the attributes in the records to be processed are queried in the database. If a query result exists, that is, a matching data record exists, it means that the record to be processed has appeared in the past and is normal, and can be directly used to form the second batch of files. If no query result exists, that is, no matching data record exists, it means that the record to be processed has not appeared in the past processing and is likely an abnormal record, and the verification of the record to be processed fails.

[0105] In alternative implementations, such as Figure 9 As shown, the method further includes:

[0106] S500: For pending records that fail verification, form a first batch file based on the pending records that pass verification, form a second batch file based on the first batch file, and transmit it to the downstream application server.

[0107] Specifically, this application does not intercept the first batch of files containing records that fail verification and wait for manual processing. Instead, it stores the records that fail verification separately and waits for manual processing. For the records that pass verification, they can be directly used to generate the second batch of files and transmitted to the downstream application server in a timely manner.

[0108] The following example will further illustrate this application. We will use the first batch file named APPNAME-FILENAME.BIN as an example:

[0109] Step S301: Before starting to process the first batch of files, it is necessary to predefine the file format of the first batch of files transmitted by one or more upstream application servers, and configure information such as the growth rate of each file, including file name, application name, growth rate threshold, processing status and batch information.

[0110] Assuming the parameters of the first batch file APPNAME-FILENAME.BIN are as shown in Table 4, the batch file processing device needs to process the APPNAME-FILENAME.BIN file received from the upstream application server. The application source of the file is F-APP. The percentage increase or decrease of the number of pending records in the first batch of files being processed relative to the number of pending records in the first batch of files processed in the previous day is greater than 5% or less than -5%, then the change rate threshold is triggered. The processing status is 1, indicating that this first batch of files is normal, and the processing batch is 001.

[0111] Table 4

[0112]

[0113] Step S302: The batch file processing device receives one or more first batch files sent by the upstream application server. The first batch files define the fields to be processed according to the rules agreed upon with the downstream application server. Assume that the data format of the APPNAME-FILENAME.BIN file has four fields, namely field 1, field 2, field 3 and field 4, each separated by a comma. Field 1 is the field that needs to be queried and associated with the data verification server 3. Field 1 can be used as the primary key of the file, or field 1 can be combined with other fields to form a primary key. Assume that part of the data to be processed is as follows:

[0114] 625678,6,abcd,010101

[0115] 985678,6,efgh,020301

[0116] After performing an integrity check on the received APPNAME-FILENAME.BIN file, the batch file processing device calculates the number of records to be processed. Then, it obtains the number of records to be processed in the first batch of files from the previous day. Finally, it calculates the growth rate of the number of records to be processed in the APPNAME-FILENAME.BIN file. If the growth rate exceeds the change rate threshold of 5%, the device assembles the values ​​of the fields to be verified and the corresponding attributes of the current records to be processed into a message and sends it to the data verification server. Otherwise, it forms a second batch of files based on the first batch of files and transmits it to the downstream application server.

[0117] Step S303: The data verification server receives data from the batch file processing device that exceeds the rate of change threshold and needs to be verified, as shown in Table 5. The data verification server uses the data in field 1 (e.g., 625678) in the message as input conditions and queries the stored original data in the local database or file system. If the corresponding data is found, it then matches the information in field 2 (e.g., 6), field 3 (e.g., abcd), and field 4 (e.g., 010101), or matches only other fields as needed. If the matching is successful, the verification is successful. If the corresponding data is found but other attributes are incorrect, the attributes of the current data are considered suspicious, and the data verification fails. In this embodiment, for pending records with abnormal attribute values, the attribute values ​​need to be updated and modified. The modified pending records can be generated together with the verified pending records to form a second batch of files.

[0118] Step S304: After the batch file processing device completes the data verification of one or more first batch files (such as APPNAME-FILENAME.BIN), it backs up the data of the last processed APPNAME-FILENAME.BIN. If there are multiple first batch files, the data of other first batch files need to be backed up. Then, the data marked as "successful" or "suspicious" in one or more first batch files is processed and inserted or updated into the database of the batch file processing device. The batch processing time of the records imported into the database is modified to the current batch processing time, that is, the imported records to be processed are valid data that have passed the automatic verification and can be used.

[0119] Step S305: After the batch file processing device completes the data entry of all the first batch files, it checks the records in the batch file processing device database. If there is a record whose processing time is the processing time of the batch on the same day, it means that the current record is the latest data. The second batch file is then regenerated according to the batch date of the day and the data format agreed upon by the upstream and downstream applications for downstream use.

[0120] Based on the same principle, this application also discloses a batch file processing device. For example... Figure 10As shown, the device includes a file receiving and processing unit 11, a data loading and processing unit 12, and a file generating and processing unit 13.

[0121] The file receiving and processing unit 11 is used to receive the first batch of files transmitted by the upstream application server.

[0122] The data loading and processing unit 12 is used to determine the number of pending records in the first batch file and the number of pending records in the historical first batch file.

[0123] The file generation and processing unit 13 is used to determine whether the rate of change of the number of records to be processed in the first batch of files relative to the number of historical records to be processed exceeds a preset rate of change threshold. If so, the first batch of files is validated based on preset rules. If the validation passes, a second batch of files is generated based on the first batch of files and transmitted to the downstream application server.

[0124] Since the principle by which this device solves the problem is similar to the methods described above, the implementation of this device can be found in the implementation of the methods, and will not be repeated here.

[0125] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0126] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0127] Those skilled in the art will understand that the embodiments of this application can be provided as methods, systems, or computer programs, producing the systems, apparatuses, modules, or units described in the above embodiments. Specifically, they can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer device; specifically, a computer device can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0128] In a typical example, the computer device specifically includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method executed by the client as described above, or the method executed by the server as described above.

[0129] The following is for reference. Figure 11It shows a schematic diagram of the structure of a computer device 600 suitable for implementing the embodiments of this application.

[0130] like Figure 11 As shown, the computer device 600 includes a central processing unit (CPU) 601, which can perform various appropriate tasks and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0131] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal feedback (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 606 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed in storage section 608 as needed.

[0132] In particular, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611.

[0133] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0134] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0135] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0141] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0142] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A batch file processing method, characterized in that, include: Receive the first batch of files transmitted from the upstream application server; Determine the number of pending records in the first batch file, and determine the historical number of pending records in the first batch file. Determine whether the rate of change of the number of pending records in the first batch of files relative to the historical number of pending records exceeds a preset rate of change threshold. If so, perform data verification on the first batch of files based on preset rules. If the verification passes, form a second batch of files based on the first batch of files and transmit them to the downstream application server. Specifically, the data verification of the first batch of files based on preset rules includes: Based on preset rules, extract the fields to be verified and field attributes of each record to be processed in the first batch file; The query results are obtained by querying the validation database based on the field to be validated and the field attributes. If a query result exists, the corresponding record to be processed passes the verification; otherwise, the corresponding record to be processed fails the verification.

2. The batch file processing method according to claim 1, characterized in that, The determination of the number of historical pending records for the first batch of historical files includes: Extract the filenames of the first batch of files; Based on the filenames, obtain the first batch of historical files within a preset time period before the current time; The number of pending records in the first batch of historical files is determined to obtain the total number of pending historical records.

3. The batch file processing method according to claim 1, characterized in that, Before determining whether the rate of change of the number of records to be processed in the first batch of files relative to the historical number of records to be processed exceeds a preset rate of change threshold: The configuration information is obtained by determining the correspondence between the filenames of the first batch of files and the preset change rate threshold.

4. The batch file processing method according to claim 3, characterized in that, Further includes: Extract the filenames of the first batch of files; The change rate threshold corresponding to the filename of the first batch of files is determined from the preset configuration information based on the filename.

5. The batch file processing method according to claim 1, characterized in that, If the rate of change of the number of pending records in the first batch of files relative to the historical number of pending records does not exceed a preset rate of change threshold, the first batch of files is stored. Once the preset conditions are met, a second batch of files is generated based on the stored first batch of files, and the second batch of files is transmitted to the downstream application server for processing.

6. The batch file processing method according to claim 5, characterized in that, The step of generating a second batch of files based on the stored first batch of files after the preset conditions are met specifically includes: Once the preset conditions are met, feature values ​​corresponding to the downstream file features in the first batch of files are extracted according to the preset downstream file rules. Based on the downstream file characteristics and corresponding feature values, a first batch of files is generated according to the downstream file rules, and the second batch of files is transmitted to the downstream application server for processing.

7. The batch file processing method according to claim 5, characterized in that, The preset condition is that the file size of all the first batch of stored files reaches a preset value at preset time intervals.

8. The batch file processing method according to claim 1, characterized in that, Further includes: For pending records that fail verification, a first batch of files is generated based on the pending records that pass verification. A second batch of files is then generated based on the first batch of files and transmitted to the downstream application server.

9. A batch file processing device, characterized in that, include: The file receiving and processing unit is used to receive the first batch of files transmitted by the upstream application server. The data loading and processing unit is used to determine the number of pending records in the first batch file and the historical number of pending records in the first batch file. The file generation and processing unit is used to determine whether the rate of change of the number of records to be processed in the first batch of files relative to the number of historical records to be processed exceeds a preset rate of change threshold. If so, the first batch of files is validated based on a preset rule. If the validation passes, a second batch of files is generated based on the first batch of files and transmitted to the downstream application server. The file generation and processing unit is further configured to extract the fields to be verified and field attributes of each record to be processed in the first batch of files based on preset rules. The query results are obtained by querying the verification database based on the field to be verified and the field attribute; if a query result exists, the corresponding record to be processed passes the verification; if no query result exists, the corresponding record to be processed fails the verification.

10. A batch file processing system, characterized in that, This includes upstream application servers, batch file processing devices, and downstream application servers; The batch file processing device is used to receive a first batch of files transmitted from an upstream application server; determine the number of pending records in the first batch of files, and determine the historical number of pending records in the first batch of files; determine whether the rate of change of the number of pending records in the first batch of files relative to the historical number of pending records exceeds a preset rate of change threshold; if so, perform data verification on the first batch of files based on preset rules; if the verification passes, form a second batch of files based on the first batch of files and transmit them to the downstream application server; wherein, the data verification on the first batch of files based on preset rules specifically includes: extracting the fields to be verified and field attributes of each pending record in the first batch of files based on preset rules; querying the verification database based on the fields to be verified and the field attributes to obtain query results; if a query result exists, the corresponding pending record passes the verification; if no query result exists, the corresponding pending record fails the verification.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images