Batch file verification method, apparatus, device, medium, and program product

Abstract

The disclosure provides a batch file verification method, which can be applied to the fields of big data and financial technology. The method comprises the following steps: determining table data of a first database; synchronizing the table data to one or more second databases; performing scheduling execution on the table data in the first database to obtain a first batch file, and performing scheduling execution on the table data in the second database to obtain a second batch file; and comparing the first batch file with the second batch file to determine that the contents of the first batch file and the second batch file are consistent; wherein the first database and the second database are different databases, and the table data is batch data. The disclosure also provides a batch file verification device, equipment, storage medium and program product.

Description

Technical Field

[0001] This disclosure relates to the fields of big data and finance, and specifically to a batch file verification method, apparatus, device, medium, and program product. Background Technology

[0002] During the process of database localization, to ensure that business functional logic and upstream and downstream interactions remain unchanged, it is necessary to verify the consistency of batch files before and after the transformation. Each business system has a large number of batch files. Currently, the full-process verification of batch files is mainly done manually. To ensure the consistency of the generated batch files, testers need to manually verify the databases before and after the transformation.

[0003] Comparing individual batch files is time-consuming in terms of initial data preparation, and differences between batch files require visual inspection and manual recording, a tedious process. Furthermore, the volume of batch files across various business systems is substantial, necessitating significant manpower for batch file verification. Additionally, database localization transformation projects typically have short cycles, leaving testers with very limited testing time. Within this limited time, testers can only select key batches for verification, resulting in incomplete coverage and potential omissions, leading to production issues.

[0004] In addition, preparing the foundational data requires testers to have a grasp of batch logic and to clearly identify which business tables are involved in the batch processing in order to prepare the relevant basic data. Because the transformation involves two heterogeneous databases, testers need to understand the differences between the databases, including data types, system functions, SQL syntax, etc., thus presenting a certain business and technical threshold for testers. Summary of the Invention

[0005] In view of the above problems, this disclosure provides a method, apparatus, device, medium and program product for batch file verification based on database transformation to improve testing efficiency, and to at least partially solve the above technical problems.

[0006] According to a first aspect of this disclosure, a batch file verification method is provided, comprising: determining table data in a first database; synchronizing the table data to one or more second databases; scheduling and executing the table data in the first database to obtain a first batch of files, and scheduling and executing the table data in the second database to obtain a second batch of files; and comparing the first batch of files and the second batch of files to determine that the contents of the first batch of files and the second batch of files are consistent; wherein the first database and the second database are different databases, and the table data is batch data.

[0007] According to embodiments of this disclosure, determining the table data of the first database includes: determining the batch framework information of the business system, the batch framework information including the stored procedure mapping logic of a batch or batch group; determining the access data table of the first database according to the stored procedure mapping logic; and parsing the access data table to obtain the table data.

[0008] According to embodiments of this disclosure, in a first database, scheduling and executing table data to obtain a first batch of files, and in a second database, scheduling and executing table data to obtain a second batch of files includes: filtering table data to obtain intermediate process data; generating intermediate process files based on the intermediate process data; or scheduling and executing table data corresponding to a batch group to obtain result data; generating result files based on the result data; wherein the first batch file and the second batch file are both intermediate process files or result files.

[0009] According to embodiments of this disclosure, scheduling and executing table data corresponding to a batch group to obtain result data includes: calling a shell program of a batch scheduling management tool; using the shell program to schedule and execute table data corresponding to the batch group to obtain result data; wherein the shell program includes a command parser that interacts with the operating system.

[0010] According to embodiments of this disclosure, comparing a first batch of files with a second batch of files to determine that the contents of the first batch of files and the second batch of files are consistent includes: comparing the filenames of the first batch of files and the second batch of files; calculating the MD5 values ​​of the first batch of files and the second batch of files respectively when the filenames of the first batch of files and the second batch of files are the same; and determining that the contents of the first batch of files and the second batch of files are consistent when the MD5 values ​​of the first batch of files and the second batch of files are the same.

[0011] According to embodiments of this disclosure, the batch framework information further includes batch file structure information, and the method further includes: when the MD5 values ​​of the first batch files and the second batch files are different, parsing the first batch files and the second batch files respectively according to the batch file structure information; comparing the file contents of the parsed first batch files and the second batch files to determine the differences.

[0012] According to embodiments of this disclosure, comparing a first batch of files with a second batch of files to determine that the contents of the first batch of files and the second batch of files are consistent includes: establishing a batch comparison platform; and comparing the first batch of files with the second batch of files through the batch comparison platform.

[0013] According to embodiments of this disclosure, the batch file verification method further includes: displaying the comparison results of the first batch of files and the second batch of files through a batch comparison platform.

[0014] A second aspect of this disclosure provides a batch file verification apparatus, comprising: a determining module for determining table data in a first database; a synchronization module for synchronizing the table data to one or more second databases, wherein the first database and the second databases are different databases, and the table data is batch data; a scheduling module for scheduling and executing the table data in the first database to obtain a first batch of files, and scheduling and executing the table data in the second database to obtain a second batch of files; and a comparison module for comparing the first batch of files and the second batch of files to determine that the contents of the first batch of files and the second batch of files are consistent.

[0015] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods of any of the above embodiments.

[0016] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods of any of the above embodiments.

[0017] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the methods of any of the above embodiments.

[0018] Compared with existing technologies, the batch file verification method, apparatus, electronic device, storage medium, and program product provided in this disclosure have at least the following beneficial effects:

[0019] (1) The batch file verification method disclosed herein synchronizes table data in multiple different databases, ensuring that the underlying data of the source database and the target database are identical. By comparing the batch files generated through the same scheduling execution, batch file content verification can be achieved during the transformation between different databases. The method disclosed herein can be implemented by establishing an automated platform, thereby improving the efficiency of batch file verification.

[0020] (2) The batch file verification method disclosed herein directly extracts the batch stored procedure mapping logic from the batch framework information of the business system, and then determines the table data based on the stored procedure mapping logic. The batch file generation process based on the table data does not require testers to pay attention to the batch logic of the business, which lowers the technical threshold for testing and improves the adaptability of the verification method.

[0021] (3) The batch file verification method disclosed herein can obtain table data that can be intermediate process data or execution result data, which can achieve full coverage of table data and avoid the occurrence of production problems after database transformation. Attached Figure Description

[0022] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0023] Figure 1 The illustration schematically depicts application scenarios of batch file verification methods, apparatus, devices, media, and program products according to embodiments of the present disclosure;

[0024] Figure 2 A flowchart illustrating a batch file verification method according to an embodiment of the present disclosure is shown schematically.

[0025] Figure 3 A flowchart illustrating a method for determining table data in a first database according to an embodiment of the present disclosure is shown.

[0026] Figure 4 A flowchart illustrating a method for generating batch files according to an embodiment of the present disclosure is shown schematically.

[0027] Figure 5 A flowchart illustrating a method for generating result data according to an embodiment of the present disclosure is shown schematically.

[0028] Figure 6 A flowchart illustrating a batch comparison method according to an embodiment of the present disclosure is shown schematically;

[0029] Figure 7 A flowchart illustrating a batch comparison method according to another embodiment of the present disclosure is shown schematically;

[0030] Figure 8A A flowchart illustrating a batch comparison method according to yet another embodiment of the present disclosure is shown schematically; Figure 8B A schematic block diagram of a bulk file apparatus according to an embodiment of the present disclosure is shown. Figure 8C A flowchart illustrating a batch file verification method according to another embodiment of the present disclosure is shown schematically;

[0031] Figure 9 A flowchart illustrating a batch file verification method according to yet another embodiment of the present disclosure is shown schematically;

[0032] Figure 10 A schematic diagram illustrating a bulk document verification apparatus according to another embodiment of the present disclosure is shown; and

[0033] Figure 11 A block diagram schematically illustrates an electronic device suitable for implementing a batch file verification method according to an embodiment of the present disclosure. Detailed Implementation

[0034] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0036] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0037] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0038] This disclosure provides a batch document verification method, apparatus, device, medium, and program product, which can be used in the financial field or other fields. It should be noted that the batch document verification method, apparatus, device, medium, and program product of this disclosure can be used in the financial field, as well as in any field other than the financial field; the application field of the batch document verification method, apparatus, device, medium, and program product of this disclosure is not limited.

[0039] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0040] This disclosure provides a batch file verification method, comprising: determining table data in a first database; synchronizing the table data to one or more second databases; scheduling and executing the table data in the first database to obtain a first batch of files, and scheduling and executing the table data in the second database to obtain a second batch of files; and comparing the first batch of files and the second batch of files to determine that their contents are consistent. The first database and the second database are different databases, and the table data is batch data.

[0041] Figure 1 The illustration schematically depicts an application scenario of a batch file verification method, apparatus, device, medium, and program product according to embodiments of the present disclosure.

[0042] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and servers 105 and 106. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and servers 105 and 106. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0043] Users can use terminal devices 101, 102, and 103 to interact with servers 105 and 106 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as database maintenance applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0044] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0045] Servers 105 and 106 can be servers providing various services, such as backend management servers supporting websites browsed by users using terminal devices 101, 102, and 103 (this is just an example). The backend management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated based on user requests) to the terminal devices. Specifically, server 105 is a database server running at least two different types of databases, and can respond to requests from terminal devices 101, 102, and 103 by converting these databases. Server 106 is a server running a business system, used to provide batch framework information to server 105.

[0046] It should be noted that the batch file verification method provided in this embodiment can generally be executed by servers 105 and 106. Correspondingly, the batch file verification device provided in this embodiment can generally be located in servers 105 and 106. The batch file verification method provided in this embodiment can also be executed by a server or server cluster that is different from servers 105 and 106 and can communicate with terminal devices 101, 102, 103 and / or servers 105 and 106. Correspondingly, the batch file verification device provided in this embodiment can also be located in a server or server cluster that is different from servers 105 and 106 and can communicate with terminal devices 101, 102, 103 and / or servers 105 and 106.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0048] Entegor: A batch job scheduling and management tool that reduces the risks of manual operation through automation.

[0049] Shell: A command interpreter software written in C that interacts with the operating system. It receives user input commands and then calls the corresponding applications. Shell is also a programming language. As a command language, it interactively interprets and executes user-input commands or automatically interprets and executes a pre-defined series of commands. Shell has two modes of command execution: interactive and batch. Interactive: The shell interprets and executes user commands one by one. Batch: The user writes a shell script containing many commands, which the shell executes all at once, instead of typing them one by one.

[0050] Batch processing refers to the unified processing of multiple data entries in a single task, improving system efficiency through the unified processing of multiple records. For example, in interest calculation, a single batch processing operation can calculate the interest for all accounts under a specific branch or bank.

[0051] Batch group: refers to the combination of multiple individual batches into a complete function.

[0052] Batch framework: refers to the basic environment that provides the operation of batch processing, including concurrent processing of batches and scheduling of pre- and post-dependent dependencies between batches.

[0053] Batch files: refers to the output of batch processing results or intermediate processing data in the form of files for use by other systems.

[0054] Batch comparison tool: Batch functions are mainly used for database operations. After database transformation and upgrade, it is necessary to verify the correctness of batch function execution under different databases before and after the transformation. The batch comparison tool is a test and verification tool for batch functions and batch files before and after the transformation.

[0055] Batch comparison platform: This refers to the testing tools that are built and provided to testers for front-end operation to achieve batch comparison functions.

[0056] Foundation data: refers to the basic data and business data required for batch operation, which can be understood as test data in the test environment.

[0057] The following will be based on Figure 1 The described scene, through Figures 2-9 The batch file verification method of the disclosed embodiments is described in detail.

[0058] Figure 2 A flowchart illustrating a batch file verification method according to an embodiment of the present disclosure is shown schematically.

[0059] like Figure 2 As shown, the batch file verification method of this embodiment includes operations S210 to S240, and the batch file verification method can be executed by a computer program on corresponding computer hardware.

[0060] In operation S210, the table data of the first database is determined.

[0061] In operation S220, table data is synchronized to one or more second databases. The first and second databases are different databases, and the table data is batch data.

[0062] In operation S230, table data is scheduled and executed in the first database to obtain the first batch of files, and table data is scheduled and executed in the second database to obtain the second batch of files.

[0063] In operation S240, the first batch of files and the second batch of files are compared to determine that the contents of the first batch of files and the second batch of files are consistent.

[0064] For example, a company is transforming its existing MySQL relational database to a MongoDB non-relational database. First, the table data in the MySQL database is identified, including user information, order information, and product information. Then, this table data is synchronized to the MongoDB database, creating corresponding collections such as "Users," "Orders," and "Products." Next, the table data in the MySQL database is processed, resulting in the first batch of files containing the MySQL table data. Simultaneously, the table data in the MongoDB database is also processed, resulting in the second batch of files containing the MongoDB collection data. Finally, the first and second batches of files are compared to determine if the content of the generated MongoDB collection data after the transformation is consistent with the original MySQL table data. If the contents of the two files are completely identical, it indicates that the data has not changed during the transformation process, and the table data in the transformed MongoDB database is consistent with the data in the original MySQL database. This batch file verification method ensures that the content of the batch files generated after the transformation remains unchanged during the transformation process, thus guaranteeing the accuracy and stability of the database transformation.

[0065] For example, consider a telecommunications operator where user account information is stored in an Oracle database, while real-time call logs are stored in a Redis database. First, the account table data in the Oracle database is determined, including user ID, name, balance, and other information. Then, the account table data is synchronized to the Redis database, creating a key-value pair collection named "Accounts" and storing the account information within it. Next, the account table data in the Oracle database is processed, resulting in the first batch of files containing the Oracle account data. Simultaneously, the account table data in the Redis database is also processed, resulting in the second batch of files containing the Redis key-value pair collection. Finally, the first and second batches are compared, checking if fields such as user ID, name, and balance for each account are consistent. If all account information is completely identical, then the contents of the first and second batches are confirmed to be consistent, meaning the account data in the Oracle and Redis databases is consistent. This batch file verification method allows the telecommunications operator to verify account data consistency and ensure data synchronization between the two databases, even though Oracle and Redis are different types of databases.

[0066] For example, a company might be migrating its existing Oracle relational database to Gauss DB. First, data analysis is performed to determine the table structure, fields, and data in the Oracle database. Then, the Oracle table structure and fields are mapped to the Gauss DB database. This includes creating the corresponding tables and columns and ensuring that data types and constraints match. Once mapping is complete, ETL tools or data migration tools can be used to migrate the data from the Oracle database to the Gauss DB database. This process can be done through batch import or row-by-row insertion. After the data migration is complete, it's necessary to verify the consistency between the data in the transformed Gauss DB database and the original Oracle database. A random sample of data can be selected for comparison to verify the accuracy and consistency of the records. After successful data verification, integrity and performance tests can be performed to ensure the stability and performance of the Gauss DB database when processing data. Through these steps, the company can successfully migrate data from its Oracle database to Gauss DB, ensuring data accuracy and consistency throughout the process. This allows them to smoothly migrate from Oracle to Gauss DB, enjoying higher performance and better data management capabilities.

[0067] Figure 3 A flowchart illustrating a method for determining table data in a first database according to an embodiment of the present disclosure is shown.

[0068] According to embodiments of this disclosure, such as Figure 3 As shown, for example, table data of the first database is determined by operations S311 to S313.

[0069] In operation S311, the batch framework information of the business system is determined. The batch framework information includes the stored procedure mapping logic of a batch or a batch group.

[0070] In operation S312, the access data table of the first database is determined according to the stored procedure mapping logic.

[0071] In operation S313, the access data table is parsed and the table data is obtained.

[0072] For example, due to differences in the batch processing frameworks of various business systems, it is necessary to collect batch processing framework information from these systems in advance. This includes information such as the stored procedure mapping logic under batch processing and batch groups, the determination of batch execution status, the directory where batch files are stored on the server after generation, and the batch file structure information. Taking Oracle database as an example, its batch processing logic is basically implemented through stored procedures. By accessing Oracle views and parsing stored procedures, all tables accessed by the stored procedure can be obtained. Then, this table data is exported from the production-like Oracle database. The exported table data is then imported into the Oracle database to be batch tested and the target database for transformation, ensuring that the test databases before and after the transformation have the same underlying data, guaranteeing the generation of identical batch files while maintaining the same logic.

[0073] For example, Oracle's base view USER_DEPENDENCIES stores the objects and accessed tables that a stored procedure (one or more stored procedures can constitute a stored procedure) depends on. By parsing the stored procedure's code text, the accessed tables within a single stored procedure can be determined. For instance, stored procedures a1, a2, and a3 form stored procedure A. The USER_DEPENDENCIES view provides access to tables b1, b2, and b3 of A. Parsing and splitting stored procedure A reveals that stored procedure a1 accesses table b1, a2 accesses table b2, and a3 accesses table b3.

[0074] Figure 4 A flowchart illustrating a method for generating batch files according to an embodiment of the present disclosure is shown schematically.

[0075] According to embodiments of this disclosure, such as Figure 4 As shown, for example, by operating S431 to S434, table data can be scheduled and executed to obtain batch files.

[0076] In operation S431, the table data is filtered to obtain intermediate process data.

[0077] When operating S432, an intermediate process file is generated based on the intermediate process data. Or

[0078] In operation S433, the table data corresponding to the batch group is scheduled and executed to obtain the result data.

[0079] In operation S434, result files are generated based on the result data. The first batch file and the second batch file can both be intermediate process files or result files.

[0080] For example, an e-commerce company might have a large user data table. It can filter user data from an Oracle database based on specific criteria and pass the filtered results to a batch group for further business processing. First, the structure and fields of the user data table are analyzed, and filtering criteria are defined. For example, filtering for user data with purchase records within the past month. Then, SQL queries can be used to execute the filtering operation in the Oracle database, extracting user data that meets the criteria as intermediate process data. This intermediate process data will only contain user data under specific conditions. After filtering, the intermediate process data can be passed to the batch group for subsequent business processing. The intermediate process data can be transmitted in file format, such as CSV or JSON files. A batch group is a complete business function composed of multiple single batch tasks. In this embodiment, the batch group can use the received intermediate process data as input and execute corresponding operations according to business logic, such as calculating the user's total spending and generating user purchase reports. After the batch group completes, it will generate result data. This result data can be statistical analysis reports, user purchase record updates, etc. Through these steps, the e-commerce company can filter user data from the Oracle database based on specific criteria and further process and analyze the filtered results by calling the batch group. Therefore, it can improve the efficiency of data processing and business functions, and generate result data that meets business needs.

[0081] Figure 5 A flowchart illustrating a method for generating result data according to an embodiment of the present disclosure is shown schematically.

[0082] According to embodiments of this disclosure, such as Figure 5 As shown, for example, by operating S5331 to S5332, the table data corresponding to the batch group is scheduled and executed to obtain the result data.

[0083] When operating S5331, the shell program of the batch scheduling management tool is invoked.

[0084] In operation S5332, a shell program is used to schedule and execute the table data corresponding to the batch group to obtain the result data. The shell program includes a command parser that interacts with the operating system.

[0085] For example, Entegor can be used to schedule batch execution in an Oracle database. By directly scheduling the pre-written shell file under Entegor, batch or batch group execution can be achieved, and the same method can be used to schedule batch execution in the target database after transformation.

[0086] For example, a bank uses the Entegor batch job management system, which has already written some shell scripts to execute specific batch tasks or batch groups. First, corresponding shell scripts can be written for different batch tasks or batch groups. These shell scripts contain code and commands that implement specific business logic. Then, through the Entegor batch job management system, the scheduling parameters of the scripts can be configured, such as execution time and execution frequency. This associates batch tasks or batch groups with specific time points or frequencies. After configuration, the Entegor system will trigger the corresponding shell scripts to execute at the specified time points or frequencies according to the set scheduling parameters. Thus, batch tasks or batch groups will be scheduled and executed according to the predetermined plan. After execution, the Entegor system will record the execution results, including success or failure information. The execution status of each batch task or batch group can be viewed through the system's monitoring interface or logs. If errors or anomalies occur, appropriate measures can be taken promptly. Through the above steps, the Entegor batch job management system can directly schedule the written shell files, achieving automated execution of batch tasks or batch groups. This can greatly reduce the workload of manual intervention and improve the efficiency and accuracy of batch jobs.

[0087] It's important to note that besides directly executing shell files through a scheduling system, there are other methods to achieve batch scheduling of database operations. For example, using the database's built-in scheduling functionality: Many database management systems (such as Oracle, MySQL, SQL Server, etc.) provide built-in scheduling capabilities, allowing you to execute batch operations by creating scheduled tasks or jobs. These scheduling capabilities typically offer flexible scheduling options, such as specifying execution time, repeating execution, and dependencies. Many third-party scheduling tools are also available for batch database scheduling, such as Autosys, Control-M, and Cron. These tools offer more complex scheduling features, such as distributed scheduling, fault handling, and logging. If the database does not provide built-in scheduling capabilities or third-party scheduling tools are unavailable, you can write scripts or programs to achieve batch scheduling. For example, you can write a scheduled task using programming languages ​​such as Python or Java to execute batch operations through a database connection. Finally, you can use ETL tools: ETL (Extract, Transform, and Load) tools are commonly used for data integration and data processing. Some ETL tools have built-in scheduling capabilities, making it easy to execute batch operations. For example, you can use ETL tools to create workflows and schedule their execution. Regardless of the method chosen, the flexibility, reliability, and security of scheduling must be considered. At the same time, it is essential to ensure appropriate monitoring and logging during batch scheduling execution to promptly identify and resolve issues.

[0088] Figure 6 A flowchart illustrating a batch comparison method according to an embodiment of the present disclosure is shown schematically.

[0089] According to embodiments of this disclosure, such as Figure 6 As shown, for example, by operating S641 to S643, the first batch of files and the second batch of files are compared to determine that the contents of the first batch of files and the second batch of files are consistent.

[0090] In operation S641, the filenames of the first batch of files are compared with those of the second batch of files.

[0091] In operation S642, assuming the filenames of the first batch of files and the second batch of files are the same, calculate the MD5 values ​​of the first batch of files and the second batch of files respectively.

[0092] In operation S643, if the MD5 values ​​of the first batch of files and the second batch of files are the same, it is determined that the contents of the first batch of files and the second batch of files are consistent.

[0093] For example, based on the collected batch framework information, it can be determined whether the database batch processing before and after the transformation has been successfully executed. After the batch processing is complete, the server that generated the corresponding batch files is logged in to retrieve the batch files, and it is determined whether batches with the same name have been generated before and after the transformation. If batches with the same name have been generated, the MD5 value of the batch files is calculated to determine whether the content of the generated batch files is consistent. If the MD5 values ​​are consistent, it is determined that the content of the first batch file and the second batch file are consistent. In the case of identical filenames, by calculating the MD5 value of the batch files, the content of the entire batch file can be compared, achieving fast file consistency comparison.

[0094] Figure 7 A flowchart illustrating a batch comparison method according to another embodiment of the present disclosure is shown.

[0095] According to embodiments of this disclosure, such as Figure 7 As shown, for example, operations S741 to S742 are used to compare the first batch of files with the second batch of files to determine the differences.

[0096] When operating S741, if the MD5 values ​​of the first batch of files and the second batch of files are different, the first batch of files and the second batch of files are parsed separately according to the batch file structure information.

[0097] In operation S742, the file contents of the first batch of parsed files and the second batch of files are compared to determine the differences.

[0098] For example, MD5 values ​​can compare the entire file content, enabling rapid consistency checks. If files are inconsistent, a detailed comparison of the file content is required to identify the specific discrepancies. In cases of inconsistent MD5 values, the batch files need to be parsed according to their structure, and their contents compared in detail to identify and record the differences. Each line in the batch file represents a record, and each record contains only a value, not a key. These records are concatenated using delimiters to obtain a fixed-length file. The file structure represents the composition of each record, such as the length of each value and the delimiters used to concatenate values. In other words, the file structure can be used to split a single line record into corresponding key-value pairs. By analyzing the key-value pairs for each line, the differences between the first and second batches of files can be quickly identified, allowing for relevant optimizations and adjustments to improve database conversion efficiency.

[0099] Figure 8A A flowchart illustrating a batch comparison method according to yet another embodiment of the present disclosure is shown. Figure 8B A schematic block diagram of a bulk file apparatus according to an embodiment of the present disclosure is shown. Figure 8C A flowchart illustrating a batch file verification method according to another embodiment of the present disclosure is shown schematically.

[0100] According to embodiments of this disclosure, such as Figure 8A As shown, for example, by operating S841 to S842, the first batch of files and the second batch of files are compared to determine that the contents of the first batch of files and the second batch of files are consistent.

[0101] When operating S841, establish a batch comparison platform.

[0102] When operating S842, the first batch of files is compared with the second batch of files through a batch comparison platform.

[0103] For example, such as Figure 8BAs shown, the batch file device 800 includes an information acquisition module 810, a data preparation module 820, a batch scheduling module 830, a batch comparison module 840, and a comparison platform module 850. The information acquisition module collects batch framework information from the business system and inputs it into the system, providing logical support for subsequent batch execution and comparison. The data preparation module is responsible for parsing the data tables involved in the stored procedures in the batch and preparing the underlying data for the business data tables. The batch scheduling module schedules the Shell files to ensure consistent batch execution before and after the transformation. The batch comparison module compares the batch-generated files to determine the differences and whether there are any problems with the transformed batch logic. The comparison platform module implements the functions of inputting collected information, initiating batch comparison tasks, and visualizing the results.

[0104] For example, such as Figure 8C As shown, the batch file verification method includes: analyzing the batch framework of the business system to obtain batch framework information and inputting it into the batch comparison tool; parsing the stored procedures corresponding to the batch to obtain the business tables involved in the batch, exporting the table data from the production-like environment, and synchronizing the exported data to the database to be compared to prepare the foundational data; automatically scheduling the batch by executing the shell file on the server; determining the batch execution status based on the information collected by the batch framework, comparing the files of the completed batch, obtaining the differences, and determining the correctness of the transformed logic; and initiating a comparison task in the batch comparison tool system and displaying the detailed results of the comparison task.

[0105] Figure 9 A flowchart illustrating a batch file verification method according to yet another embodiment of the present disclosure is shown.

[0106] According to embodiments of this disclosure, such as Figure 9 As shown, for example, the comparison results of batch files can be displayed by operating S910.

[0107] When operating the S910, the comparison results of the first batch of files and the second batch of files are displayed through the batch comparison platform. Visualizing the comparison results of batch files provides a high degree of intuitiveness and makes it easier to pinpoint potential problems.

[0108] Based on the above-described batch file verification method, this disclosure also provides a batch file verification device. The following will be combined with... Figure 10 This batch file verification device is described in detail.

[0109] Figure 10 A schematic block diagram of a batch document verification apparatus according to another embodiment of the present disclosure is shown.

[0110] like Figure 10 As shown, the batch file verification device 1000 of this embodiment includes, for example, a determination module 1010, a synchronization module 1020, a scheduling module 1030, and a comparison module 1040.

[0111] The determining module 1010 is used to determine the table data of the first database. In one embodiment, the determining module 1010 can be used to perform the operation S210 described above, which will not be repeated here.

[0112] The synchronization module 1020 is used to synchronize table data to one or more second databases, wherein the first database and the second database are different databases, and the table data is batch data. In one embodiment, the synchronization module 1020 can be used to perform the operation S220 described above, which will not be repeated here.

[0113] The scheduling module 1030 is used to schedule and execute table data in the first database to obtain a first batch of files, and in the second database to schedule and execute table data to obtain a second batch of files. In one embodiment, the scheduling module 1030 can be used to execute the operation S230 described above, which will not be repeated here.

[0114] The comparison module 1040 is used to compare the first batch of files with the second batch of files to determine that the contents of the first batch of files and the second batch of files are consistent. In one embodiment, the comparison module 1040 can be used to perform the operation S240 described above, which will not be repeated here.

[0115] According to embodiments of this disclosure, any plurality of modules among the determining module 1010, synchronization module 1020, scheduling module 1030, and comparison module 1040 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the determining module 1010, synchronization module 1020, scheduling module 1030, and comparison module 1040 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the determination module 1010, synchronization module 1020, scheduling module 1030 and comparison module 1040 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0116] Figure 11 A block diagram schematically illustrates an electronic device suitable for implementing a batch file verification method according to an embodiment of the present disclosure.

[0117] like Figure 11 As shown, an electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage portion 1108 into a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0118] RAM 1103 stores various programs and data required for the operation of electronic device 1100. Processor 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Processor 1101 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1102 and / or RAM 1103. It should be noted that the programs may also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0119] According to embodiments of this disclosure, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to a bus 1104. The electronic device 1100 may also include one or more of the following components connected to the I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1110 as needed so that computer programs read from it can be installed into the storage section 1108 as needed.

[0120] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the batch file verification method according to the embodiments of this disclosure.

[0121] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1102 and / or RAM 1103 and / or one or more memories other than ROM 1102 and RAM 1103 described above.

[0122] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the batch file verification method provided in the embodiments of this disclosure.

[0123] When the computer program is executed by the processor 1101, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0124] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1109, and / or installed from the removable medium 1111. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0125] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by processor 1101, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0126] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0128] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0129] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A batch file verification method, characterized in that, include: Determine the table data in the first database; Synchronize the table data to one or more second databases; In the first database, the table data is scheduled and executed to obtain a first batch of files; and in the second database, the table data is scheduled and executed to obtain a second batch of files. as well as The first batch of files and the second batch of files are compared to determine that the contents of the first batch of files and the second batch of files are consistent. Wherein, the first database and the second database are different databases, and the table data is batch data; The determination of table data in the first database includes: Determine the batch framework information of the business system, wherein the batch framework information includes the stored procedure mapping logic for a batch or a batch group; The access data table of the first database is determined according to the stored procedure mapping logic; Parse the access data table to obtain the table data; The steps of scheduling and executing the table data in the first database to obtain a first batch of files, and scheduling and executing the table data in the second database to obtain a second batch of files, include: The table data is filtered to obtain intermediate process data; Based on the intermediate process data, generate an intermediate process file; or The table data corresponding to the batch group is scheduled and executed to obtain the result data; Generate a result file based on the obtained data; Wherein, the first batch file and the second batch file are both the intermediate process file or the result file.

2. The method according to claim 1, characterized in that, The scheduled execution of the table data corresponding to the batch group yields the following result data: A shell program that calls a batch scheduling management tool; The shell program is used to schedule and execute the table data corresponding to the batch group to obtain the result data; The shell program includes a command parser that interacts with the operating system.

3. The method according to claim 1, characterized in that, The step of comparing the first batch of files with the second batch of files to determine that the contents of the first batch of files and the second batch of files are consistent includes: Compare the filenames of the first batch of files with those of the second batch of files; Calculate the MD5 values ​​of the first batch of files and the second batch of files if their filenames are the same; and If the MD5 values ​​of the first batch of files and the second batch of files are the same, it is determined that the contents of the first batch of files and the second batch of files are consistent.

4. The method according to claim 3, characterized in that, The batch framework information also includes batch file structure information, and the method further includes: If the MD5 values ​​of the first batch of files and the second batch of files are different, the first batch of files and the second batch of files are parsed respectively according to the batch file structure information; Compare the contents of the first batch of files and the second batch of files after parsing to identify the differences.

5. The method according to claim 1, characterized in that, The step of comparing the first batch of files with the second batch of files to determine that the contents of the first batch of files and the second batch of files are consistent includes: Establish a batch comparison platform; The first batch of files and the second batch of files are compared using the batch comparison platform.

6. The method according to claim 5, characterized in that, Also includes: The comparison results between the first batch of files and the second batch of files are displayed using the batch comparison platform.

7. A batch file verification device, characterized in that, include: The determination module is used to determine table data in a first database, including: determining batch framework information of a business system, wherein the batch framework information includes stored procedure mapping logic for batches or batch groups; determining access data tables of the first database according to the stored procedure mapping logic; and parsing the access data tables to obtain the table data. The synchronization module is used to synchronize the table data to one or more second databases, wherein the first database and the second database are different databases, and the table data is batch data; A scheduling module is configured to schedule and execute table data in the first database to obtain a first batch of files, and to schedule and execute table data in the second database to obtain a second batch of files. The module includes: filtering the table data to obtain intermediate process data; generating intermediate process files based on the intermediate process data; or scheduling and executing table data corresponding to the batch group to obtain result data; and generating result files based on the result data. The first batch file and the second batch file are both either the intermediate process files or the result files. The comparison module is used to compare the first batch of files with the second batch of files to determine whether the contents of the first batch of files and the second batch of files are consistent.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images