Data processing method, system, device, equipment and medium

Abstract

The invention provides a data processing method, system and device, equipment and a medium, and relates to the field of data processing. The method comprises the steps of receiving a second file from a second server, obtaining first source data in a first database, executing a batch processing program on the first source data to obtain a first file, comparing the first file with the second file, if the comparison result is consistent, determining that the first source data is successfully migrated, and if the comparison result is inconsistent, determining that the first source data is not successfully migrated. If yes, it is determined that migration of the first source data is wrong, the second file is obtained by obtaining second source data from a second database by a second server and executing a batch processing program on the second source data, and the first source data is obtained by migrating the second source data in the second database from the second database to the first database. According to the method, the problem that in the prior art, when data are migrated between two databases, it cannot be guaranteed that the data are completely consistent in the two databases before and after migration is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data processing method, system, device, equipment and medium. Background Art

[0002] With the continuous development of databases, databases are used more and more widely in the financial industry. More and more companies and institutions choose to use databases to store their business information, among which the most involved is database migration operation.

[0003] In the database migration operation, it is necessary to ensure that all data is migrated from the old database to the new database, and the data before and after the migration remain completely consistent. Especially in the financial industry, it is particularly important to ensure the correctness of the migration of payment systems, amounts, account numbers, and time. Even small data differences may lead to payment errors, which in turn cause serious business problems.

[0004] Therefore, how to ensure that data is completely consistent before and after migration is an urgent problem that needs to be solved. Summary of the invention

[0005] Embodiments of the present application provide a data processing method, system, apparatus, device and medium for solving the technical problem of inaccurate data migration when migrating data between two databases in the prior art.

[0006] In a first aspect, an embodiment of the present invention provides a data processing method, which is applicable to a first server, and the method includes: receiving a second file from a second server, the second file is obtained by the second server acquiring second source data in a second database and executing a batch processing program on the second source data, acquiring first source data in a first database, and executing a batch processing program on the first source data to obtain a first file, the first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database, and comparing the first file and the second file, if the comparison results are consistent, it is determined that the migration of the first source data is successful, if the comparison results are inconsistent, it is determined that the migration of the first source data is erroneous.

[0007] Through the above steps, the accuracy of data migration can be verified by re-executing the processing flow based on the actual data of the original system, that is, the source data is re-run through the application to generate a result file, and then the result file is compared to determine whether the data in the first database and the data in the second database after migration are consistent. Based on this, before the application is officially put into use, the comprehensive processing and comparison verification of key files can be achieved, which can not only effectively ensure the accuracy of data migration, but also greatly improve the stability of the application. This solution belongs to a preventive verification process, which can detect and solve potential data inconsistency problems early, thereby minimizing the risks that may be caused by data migration. In addition, the present application is a result file after the comparison application is re-run, which can be directly sent to the downstream data for processing. If the comparison results are consistent, it can ensure that the file sent to the downstream data is accurate. Compared with the comparison of source data in the prior art, the present application avoids the problem that even if the comparison results of the source data are consistent, the result files obtained after processing are different, which leads to errors in the files sent to the downstream data.

[0008] In a possible embodiment, for any one of the first file and the second file: it includes one or more of a confession file, an accounting file and other files, the confession file is a file used to transmit, store or exchange data between different databases, the accounting file is a file related to transaction accounts, and the other files are files other than the confession file and the financial file.

[0009] Through the above scheme, the accuracy of data migration can be ensured by comparing files. Compared with the comparison of source data in the prior art, this application compares the source data after re-running the application. The comparison result is more accurate, and there are different comparison processing methods for different file types, which improves the efficiency of the comparison.

[0010] In a possible embodiment, the first file includes a first data supply file, the second file includes a second data supply file, and the first file and the second file are compared, including: comparing the first data supply file and the second data supply file line by line to obtain difference fields, and determining whether the difference fields affect the processing of downstream data components. If so, determining that the comparison results are inconsistent; if not, determining that the comparison results are consistent, wherein the downstream data component is a component located in a layer below the layer where the first server is located, and is used to receive and process data from the first server.

[0011] Through the above scheme, we can focus on the differences in all the comparison fields in the difference result file and evaluate whether these differences will have a substantial impact on the loading and processing of downstream data components. If it is determined that these differences will not cause any impact, the system will adopt a flexible processing strategy and treat them as no difference, thereby avoiding unnecessary alarms and processing. This intelligent judgment mechanism greatly improves the efficiency and accuracy of the system.

[0012] In a possible embodiment, the first file includes a first accounting file, the second file includes a second accounting file, and the first file and the second file are compared, including: comparing the first accounting file and the second accounting file line by line, obtaining a difference field, and determining whether the difference field belongs to a key field, if so, determining that the comparison result is inconsistent, and if not, determining that the comparison result is consistent, wherein the key field includes at least one of the amount, account number, and time. In this way, by performing a comprehensive comparison on a large number of batch accounting processing files, a wider range of business scenarios and data types can be covered, ensuring the comprehensiveness and reliability of the verification results.

[0013] In a possible embodiment, the second file is a compressed file. After receiving the second file from the second server, it also includes: storing the second file in a setting file directory, the setting file directory is configured to automatically decompress files existing inside, importing the decompressed file corresponding to the second file into the first database, and comparing the first file and the second file, including: comparing the decompressed files corresponding to the first file and the second file.

[0014] Through the above scheme, the second file is placed in the set file directory for decompression, and the decompressed second file is imported into the first database. During the data transmission process, the file is compressed and decompressed, which can reduce the file size and thus speed up the file transmission efficiency. In addition, placing the file in the set file directory for decompression can better organize and manage data for subsequent processing and searching. The present application can realize automatic file decompression, reduce labor costs and improve efficiency.

[0015] In a possible embodiment, the first file includes multiple first sub-files, and the second file includes multiple second sub-files. After receiving the second file from the second server, it also includes: obtaining a preset configuration file from the first database, the configuration file includes identifiers of multiple second sub-files arranged in sequence according to a set order, and the set order is consistent with the order of the multiple first sub-files in the first file; according to the preset configuration file, the multiple second sub-files in the second file are imported into the first database according to the set order, and the decompressed files corresponding to the first file and the second file are compared, including: comparing the first sub-file and the second sub-file located at the same position in the first file and the second file imported from the first database.

[0016] Through the above scheme, by using a preset configuration file, it can be ensured that the sub-files at corresponding positions in the first file and the second file are correctly compared, and by automatically importing and comparing files in a set order, no manual intervention is required, which reduces labor costs and improves the efficiency of data import.

[0017] In a second aspect, an embodiment of the present invention provides a data processing method, which is applicable to a second server, and the method includes: obtaining second source data in a second database, and executing a batch processing program on the second source data to obtain a second file. Sending the second file to a first server, the second file is used by the first server to obtain the first source data in the first database, and executing a batch processing program on the first source data to obtain a first file, and comparing the first file with the second file, the first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database.

[0018] Through the above steps, the source data is re-run through the application to generate a result file, and the result file is compared to determine whether the data in the first database and the data in the second database after migration are consistent. In this way, the accuracy of data migration from the second database to the first database can be ensured.

[0019] In a possible embodiment, for any file in the first file and the second file: including one or more of the data supply file, the accounting file and other files, the data supply file is a file used to transmit, store or exchange data between the first database and the second database, the accounting file is a file related to transaction accounts, and the other files are files other than the data supply file and the accounting file.

[0020] Through the above scheme, the accuracy of data migration can be ensured by comparing files. Compared with the comparison of source data in the prior art, this application compares the source data after re-running the application. The comparison result is more accurate, and there are different comparison processing methods for different file types, which improves the efficiency of the comparison.

[0021] In a possible embodiment, sending the second file to the first server includes: obtaining a preset correspondence table from a second database, the preset correspondence table containing correspondences between identifiers of multiple files and path information of the multiple files, determining the path information corresponding to the second file according to the preset correspondence table, compressing the second file stored in the path indicated by the path information, and sending the compressed second file to the first server.

[0022] Through the above solution, compressing files can reduce the size of files, thereby speeding up file transfer. By identifying the file path in the configuration file to compress files, an automated file processing process is implemented, reducing human intervention and errors.

[0023] In a third aspect, an embodiment of the present invention provides a data processing method, the method comprising: obtaining second source data in a second database, and executing a batch processing program on the second source data to obtain a second file, obtaining first source data from a first database, and executing a batch processing program on the first source data to obtain a first file, the first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database, and comparing the first file and the second file, if the comparison results are consistent, it is determined that the migration of the first source data is successful, if the comparison results are inconsistent, it is determined that the migration of the first source data is erroneous.

[0024] Through the above steps, the accuracy of data migration can be verified by re-executing the processing flow based on the actual data files of the original system. The present invention comprehensively processes and compares and verifies key files before the application is officially put into use. This method can not only effectively ensure the accuracy of data migration, but also greatly improve the stability of the application. Through this preventive verification process, potential data inconsistency problems can be discovered and solved early, thereby minimizing the risks that may be caused by data migration. In addition, the present application is a result file after the comparison application is re-run. The file can be directly sent to the downstream data for processing. If the comparison results are consistent, it can be ensured that the file sent to the downstream data is accurate. Compared with the comparison of source data in the prior art, the present application avoids the problem that even if the source data comparison results are consistent, the result files obtained after processing are different, which leads to errors in the files sent to the downstream data.

[0025] In a fourth aspect, the present application provides a data processing system, the system comprising: a product service layer, a first server and a second server are provided in the product service layer, a first database is provided in the first server, a second database is provided in the second server, the first server is connected to the second server, the second server is used to obtain second source data from the second database, and execute a batch program on the second source data to obtain a second file, and send it to the first server, the first server is used to obtain the first source data from the first database after receiving the second file, and execute a batch program on the first source data to obtain a first file, compare the first file and the second file, if the comparison results are consistent, it is determined that the second server has successfully migrated the first source data, if the comparison results are inconsistent, it is determined that the migration of the first source data has failed, wherein the first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database.

[0026] Through the above scheme, the architecture of the data processing system can be simplified. In the existing system architecture, two independent data processing systems (also called application systems) are set up, and complete data processing environments are built separately. A first server is set up in one application system, and a second server is set up in another application system. The two servers interact between the two application systems to verify the data. This method requires that all layers of data in each application system must be rerun before data comparison can be achieved. The steps are relatively cumbersome. This scheme does not require the construction of two independent application systems. It only requires the creation of a first server on the product server where the second server is located on the original application system. Through the interaction between the first server and the second server in the same layer, the scheme of automatically pulling data, automatically rerunning applications, and automatically comparing can be achieved. It does not require the rerunning of upper and lower layer applications. The steps are simple and easy to implement.

[0027] In a possible embodiment, the data processing system also includes an external integration layer and a data integration layer. The external integration layer is provided with a third database, and the data integration layer is provided with a fourth database. The second server is connected to the third database and the fourth database respectively. The second server is also used to obtain data from the third database, generate files after batch processing the data, and send the files to the fourth database.

[0028] In a fifth aspect, the present application provides a data processing device, the device comprising:

[0029] A receiving unit, configured to receive a second file from a second server, where the second file is obtained by the second server acquiring second source data in a second database and executing a batch processing program on the second source data;

[0030] an acquisition unit, configured to acquire first source data in a first database, and execute a batch processing program on the first source data to obtain a first file; the first source data in the first database is obtained by migrating second source data in a second database from the second database to the first database;

[0031] The comparison unit is used to compare the first file and the second file. If the comparison results are consistent, it is determined that the migration of the first source data is successful. If the comparison results are inconsistent, it is determined that the migration of the first source data is wrong.

[0032] In a possible embodiment, for any one of the first file and the second file: it includes one or more of a confession file, an accounting file and other files, the confession file is a file used to transmit, store or exchange data between different databases, the accounting file is a file related to transaction accounts, and the other files are files other than the confession file and the financial file.

[0033] In a possible embodiment, the first file includes a first data supply file, and the second file includes a second data supply file. The comparison unit is specifically used to: compare the first data supply file and the second data supply file line by line to obtain difference fields, and determine whether the difference fields affect the processing of downstream data components. If so, it is determined that the comparison results are inconsistent; if not, it is determined that the comparison results are consistent, wherein the downstream data component is a component located in a layer below the layer where the first server is located, and is used to receive and process data from the first server.

[0034] In a possible embodiment, the first file includes a first accounting file, the second file includes a second accounting file, and the comparison unit is specifically used to: compare the first accounting file and the second accounting file line by line, obtain the difference field, and determine whether the difference field belongs to the key field, if it does, determine that the comparison result is inconsistent, and if it does not, determine that the comparison result is consistent, wherein the key field includes at least one of the amount, account number, and time. In this way, by performing a comprehensive comparison on a large number of batch accounting processing files, a wider range of business scenarios and data types can be covered, ensuring the comprehensiveness and reliability of the verification results.

[0035] In a possible embodiment, the second file is a compressed file. After receiving the second file from the second server, the receiving unit is further used to: store the second file in a setting file directory, the setting file directory is configured to automatically decompress files existing inside, and import the decompressed file corresponding to the second file into the first database. The comparison unit is specifically used to: compare the decompressed files corresponding to the first file and the second file.

[0036] In a possible embodiment, the first file includes multiple first sub-files, and the second file includes multiple second sub-files. After receiving the second file from the second server, the receiving unit is further used to: obtain a preset configuration file from the first database, the configuration file includes identifiers of multiple second sub-files arranged in sequence according to a set order, the set order is consistent with the order of the multiple first sub-files in the first file, and according to the preset configuration file, the multiple second sub-files in the second file are imported into the first database according to the set order; the comparison unit is specifically used to: compare the first sub-file and the second sub-file located at the same position in the first file and the second file imported from the first database.

[0037] In a sixth aspect, the present application provides a data processing device, the device comprising:

[0038] A processing unit, used for acquiring second source data in a second database, and executing a batch processing program on the second source data to obtain a second file;

[0039] A sending unit is used to send a second file to a first server, the second file is used by the first server to obtain first source data in a first database, and execute a batch processing program on the first source data to obtain a first file, and compare the first file with the second file. The first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database.

[0040] In a possible embodiment, for any file in the first file and the second file: including one or more of the data supply file, the accounting file and other files, the data supply file is a file used to transmit, store or exchange data between the first database and the second database, the accounting file is a file related to transaction accounts, and the other files are files other than the data supply file and the accounting file.

[0041] In a possible embodiment, the processing unit is specifically used to: obtain a preset correspondence table from a second database, the preset correspondence table containing correspondence between identifiers of multiple files and path information of multiple files, determine the path information corresponding to the second file according to the preset correspondence table, compress the second file stored in the path indicated by the path information, and send the compressed second file to the first server.

[0042] In a seventh aspect, the present application provides a data processing device, including:

[0043] a processing unit, configured to obtain second source data in the second database, execute a batch processing program on the second source data to obtain a second file, obtain first source data from the first database, execute a batch processing program on the first source data to obtain a first file, wherein the first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database;

[0044] The comparison unit is used to compare the first file and the second file. If the comparison results are consistent, it is determined that the migration of the first source data is successful. If the comparison results are inconsistent, it is determined that the migration of the first source data is wrong.

[0045] In an eighth aspect, the present application provides an electronic device, including:

[0046] A memory for storing program instructions;

[0047] The processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the methods in the first aspect according to the obtained program instructions.

[0048] In a ninth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes any one of the methods described in the first aspect.

[0049] In a tenth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes any one of the methods described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a data processing system provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of another data processing system provided in an embodiment of the present application;

[0052] Figure 3 A flowchart of a data processing method provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of a process for file comparison provided in an embodiment of the present application;

[0054] Figure 5 A flowchart of another data processing method provided in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of the structure of an application system provided in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of a data processing device provided by an embodiment of the present invention;

[0057] Figure 8 A schematic diagram of another data processing device provided by an embodiment of the present invention;

[0058] Fig. 9 A schematic diagram of another data processing device provided by an embodiment of the present invention;

[0059] Fig.10 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0061] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0062] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0063] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0064] Before introducing the data processing method provided in the embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced in detail below.

[0065] First, some terms in this application are explained. It should be noted that these explanations are for the convenience of understanding by those skilled in the art and do not constitute a limitation on the scope of protection claimed in this application.

[0066] 1. Oracle database.

[0067] Oracle Database is a widely used relational database management system. It supports large-scale transaction processing, data warehouses, and recently, cloud database services. Oracle Database is known for its stability, scalability, and security, and is suitable for processing large amounts of data and complex application systems.

[0068] 2. GoldenDB database.

[0069] GoldenDB database (GDB for short) is a distributed database management system widely used in China's financial industry. GoldenDB is designed to meet the financial industry's needs for high concurrency, high availability and high performance, and is particularly suitable for financial institutions such as banks, securities, and insurance.

[0070] 3. Data comparison.

[0071] Compare the formats of data files generated by database applications and scripts to identify inconsistencies in data generated by applications and scripts due to differences in heterogeneous databases.

[0072] 4. Data migration.

[0073] After exporting the data from the source database, use the file transfer tool to transfer the data file from the source database to the machine where the target database is located, and then import the exported source data file into the target database environment.

[0074] 5. DBTOOLS.

[0075] The data import tool provided by the GoldenDB manufacturer can import database text data exported in a specified format into the GoldenDB database and generate a result log of the imported data.

[0076] 6. Fserver.

[0077] A tool for point-to-point file transfer that can pull the corresponding files from the target server or push the corresponding files from the source server to the target server according to the specified file server path and file name.

[0078] 7. Batch processing of accounts.

[0079] A method of processing accounting details in the form of files. Because there are many details in the file, abnormalities in the batch processing program will often cause problems in the processing of details in the entire file.

[0080] 8. Data download tool.

[0081] In the process of database management and data migration, a data downgrade tool usually refers to a tool used to extract, transform and load data. It is responsible for extracting data from one database system (source database), possibly performing some preprocessing or transformation, and then loading it into another environment (target database or file system).

[0082] 9. Supply number.

[0083] Data provision refers to the process of providing data from one system or database to another system or user for processing, analysis or use. This process involves the extraction, processing, transmission and reception of data, and is a key link to ensure the smooth flow and use of data between different systems and departments. In scenarios such as finance, internal enterprise data sharing and data analysis, data provision is a very important operation that ensures the timeliness, accuracy and completeness of data.

[0084] See also Figure 1 , a system diagram of data processing is given as an example. Among them, the external integration layer is mainly responsible for connecting with external systems and interacting with data. By integrating different external interfaces, the external integration layer can achieve docking with third-party services or systems and provide external data support for the system; the product service layer is mainly responsible for generating various business files, such as credit card files, personal loan files, acquisition files, etc.; the data integration layer is used to store and manage a large amount of raw data from different data sources, and integrate and analyze these data. The product service layer receives data from the external integration layer and sends the processed data to the data integration layer, where the product service layer can include an Oracle deployment unit or a GDB deployment unit.

[0085] If the data of the product service layer is stored in an Oracle deployment unit, when you want to migrate the data in the Oracle deployment unit to the GDB deployment unit, you will face the challenges of syntax and tool changes. If errors occur in the data migration process, it may bring potential risks to the actual use of the application. Therefore, you need to compare the data before and after the migration.

[0086] At present, data comparison is mainly a dual-machine parallel verification method, that is, the same transaction message data is transmitted to two independent systems for processing at the same time, and then the processing results of the two systems are compared. If the results are consistent, it means that the status before and after the data migration is consistent. However, when this method is applied to a system that does not save the original transaction message, the following two problems will arise:

[0087] First, building a dual-machine parallel environment is complex in itself. In order to ensure that the simulated transaction messages can be executed accurately, a comprehensive analysis of the system is required. This includes not only an in-depth understanding of the system architecture and data flow, but also a detailed understanding of the system's business logic and processing rules. Such analysis is large and complex, requiring the team to have high technical capabilities and business understanding.

[0088] Second: Even if a dual-machine parallel environment is successfully built, the results of the simulation verification may not cover all trading scenarios. Real trading systems often contain a large number of transaction types and processing logic, and it is difficult for the simulation environment to fully reproduce all possible trading scenarios.

[0089] In addition, the current data migration process mainly relies on manual operations, that is, manually executing the migration script or using the migration tool to export the specified data to the file disk, and then transferring the comparison file from the source machine to the target machine. Finally, the operator manually imports the comparison data and verifies whether the import result meets the requirements. This manual intervention method has the following four problems:

[0090] First: The whole process is time-consuming and inefficient.

[0091] Second: Since data migration usually needs to be carried out during the low trading peak period in the early morning, it increases the workload of operators.

[0092] Third: In order to increase the number and reliability of comparison samples, data migration requires repeating the same operations every day, further increasing labor costs.

[0093] Fourth: Due to the differences in processing time of different business systems, the processing time required for various data processing modules or subsystems is also different. In addition, the data result files generated by the application are scattered and stored in different file disk paths. When manually performing the comparison, a large number of files need to be processed. This not only increases the complexity of the operation, but also increases the risk of errors. There may be omissions in the comparison, that is, some files that need to be compared are ignored; or mismatches may occur, that is, files or data that should not be compared are mistakenly compared. These potential errors may affect the accuracy and completeness of data migration, which may cause subsequent business problems.

[0094] Based on this, the present invention provides a data processing method, please refer to Figure 2 , a schematic diagram of migrating data from a second database (such as an Oracle database) to a first database (such as a GDB database) is given as an example. In order to overcome the above problems, the present invention builds two deployment units at the service layer. Compared with building a dual-machine parallel environment in the prior art, the present invention can more efficiently utilize existing hardware resources without the need to establish two complete system architectures.

[0095] The present application scheme is described in detail below in conjunction with the accompanying drawings.

[0096] See also Figure 3 , a flow chart of a data processing method is shown, the method is applicable to a first server and a second server, the first server is a target server for data migration, and the second server is a source server for data migration. It can be understood that the first server or the second server can be any server with data processing capabilities, such as a tower server, a rack server, etc., without limitation. The method specifically includes the following steps:

[0097] Step 310: The second server obtains the second source data in the second database, and executes a batch processing program on the second source data to obtain a second file.

[0098] The second source data is data before being processed, and the second server processes the second source data in the second database through a batch processing program to obtain a second file. The batch processing program is a computer program that processes data by automatically executing a series of tasks, such as updating transaction records, backing up data, etc., without limitation.

[0099] Optionally, for any file in the second file: including one or more of the confession file, the accounting file and other files, the confession file is a file used to transmit, store or exchange data between different databases, the accounting file is a file related to transaction accounts, and the other files are files other than the confession file and the financial file

[0100] Optionally, the second server can obtain a preset correspondence table from the second database, the preset correspondence table containing the correspondence between the identifiers of multiple files and the path information of the multiple files, determine the path information corresponding to the second file according to the preset correspondence table, and compress the second file stored in the path indicated by the path information.

[0101] Since the compressed file contains various transaction data and contract data generated by the application system, these data are usually stored in different file paths on the disk. In order to efficiently process these scattered data, an automatic task mechanism can be adopted. The automatic task will pre-read a preset correspondence table, which lists in detail the file paths corresponding to the data files to be compressed. Based on this configuration information, the system can automatically identify the exact storage location of each file and compress them into a single file. This centralized compression strategy not only simplifies the subsequent file management process, but also significantly improves the efficiency and reliability of data transmission.

[0102] Through the above solution, compressing files can reduce the size of files, thereby speeding up file transfer. By identifying the file path in the configuration file to compress files, an automated file processing process is implemented, reducing human intervention and errors.

[0103] Step 320: The second server sends the second file to the first server.

[0104] The second file is used by the first server to obtain the first source data in the first database, and execute the batch processing program on the first source data to obtain the first file, and compare the first file with the second file. The first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database.

[0105] Optionally, when the first database is a GDB database and the second database is an Oracle database, in step 320, Oracle's data download tool can be used to download the business data from the second database. In this way, the data in the Oracle database can be extracted. Compared with traditional manual operation methods, this automated processing method not only improves the efficiency of the verification process, but also reduces the possibility of human errors.

[0106] Optionally, the second file is a compressed file. After the first server receives the second file from the second server, it also includes: storing the second file in a setting file directory, configuring the setting file directory to automatically decompress files existing inside, and importing the decompressed file corresponding to the second file into the first database.

[0107] Optionally, a setting file directory can be pre-set by a script, and the setting file directory has the following characteristics: when there are files in the directory, the files can be automatically decompressed. In this way, when the second file is stored in the setting file directory, the files can be automatically decompressed, and then the decompressed files can be imported into the first database using a tool.

[0108] Optionally, you can use an automatic decompression tool to decompress the file, so that you can use the tool to improve efficiency and reduce manual intervention.

[0109] Through the above scheme, the second file is placed in the set file directory for decompression, and the decompressed second file is imported into the first database. During the data transmission process, the file is compressed and decompressed, which can reduce the file size and thus speed up the file transmission efficiency. In addition, placing the file in the set file directory for decompression can better organize and manage data for subsequent processing and searching. The present application can realize automatic file decompression, reduce labor costs and improve efficiency.

[0110] Optionally, after the first server receives the second file, it needs to import the second file into the first database. During the data import process, an intelligent error handling mechanism can be used. That is, if a record cannot be successfully imported during the import process, the system will automatically skip the erroneous data to ensure that the entire import process will not be interrupted due to the failure of a single record, and continue to process subsequent data records to ensure the continuity and efficiency of the import process.

[0111] At the same time, in order to ensure data quality and traceability, each failed import record can be recorded in detail in a special data loading log. This log not only contains the specific information of the failed record, but also includes a preliminary analysis of the cause of the failure. In this way, it can provide reference materials for subsequent data comparison and abnormal analysis, making it easier for technicians to troubleshoot and analyze problems when abnormal comparison results are found.

[0112] Step 330: The first server obtains the first source data in the first database, and executes a batch processing program on the first source data to obtain a first file.

[0113] The first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database. That is, the first source data is migrated from the second database to the first database, and the first database regenerates the result file (i.e., the first file) with the first source data through the application according to the original file generation process.

[0114] Optionally, after being processed by a batch processing program, the first file may be one or more of a confession file, an accounting file, and other files. A confession file is a file used to transmit, store or exchange data between different databases, an accounting file is a file related to transaction accounts, and other files are files other than confession files and financial files.

[0115] In this way, the source data is processed to obtain a result file. When the data is subsequently compared, by comparing the result file instead of the source data, it is possible to more accurately verify whether the data is consistent during the migration process.

[0116] Step 340: The first server compares the first file and the second file to determine whether the comparison results are consistent. If the comparison results are consistent, step 350 is executed; if the comparison results are inconsistent, step 360 is executed.

[0117] Optionally, before the comparison, the number of data items in the first source data and the number of data items in the second source data can be compared to ensure that the data volume of the first source data is consistent with the data volume of the second source data, and then the first file and the second file can be compared to ensure that the format and content of the first file and the second file are consistent. In this way, through the double verification mechanism, the integrity of the data can be verified first, and then the quality of the data can be verified, thereby improving the reliability of data comparison.

[0118] Optionally, if the second server compresses the second file in the above step 310, the second server sends the compressed second file to the first server in step 320, the first server decompresses the second file, and then compares the decompressed files corresponding to the first file and the second file in step 340.

[0119] Optionally, the first file includes a plurality of first sub-files, and the second file includes a plurality of second sub-files. Figure 4 A schematic diagram of a process of comparing a first file with a second file is shown. Comparing the decompressed files corresponding to the first file and the second file may include the following steps 410 to 430:

[0120] Step 410: The first server obtains a preset configuration file from the first database.

[0121] Among them, the configuration file contains identifiers of multiple second sub-files arranged in sequence according to a set order, wherein the set order is consistent with the order of multiple first sub-files in the first file, so that it can be ensured that in subsequent comparisons, the order of sub-files in the first file and the order of sub-files in the second file are one-to-one corresponding.

[0122] Step 420: The first server imports the plurality of second sub-files in the second file into the first database in a set order according to a preset configuration file.

[0123] Step 430: The first server compares the first subfile and the second subfile located at the same position in the first file and the second file imported from the first database.

[0124] In this way, files at corresponding locations can be compared to avoid mistakenly comparing files that should not be compared.

[0125] Optionally, multiple first subfiles and multiple second subfiles can be sorted, and after sorting, the first subfiles and second subfiles in the same order have the same name, and for the first subfiles and second subfiles in the same sorting order, the contents of the first subfiles and the second subfiles are compared field by field. In this way, sorting the first subfiles and the second subfiles can lay the foundation for subsequent comparison. By comparing field by field, this refined comparison can capture minor differences that may be overlooked, so that each piece of data can be carefully reviewed, which is convenient for subsequent analysis and review.

[0126] Through the above steps 410 to 430, by using a preset configuration file, it can be ensured that the sub-files at corresponding positions in the first file and the second file are correctly compared, and by automatically importing and comparing files in a set order, no manual intervention is required, which reduces labor costs and improves the efficiency of data import.

[0127] The above describes a method for comparing the decompressed first file and the second file. The following describes the processing methods for comparing different files in the process of comparing the first file and the second file:

[0128] (a) For the data supply file, determine the difference fields between the first file and the second file, and if the difference fields do not affect the processing of the downstream data component, ignore the difference fields.

[0129] In a possible embodiment, the first file includes a first data supply file, the second file includes a second data supply file, and the first file and the second file are compared, including: comparing the first data supply file and the second data supply file line by line to obtain difference fields, and determining whether the difference fields affect the processing of downstream data components. If so, determining that the comparison results are inconsistent; if not, determining that the comparison results are consistent, wherein the downstream data component is a component located in a layer below the layer where the first server is located, and is used to receive and process data from the first server.

[0130] In this way, we can focus on the differences in all the comparison fields in the difference result file and evaluate whether these differences will have a substantial impact on the loading and processing of downstream data components. If it is determined that these differences will not cause any impact, the system will adopt a flexible processing strategy and treat them as no differences, thereby avoiding unnecessary alarms and processing. This intelligent judgment mechanism greatly improves the efficiency and accuracy of the system.

[0131] (b) For the batch accounting files, the difference fields between the first file and the second file are determined. If the difference fields are preset important fields, the comparison results are inconsistent.

[0132] In a possible embodiment, the first file includes a first accounting file, the second file includes a second accounting file, and the first file and the second file are compared, including: comparing the first accounting file and the second accounting file line by line, obtaining a difference field, and determining whether the difference field belongs to a key field, if so, determining that the comparison result is inconsistent, and if not, determining that the comparison result is consistent, wherein the key field includes at least one of the amount, account number, and time. In this way, by performing a comprehensive comparison on a large number of batch accounting processing files, a wider range of business scenarios and data types can be covered, ensuring the comprehensiveness and reliability of the verification results.

[0133] For batch accounting files, we mainly focus on the core transaction fields in the difference result files, such as amount, account number and time, which are critical to financial accuracy. Since batch accounting transaction files must be completely consistent before and after migration, any minor differences may have a significant impact on the processing results of downstream accounting systems, which may lead to serious financial problems. Therefore, if differences are found in these key fields, the comparison results are judged to be inconsistent.

[0134] By comprehensively comparing a large number of batch accounting processing files, a wider range of business scenarios and data types can be covered, ensuring the comprehensiveness and reliability of the verification results.

[0135] (c) For other files that do not fall into the above two categories, the system adopts a more flexible comparison strategy. At this stage, the focus is on the comparison fields in the difference result files to evaluate whether the difference content is as expected. This approach allows the system to maintain sufficient flexibility when processing non-standardized or special-purpose files while still ensuring overall data quality and consistency.

[0136] Through this multi-level, all-round comparison strategy, the system can not only fully capture and analyze data differences, but also take corresponding treatment measures based on different types of data and potential impacts. This refined comparison process greatly improves the accuracy and reliability of data migration.

[0137] Optionally, you can format the comparison results and output the result file. This formatted output not only improves the readability of the results, but also facilitates the rapid location of potential problems.

[0138] Step 350: The first server determines that the first source data is successfully migrated.

[0139] If the comparison between the first file and the second file shows that the results are consistent, it means that the first source data is consistent before and after the migration, that is, the migration is successful.

[0140] Step 360: The first server determines that migration of the first source data fails.

[0141] If the comparison results of the first file and the second file are inconsistent, it means that an error occurred in the migration process of the first source data, and the first source data is inconsistent before and after the migration.

[0142] Optionally, if the comparison between the first file and the second file shows inconsistent results, the system will confirm the comparison results and quickly repair the difference data in the first database. After the repair, the system automatically verifies the results without additional manual operation. In this way, the comparison process can be simplified and the overall efficiency can be greatly improved.

[0143] Through the above steps 310-360, the accuracy of data migration can be verified by re-executing the processing flow based on the actual data files of the original system. The present invention comprehensively processes and compares and verifies key files before the application is officially put into use. This method can not only effectively ensure the accuracy of data migration, but also greatly improve the stability of the application. Through this preventive verification process, potential data inconsistency problems can be discovered and solved early, thereby minimizing the risks that may be caused by data migration. In addition, the present application is a result file after the comparison application is re-run. The file can be directly sent to the downstream data for processing. If the comparison results are consistent, it can be ensured that the file sent to the downstream data is accurate. Compared with the comparison of source data in the prior art, the present application avoids the problem that even if the comparison results of the source data are consistent, the result files obtained after processing are different, which leads to errors in the files sent to the downstream data.

[0144] Based on the same inventive concept, the present application may also provide a data processing method, which may be executed by a data processing system, wherein the data processing system includes the above-mentioned first server and second server.

[0145] See also Figure 5 , a flow chart of a data processing method is shown, which specifically includes the following steps:

[0146] Step 510: Acquire second source data in a second database, and execute a batch processing program on the second source data to obtain a second file.

[0147] Step 520: Acquire first source data from the first database, and execute a batch processing program on the first source data to obtain a first file.

[0148] Here, the data processing system can connect the second database and the first database. After determining to migrate the second source data in the second database to the first database, the data processing system can obtain the first source data from the first database and execute a batch processing program on the first source data to obtain a first file.

[0149] Step 530: Compare the first file and the second file to determine whether the comparison results are consistent. If the comparison results are consistent, execute step 540; if the comparison results are inconsistent, execute step 550.

[0150] Step 540: Determine whether the first source data is successfully migrated.

[0151] Step 550: Determine that a first source data migration error occurs.

[0152] Here, the data processing system can compare the first file and the second file after the application is re-run. Compared with the comparison of source data in the prior art, the present application compares the source data after the application is re-run. The comparison result is more accurate, and there are different comparison processing methods for different file types, which improves the efficiency of the comparison.

[0153] It can be understood that the above steps 510 to 550 can refer to the above steps 310 to 360, and will not be repeated here one by one.

[0154] The above introduces the method of data processing. The following takes data comparison on day T as an example to illustrate the above method.

[0155] First, let’s introduce the overall architecture of this solution. Figure 6 A schematic diagram of the structure of a data processing system is shown, and the data processing system can perform data processing. The system includes: a product service layer (i.e. Figure 6 P6 in the product service layer), a first server and a second server are provided in the product service layer, a first database is provided in the first server, a second database is provided in the second server, the first server is connected to the second server, when migrating data from the second server to the first server, the second server is used to obtain the second source data from the second database, and execute a batch program on the second source data to obtain a second file (such as generating a credit card file, a personal loan file, a collection file, etc.), and send it to the first server, the first server is used to obtain the first source data from the first database after receiving the second file, and execute a batch program on the first source data to obtain a first file (such as generating a credit card file, a personal loan file, a collection file, etc.), compare the first file with the second file, if the comparison results are consistent, it is determined that the second server has successfully migrated the first source data, if the comparison results are inconsistent, it is determined that the migration of the first source data has failed, wherein the first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database.

[0156] Optionally, the application system also includes an external integration layer (i.e. Figure 6 P5 in the Figure 6P7), the external integration layer is provided with a third database, the data integration layer is provided with a fourth database, the second server is connected to the third database and the fourth database respectively, the second server is also used to obtain data from the third database, generate files after batch processing the data, and send the files to the fourth database.

[0157] It can be understood that the fourth database in the data integration layer is the downstream component of the second server, that is, the files generated by the second server are sent to the fourth database. There can be multiple fourth databases without limitation. The fourth database in the data integration layer sends data to the management and analysis layer (i.e. Figure 6 The management and analysis layer can provide management and analysis services for data files. The third database in the external integration layer can be obtained from foreign financial institutions (i.e. Figure 6 P10) in the product service layer obtains the data file, and then the third database sends the processed data file to the second server in the product service layer.

[0158] Optionally, the data processing system also includes basic components such as security, message center, and file transfer (i.e. Figure 6 P0 in the system is used to ensure the security of system operation and achieve efficient data transmission. These components together build a stable and reliable application environment to ensure the smooth progress of the entire data comparison process.

[0159] Optionally, the data processing system may also include a channel integration layer (i.e. Figure 6 P1 and P2 in the figure are used to receive transaction requests from mobile phones, WeChat, switching virtual circuits (SCVs), and employee channels. These transaction requests are sent to the user service integration layer (i.e. Figure 6 P3 in the application integration layer (i.e. Figure 6 The application service layer also receives transaction requests from branches (i.e. Figure 6 P9), after processing these transaction requests through the service release in the application integration layer, the transaction data is sent to the second server, and the second server generates a storage debit card file based on the transaction data.

[0160] Through the above scheme, the architecture of the data processing system can be simplified. In the existing system architecture, two independent data processing systems (also called application systems) are set up, and complete data processing environments are built separately. A first server is set up in one application system, and a second server is set up in another application system. The two servers interact between the two application systems to verify the data. This method requires that all layers of data in each application system must be rerun before data comparison can be achieved. The steps are relatively cumbersome. This scheme does not require the construction of two independent application systems. It only requires the creation of a first server on the product server where the second server is located on the original application system. Through the interaction between the first server and the second server in the same layer, the scheme of automatically pulling data, automatically rerunning applications, and automatically comparing can be achieved. It does not require the rerunning of upper and lower layer applications. The steps are simple and easy to implement.

[0161] The above introduces the example system architecture, and the following introduces the complete process of data processing. It can be understood that the time of the following scheme is only used as an example and is not intended to limit the scheme of this application.

[0162] First, before data comparison, you need to ensure the security of the system during the subsequent data comparison process. For example, you can do the following:

[0163] 1. After the data file is generated, the file sending instruction will no longer be automatically called to send the data to the downstream components. That is, the data file will not be automatically transmitted to other systems or services after it is generated.

[0164] 2. Process the system deployment and configuration. For example, the system does not start the registration center and the service catalog does not register.

[0165] 3. Turn off the application's local security and outbound call configuration to reduce system complexity and potential risks.

[0166] 4. Simplify system services. For example, when scanning configuration files, all scanning services except the data supply task are deregistered. At the same time, all online services are also deregistered in the service registration configuration file. These measures ensure that the application is not visible to the outside world and enhance the security and isolation of the system.

[0167] After that, the data is processed. Assuming that the data comparison is performed on day T, the specific steps include:

[0168] Step 1: The second server obtains the second source data from the second database, and executes a batch processing program on the second source data to obtain a second file.

[0169] This step may refer to the above step 310 and will not be described in detail here.

[0170] Step 2: The second server sends the second file to the first server.

[0171] This step may refer to the above step 320 and will not be described again here.

[0172] Step three: The first server obtains the first source data in the first database, and executes a batch processing program on the first source data to obtain a first file.

[0173] This step can be divided into the following four parts:

[0174] Step 1: At 4:00 on T-day, the first database obtains all data files generated by the application of the second database at 2:00 on T-day through an automatic task.

[0175] Among them, the first database can be a database in a localized environment, such as a GDB database, and the second database can be a database in a non-localized environment, such as an Oracle database. At 4:00 on T day, the first database (such as a GDB database) will comprehensively organize the application batch files (including data on T-1 day) generated by the second database (such as an ORACLE database) at 2:00 on T day, that is, the system will identify the files that need to be compared, and use a special script tool to compress these files. The specific compression processing steps can be found in the above step 310, which will not be repeated here.

[0176] Step 2: At 6:00 on T day, the data file is automatically parsed using the data loading tool through an automatic task, and is simultaneously loaded into the first database (eg, the localized database).

[0177] For example, at 6:00 on T day, the compressed data file of the ORACLE application server is pulled from the first server (eg, GDB application server) by using the script tool Fserver, placed in the file directory specified by the first database (eg, localized environment database GDB) and automatically decompressed.

[0178] Step 3: At 8:00 on T day, the first server (for example, GDB application server) uses a script tool to decompress the pulled data file and uses the DBTOOLS tool to parse and store it in the database.

[0179] At 8:00 on T day, the GDB application server starts processing the decompressed data files through a special script tool. This process follows the set file directory and uses the DBTOOLS tool to accurately parse the data and import it into the database. The steps of decompressing the file into the database can be referred to in the above step 320, which will not be repeated here.

[0180] Step 4: After the data is loaded successfully and verified, the parallel environment data supply script and data supply task are started through automatic tasks at 10:00 on T day to generate the same batch of data supply files and accounting batch files in the domestic parallel verification environment.

[0181] In order to achieve more accurate data comparison, the data processing time can be adjusted. For example, the data generation task time can be synchronously modified to between 10:00 and 11:00 every day.

[0182] By making necessary modifications to the application code on the GDB application server, the system starts executing the application batch program on the GDB server on T-2 at 10:00 on T day. This process can generate two types of files: batch data supply files and batch accounting processing files. Batch data supply files contain various data that the system needs to provide to other components or systems, while batch accounting processing files involve various financial and accounting-related processing results within the system.

[0183] The data query time is adjusted to the day before the original data query time because the data content in the data file obtained from the non-localized environment is data up to a certain time the day before the file was generated. In order to ensure the integrity of the data comparison, the localized environment needs to re-execute the data supply date for one day. In this way, it can ensure that the data on both sides of the comparison are data at the same time point, avoiding inconsistent comparisons caused by data time differences.

[0184] Step 4: Compare the first file and the second file to determine whether the comparison results are consistent. If they are consistent, execute step 5; if they are inconsistent, execute step 6.

[0185] At 14:00 on T day, the system uses automatic data comparison tools through automatic tasks to conduct a comprehensive comparison and verification of data supply files and accounting batches in domestic and non-domestic environments.

[0186] The system automatically starts the comparison script and begins the data comparison process. The comparison script first locates and extracts the original data file and the target data file according to the pre-configured configuration file, and transfers them to a dedicated comparison result directory. Then, the comparison script uses a sorting algorithm to systematically sort the file contents according to the file name, laying the foundation for the subsequent line-by-line comparison.

[0187] In this automated process, the script scans and compares the file contents line by line. If any differences are found during the comparison process, the relevant table names are recorded in the relevant failed comparison result files for subsequent in-depth analysis and processing; if the entire comparison process is completed smoothly and no differences are found, the comparison results are recorded in the successful comparison result files. Through this series of automated tasks, scheduled tasks are automatically executed to pull, parse and compare pre-configured target files, and then the results are recorded in the log. This approach simplifies the comparison steps, saves time, improves quality and efficiency, ensures data consistency between the localized environment and the non-localized environment, while minimizing human intervention and improving verification efficiency and accuracy.

[0188] Step 5: Confirm that the first source data is successfully migrated.

[0189] Step 6: Determine that the first source data migration has errors.

[0190] In this way, the source data is rerun through the application to generate a result file, and the result file is compared to determine whether the data in the first database and the data in the second database after migration are consistent. In this way, the accuracy of data migration from the second database to the first database can be ensured.

[0191] Through the above steps, the accuracy of data migration can be verified by re-executing the processing flow based on the actual data of the original system, that is, the source data is re-run through the application to generate a result file, and then the result file is compared to determine whether the data in the first database and the data in the second database after migration are consistent. Based on this, before the application is officially put into use, the comprehensive processing and comparison verification of key files can be achieved, which can not only effectively ensure the accuracy of data migration, but also greatly improve the stability of the application. This solution belongs to a preventive verification process, which can detect and solve potential data inconsistency problems early, thereby minimizing the risks that may be caused by data migration. In addition, the present application is a result file after the comparison application is re-run, which can be directly sent to the downstream data for processing. If the comparison results are consistent, it can ensure that the file sent to the downstream data is accurate. Compared with the comparison of source data in the prior art, the present application avoids the problem that even if the comparison results of the source data are consistent, the result files obtained after processing are different, which leads to errors in the files sent to the downstream data.

[0192] Based on the same inventive concept, an embodiment of the present application provides a data processing device, which can execute the data processing method introduced in the above content.

[0193] Please refer to Figure 7 , a schematic diagram of the structure of a data processing device provided in an embodiment of the present application is given, such as Figure 7As shown, the data processing device includes:

[0194] The receiving unit 701 is used to receive a second file from a second server, where the second file is obtained by the second server acquiring second source data in a second database and executing a batch processing program on the second source data.

[0195] The acquisition unit 702 is used to acquire the first source data in the first database and execute a batch processing program on the first source data to obtain a first file; the first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database.

[0196] The comparison unit 703 is used to compare the first file and the second file. If the comparison result is consistent, it is determined that the migration of the first source data is successful. If the comparison result is inconsistent, it is determined that the migration of the first source data is wrong.

[0197] In a possible embodiment, for any one of the first file and the second file: it includes one or more of a confession file, an accounting file and other files, the confession file is a file used to transmit, store or exchange data between different databases, the accounting file is a file related to transaction accounts, and the other files are files other than the confession file and the financial file.

[0198] In a possible embodiment, the first file includes a first data supply file, and the second file includes a second data supply file. The comparison unit 703 is specifically used to: compare the first data supply file and the second data supply file line by line to obtain difference fields, and determine whether the difference fields affect the processing of downstream data components. If so, it is determined that the comparison results are inconsistent; if not, it is determined that the comparison results are consistent, wherein the downstream data component is a component located in a layer below the layer where the first server is located, and is used to receive and process data from the first server.

[0199] In a possible embodiment, the first file includes a first accounting file, and the second file includes a second accounting file. The comparison unit 703 is specifically used to compare the first accounting file and the second accounting file line by line, obtain the difference field, and determine whether the difference field belongs to the key field. If it does, it is determined that the comparison result is inconsistent, and if it does not, it is determined that the comparison result is consistent, wherein the key field includes at least one of the amount, account number, and time. In this way, by performing a comprehensive comparison on a large number of batch accounting processing files, a wider range of business scenarios and data types can be covered, ensuring the comprehensiveness and reliability of the verification results.

[0200] In a possible embodiment, the second file is a compressed file. After receiving the second file from the second server, the receiving unit 701 is further used to: store the second file in a setting file directory, the setting file directory is configured to automatically decompress files when there are files inside, and import the decompressed file corresponding to the second file into the first database. The comparison unit 703 is specifically used to: compare the decompressed files corresponding to the first file and the second file.

[0201] In a possible embodiment, the first file includes multiple first sub-files, and the second file includes multiple second sub-files. After receiving the second file from the second server, the receiving unit 701 is further used to: obtain a preset configuration file from the first database, the configuration file includes identifiers of multiple second sub-files arranged in sequence according to a set order, and the set order is consistent with the order of the multiple first sub-files in the first file; according to the preset configuration file, the multiple second sub-files in the second file are imported into the first database according to the set order; the comparison unit 703 is specifically used to: compare the first sub-file and the second sub-file located at the same position in the first file and the second file imported from the first database.

[0202] Based on the same inventive concept, an embodiment of the present application provides a data processing device, which can execute the data processing method introduced in the above content.

[0203] Please refer to Figure 8 , a schematic diagram of the structure of a data processing device provided in an embodiment of the present application is given, such as Figure 8 As shown, the data processing device includes:

[0204] The processing unit 801 is used to obtain second source data in the second database and execute a batch processing program on the second source data to obtain a second file.

[0205] The sending unit 802 is used to send a second file to the first server. The second file is used by the first server to obtain the first source data in the first database, execute a batch processing program on the first source data to obtain the first file, and compare the first file and the second file. The first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database.

[0206] In a possible embodiment, for any file in the first file and the second file: including one or more of the data supply file, the accounting file and other files, the data supply file is a file used to transmit, store or exchange data between the first database and the second database, the accounting file is a file related to transaction accounts, and the other files are files other than the data supply file and the accounting file.

[0207] In a possible embodiment, the processing unit 801 is specifically used to: obtain a preset correspondence table from the second database, the preset correspondence table containing the correspondence between the identifiers of multiple files and the path information of the multiple files, determine the path information corresponding to the second file according to the preset correspondence table, compress the second file stored in the path indicated by the path information, obtain the second file, and send the compressed second file to the first server.

[0208] Based on the same inventive concept, an embodiment of the present application provides a data processing device, which can execute the data processing method introduced in the above content.

[0209] Please refer to Fig. 9 , a schematic diagram of the structure of a data processing device provided in an embodiment of the present application is given, such as Fig. 9 As shown, the data processing device includes:

[0210] The processing unit 901 is used to obtain the second source data in the second database, and execute the batch processing program on the second source data to obtain the second file, obtain the first source data from the first database, and execute the batch processing program on the first source data to obtain the first file, wherein the first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database;

[0211] The comparison unit 902 is used to compare the first file and the second file. If the comparison results are consistent, it is determined that the migration of the first source data is successful. If the comparison results are inconsistent, it is determined that the migration of the first source data is wrong.

[0212] See also Fig.10 , showing a schematic diagram of the structure of an electronic device provided by an embodiment of the present application, such as Fig.10 As shown, the electronic device includes: a memory 1001 and a processor 1002, and the processor 1003 is coupled to the memory 1001. The memory 1001 is used to store program instructions, and the processor 1002 is used to call the program instructions stored in the memory 1001 and execute the above data processing method according to the obtained program.

[0213] Optionally, the electronic device may further include an interface circuit 1003, which may be a transceiver or an input / output interface. The input / output interface is used to input and / or output information, where output may be understood as sending and input may be understood as receiving. The processor 1002 may communicate with other devices in the test device or other devices other than the test device through the interface circuit 1003 to obtain information required for executing the above data processing method.

[0214] When the electronic device 1000 is used to implement Figure 3When the method shown is performed, the processor 1002 is used to implement the functions of the above-mentioned receiving unit 701, the obtaining unit 702, and the comparing unit 703.

[0215] It is understandable that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0216] The memory in the embodiments of the present application can be a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a register, a hard disk, a mobile hard disk, a compact disc read-only memory (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. The storage medium can also be a component of the processor.

[0217] Based on the same inventive concept, the embodiment of the present application provides a computer-readable storage medium, and the computer program product includes: computer program code, when the computer program code is run on a computer, the computer executes any of the data processing methods discussed above. Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the data processing method, the implementation of the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0218] Based on the same inventive concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the data processing methods discussed above. Since the principle of solving the problem by the above computer program product is similar to that of the data processing method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0219] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0220] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0221] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0222] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of user-operated steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0223] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: Applicable to the first server, the method comprises: receiving a second file from a second server, where the second file is obtained by the second server acquiring second source data in a second database and executing a batch processing program on the second source data; Acquire first source data in a first database, and execute a batch processing program on the first source data to obtain a first file; the first source data in the first database is obtained by migrating second source data in the second database from the second database to the first database; The first file and the second file are compared. If the comparison results are consistent, it is determined that the migration of the first source data is successful. If the comparison results are inconsistent, it is determined that the migration of the first source data is wrong.

2. The method according to claim 1, characterized in that For any file in the first file and the second file: including one or more of the data supply file, the accounting file and other files; the data supply file is a file used to transmit, store or exchange data between different databases, the accounting file is a file related to transaction accounts, and the other files are files other than the data supply file and the accounting file.

3. The method according to claim 1, characterized in that The first file includes a first supply data file, and the second file includes a second supply data file; The comparing the first file and the second file includes: Comparing the first supply data file and the second supply data file line by line to obtain difference fields; Determine whether the difference field affects the processing of the downstream data component, if so, determine that the comparison result is inconsistent, and if not, determine that the comparison result is consistent; The downstream data component is a component located in a layer below the layer where the first server is located, and is used to receive and process data from the first server.

4. The method according to claim 1, characterized in that The first file includes a first accounting file, and the second file includes a second accounting file; The comparing the first file and the second file includes: Comparing the first accounting file and the second accounting file line by line to obtain difference fields; Determine whether the difference field belongs to a key field, if so, determine that the comparison result is inconsistent, if not, determine that the comparison result is consistent; The key fields include at least one of amount, account number and time.

5. The method according to claim 1, characterized in that The second file is a compressed file; After receiving the second file from the second server, the method further includes: storing the second file in a setting file directory, wherein the setting file directory is configured to automatically decompress files when there are files in the directory; Importing the decompressed file corresponding to the second file into the first database; The comparing the first file and the second file includes: Compare the decompressed files corresponding to the first file and the second file.

6. The method according to any one of claims 1 to 5, characterized in that The first file includes a plurality of first sub-files, and the second file includes a plurality of second sub-files; After receiving the second file from the second server, the method further includes: Acquire a preset configuration file from the first database, wherein the preset configuration file includes identifiers of a plurality of second sub-files arranged in sequence according to a set sequence, and the set sequence is consistent with an order of the plurality of first sub-files in the first file; According to the preset configuration file, importing a plurality of second sub-files in the second file into the first database in the set order; The comparing the decompressed files corresponding to the first file and the second file includes: Compare a first subfile and a second subfile located at the same position in the first file and the second file imported into the first database.

7. A data processing method, characterized in that: Applicable to the second server, the method comprises: Acquire second source data in a second database, and execute a batch processing program on the second source data to obtain a second file; The second file is sent to the first server, where the second file is used by the first server to obtain the first source data in the first database, and execute a batch processing program on the first source data to obtain the first file, and compare the first file with the second file. The first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database.

8. The method according to claim 7, characterized in that For any file in the first file and the second file: including one or more of the data supply file, the accounting file and other files; the data supply file is a file used to transmit, store or exchange data between the first database and the second database, the accounting file is a file related to transaction accounts, and the other files are files other than the data supply file and the accounting file.

9. The method according to claim 7 or 8, characterized in that The sending the second file to the first server includes: Acquire a preset correspondence table from the second database, wherein the preset correspondence table includes correspondences between identifiers of a plurality of files and path information of the plurality of files; According to the preset correspondence table, the path information corresponding to the second file is determined, the second file stored in the path indicated by the path information is compressed, and the compressed second file is sent to the first server.

10. A data processing method, characterized in that: The method comprises: Acquire second source data in a second database, and execute a batch processing program on the second source data to obtain a second file; Acquire first source data from a first database, and execute a batch processing program on the first source data to obtain a first file; the first source data in the first database is obtained by migrating second source data in the second database from the second database to the first database; The first file and the second file are compared. If the comparison results are consistent, it is determined that the migration of the first source data is successful. If the comparison results are inconsistent, it is determined that the migration of the first source data is wrong.

11. A data processing system, characterized in that: It comprises a product service layer, wherein a first server and a second server are arranged in the product service layer, a first database is arranged in the first server, a second database is arranged in the second server, and the first server is connected to the second server; The second server is used to obtain second source data from a second database, execute a batch processing program on the second source data, obtain a second file, and send the second file to the first server; The first server is used to obtain first source data from the first database after receiving the second file, execute a batch processing program on the first source data to obtain a first file, compare the first file with the second file, and if the comparison results are consistent, determine that the second server has successfully migrated the first source data; if the comparison results are inconsistent, determine that the migration of the first source data has failed; wherein the first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database.

12. The system according to claim 11, characterized in that The data processing system further comprises an external integration layer and a data integration layer, the external integration layer is provided with a third database, the data integration layer is provided with a fourth database, and the second server is connected to the third database and the fourth database respectively; The second server is further used to obtain data from the third database, generate files after batch processing the data, and send the files to the fourth database.

13. A data processing device, characterized in that: include: A receiving unit, configured to receive a second file from a second server, wherein the second file is obtained by the second server acquiring second source data of a second database and executing a batch processing program on the second source data; An acquisition unit, used for acquiring first source data in a first database, and executing a batch processing program on the first source data to obtain a first file; The first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database; The comparison unit is used to compare the first file and the second file. If the comparison results are consistent, it is determined that the migration of the first source data is successful; if the comparison results are inconsistent, it is determined that the migration of the first source data is wrong.

14. A data processing device, characterized in that: include: A processing unit, configured to obtain second source data in a second database, and execute a batch processing program on the second source data to obtain a second file; A sending unit is used to send the second file to the first server, the second file is used by the first server to obtain the first source data in the first database, and execute a batch processing program on the first source data to obtain the first file, and compare the first file and the second file. The first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database.

15. A data processing device, characterized in that: include: A processing unit, configured to obtain second source data in a second database, and execute a batch processing program on the second source data to obtain a second file; Acquire first source data from a first database, and execute a batch processing program on the first source data to obtain a first file; The first source data in the first database is obtained by migrating the second source data in the second database from the second database to the first database; The comparison unit is used to compare the first file and the second file. If the comparison results are consistent, it is determined that the migration of the first source data is successful. If the comparison results are inconsistent, it is determined that the migration of the first source data is wrong.

16. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in any one of claims 1-10 according to the obtained program instructions.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 10.

18. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 10.

Images