Multi-source data consistency processing method based on data comparison table

Abstract

The present application relates to a multi-source data consistency processing method, device, computer equipment, computer-readable storage medium and computer program product based on a data comparison table, which can be used in the field of big data technology. The method includes: establishing a corresponding data comparison table for the source system; performing data conversion processing on the original data to obtain the target data, and updating the target data to the source system field set in the data comparison table; updating the data change information to the master data platform field set in the data comparison table; performing data comparison processing on the source system field set and the master data platform field set according to the corresponding relationship to obtain the data comparison result; determining the data to be checked in the data comparison table according to the data comparison result; checking the data to be checked to obtain the checking result; and performing data consistency update processing on the master data platform or the source system according to the checking result. The use of this method can improve the efficiency of data consistency processing.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a multi-source data consistency processing method, device, computer equipment, computer-readable storage medium and computer program product based on a data comparison table. Background Art

[0002] With the development of big data technology, the demand for multi-source data management and integration is growing. In order to meet this demand, master data management (MDM) technology has emerged. Master data management technology can provide a unified data view, but in practical applications, how to ensure data consistency between multiple source systems and the master data platform has become an important research direction.

[0003] Traditional technology usually performs multi-source data consistency processing through manual comparison and verification; however, data consistency processing through this method requires a lot of manual processing time, resulting in low efficiency of data consistency processing. Summary of the invention

[0004] Based on this, it is necessary to provide a multi-source data consistency processing method, device, computer equipment, computer-readable storage medium and computer program product based on a data comparison table that can improve the efficiency of data consistency processing in response to the above technical problems.

[0005] In a first aspect, the present application provides a multi-source data consistency processing method based on a data comparison table. The method comprises:

[0006] For the source system, a corresponding data comparison table is established; the data comparison table is used to record the correspondence between the data fields of the source system and the data fields of the main data platform;

[0007] Extracting original data from the source system, performing data conversion processing on the original data to obtain target data, and updating the target data to the source system field set in the data comparison table;

[0008] Acquire data change information of the master data platform, and update the data change information to the master data platform field set in the data comparison table;

[0009] According to the corresponding relationship, data comparison processing is performed on the source system field set and the main data platform field set to obtain a data comparison result;

[0010] According to the data comparison result, determine the data to be checked in the data comparison table;

[0011] Performing verification processing on the data to be verified to obtain a verification processing result;

[0012] According to the verification result, data consistency update processing is performed on the master data platform or the source system.

[0013] In one embodiment, performing data consistency update processing on the master data platform or the source system according to the verification processing result includes:

[0014] Determine corresponding processing method information according to the verification processing result;

[0015] According to the processing method information, data consistency update processing is performed on the main data platform or the source system.

[0016] In one embodiment, performing data consistency update processing on the master data platform or the source system according to the processing mode information includes:

[0017] When the processing method information indicates updating the master data platform, updating the data of the master data platform;

[0018] When the processing method information indicates that the source system is to be updated, data of the source system is updated.

[0019] In one of the embodiments, the data comparison result includes at least mapped and the fields are completely consistent, mapped but the fields are not completely consistent, and unmapped;

[0020] Determining the data to be checked in the data comparison table according to the data comparison result includes:

[0021] According to the data comparison result, the data corresponding to the mapped but incompletely consistent fields and the data corresponding to the unmapped fields are determined;

[0022] The data corresponding to the mapped fields but not completely consistent with each other and the data corresponding to the unmapped fields are both used as the data to be checked.

[0023] In one embodiment, after performing data consistency update processing on the master data platform or the source system according to the verification processing result, the method further includes:

[0024] Acquire data update information corresponding to the data consistency update process;

[0025] The data comparison table is updated according to the data update information.

[0026] In one embodiment, performing data conversion processing on the original data to obtain target data includes:

[0027] Performing data conversion processing on the original data to obtain converted data consistent with the data format of the main data platform;

[0028] The converted data is used as the target data.

[0029] In a second aspect, the present application also provides a multi-source data consistency processing device based on a data comparison table. The device comprises:

[0030] A data establishment module is used to establish a corresponding data comparison table for the source system; the data comparison table is used to record the corresponding relationship between the data fields of the source system and the data fields of the main data platform;

[0031] A data conversion module, used to extract original data from the source system, perform data conversion processing on the original data to obtain target data, and update the target data to the source system field set in the data comparison table;

[0032] An information acquisition module, used to acquire data change information of the master data platform, and update the data change information to the master data platform field set in the data comparison table;

[0033] A data comparison module, used to perform data comparison processing on the source system field set and the main data platform field set according to the corresponding relationship to obtain a data comparison result;

[0034] A data determination module, used to determine the data to be checked in the data comparison table according to the data comparison result;

[0035] A data checking module is used to check the data to be checked and obtain a checking result;

[0036] The system update module is used to perform data consistency update processing on the master data platform or the source system according to the verification processing result.

[0037] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0038] For the source system, a corresponding data comparison table is established; the data comparison table is used to record the correspondence between the data fields of the source system and the data fields of the main data platform;

[0039] Extracting original data from the source system, performing data conversion processing on the original data to obtain target data, and updating the target data to the source system field set in the data comparison table;

[0040] Acquire data change information of the master data platform, and update the data change information to the master data platform field set in the data comparison table;

[0041] According to the corresponding relationship, data comparison processing is performed on the source system field set and the main data platform field set to obtain a data comparison result;

[0042] According to the data comparison result, determine the data to be checked in the data comparison table;

[0043] Performing verification processing on the data to be verified to obtain a verification processing result;

[0044] According to the verification result, data consistency update processing is performed on the master data platform or the source system.

[0045] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0046] For the source system, a corresponding data comparison table is established; the data comparison table is used to record the correspondence between the data fields of the source system and the data fields of the main data platform;

[0047] Extracting original data from the source system, performing data conversion processing on the original data to obtain target data, and updating the target data to the source system field set in the data comparison table;

[0048] Acquire data change information of the master data platform, and update the data change information to the master data platform field set in the data comparison table;

[0049] According to the corresponding relationship, data comparison processing is performed on the source system field set and the main data platform field set to obtain a data comparison result;

[0050] According to the data comparison result, determine the data to be checked in the data comparison table;

[0051] Performing verification processing on the data to be verified to obtain a verification processing result;

[0052] According to the verification result, data consistency update processing is performed on the master data platform or the source system.

[0053] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0054] For the source system, a corresponding data comparison table is established; the data comparison table is used to record the correspondence between the data fields of the source system and the data fields of the main data platform;

[0055] Extracting original data from the source system, performing data conversion processing on the original data to obtain target data, and updating the target data to the source system field set in the data comparison table;

[0056] Acquire data change information of the master data platform, and update the data change information to the master data platform field set in the data comparison table;

[0057] According to the corresponding relationship, data comparison processing is performed on the source system field set and the main data platform field set to obtain a data comparison result;

[0058] According to the data comparison result, determine the data to be checked in the data comparison table;

[0059] Performing verification processing on the data to be verified to obtain a verification processing result;

[0060] According to the verification result, data consistency update processing is performed on the master data platform or the source system.

[0061] The above-mentioned multi-source data consistency processing method, device, computer equipment, computer-readable storage medium and computer program product based on the data comparison table establish a corresponding data comparison table for the source system; the data comparison table is used to record the correspondence between the data fields of the source system and the data fields of the main data platform; extract the original data from the source system, perform data conversion processing on the original data to obtain the target data, and update the target data to the source system field set in the data comparison table; obtain the data change information of the main data platform, and update the data change information to the main data platform field set in the data comparison table; according to the corresponding relationship, perform data comparison processing on the source system field set and the main data platform field set to obtain data comparison results; according to the data comparison results, determine the data to be checked in the data comparison table; check the data to be checked to obtain the check processing result; according to the check processing result, perform data consistency update processing on the main data platform or the source system. This solution achieves structured mapping of data on both ends by establishing a data comparison table to record the field correspondence between the source system and the master data platform; through automated data conversion, change synchronization and comparison processing, it is conducive to quickly identifying data inconsistencies; finally, the data is automatically updated according to the verification results, which is conducive to timely elimination of data differences; this series of automated processing flows is conducive to improving the efficiency of data consistency processing and maintaining data consistency in a multi-source data environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 It is a flowchart of a method for processing consistency of multi-source data based on a data comparison table in one embodiment;

[0064] Figure 2 A schematic diagram of an overall framework in one embodiment;

[0065] Figure 3 A schematic diagram of field comparison relationship in an embodiment;

[0066] Figure 4 A schematic diagram of the operational relationship between a data comparison table and components in an embodiment;

[0067] Figure 5 A schematic diagram of a data comparison and processing flow in an embodiment;

[0068] Figure 6 It is a structural block diagram of a multi-source data consistency processing device based on a data comparison table in one embodiment;

[0069] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0071] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0072] In an exemplary embodiment, Figure 1As shown, a multi-source data consistency processing method based on a data comparison table is provided. This embodiment uses the method applied to a terminal as an example for illustration; it can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers, etc.; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. In this embodiment, the method includes the following steps:

[0073] Step S101, establishing a corresponding data comparison table for the source system; the data comparison table is used to record the correspondence between the data fields of the source system and the data fields of the main data platform.

[0074] Step S102, extracting original data from the source system, performing data conversion processing on the original data to obtain target data, and updating the target data to the source system field set in the data comparison table.

[0075] Step S103, obtaining data change information of the master data platform, and updating the data change information to the master data platform field set in the data comparison table.

[0076] Step S104: performing data comparison processing on the source system field set and the main data platform field set according to the corresponding relationship to obtain a data comparison result.

[0077] Step S105, determining the data to be checked in the data comparison table according to the data comparison result.

[0078] Step S106, verify the data to be verified and obtain the verification result.

[0079] Step S107: According to the verification result, data consistency update processing is performed on the main data platform or the source system.

[0080] Among them, the data comparison table can be a table structure used to record the correspondence between the source system data fields and the main data platform data fields. For example, the data comparison table can include the source system field set, the main object field set, the comparison status and the verification processing status.

[0081] The source system may be a business-end system that maintains master data, for example, the source system may be a business system that maintains personnel information, such as a human resources system.

[0082] The master data platform may be a platform that serves as a trusted data source for data coordination. For example, the master data platform may be a system that stores and manages master object data.

[0083] The original data may be unprocessed data extracted from a source system. For example, the original data may be source system mirror data extracted in batches through data integration technology at a data center.

[0084] Among them, data conversion processing can be the process of cleaning and converting raw data. For example, data conversion processing can include logical operations such as data format conversion, field merging and splitting, etc.

[0085] The target data may be data that conforms to a specific format and requirements after data conversion processing. For example, the target data may be data that can be effectively compared with the data on the main data platform after cleaning and conversion.

[0086] The source system field set may be a part of the data comparison table that records the source system data fields. For example, the source system field set may include the system primary key, business handshake fields, and general fields of the source system.

[0087] The master data platform field set may be a part of the data comparison table that records the master data platform data fields. For example, the master data platform field set may include the system primary key of the master object, business handshake fields, and general fields.

[0088] The data comparison process may be a process of comparing the consistency of the source system field set and the master data platform field set. For example, the data comparison process may include steps such as intelligent matching mapping and field consistency comparison.

[0089] The data to be checked may be data records that need to be verified and processed. For example, the data to be checked may include records that have been mapped but whose fields are not completely consistent and records that have not been mapped.

[0090] The verification process may be a process of confirming and processing the data to be verified, for example, the verification process may include steps such as matching and confirming the processing method.

[0091] Among them, the data consistency update processing can be the process of updating the data of the main data platform or source system according to the verification processing results. For example, the data consistency update processing can include operations such as main object change, main object addition, or source system data change.

[0092] Optionally, the terminal establishes a corresponding data comparison table for the source system, which is used to record the correspondence between the data fields of the source system and the data fields of the master data platform. The terminal extracts the original data from the source system, performs data conversion processing on the original data to obtain the target data, and updates the target data to the source system field set in the data comparison table. The terminal obtains the data change information of the master data platform, and updates the data change information to the master data platform field set in the data comparison table. The terminal performs data comparison processing on the source system field set and the master data platform field set according to the corresponding relationship to obtain the data comparison result. The terminal determines the data to be checked in the data comparison table based on the data comparison result, performs verification processing on the data to be checked, and obtains the verification processing result. The terminal performs data consistency update processing on the master data platform or the source system based on the verification processing result.

[0093] For example, the terminal establishes a personnel data comparison table for the human resources system, which records the correspondence between the personnel data fields of the human resources system and the personnel master object data fields of the master data platform. The terminal extracts the original personnel data from the human resources system, cleans and converts the original data to obtain the target data, and updates the target data to the human resources system field set in the personnel data comparison table. The terminal obtains the change information of the personnel master object of the master data platform, and updates the change information to the master data platform field set in the personnel data comparison table. Based on the correspondence, the terminal compares the human resources system field set and the master data platform field set to obtain the comparison result. Based on the comparison result, the terminal determines the data records that need to be checked, checks and processes these records, and obtains the processing result. Based on the processing result, the terminal updates the personnel master object of the master data platform or the personnel data of the human resources system for consistency.

[0094] In the above-mentioned multi-source data consistency processing method based on the data comparison table, a corresponding data comparison table is established for the source system; the data comparison table is used to record the correspondence between the data fields of the source system and the data fields of the master data platform; the original data is extracted from the source system, and the original data is converted to obtain the target data, and the target data is updated to the source system field set in the data comparison table; the data change information of the master data platform is obtained, and the data change information is updated to the master data platform field set in the data comparison table; according to the corresponding relationship, the source system field set and the master data platform field set are compared and processed to obtain the data comparison result; according to the data comparison result, the data to be checked in the data comparison table is determined; the data to be checked is checked to obtain the checking result; according to the checking result, the master data platform or the source system is updated for data consistency. This solution achieves structured mapping of data on both ends by establishing a data comparison table to record the field correspondence between the source system and the master data platform; through automated data conversion, change synchronization and comparison processing, it is conducive to quickly identifying data inconsistencies; finally, the data is automatically updated according to the verification results, which is conducive to timely elimination of data differences; this series of automated processing flows is conducive to improving the efficiency of data consistency processing and maintaining data consistency in a multi-source data environment.

[0095] In an exemplary embodiment, data consistency update processing is performed on the master data platform or source system based on the verification processing results, which specifically includes the following contents: determining corresponding processing method information based on the verification processing results; and performing data consistency update processing on the master data platform or source system based on the processing method information.

[0096] The verification processing result may be a result obtained after confirming and processing the data to be verified. For example, the verification processing result may include processing results of steps such as matching and confirming processing methods.

[0097] The processing method information may be a data update method determined after verification and processing. For example, the processing method information may include options such as changing the main object, adding a new main object, changing the source system, or not processing.

[0098] Optionally, the terminal determines the corresponding processing method information based on the verification processing result. The terminal analyzes the data difference content in the verification processing result and determines the processing method according to the preset rules, such as changing the main object, adding a new main object, changing the source system, or not processing. The terminal performs data consistency update processing on the main data platform or source system based on the determined processing method information. The terminal calls the corresponding interface based on the processing method information, such as calling the component to initiate the main object change or addition process, or calling the source system interface to initiate the data source system change process.

[0099] The technical solution provided in this embodiment is conducive to flexibly handling different types of data inconsistencies by determining the processing method information according to the verification processing results, and updating the data consistency of the main data platform or source system based on the information, thereby facilitating the dynamic consistency management and control of multi-source data, improving data quality, reducing manual intervention, and enhancing the automation and intelligence of data management.

[0100] In an exemplary embodiment, data consistency update processing is performed on the master data platform or the source system according to the processing method information, which specifically includes the following contents: when the processing method information indicates updating the master data platform, the data of the master data platform is updated; when the processing method information indicates updating the source system, the data of the source system is updated.

[0101] Among them, updating the data of the master data platform may be changing or adding a master object in the master data platform. For example, updating the data of the master data platform may be calling a component to initiate a master object change or addition process.

[0102] Here, updating the data of the source system may be performing a change operation on the data in the source system. For example, updating the data of the source system may be invoking a source system interface to initiate a data source system change process.

[0103] Optionally, the terminal determines the target system for data consistency update based on the processing method information. When the processing method information indicates that the master data platform is updated, the terminal calls the component to initiate the master object change process to update the data of the source system to the master data platform. When the processing method information indicates that the source system is updated, the terminal calls the source system interface to initiate the data source system change process to update the data of the master data platform to the source system. After completing the data update, the terminal triggers the data comparison component to re-execute the comparison process to confirm data consistency.

[0104] The technical solution provided in this embodiment, by flexibly selecting to update the data of the main data platform or the source system according to the processing method information, is conducive to adopting corresponding update strategies for different data inconsistency situations, thereby facilitating the dynamic consistency management and control of multi-source data, improving data quality and reliability, and maintaining data synchronization between the source system and the main data platform, providing an accurate basis for subsequent data analysis and decision-making.

[0105] In an exemplary embodiment, the data to be checked in the data comparison table is determined based on the data comparison results, which specifically includes the following contents: based on the data comparison results, the data corresponding to the mapped but incompletely consistent fields and the data corresponding to the unmapped are determined; the data corresponding to the mapped but incompletely consistent fields and the data corresponding to the unmapped are both used as the data to be checked.

[0106] The data comparison results include at least mapped and fields are completely consistent, mapped but fields are not completely consistent, and not mapped. For example, the data comparison results may be the result obtained after comparing the source system field set and the master data platform field set. For example, the data comparison results may include the three situations of mapped and fields are completely consistent, mapped but fields are not completely consistent, and not mapped.

[0107] Among them, mapped and fields are completely consistent can be a situation where a mapping relationship has been established between the objects in the source system and the master data platform and all field values ​​are exactly the same. For example, mapped and fields are completely consistent can be a situation where a corresponding relationship has been established between the personnel record in the human resources system and the personnel master object in the master data platform and all field values ​​are the same.

[0108] Among them, mapped but fields are not completely consistent can be a situation where the objects in the source system and the master data platform have established a mapping relationship but there are inconsistent field values. For example, mapped but fields are not completely consistent can be a situation where the personnel record in the human resources system and the personnel master object in the master data platform have established a corresponding relationship but some field values ​​are different, such as inconsistent gender fields.

[0109] Among them, unmapped may be a situation where no corresponding record is found in the master data platform for an object existing in the source system. For example, unmapped may be a situation where no corresponding record is found in the personnel master object table of the master data platform for a personnel record existing in the human resources system.

[0110] Optionally, the terminal analyzes the data comparison results and divides the results into three types: mapped and the fields are completely consistent, mapped but the fields are not completely consistent, and unmapped. The terminal filters out two types of data: data corresponding to mapped but the fields are not completely consistent and data corresponding to unmapped. The terminal marks the data corresponding to mapped but the fields are not completely consistent and the data corresponding to unmapped as data to be verified, and stores the data to be verified in a list of data to be verified for subsequent verification processing.

[0111] The technical solution provided in this embodiment, by classifying the data comparison results into mapped and completely consistent fields, mapped but incompletely consistent fields, and unmapped, and using the latter two types of data as data to be verified, is conducive to accurately identifying data inconsistencies, thereby facilitating improving the efficiency and accuracy of data consistency management, reducing unnecessary verification work, and realizing intelligent dynamic management and control of multi-source data.

[0112] In an exemplary embodiment, after performing data consistency update processing on the main data platform or source system according to the verification processing results, it also includes the following content: obtaining data update information corresponding to the data consistency update processing; updating the data comparison table according to the data update information.

[0113] The data update information may be information describing the result of the data consistency update process. For example, the data update information may include the updated object identifier, the updated field name, the field values ​​before and after the update, and the like.

[0114] Among them, the update processing can be the process of modifying the data comparison table accordingly according to the data update information. For example, the update processing can include updating the values ​​of fields such as the main object field set, comparison status and / or verification processing status in the data comparison table.

[0115] Optionally, after completing the data consistency update process for the master data platform or source system, the terminal obtains data update information through the master object change synchronization component. Based on the acquired data update information, the terminal calls the data comparison component to update the data comparison table. The terminal updates the master object field set in the data comparison table to make it consistent with the latest data in the master data platform. The terminal triggers the data comparison component to re-execute the comparison process, update the comparison status field, and ensure that the data comparison table reflects the latest data consistency status.

[0116] The technical solution provided in this embodiment, by obtaining data update information processed by data consistency update and updating the data comparison table accordingly, is conducive to maintaining real-time synchronization of the data comparison table with the main data platform and the source system, thereby facilitating timely reflection of the latest data consistency status, realizing dynamic consistency management and control of multi-source data, and improving the efficiency and accuracy of data management.

[0117] In an exemplary embodiment, data conversion processing is performed on the original data to obtain target data, which specifically includes the following contents: data conversion processing is performed on the original data to obtain converted data consistent with the data format of the main data platform; and the converted data is used as the target data.

[0118] The data conversion process may be a process of cleaning and converting the original data. For example, the data conversion process may include operations such as data format conversion and / or field merging and splitting.

[0119] The converted data may be an intermediate result obtained after data conversion processing. For example, the converted data may be data obtained after format conversion, merging or splitting of fields in the original personnel data of the human resources system.

[0120] The data format of the master data platform may be a standard data model format defined in the master data platform. For example, the data format of the master data platform may be a field set and a format of a field set defined in a personnel master object model.

[0121] Optionally, the terminal obtains the original data from the original data table, and cleans and converts the original data according to the preset conversion rules. The terminal adjusts the format of the converted data to be consistent with the data format of the main data platform, and generates conversion data. The terminal stores the generated conversion data as the target data in the corresponding data table for subsequent data comparison and processing operations.

[0122] The technical solution provided in this embodiment converts the original data to make it consistent with the data format of the main data platform, which is conducive to eliminating the data format differences between the source system and the main data platform, thereby facilitating accurate data comparison and comparison, and improving the efficiency and accuracy of multi-source data consistency management and control.

[0123] The following is an application example to illustrate the multi-source data consistency processing method based on the data comparison table provided in this application. This application example uses the method applied to a terminal as an example.

[0124] 1.1 Pain points:

[0125] There is multiple management of master data, and the pain points are as follows:

[0126] 1) There are multiple entries in common fields, such as the name of the unit, credit code, etc. There are inconsistent data.

[0127] 2) Since it is impossible to achieve OneID (unified identifier) ​​for the main object, it is not conducive to global data sharing and analysis.

[0128] 1.2 Scope of application:

[0129] Taking eDam (data management platform) as a maintenance source, if there are multiple sources of maintenance data, this technical solution can be applied (not limited to master data), as follows:

[0130] 1) There are multiple business source systems that maintain the same object (generally master data), such as organizational personnel. In this case, eDam is needed as a trusted data source for coordination;

[0131] 2) It is difficult to maintain and manage data in the business source system, and it needs to be extracted to eDam for maintenance and management. In this case, both the business source system and eDam have the right to modify the data.

[0132] 1.3 Solution Description:

[0133] Since the problem of master data consistency is more representative, this application example uses the master data scenario as the example and uses the master data platform to represent eDam.

[0134] 2 Overall framework:

[0135] Overall framework reference Figure 2 , including: data source system, system A, system B, batch data collection, original data table of system A, data conversion component of system A, data comparison table of system A, data comparison table component of system A, original data table of system B, data conversion component of system B, data comparison table of system B, data comparison table component of system B, source system change request, master data platform change request, master data terminal, master object component, master object table, master object change synchronization component, B system operation and maintenance personnel, master data operation personnel, maintenance of master objects, A system operation and maintenance personnel, monitoring and processing of data differences, data comparison component, scanning of data differences.

[0136] (1) Data source system:

[0137] The data source system is the business-end system that maintains master data. It generally extracts data in batches into the original data table through data integration technology at the data center.

[0138] (2) Original data table:

[0139] The original database table of the data center is consistent with the table structure of the source system and stores a mirror image of the source system.

[0140] (3) Data conversion component:

[0141] Due to the inconsistency between the source system and the master object model, data conversion is required to ensure that the source and master data can be effectively compared. Data conversion includes logic such as data format conversion, field merging and splitting, and generally runs on a fixed cycle.

[0142] (4) Data comparison table:

[0143] Each source system has a separate data comparison table to store the mapping relationship with the main object, comparison status and manual processing status.

[0144] (5) Data comparison component:

[0145] The data comparison component is responsible for scanning the consistency of the source system data and the master data platform data in the data comparison table, providing a description of the comparison situation, and can also automatically match object mapping relationships based on field matching rules.

[0146] (6) Data comparison table_R component (data comparison table component):

[0147] System operation and maintenance personnel can manually check the comparison status through the data comparison table_R component, and request to initiate a change request to the source system or master object.

[0148] (7) Main object_R component (main object component):

[0149] A tool for master data operators to maintain master data, with the functions of adding and modifying master objects.

[0150] (8) Main object table:

[0151] Stores primary objects and provides primary object query services.

[0152] 3Data comparison table:

[0153] 3.1 Sort out the field correspondence:

[0154] Field mapping reference Figure 3 , including: source system field list, serial number (1, 2, 3, 4), fields (o_id, o_a, o_b, o_c), primary object field list, serial number (1, 2, 3, 4), fields (m_id, m_a, m_b, m_d), field mapping relationship, serial number (1, 2, 3), fields (o_id, o_a, o_b), primary object fields (m_id, m_a, m_b), field categories (system primary key, business handshake field and general field).

[0155] Since the data models of the source system and the master data platform are inconsistent, it is necessary to sort out the data field mapping relationship between the two ends. The field categories are divided into:

[0156] (1) System primary key: generally refers to the unique identifier of an object assigned at the technical level, such as a UUID (universally unique identifier) ​​or a serial number.

[0157] (2) Business handshake field: generally refers to a business field that can uniquely identify an object. It can be one or more fields, such as: personnel attribute information, file name, and code.

[0158] The business handshake field is an important basis for matching unified objects between multiple systems, and can be matched intelligently with the help of technical means.

[0159] (3) General fields: general fields that cannot identify the uniqueness of an object, such as amount.

[0160] The field mapping relationship only includes the overlapping parts of the data fields in the source system and the master data platform.

[0161] 3.2 Data comparison table structure:

[0162] Table 1 (Data comparison table structure)

[0163]

[0164] Based on the field comparison relationship, a data comparison table is further constructed, including the source system field set, the main object field set, the comparison status, and the manual verification processing status. Each record in the data comparison table represents the mapping relationship of an object.

[0165] (1) The source system field set comes from the field mapping relationship.

[0166] (2) The main object field set comes from the field mapping relationship.

[0167] (3) Comparison situation

[0168] There are three types of comparison categories: 1) Unmapped: an object exists in the source system but is not found in the primary object table. In this case, the primary object field set is empty. 2) Mapped, fields are completely consistent: an object exists in the source system and the corresponding primary object is found. A mapping relationship is established between the source system and the system primary key field of the primary object, and all field values ​​are consistent. 3) Mapped, fields are not completely consistent: an object exists in the source system and the corresponding primary object is found. A mapping relationship is established between the source system and the system primary key field of the primary object, but there are fields with inconsistent values.

[0169] Difference content: When the comparison category is "Mapped, fields are not completely consistent", further record the difference between the source system and the primary object fields in the following format:

[0170] Source system field a name: source system field value, primary object field value; source system field b name: source system field value, primary object field value.

[0171] (4) Manual verification and processing:

[0172] Manual verification flags include: pending verification, verified, and situations that require manual verification include:

[0173] Not mapped;

[0174] The mapping relationship that the system automatically matches for the first time needs to be manually checked;

[0175] When the mapped fields are not completely consistent, manual verification is required when the difference content changes compared to the last time.

[0176] The processing methods include: changing the primary object, adding a new primary object, changing the source system and / or not processing.

[0177] 3.3 The relationship between the data comparison table and the component operation:

[0178] Data comparison table and component operation relationship reference Figure 4, including: data comparison table, serial number, source system field set, o_id, o_a, o_b, primary object field set, m_id, m_a, m_b, comparison situation, comparison category, difference content, manual verification processing situation, manual verification flag, processing method, 1, o_1, a_1, b_1, unmapped, verified, change primary object, 2, o_2, a_2, b_2, m_2, a_2, b_2, mapped, fields are completely consistent, 3, o_3, a_3, b_3, m_3, a_31, b_3, mapped, fields are not completely consistent, o_a: a_3, a_31, verified, change source system, source system data conversion component, primary object change synchronization component, data comparison component, data comparison table_R component, original data table, primary object table.

[0179] The data comparison table is maintained by various components in a collaborative manner:

[0180] (1) The source system data conversion component converts the original data table and writes it into the source system field set.

[0181] (2) The main object change synchronization component monitors the main object table. When the value of the main object field in the data comparison table changes, the content of the corresponding main object field set is updated to maintain consistency with the main object field value.

[0182] (3) After comparing the source system field set with the primary object field set, the data comparison component writes the comparison category and difference content. If manual verification is required, the manual verification flag will be updated.

[0183] (4) After manual verification, the data comparison table R component updates the manual verification flag and processing method fields.

[0184] The data comparison table and the CU (create-update) matrix of each component are as follows:

[0185] Table 2 (Data comparison table and CU matrix table of each component)

[0186]

[0187] 4. Data comparison and processing flow logic:

[0188] Data comparison and processing flow chart reference Figure 5, including: source system data, primary object change synchronization component, data comparison component, data comparison table component, primary object component, data source system, batch acquisition of original data, real-time monitoring of primary object changes, execution of data conversion and cleaning, primary object field set written into data comparison table, source system field set written into data comparison table, traversal of data comparison table, data comparison category, unmapped, mapped, intelligent matching mapping, field consistency comparison, matching results, update comparison status, traversal of data comparison records to be checked, data comparison category, mapped, fields are not completely consistent, unmapped, manual matching, manual confirmation processing method, update manual verification processing status, matching results, unable to match, matched, change primary object, primary object change process, change source system, data source system change process, primary object new process, end of new process, return new primary object ID, record source system object ID and primary object ID mapping relationship.

[0189] The data comparison and processing flow are described as follows:

[0190] Table 3 (Data comparison and processing flow chart)

[0191]

[0192] The technical solution provided by this application example realizes the structured mapping of data on both ends by establishing a data comparison table to record the field correspondence between the source system and the master data platform; through automated data conversion, change synchronization and comparison processing, it is conducive to quickly identifying data inconsistencies; finally, the data is automatically updated according to the verification results, which is conducive to timely elimination of data differences; this series of automated processing flows is conducive to improving the efficiency of data consistency processing and maintaining data consistency in a multi-source data environment.

[0193] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0194] Based on the same inventive concept, the embodiment of the present application also provides a multi-source data consistency processing device based on a data comparison table for implementing the multi-source data consistency processing method based on a data comparison table involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the multi-source data consistency processing device based on a data comparison table provided below can be referred to the limitations of the multi-source data consistency processing method based on a data comparison table above, and will not be repeated here.

[0195] In an exemplary embodiment, Figure 6 As shown, a multi-source data consistency processing device based on a data comparison table is provided. The multi-source data consistency processing device 600 based on a data comparison table may include:

[0196] The data establishment module 601 is used to establish a corresponding data comparison table for the source system; the data comparison table is used to record the correspondence between the data fields of the source system and the data fields of the master data platform;

[0197] The data conversion module 602 is used to extract original data from the source system, perform data conversion processing on the original data to obtain target data, and update the target data to the source system field set in the data comparison table;

[0198] The information acquisition module 603 is used to acquire the data change information of the master data platform and update the data change information to the master data platform field set in the data comparison table;

[0199] The data comparison module 604 is used to perform data comparison processing on the source system field set and the main data platform field set according to the corresponding relationship to obtain a data comparison result;

[0200] The data determination module 605 is used to determine the data to be checked in the data comparison table according to the data comparison result;

[0201] The data checking module 606 is used to check the data to be checked and obtain the checking result;

[0202] The system update module 607 is used to perform data consistency update processing on the main data platform or the source system according to the verification processing result.

[0203] In an exemplary embodiment, the system update module 607 is further used to determine corresponding processing method information according to the verification processing result; and perform data consistency update processing on the main data platform or the source system according to the processing method information.

[0204] In an exemplary embodiment, the system update module 607 is also used to update the data of the main data platform when the processing method information indicates updating the main data platform; and to update the data of the source system when the processing method information indicates updating the source system.

[0205] In an exemplary embodiment, the data comparison results include at least mapped and the fields are completely consistent, mapped but the fields are not completely consistent, and unmapped; the data determination module 605 is also used to determine the data corresponding to the mapped but incompletely consistent fields and the data corresponding to the unmapped according to the data comparison results; the data corresponding to the mapped but incompletely consistent fields and the data corresponding to the unmapped are both used as data to be checked.

[0206] In an exemplary embodiment, the device 600 further includes: a data processing module, configured to obtain data update information corresponding to data consistency update processing; and perform update processing on the data comparison table according to the data update information.

[0207] In an exemplary embodiment, the data conversion module 602 is also used to perform data conversion processing on the original data to obtain conversion data consistent with the data format of the main data platform; and use the conversion data as the target data.

[0208] Each module in the multi-source data consistency processing device based on the data comparison table can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0209] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a multi-source data consistency processing method based on a data comparison table is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0210] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0211] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0212] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0213] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0214] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0215] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0216] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-source data consistency processing method based on a data comparison table, characterized in that: The method comprises: For the source system, a corresponding data comparison table is established; the data comparison table is used to record the correspondence between the data fields of the source system and the data fields of the master data platform; the source system includes the human resources system; the data comparison table includes the source system field set, the master object field set, the comparison status and the verification processing status; Extracting original data from the source system, performing data conversion processing on the original data to obtain target data, and updating the target data to the source system field set in the data comparison table; Acquire data change information of the master data platform, and update the data change information to the master object field set in the data comparison table; According to the corresponding relationship, data comparison processing is performed on the source system field set and the primary object field set to obtain a data comparison result; the data comparison result at least includes mapped and the fields are completely consistent, mapped but the fields are not completely consistent, and unmapped; the mapped and the fields are completely consistent means that the objects in the source system and the primary data platform have established a mapping relationship and all field values ​​are exactly the same; the mapped but the fields are not completely consistent means that the objects in the source system and the primary data platform have established the mapping relationship but there are inconsistent field values; the unmapped means that the object existing in the source system has no corresponding record found in the primary data platform; According to the data comparison result, the data corresponding to the mapped but incompletely consistent fields and the data corresponding to the unmapped fields are determined; The mapped data corresponding to the fields that are not completely consistent and the unmapped data are both used as the data to be checked in the data comparison table; Performing verification processing on the data to be verified to obtain a verification processing result; According to the verification result, the master data platform or the source system is updated with data consistency; the data consistency update includes master object change, master object addition or source system data change; Acquire data update information corresponding to the data consistency update process; the data update information includes an updated object identifier, an updated field name, and field values ​​before and after the update; The data comparison table is updated according to the data update information; the update process includes updating the primary object field set, the comparison status and / or the verification status in the data comparison table.

2. The method according to claim 1, characterized in that The step of performing data consistency update processing on the master data platform or the source system according to the verification processing result includes: Determine corresponding processing method information according to the verification processing result; According to the processing method information, data consistency update processing is performed on the main data platform or the source system.

3. The method according to claim 2, characterized in that The performing data consistency update processing on the master data platform or the source system according to the processing mode information includes: When the processing method information indicates updating the master data platform, updating the data of the master data platform; When the processing method information indicates that the source system is to be updated, data of the source system is updated.

4. The method according to claim 1, characterized in that: The master data platform is a platform that serves as a trusted data source for data coordination.

5. The method according to claim 1, characterized in that The raw data is unprocessed data extracted from the source system.

6. The method according to any one of claims 1 to 5, characterized in that The performing data conversion processing on the original data to obtain target data includes: Performing data conversion processing on the original data to obtain converted data consistent with the data format of the main data platform; The converted data is used as the target data.

7. A multi-source data consistency processing device based on a data comparison table, characterized in that: The device comprises: A data establishment module is used to establish a corresponding data comparison table for the source system; the data comparison table is used to record the corresponding relationship between the data fields of the source system and the data fields of the master data platform; the source system includes a human resources system; the data comparison table includes a source system field set, a master object field set, a comparison status and a verification processing status; A data conversion module, used to extract original data from the source system, perform data conversion processing on the original data to obtain target data, and update the target data to the source system field set in the data comparison table; An information acquisition module, used to acquire data change information of the master data platform, and update the data change information to the master object field set in the data comparison table; A data comparison module is used to perform data comparison processing on the source system field set and the primary object field set according to the corresponding relationship to obtain data comparison results; the data comparison results at least include mapped and the fields are completely consistent, mapped but the fields are not completely consistent, and unmapped; the mapped and the fields are completely consistent means that the objects in the source system and the primary data platform have established a mapping relationship and all field values ​​are exactly the same; the mapped but the fields are not completely consistent means that the objects in the source system and the primary data platform have established the mapping relationship but there are inconsistent field values; the unmapped means that the object existing in the source system has no corresponding record found in the primary data platform; A data determination module is used to determine the data corresponding to the mapped but incompletely consistent fields and the data corresponding to the unmapped fields according to the data comparison result; and use the data corresponding to the mapped but incompletely consistent fields and the data corresponding to the unmapped fields as the data to be checked in the data comparison table; A data checking module is used to check the data to be checked and obtain a checking result; A system update module, used to perform data consistency update processing on the master data platform or the source system according to the verification processing result; the data consistency update processing includes master object change, master object addition or source system data change; A data processing module is used to obtain data update information corresponding to the data consistency update processing; the data update information includes the updated object identifier, the updated field name and the field values ​​before and after the update; according to the data update information, the data comparison table is updated; the update processing includes updating the main object field set, the comparison status and / or the verification processing status in the data comparison table.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images