Master data integration method, system and equipment based on index weighting and storage medium

Abstract

The embodiment of the invention provides a main data integration method and system based on index weighting, electronic equipment and a storage medium, and belongs to the technical field of data processing. According to the scheme, an evaluation system of main data is determined, and the evaluation system comprises evaluation indexes and corresponding weight values; obtaining a multi-source data set, wherein the multi-source data set comprises main data of a plurality of data sources; constructing a standard model, and performing data relation mapping according to the multi-source data set to obtain a data mapping relation; performing field scanning detection on the multi-source data set based on the standard model and the data mapping relation to obtain a scanning result, and performing weighted calculation on the scanning result according to the evaluation system to obtain a comprehensive scoring result; exporting a field full association table by referring to the comprehensive scoring result, and performing strategy analysis on the field full association table to determine a merging strategy; and integrating the multi-source data set according to the merging strategy. According to the scheme, the accuracy and efficiency of master data integration can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a master data integration method, system, device and storage medium based on indicator weighting. Background Art

[0002] In the process of enterprise informatization construction, master data is the core data asset, and its quality is directly related to the efficiency of business processes and the accuracy of decision-making. The goal of master data integration is to merge master data from different business systems or data sources into a unified data model or data set to improve the integrity, accuracy and availability of data, thereby supporting deeper analysis, decision-making and application development.

[0003] Master data describes the key attributes of business entities, and these data are often stored in multiple unintegrated systems or databases. At present, the integration method of master data often adopts manual judgment or simple data matching algorithms. Due to the lack of quantitative evaluation and optimization of data quality, it not only increases the complexity of data management and maintenance, but also affects the accuracy and completeness of master data. When faced with master data from different sources and with various definitions, it is easy to have problems of poor data quality and low integration efficiency. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a master data integration method, system, electronic device and storage medium based on indicator weighting, aiming to improve the accuracy and efficiency of master data integration.

[0005] To achieve the above purpose, one aspect of an embodiment of the present application proposes a master data integration method based on index weighting, the method comprising:

[0006] Determine an evaluation system for master data, wherein the evaluation system includes evaluation indicators and corresponding weight values;

[0007] Acquire a multi-source data set, the multi-source data set including master data from multiple data sources;

[0008] Constructing a standard model, and performing data relationship mapping according to the multi-source data sets to obtain a data mapping relationship;

[0009] Performing field scanning detection on the multi-source data set based on the standard model and the data mapping relationship to obtain scanning results, and performing weighted calculation on the scanning results according to the evaluation system to obtain a comprehensive scoring result;

[0010] Deriving a field full association table with reference to the comprehensive scoring result, and performing a strategy analysis on the field full association table to determine a merging strategy;

[0011] The multi-source data sets are integrated according to the merging strategy.

[0012] In some embodiments, the determining of the evaluation system of the master data comprises the following steps:

[0013] Matching degree, repetition rate, null value rate, similarity and sensitivity are determined as evaluation indicators;

[0014] Determine corresponding weight values ​​for the evaluation indicators through an expert scoring mechanism;

[0015] A weighted calculation formula is determined according to the evaluation index and the weight value to obtain an evaluation system for the master data.

[0016] In some embodiments, the constructing of the standard model and performing data relationship mapping according to the multi-source data sets to obtain the data mapping relationship includes the following steps:

[0017] Performing data analysis on the multi-source data set to determine entity elements of the master data;

[0018] Based on the entity elements, a standard model with a unified format and structure is established according to a preset entity standard;

[0019] The fields in the master data are matched with the corresponding fields in the standard model to determine the data mapping relationship.

[0020] In some embodiments, the performing field scanning detection on the multi-source data set based on the standard model and the data mapping relationship to obtain a scanning result, and performing weighted calculation on the scanning result according to the evaluation system to obtain a comprehensive scoring result, includes the following steps:

[0021] Based on the standard model and the data mapping relationship, the master data of the same data source is scanned and detected, and the data type, length, format, value range and content of the fields in the master data are checked to obtain a scanning result;

[0022] A weighted calculation is performed on each field in the scan result according to the evaluation system to obtain a comprehensive scoring result.

[0023] In some embodiments, the deriving of a field-wide association table with reference to the comprehensive scoring result, and performing a strategy analysis on the field-wide association table to determine a merging strategy, comprises the following steps:

[0024] Performing association analysis on the master data between multiple data sources to extract associated fields, and sorting the data sources to which the associated fields belong in descending order according to the comprehensive scoring result to obtain a field full association table, wherein the rows of the field full association table represent associated data, and the columns of the field full association table represent the data sources to which the associated data belong;

[0025] The data source with the highest comprehensive score in the field full association table is determined as a primary data source, and the remaining data sources in the field full association table except the primary data source are determined as reference data sources;

[0026] A merging strategy is determined by performing a strategy analysis on the field full association table according to the primary data source.

[0027] In some embodiments, the step of performing a strategy analysis on the field full association table according to the primary data source and the reference data source to determine a merging strategy comprises the following steps:

[0028] Performing a strategic analysis on the field full association table according to the primary data source and the reference data source to determine whether field data exists in the primary data source;

[0029] When the primary data source has field data, determining whether there is a data conflict between the field data of the primary data source and the field data of the reference data source;

[0030] When there is no data conflict between the field data of the primary data source and the field data of the reference data source, determining the merging strategy to adopt the field data of the primary data source;

[0031] When there is a data conflict between the field data of the primary data source and the field data of the reference data source, determining a merging strategy to perform data cleaning based on a preset first conflict rule, and adopting the field data of the data source that meets the specification after data cleaning;

[0032] Alternatively, when the primary data source does not have field data, determining whether there is a data conflict in the field data between the reference data sources;

[0033] When there is no data conflict in the field data between the reference data sources, the merging strategy is determined to check the usage of the field data of the reference data source; when the usage is newly added, the field data of the reference data source is used to supplement the data of the main data source;

[0034] When there is a data conflict in the field data between the reference data sources, the merging strategy is determined to perform data verification based on a preset second conflict rule, and the field data that has been verified to be correct is used to supplement the data of the primary data source.

[0035] In some embodiments, the master data integration method based on index weighting further includes the following steps:

[0036] Obtaining verification results by performing quality checks, data comparisons, business verifications, and compliance checks on the integrated master data;

[0037] The evaluation system and the merging strategy are adjusted according to the verification result.

[0038] To achieve the above purpose, another aspect of the embodiment of the present application proposes a master data integration system based on index weighting, the system comprising:

[0039] The first module is used to determine the evaluation system of the master data, wherein the evaluation system includes evaluation indicators and corresponding weight values;

[0040] A second module is used to obtain a multi-source data set, wherein the multi-source data set includes master data from multiple data sources;

[0041] The third module is used to build a standard model and perform data relationship mapping according to the multi-source data set to obtain a data mapping relationship;

[0042] The fourth module is used to perform field scanning detection on the multi-source data set based on the standard model and the data mapping relationship to obtain scanning results, and perform weighted calculation on the scanning results according to the evaluation system to obtain a comprehensive scoring result;

[0043] The fifth module is used to derive a field full association table with reference to the comprehensive scoring result, and to perform a strategy analysis on the field full association table to determine a merging strategy;

[0044] The sixth module is used to integrate the multi-source data sets according to the merging strategy.

[0045] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0046] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0047] The embodiments of the present application include at least the following beneficial effects: The present application provides a master data integration method, system, electronic device and storage medium based on indicator weighting. The scheme determines an evaluation system for master data, wherein the evaluation system includes evaluation indicators and corresponding weight values; it can realize quantitative evaluation of data and improve the objectivity and accuracy of data integration.

[0048] By acquiring a multi-source data set, the multi-source data set includes master data from multiple data sources; constructing a standard model, and mapping data relationships based on the multi-source data set to obtain a data mapping relationship; performing field scanning and detection on the multi-source data set based on the standard model and the data mapping relationship to obtain a scanning result, and performing weighted calculation on the scanning result according to the evaluation system to obtain a comprehensive scoring result; deriving a field full association table with reference to the comprehensive scoring result, and performing a strategic analysis on the field full association table to determine a merging strategy; integrating the multi-source data sets according to the merging strategy; being able to use automatic scanning and detection to replace manual exploration, and using the result of weighted summation of indicators to guide the integration of master data, thereby improving the efficiency of master data integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flowchart of the steps of the master data integration method based on indicator weighting provided in an embodiment of the present application;

[0050] Figure 2 It is a schematic diagram of a process for generating a field full association table provided in an embodiment of the present application;

[0051] Figure 3 It is a schematic diagram of the evaluation system and weight setting provided in the embodiment of the present application;

[0052] Figure 4 is a schematic diagram of the standard model data mapping relationship provided in the embodiment of the present application;

[0053] Figure 5 It is a schematic diagram of the comprehensive scoring result of the toll station name provided in the embodiment of the present application;

[0054] Figure 6 It is a schematic diagram of a field full association table provided in an embodiment of the present application;

[0055] Figure 7 This is a schematic diagram of the effect of integrating multi-source master data fields provided by an embodiment of the present application;

[0056] Figure 8 is a flowchart of the steps of a master data integration method based on indicator weighting provided by another embodiment of the present application;

[0057] Fig. 9 It is a flowchart of the steps of establishing an indicator evaluation system and determining indicator weights provided in an embodiment of the present application;

[0058] Fig.10 It is a flowchart of the steps of building a standard model and matching mapping relationships provided in an embodiment of the present application;

[0059] Fig.11 It is a flowchart of performing field scanning detection and outputting indicator scoring results provided by an embodiment of the present application;

[0060] Fig.12 It is a flowchart of the steps of generating a field full association table and performing field merging strategy analysis provided by an embodiment of the present application;

[0061] Fig.13 It is a structural diagram of a master data integration system based on index weighting provided in an embodiment of the present application;

[0062] Fig.14 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0064] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0065] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0067] Before describing the embodiments of the present application in detail, some related technologies involved in the embodiments of the present application are first described.

[0068] There are many methods for master data integration, and there is no unified standard. Currently, master data integration methods often rely on manual judgment or simple data matching algorithms. These methods lack quantitative evaluation and optimization mechanisms for data quality. Since master data is usually stored in multiple unintegrated systems or databases, how to identify and integrate similar or identical objects has become a key issue that needs to be addressed. Master data integration mainly faces the following challenges:

[0069] First, data redundancy and inconsistency. Different business systems may independently access the same core data, which often results in data duplication and inconsistency. For example, the same toll station may have different identifiers or information in different systems, which increases the complexity of data management and maintenance.

[0070] Secondly, data quality is low. Errors, missing or outdated information in master data will directly affect the execution of business processes and the accuracy of decision-making. Low-quality master data may lead to reduced customer satisfaction, supply chain disruptions, inaccurate financial reports and other consequences. For example, the academic qualifications of personnel are filled in manually, which leads to some problems, such as multiple different descriptions of the same major, the use of professional abbreviations, and the inability to find corresponding items in the national professional code for related professional descriptions.

[0071] Third, there is a lack of quantitative evaluation mechanism. In the traditional master data integration process, there is a lack of quantitative evaluation and optimization mechanism for data quality. Traditional methods often rely on manual judgment and operation, rely on the subjective experience of technicians, and lack intuitive indicators to support accurate decision-making.

[0072] Finally, there is a lack of unified data standards. The data formats are diverse, making it difficult to obtain business information; data models vary significantly; there are many sources of the same data, and there is a lack of accurate data identification and indicator definition, resulting in inconsistent data calibers and large differences in statistical indicators, making data merging complicated.

[0073] In view of this, a master data integration method based on indicator weighting is provided in an embodiment of the present application. The scheme determines an evaluation system for master data, wherein the evaluation system includes evaluation indicators and corresponding weight values; it is capable of achieving quantitative evaluation of data and improving the objectivity and accuracy of data integration.

[0074] By acquiring a multi-source data set, the multi-source data set includes master data from multiple data sources; constructing a standard model, and mapping data relationships based on the multi-source data set to obtain a data mapping relationship; performing field scanning and detection on the multi-source data set based on the standard model and the data mapping relationship to obtain a scanning result, and performing weighted calculation on the scanning result according to the evaluation system to obtain a comprehensive scoring result; deriving a field full association table with reference to the comprehensive scoring result, and performing a strategic analysis on the field full association table to determine a merging strategy; integrating the multi-source data sets according to the merging strategy; being able to use automatic scanning and detection to replace manual exploration, and using the result of weighted summation of indicators to guide the integration of master data, thereby improving the efficiency of master data integration.

[0075] The master data integration method based on index weighting provided in the embodiment of the present application relates to the field of data processing technology. The master data integration method based on index weighting provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the master data integration method based on index weighting, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as personal information, financial data, and health records, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0078] Figure 1 is an optional flow chart of the master data integration method based on indicator weighting provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0079] Step S101 : determining an evaluation system for master data, wherein the evaluation system includes evaluation indicators and corresponding weight values.

[0080] Specifically, for master data from different sources, with diverse definitions, repeated fields and inconsistent data, in order to facilitate effective evaluation and analysis during the data integration process, it is necessary to determine the evaluation system of the master data, where the evaluation system includes evaluation indicators and corresponding weight values. In order to build a comprehensive and scientific evaluation system, it is necessary to first clarify the purpose of the evaluation. In the process of master data integration, the evaluation system should take the accuracy and efficiency of master data integration as the evaluation purpose. Then, according to the evaluation purpose, multiple dimensions are set to construct the evaluation system, and the combination of multiple dimensions comprehensively covers all aspects of the evaluation, thereby ensuring the accuracy and reliability of the evaluation results. After determining the evaluation indicators, the corresponding weight values ​​are assigned to each weight based on the importance and influence of each evaluation indicator, and a weighted calculation formula is formulated to improve the evaluation system.

[0081] For example, after comprehensively considering credibility and importance, five aspects, namely, matching degree, repetition rate, null value rate, similarity and sensitivity, are selected as evaluation indicators, different weight values ​​are assigned to the five indicators, and each indicator is multiplied by its corresponding weight to formulate a weighted calculation formula.

[0082] It is understandable that the evaluation indicators and corresponding weight values ​​in the evaluation system can select different dimensions and assign weight values ​​according to the evaluation purpose. This application is only exemplary and does not impose specific limitations.

[0083] Step S102: acquiring a multi-source data set, where the multi-source data set includes master data from multiple data sources.

[0084] Specifically, a multi-source data set is obtained, which includes master data from multiple data sources. Each independently maintained business system is used as a data source, and master data to be integrated from different business systems is collected to obtain a multi-source data set including master data from multiple data sources.

[0085] Step S103, constructing a standard model, and performing data relationship mapping according to the multi-source data sets to obtain a data mapping relationship.

[0086] Specifically, a standard model is constructed, and data relationship mapping is performed based on multi-source data sets to obtain data mapping relationships. In order to effectively integrate data from different systems, after obtaining multi-source data sets, the multi-source data sets are analyzed to clarify the format and content of the data sources, so as to define the standard model of master data and establish the format and structure that conforms to entity standards. Data relationship mapping is performed based on the data elements of multi-source data sets. By identifying key fields and attributes and mapping these fields and attributes to the constructed standard model, it is ensured that the master data of different data sources follow the same model standard, providing a basic guarantee for the consistency and comparability of the master data.

[0087] Step S104, based on the standard model and the data mapping relationship, field scanning detection is performed on the multi-source data set to obtain scanning results, and weighted calculation is performed on the scanning results according to the evaluation system to obtain a comprehensive scoring result.

[0088] Specifically, based on the standard model and data mapping relationship, the multi-source data set is scanned and detected to obtain the scanning results, and the scanning results are weighted according to the evaluation system to obtain the comprehensive scoring results. In practical applications, data often comes from multiple different systems or platforms, and these data may differ in format, structure and content. For multi-source data sets with different sources and diverse definitions, the standard model and data mapping relationship are used to perform field scanning and detection on the multi-source data set, which can ensure that the data from different systems and data sources have a consistent format and structure, so that field scanning and detection can be implemented in an automated manner, improving the efficiency and accuracy of field scanning and detection, and reducing the tediousness and errors caused by manual operations.

[0089] Optionally, automatic scanning detection is used to replace manual detection. According to the pre-set standard model and the corresponding mapping relationship, the scanner can automatically perform model matching and data compliance scanning tasks for specific field information in the same data source. In this process, the scanner will automatically score the scanning results according to the evaluation system, evaluate the overall compliance of the data, and obtain a comprehensive scoring result.

[0090] After obtaining the scan results, the scan results obtained from each data source are weighted according to the evaluation system to obtain a comprehensive score result. The weighted scoring method can effectively determine the main source of data fields from multiple different data sources to provide support for subsequent data integration decisions. For example, data with higher comprehensive scores can be integrated first, or a preset scoring threshold can be set to consider data with a comprehensive score below the scoring threshold as unqualified data.

[0091] Step S105 , deriving a field full association table with reference to the comprehensive scoring result, and performing a strategy analysis on the field full association table to determine a merging strategy.

[0092] Specifically, refer to the comprehensive scoring results to export the field full association table, and perform a strategic analysis on the field full association table to determine the merging strategy. Use tools or scripts to export the required data fields from various data sources, and perform an association analysis on the data fields. Refer to the comprehensive scoring results to select a main data source from multiple data sources, and use the field data of the main data source as a benchmark to perform a horizontal comparison of multi-source field data, analyze the similarities between fields, confirm the common fields that appear repeatedly in multiple data sources, and the unique fields that exist specifically in a single data source, obtain the association results of the field data, and use the association results of the field data to determine the data merging strategy.

[0093] Reference Figure 2 , Figure 2 The steps of inputting and sorting records in multiple tables are shown in the figure. In the step of sorting records, the specific values ​​of each evaluation indicator are calculated according to the evaluation system, and the comprehensive score result of the evaluation object is calculated based on the weight value of the indicator, and the records are sorted according to the comprehensive score result.

[0094] Exemplarily, first, a table input step is performed according to obtaining the data table 1 to be processed, and a sorting and recording step is performed on the input data table 1. Next, a new data table 2 to be processed is introduced to perform table input 2 step, and a sorting and recording step 2 is performed on the input data table 2. Then, the record table connection step is performed, and the field values ​​of the sorted records of the data table 1 and the data table 2 are compared, the similarity between the fields is analyzed, and the associated fields are connected to form a more comprehensive record set. The connected record set is sorted and recorded 3 steps are performed. In this step, the data sources of the same field in the record set are sorted according to the comprehensive scoring results. Data table 3 is introduced in sequence to perform table input 3 step, sorting record 4 step, record set connection 2 step, sorting record 5 step, table data 4 is introduced to perform table input 4 step, sorting record 6 step, record set connection 3 step, to achieve full association of data fields of data table 1, data table 2, data table 3 and data table 4, and finally the field full association table is exported in the form of a table.

[0095] Step S106: Integrate the multi-source data sets according to the merging strategy.

[0096] Specifically, the multi-source data sets are integrated according to the merging strategy. According to the merging strategy obtained by the strategy analysis, the master data of multiple data sources in the multi-source data sets are integrated.

[0097] It should be noted that before integrating the master data, in order to ensure that the final generated data table has high quality and high availability, a series of preprocessing steps are required for the original master data. These steps include data cleaning and deduplication, etc., in order to ensure the standardization and availability of the data. Through these preprocessing steps, redundant information and erroneous data in the data can be removed to ensure the accuracy and consistency of the data, thus providing a solid foundation for subsequent data analysis and processing.

[0098] In some embodiments, step S101 may include but is not limited to steps S201 to S203.

[0099] Step S201, determining matching degree, repetition rate, null value rate, similarity and sensitivity as evaluation indicators.

[0100] For example, refer to Figure 3 , the matching degree, repetition rate, null value rate, similarity and sensitivity are determined as evaluation indicators. Among them, the matching degree is used to evaluate the matching degree between master data in different data sources, the repetition rate is used to count the number of repeated records of master data in the same data source; the null value rate is used to count the proportion of null values ​​in the master data in the same data source, such as empty strings, NULL, "empty", etc.; similarity is used to describe the text similarity relative to other data source fields; sensitivity is used to evaluate sensitive information in the data, such as personal information, financial data, and health records.

[0101] Furthermore, after determining the evaluation index, a corresponding calculation strategy is formulated for each evaluation index to quantify the specific value of the index. For example, the calculation strategy for the repetition rate can calculate the repetition rate based on the ratio of the number of repeated records of the master data in the same data source to the total number of records of all the master data in the data source.

[0102] Step S202: Determine corresponding weight values ​​for evaluation indicators through an expert scoring mechanism.

[0103] Furthermore, in order to ensure the scientificity and objectivity of the evaluation results, the corresponding weight values ​​can be determined for the evaluation indicators through the expert scoring mechanism. By inviting experts in related fields to score and comprehensively consider various opinions and suggestions, a reasonable weight value can be given to each evaluation indicator. These weight values ​​can reflect the importance of different indicators in the comprehensive evaluation, ensuring that the evaluation results can truly reflect the actual performance of the evaluation object.

[0104] Step S203, determining a weighted calculation formula according to the evaluation index and the weight value to obtain an evaluation system for the master data.

[0105] Specifically, after calculating the specific values ​​of each evaluation index and determining the weight value, a weighted calculation formula is determined according to the evaluation index and the weight value to obtain the evaluation system of the master data. For example, according to the evaluation index and the weight value, the weighted formula is determined as: (1-((1-matching degree)*weight+repetition rate*weight+null value rate*weight+similarity*weight+sensitivity*weight))*100.

[0106] In the evaluation system, the higher the score calculated by the weighted calculation formula based on the evaluation indicators and weight values, the better the evaluation object performs in the comprehensive evaluation. Through the evaluation system of master data, it is possible to effectively identify evaluation objects with outstanding performance, thereby providing favorable support for subsequent decision-making.

[0107] In this embodiment, by constructing an evaluation system, it is possible to achieve quantitative evaluation of master data and improve the objectivity and accuracy of data integration.

[0108] In some embodiments, step S103 may include but is not limited to steps S301 to S303.

[0109] Step S301, performing data analysis on multi-source data sets to determine entity elements of master data.

[0110] Specifically, we analyze multi-source data sets to identify the entity elements of master data. We conduct in-depth analysis and understanding of entity tables from various systems, review the entity tables of each system one by one, and identify the key fields. This process usually involves entity elements such as data entity attributes, data types, lengths, formats, and contents.

[0111] Step S302: Based on the entity elements, a standard model with a unified format and structure is established according to preset entity standards.

[0112] Specifically, based on the entity elements, a standard model with a unified format and structure is established according to the preset entity standards. In this step, a standard model with a unified format and structure is established based on the entity elements obtained by data analysis and the preset entity standards. Figure 4 , Taking toll station data as an example, the standard model defines the toll station name (stat ion_name, text type, length 50), the number of toll plazas (pl aza_count, integer type, length 2), the toll station type (station_type, text type, length 2), etc.

[0113] Step S303, matching the fields in the master data with the corresponding fields in the standard model to determine the data mapping relationship.

[0114] In step S303 of some embodiments, fields in the master data are matched with corresponding fields in the standard model to determine a data mapping relationship, wherein the data mapping relationship includes direct mapping, conversion mapping and complex mapping.

[0115] Direct mapping is the most commonly used mapping form, which only requires the entity element to be directly mapped to a model field without additional conversion or calculation.

[0116] Conversion mapping requires some form of processing or transformation of the data before it can be performed, such as converting text types to floating-point types or converting names to code values.

[0117] Complex mapping involves complex conversion rules, such as mapping two entity elements after merging them to a model field or mapping a part of an entity element after splitting it to a model field.

[0118] In order to better understand the mapping relationship, a mapping relationship table can be established to show the correspondence between the entity tables of each system and the standard model. By referring to the mapping relationship table, users can clearly see the mapping relationship between different data sources, thereby better performing data integration and analysis.

[0119] For example, refer to Figure 4 For the master data in the two data sources, the provincial toll collection interface and the department platform toll collection interface, the toll station name field has different names or formats. The toll station name field of the provincial toll collection interface is "name", and the toll station name field of the department platform toll collection is "sfzmc", but in the standard model, they are all the same as "stat ion_name".

[0120] By building a standard model and establishing data mapping relationships, we can ensure the coordination and integration between different data sources, achieve data consistency and comparability, and provide a basis for subsequent data analysis and decision-making.

[0121] In some embodiments, step S104 may include but is not limited to steps S401 to S402.

[0122] Step S401 , based on the standard model and data mapping relationship, field scanning and detection is performed on the master data of the same data source, and the data type, length, format, value range and content of the fields in the master data are checked to obtain the scanning result.

[0123] Specifically, based on the standard model and data mapping relationship, the fields of the master data of the same data source are scanned and detected, and the data type, length, format, value range and content of the fields in the master data are checked to obtain the scanning results. The fields in the master data of the same data source are scanned and detected one by one, and it is checked whether the data type, length and format of the fields in the master data match the data type, length and format defined in the standard model after being mapped to the standard model according to the data mapping relationship, and whether the value range of the fields in the master data is compliant and the content is accurate. The scanning results are obtained by checking whether the data type, length, format, value range and content of the fields in the master data meet the requirements of the standard model.

[0124] Step S402: perform weighted calculation on each field in the scan result according to the evaluation system to obtain a comprehensive scoring result.

[0125] Specifically, a weighted calculation is performed on each field in the scan result according to the evaluation system to obtain a comprehensive score result. Based on the evaluation system that has been constructed and the determined indicator weights, a detailed weighted score is performed on each field in the scan result to calculate a comprehensive score result.

[0126] For example, refer to Figure 5 Based on the five indicators of matching degree, repetition rate, null value rate, similarity and sensitivity and their corresponding weights, the toll station name fields in the four data sources of provincial toll interface, department platform charging, centralized monitoring and entrance overload control are weightedly scored. Taking the provincial toll interface as an example, the toll station name field in the provincial toll interface has a matching degree of 100%, a repetition rate of 10%, a null value rate of 10%, a similarity of 20%, and a sensitivity of 0%. The comprehensive score is calculated based on the established evaluation system and the determined indicator weights: 1-((1-100%)*30%+10%*20%+10%*20%+20%*20%+20%*20%+0%*20%)=92 points.

[0127] In some embodiments, step S105 may include but is not limited to steps S501 to S503.

[0128] Step S501, perform association analysis on the master data between multiple data sources to extract associated fields, and sort the data sources to which the associated fields belong in descending order according to the comprehensive scoring results to obtain a field full association table, wherein the rows of the field full association table represent associated data, and the columns of the field full association table represent the data sources to which the associated data belongs.

[0129] Specifically, the main data between multiple data sources are analyzed for association to extract the associated fields, and the data sources to which the associated fields belong are sorted in descending order according to the comprehensive scoring results to obtain a field full association table, in which the rows of the field full association table represent the associated data, and the columns of the field full association table represent the data sources to which the associated data belongs. In the database, the associated fields refer to the data fields shared between two tables, which are used to establish the relationship between tables, and the associated data are the data connected by the associated fields. The extraction and integration of data fields usually rely on the association mechanism of primary keys or foreign keys. In this way, the required information can be extracted from multiple different data sources. In order to achieve this process, data processing tools or corresponding scripts can be used, such as Kettle, DataStage and other data processing software. In the actual data integration process, the associated fields may include fields with significant characteristics such as name and quantity. These fields may have different names or formats in different data sources. After the associated fields are obtained through association analysis, a table is established with the associated data as rows and the data sources as columns.

[0130] Optionally, for the convenience of analysis, the data sources to which the associated fields belong are sorted in descending order based on the comprehensive scoring results, and the columns of the data sources to which the associated data belongs are adjusted according to the sorting results, and the adjusted table is exported to obtain a field full association table.

[0131] For example, according to Figure 5 It can be seen that for the toll station name field, the comprehensive score of the provincial toll interface is 92 points, the comprehensive score of the department platform charging is 87 points, the comprehensive score of centralized monitoring is 79.4 points, and the comprehensive score of the entrance overload control is 75.8 points. According to the comprehensive score results, the above four data sources are sorted in descending order, and the order of arrangement is provincial toll interface, department platform charging, centralized monitoring, and entrance overload control. With the associated data as rows and the data source to which the associated data belongs as columns, a field full association table is generated in the order of arrangement, such as Figure 6 shown.

[0132] Step S502: determine the data source with the highest comprehensive score in the field full association table as the primary data source, and determine the remaining data sources in the field full association table except the primary data source as reference data sources.

[0133] Specifically, the data source with the highest comprehensive score in the field-wide association table is determined as the primary data source, and the remaining data sources in the field-wide association table except the primary data source are determined as reference data sources. In the field-wide association table, the data source with a higher comprehensive score often means that its data performs better in terms of accuracy, completeness, timeliness, etc. Determining these data sources as primary data sources can ensure higher quality of the integrated data.

[0134] At the same time, in order to improve the efficiency of data integration and supplement the missing or abnormal data in the main data source, the remaining data sources except the main data source in the field full association table are determined as reference data sources. The reference data sources can be cross-validated to supplement the missing or abnormal data in the main data source.

[0135] Step S503: Performing a strategy analysis on the field full association table according to the primary data source and the reference data source to determine a merging strategy.

[0136] Specifically, a strategy analysis is performed on the field-wide association table based on the primary data source and the reference data source to determine the merge strategy. A strategy analysis is performed on the field-wide association table based on the primary data source and the reference data source. For example, if there is no conflict in the field data between the primary data source and the reference data source, or the conflict is small and can be resolved by simple rules, the data from the primary data source is directly used. If there is a large data conflict or inconsistency, data cleaning or data verification is required according to the preset conflict resolution rules to ensure the quality and consistency of the merged data.

[0137] In some embodiments, step S503 may include but is not limited to steps S601 to S607.

[0138] Step S601 : performing a strategy analysis on the field full association table according to the primary data source to determine whether the primary data source has field data.

[0139] Specifically, a strategy analysis is performed on the field-wide association table based on the primary data source to determine whether the primary data source has field data. Considering that the existence of the primary data source will directly affect the selection of the merge strategy, a strategy analysis is first performed on the field-wide association table based on the primary data source to determine whether the primary data source has field data.

[0140] Step S602: When the primary data source has field data, it is determined whether there is a data conflict between the field data of the primary data source and the field data of the reference data source.

[0141] Specifically, when the primary data source has field data, it is further determined whether there is a data conflict between the field data of the primary data source and the field data of the reference data source. When it is confirmed that the primary data source has field data, it is checked in the field full association table whether there is a difference (such as inconsistent content or mismatched format, etc.) in the field data of the primary data source and the reference data source for the same associated field, so as to determine whether there is a data conflict between the field data of the primary data source and the field data of the reference data source.

[0142] Step S603: When there is no data conflict between the field data of the primary data source and the field data of the reference data source, a merging strategy is determined to adopt the field data of the primary data source.

[0143] Specifically, when there is no data conflict between the field data of the primary data source and the field data of the reference data source, the merge strategy is determined to use the field data of the primary data source. When the field data of the primary data source is consistent with the field data of the reference data source, it can be considered that the data is consistent. Since the primary data source is usually regarded as an authoritative data source, the merge strategy is determined to use the field data of the primary data source.

[0144] Step S604: when there is a data conflict between the field data of the primary data source and the field data of the reference data source, a merging strategy is determined to perform data cleaning based on a preset first conflict rule, and adopt the field data of the data source that meets the specification after data cleaning.

[0145] Specifically, when there is a data conflict between the field data of the primary data source and the field data of the reference data source, the merge strategy is determined to perform data cleaning based on the preset first conflict rule, and adopt the field data of the data source that meets the specifications after data cleaning. The first conflict rule is to give priority to whether the field data of the primary data source meets the specifications. If the field data of the primary data source does not meet the specifications, the field data of the primary data source is cleaned. If the field data of the primary data source meets the specifications after data cleaning, the field data of the primary data source is still used. If the field data of the primary data source still does not meet the specifications after data cleaning, consider whether the field data of the reference data source meets the specifications, and perform data cleaning on the reference data source, and adopt the field data of the data source that meets the specifications after data cleaning.

[0146] Alternatively, in step S605, when the primary data source does not have field data, it is determined whether there is a data conflict in the field data between the reference data sources.

[0147] Specifically, when the primary data source does not have field data, it is determined whether there is a data conflict in the field data between the reference data sources. When it is confirmed that the primary data source does not have field data, in the field full association table, it is checked whether there is a difference in the field data of the same association field in each reference data source to determine whether there is a data conflict in the field data between the reference data sources.

[0148] Step S606, when there is no data conflict in the field data between the reference data sources, the merging strategy is determined to check the usage of the field data of the reference data source. If the usage is new, the field data of the reference data source is used to supplement the data of the main data source.

[0149] Specifically, when there is no data conflict in the field data between the reference data sources, the merge strategy is determined to check the usage of the field data of the reference data source. When the usage is newly added, the field data of the reference data source is used to supplement the data of the primary data source. When there is no data conflict in the field data between the reference data sources, it is necessary to contact the business unit corresponding to the data source to check the usage of the field data, whether it is deleted or disabled data, or the newly added data has not been synchronized in time. When the field data is deleted or disabled, delete or stop synchronizing the field data. If the field data is confirmed to be newly added data, the data is directly used to supplement the data of the primary data source.

[0150] Step S607: when there is a data conflict in the field data between the reference data sources, the merging strategy is determined to perform data verification based on the preset second conflict rule, and the field data with correct data verification is used to supplement the main data source.

[0151] Specifically, when there is a data conflict in the field data between the reference data sources, the merging strategy is determined to perform data verification based on the preset second conflict rule, and the field data with correct data verification is used to supplement the data of the main data source. The second conflict rule is to sort the reference data sources in descending order according to the comprehensive scoring results. When there is a data conflict in the field data between the reference data sources, the field data of the reference data sources are verified in order until the field data is confirmed to be correct, and then the field data with correct data verification is used to supplement the data of the main data source.

[0152] In some embodiments, the master data integration method based on index weighting may also include but is not limited to steps S701 to S702.

[0153] Step S701, obtain the verification result by performing quality inspection, data comparison, business verification and compliance inspection on the integrated master data. The integration effect is verified by performing quality inspection, data comparison, business verification and compliance inspection on the integrated master data, so as to further optimize the integration process and form a virtuous circle.

[0154] Specifically, the verification results are obtained by performing quality checks, data comparisons, business verifications, and compliance checks on the integrated master data.

[0155] Quality checks are mainly carried out through automated scanning tools to carefully review the integrity, accuracy and consistency of the data to ensure that the integrated data meets the established standards. Once an anomaly is detected, the verification result is confirmed as a data quality anomaly.

[0156] Data comparison is to compare the latest version of data with the old version to verify whether the previously identified data problems have been properly resolved. If the problem still exists, the verification result is confirmed that the data problem has not been resolved.

[0157] Business verification is to apply the internally approved data to the actual business process to ensure that the data can meet the business needs after integration. If the data fails to meet the business needs, the verification result is confirmed as not meeting the business needs.

[0158] Compliance checks ensure that data complies with relevant laws, regulations and internal rules and regulations. Once deviations are found, the verification results are confirmed as data violations.

[0159] Step S702, adjusting the evaluation system and merging strategy according to the verification result.

[0160] Specifically, the evaluation system and merging strategy are adjusted according to the verification results. The evaluation system and merging strategy are adjusted in a targeted manner according to the verification results obtained by different verification methods.

[0161] For example, when the verification result is that the data quality is abnormal, the source data and the merging strategy need to be reviewed. For example, it can be submitted to the source department for assistance in rectification based on the source of the data source, and then evaluated after the rectification until the standards are met.

[0162] When the verification result shows that the data problem is not resolved, the problem is tracked again and resolved.

[0163] When the verification result shows that the business needs are not met, the business needs to be reorganized and the merging strategy needs to be adjusted. For example, if other source data can meet the business needs after being updated and adjusted, the source data can be used preferentially by adjusting indicators or strategies.

[0164] When the verification result shows that the data is in violation, the data will be immediately adjusted or deleted in accordance with relevant laws, regulations and internal rules and regulations.

[0165] In this embodiment, the efficiency of the master data integration method can be further improved by performing quality checks, data comparisons, business verifications, and compliance checks on the integrated master data and adjusting the evaluation system and merging strategies according to the verification results.

[0166] The following is a detailed introduction and description of the solution of the embodiment of the present invention in conjunction with a specific application example.

[0167] Reference Figure 7, obtain master data including customer number, name, and work unit from system A, system B, and system C, and use the master data integration method based on indicator weighting provided in the embodiment of the present application to integrate multi-source master data fields, and after the scanner automatically calculates the comprehensive scoring result, determine the integration strategy to use the customer number in system A, the name in system B, and the work unit in system C to integrate the master data.

[0168] Reference Figure 8 , Figure 8 It is a step flow chart of a master data integration method based on indicator weighting provided in another embodiment of the present application, which is applied to the master data integration scenario of a toll station. Figure 8 The method may include but is not limited to steps S1 to S5.

[0169] Step S1, establish an indicator evaluation system and determine the indicator weights.

[0170] Step S2, construct a standard model and perform mapping relationship matching.

[0171] Step S3, perform field scanning detection and output indicator scoring results.

[0172] Step S4, generating a field full association table and performing field merging strategy analysis.

[0173] Step S5: Implement master data merging and verification.

[0174] Specifically, first, establish an indicator evaluation system and determine the indicator weights. Fig. 9 When implementing the master data integration method based on weighted scoring of indicators, it is necessary to clarify the purpose of evaluation, which is related to the effectiveness, reliability and final application effect of the evaluation. After confirming the purpose of evaluation, the evaluation indicators are selected from the two dimensions of credibility and importance, and five core indicators covering matching degree, repetition rate, null value rate, similarity and sensitivity are obtained to ensure the comprehensiveness and scientificity of the evaluation.

[0175] For example, refer to Figure 3 The matching degree is divided into model matching degree and data matching degree. The model matching degree includes the matching of type, length, precision, and other formats. The data matching degree includes the compliance of the value range and the accuracy of the content. The matching degree is calculated by a deduction system. The initial total score is set to 100 points. For each mismatch, a certain score (for example, 10 points) is deducted until the total score is reduced to 0 points or all data are matched. Finally, 1 is subtracted from the percentage of the total score after the deduction to get the matching score.

[0176] The duplication rate is used to determine whether there are duplicate records. If a record has multiple duplicates, it will not be removed and will be counted as a duplicate record. The duplication rate is calculated by dividing the number of duplicate records by the total number of records to get the duplication rate score.

[0177] The null value rate is used to determine the null value situation of the data. The null value rate is calculated by using the ratio of the number of null value records to the total number of records to obtain the null value rate score.

[0178] Similarity is detected through the built-in algorithm of the scanner to evaluate the text similarity with other data source fields. The similarity is calculated by directly outputting the percentage from the scanner to obtain the similarity score.

[0179] Sensitivity is achieved by identifying sensitive fields in the database through sensitive word filters and analyzing the data sensitivity of the field. Automated tools can also be used for scanning, and the scanner directly outputs the percentage to obtain the sensitivity score.

[0180] After selecting the evaluation indicators, the weight of the matching degree was determined to be 30%, the weight of the repetition rate to be 20%, the weight of the null value rate to be 20%, the weight of the similarity to be 20%, and the weight of the sensitivity to be 10% through expert scoring and other methods.

[0181] Based on the evaluation indicators and indicator weights, the weighted calculation formula is set to 1 minus the sum of the above indicators and weights to obtain the evaluation score.

[0182] Optionally, the evaluation system can be continuously optimized and adjusted according to the actual situation and implementation effect. Through real-time adjustment, the evaluation system can be ensured to have strong adaptability and effectiveness.

[0183] Next, refer to Fig.10 , build a standard model and match the mapping relationship. The core goal of this step is to build a standard model that meets the entity standard. To achieve this goal, it is necessary to conduct multi-source data analysis, clarify the format and content of each data source, and define a standard model with a unified format and structure for the master data of multiple data sources. Then, design the mapping relationship to map the fields in the data source to the corresponding fields in the standard model. Subsequently, formulate mapping rules covering data conversion rules, data cleaning rules, data validation rules, etc. to ensure that the mapping rules can deal with abnormal situations and data quality issues in the data source.

[0184] At the same time, considering that building a standard model and matching mapping relationships is a complex and meticulous process, it requires selecting appropriate tools, clarifying specifications, and establishing accurate mapping relationships. Therefore, it is also necessary to regularly check and update mapping relationships and rules, and conduct sufficient verification and optimization to maintain the mapping relationship and adapt to changes in data sources and business needs.

[0185] Reference Fig.11 , import the built standard model and mapping relationship into the scanner, and perform persistence operations so that subsequent scanners can easily read it. The scanner performs field scanning detection according to the established indicator settings, checks the field's data type, length, format, value range, content and other attributes. During this process, the scanner automatically scores the scan results and automatically generates an indicator score report based on the comprehensive score results, which is exported in the form of a spreadsheet.

[0186] For example, the toll station name fields in the four data sources of provincial toll interface, department platform toll, centralized monitoring and entrance overload control are scanned and detected. The scanner automatically scores the scan results based on the established evaluation index system and the determined index weights to obtain a comprehensive scoring result, which is exported in Excel file format. The content is shown as follows: Figure 5 shown.

[0187] After the comprehensive scoring result is automatically calculated by the scanner, it can be verified based on the comprehensive scoring result. If the comprehensive scoring result is verified to be reliable, it can be applied to the subsequent data integration decision-making process to enhance the value of the data. Given that the data source may change over time, it is necessary to re-execute the field scanning detection regularly.

[0188] It should be noted that if during the verification process it is determined that the weight percentage needs to be adjusted based on the comprehensive scoring results, the specific circumstances of the data and fields must also be fully considered.

[0189] Reference Fig.12 , generate a field full association table, and analyze the field merging strategy. For each data source, sort out and list all related fields, and create an association table, where rows represent related data and columns represent the data sources to which they belong. According to the comprehensive evaluation results calculated in the previous steps, the data sources are arranged in descending order from left to right to generate a field full association table.

[0190] In-depth analysis of the field full association table, comparison of field values, and analysis of similarities between fields to identify which fields appear repeatedly in multiple data sources and which fields are specific to a single data source. For fields shared by multiple data sources, select the data source with the highest comprehensive score as the primary data source, and the remaining data sources as reference data sources. If there are differences between field values, determine the merge strategy based on the conflict resolution rules. For unique fields belonging to a single data source, after confirming that the unique fields provide valuable information, consider creating new model fields to incorporate this information.

[0191] Exemplarily, in this embodiment, the formulated merging strategy includes:

[0192] (1) The data from the primary data source exists and its content is the same as that from other data sources: directly use the data from the primary data source.

[0193] (2) The data from the primary data source exists, but the content is different from that of the data from other data sources: Determine whether the name is standardized. If it is standardized, directly use the primary source. If it is not standardized, perform data cleaning. If the cleaning is qualified, use the primary source data. If cleaning is not possible, use the standardized and second-ranked data.

[0194] (3) The data in the primary data source does not exist, but the data in other data sources exist and have the same content: check with the business unit whether the data has been deleted or disabled. If so, synchronize the data and mark it. If the data is newly added, directly use the data to supplement.

[0195] (4) If the data from the primary data source does not exist, but the data from other data sources exist and the content is different, carefully check the data and use it as the supplementary data after verification.

[0196] (5) If the data is sensitive, first merge the data and then perform data encryption.

[0197] (6) If the data is a name and the standard model is a code: the name should be converted into a code value first, and then the data should be merged.

[0198] Reference Figure 6 According to the field full association table - toll station name, the provincial toll interface is located in the first column, indicating that this data source is the main data source.

[0199] For the associated field G0006630030010, the provincial charging interface, the department platform charging, and the centralized monitoring all have this field, but the content is different from the data in other data sources. According to the merge strategy (2), it is necessary to determine whether the name is standardized. Since the name "West Main Line Station (canceled)" is not standardized, data cleaning is required. After the cleaning is qualified, the standard name is used.

[0200] For the associated field G0006630010020, the data of the main data source (provincial toll collection interface) does not exist, but the data of other data sources (departmental platform toll collection) does exist. After verifying with the business unit that the "Machang Ramp Station" has been cancelled, the data is deleted and marked according to the merge strategy (3).

[0201] For the associated field G0006630010030, the data of the main data source (provincial charging interface) exists. Although the content is different from that of other data sources, it is determined that the name "Ping An Station" meets the specification. Therefore, according to the merge strategy (2), the Ping An Station merge can be directly adopted.

[0202] For the associated field G0006630010040, the data of the primary data source (provincial toll collection interface) does not exist, but the data of other data sources (entrance overload control) does exist. After verifying with the business unit that "Huxi Ramp Station" is a newly added station, according to the merge strategy (3), this data is used for supplementation.

[0203] Finally, implement master data consolidation and verification. According to the weighted scoring results and consolidation strategy, master data is integrated. In the integration process, the data needs to be weighted and scored first to ensure that the importance and accuracy of the data are fully considered. Then, according to the scoring results, the appropriate consolidation strategy is adopted to effectively integrate the master data.

[0204] It should be noted that after implementing the master data merger, a monitoring mechanism can also be established to track the implementation of the plan and identify and solve problems in a timely manner. The monitoring mechanism can monitor various problems in the integration process in real time, such as data quality problems and system performance problems, and take timely measures to solve them. This can ensure the smooth progress of the integration process and avoid integration failures due to the accumulation of problems.

[0205] In this embodiment, by constructing an index evaluation system, weighted scoring is implemented for master data fields, and data integration is performed based on the scoring results, and then a data merging strategy is formulated, the purpose is to improve data quality and integration efficiency, and through scientific weight calculation and comprehensive scoring, the master data integration process is made more scientific and reasonable, and at the same time, a methodological basis for continuously optimizing master data quality is provided. By implementing this method, the efficiency of master data management can be significantly improved and the operational risks caused by data quality issues can be reduced. This technology is suitable for master data integration work in multiple fields such as government affairs, transportation, medical care, and education, and has broad application prospects.

[0206] Please refer to Fig.13 The embodiment of the present application further provides a master data integration system based on index weighting, which can implement the above-mentioned master data integration method based on index weighting, and the system includes:

[0207] The first module is used to determine the evaluation system of the master data, wherein the evaluation system includes evaluation indicators and corresponding weight values;

[0208] The second module is used to obtain a multi-source data set, which includes master data from multiple data sources;

[0209] The third module is used to build a standard model and perform data relationship mapping based on multi-source data sets to obtain data mapping relationships;

[0210] The fourth module is used to perform field scanning and detection on multi-source data sets based on the standard model and data mapping relationship to obtain scanning results, and to perform weighted calculation on the scanning results according to the evaluation system to obtain a comprehensive scoring result;

[0211] The fifth module is used to derive a field full association table with reference to the comprehensive scoring results, and to perform a strategy analysis on the field full association table to determine a merging strategy;

[0212] The sixth module is used to integrate multi-source data sets according to the merging strategy.

[0213] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0214] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned master data integration method based on indicator weighting when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0215] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0216] Please refer to Fig.14 , Fig.14 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0217] The processor 901 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0218] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the master data integration method based on indicator weighting in the embodiment of this application.

[0219] The input / output interface 903 is used to implement information input and output.

[0220] The communication interface 904 is used to realize the communication interaction between this device and other devices. Communication can be realized through wired methods (such as USB, network cable, etc.) or wireless methods (such as mobile network, WI FI, Bluetooth, etc.).

[0221] The bus 905 transmits information between various components of the device (eg, the processor 901 , the memory 902 , the input / output interface 903 , and the communication interface 904 ).

[0222] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0223] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the master data integration method based on indicator weighting is implemented.

[0224] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0225] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0226] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0227] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0228] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0229] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0230] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0231] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A master data integration method based on index weighting, characterized in that: The method comprises the following steps: Determine an evaluation system for master data, wherein the evaluation system includes evaluation indicators and corresponding weight values; Acquire a multi-source data set, the multi-source data set including master data from multiple data sources; Constructing a standard model, and performing data relationship mapping according to the multi-source data sets to obtain a data mapping relationship; Performing field scanning detection on the multi-source data set based on the standard model and the data mapping relationship to obtain scanning results, and performing weighted calculation on the scanning results according to the evaluation system to obtain a comprehensive scoring result; Deriving a field full association table with reference to the comprehensive scoring result, and performing a strategy analysis on the field full association table to determine a merging strategy; The multi-source data sets are integrated according to the merging strategy.

2. The method according to claim 1, characterized in that The evaluation system for determining the master data comprises the following steps: Matching degree, repetition rate, null value rate, similarity and sensitivity are determined as evaluation indicators; Determine corresponding weight values ​​for the evaluation indicators through an expert scoring mechanism; A weighted calculation formula is determined according to the evaluation index and the weight value to obtain an evaluation system for the master data.

3. The method according to claim 1, characterized in that The construction of the standard model and performing data relationship mapping according to the multi-source data sets to obtain a data mapping relationship includes the following steps: Performing data analysis on the multi-source data set to determine entity elements of the master data; Based on the entity elements, a standard model with a unified format and structure is established according to a preset entity standard; The fields in the master data are matched with the corresponding fields in the standard model to determine the data mapping relationship.

4. The method according to claim 1, characterized in that The performing field scanning detection on the multi-source data set based on the standard model and the data mapping relationship to obtain a scanning result, and performing weighted calculation on the scanning result according to the evaluation system to obtain a comprehensive scoring result, includes the following steps: Based on the standard model and the data mapping relationship, the master data of the same data source is scanned and detected, and the data type, length, format, value range and content of the fields in the master data are checked to obtain a scanning result; A weighted calculation is performed on each field in the scan result according to the evaluation system to obtain a comprehensive scoring result.

5. The method according to claim 4, characterized in that The method of deriving a field full association table with reference to the comprehensive scoring result and performing a strategy analysis on the field full association table to determine a merging strategy includes the following steps: Performing association analysis on the master data between multiple data sources to extract associated fields, and sorting the data sources to which the associated fields belong in descending order according to the comprehensive scoring result to obtain a field full association table, wherein the rows of the field full association table represent associated data, and the columns of the field full association table represent the data sources to which the associated data belong; The data source with the highest comprehensive score in the field full association table is determined as a primary data source, and the remaining data sources in the field full association table except the primary data source are determined as reference data sources; A strategy analysis is performed on the field full association table according to the primary data source and the reference data source to determine a merging strategy.

6. The method according to claim 5, characterized in that The step of performing a strategy analysis on the field full association table according to the primary data source and the reference data source to determine a merging strategy comprises the following steps: Performing a strategic analysis on the field full association table according to the primary data source to determine whether field data exists in the primary data source; When the primary data source has field data, determining whether there is a data conflict between the field data of the primary data source and the field data of the reference data source; When there is no data conflict between the field data of the primary data source and the field data of the reference data source, determining the merging strategy to adopt the field data of the primary data source; When there is a data conflict between the field data of the primary data source and the field data of the reference data source, determining a merging strategy to perform data cleaning based on a preset first conflict rule, and adopting the field data of the data source that meets the specification after data cleaning; Alternatively, when the primary data source does not have field data, determining whether there is a data conflict in the field data between the reference data sources; When there is no data conflict in the field data between the reference data sources, the merging strategy is determined to check the usage of the field data of the reference data source; when the usage is newly added, the field data of the reference data source is used to supplement the data of the main data source; When there is a data conflict in the field data between the reference data sources, the merging strategy is determined to perform data verification based on a preset second conflict rule, and the field data that has been verified to be correct is used to supplement the data of the primary data source.

7. The method according to claim 1, characterized in that The master data integration method based on index weighting also includes the following steps: Obtaining verification results by performing quality checks, data comparisons, business verifications, and compliance checks on the integrated master data; The evaluation system and the merging strategy are adjusted according to the verification result.

8. A master data integration system based on index weighting, characterized in that: The system comprises: The first module is used to determine the evaluation system of the master data, wherein the evaluation system includes evaluation indicators and corresponding weight values; A second module is used to obtain a multi-source data set, wherein the multi-source data set includes master data from multiple data sources; The third module is used to build a standard model and perform data relationship mapping according to the multi-source data set to obtain a data mapping relationship; The fourth module is used to perform field scanning detection on the multi-source data set based on the standard model and the data mapping relationship to obtain scanning results, and perform weighted calculation on the scanning results according to the evaluation system to obtain a comprehensive scoring result; The fifth module is used to derive a field full association table with reference to the comprehensive scoring result, and to perform a strategy analysis on the field full association table to determine a merging strategy; The sixth module is used to integrate the multi-source data sets according to the merging strategy.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images