Master data integration methods, apparatus, electronic devices and storage media

Abstract

This application discloses a master data integration method, apparatus, electronic device, and storage medium. The method includes: acquiring multi-source heterogeneous master data; standardizing and cleaning the multi-source heterogeneous master data to generate multiple block keys and inverted index lists, and constructing a candidate comparison set; extracting and normalizing the multi-level similarity of each pair of candidate records in the candidate comparison set, and constructing a standardized feature vector; inputting the standardized feature vector into a learning model to obtain the matching probability of each pair of candidate records, and making hierarchical decisions based on the matching probability and at least one matching threshold; constructing an undirected matching graph and generating entity clusters from the undirected matching graph; determining the data quality score of each field of each candidate record within the entity cluster, determining the optimal value of each field based on the data quality score, and generating the integrated master data. This application significantly reduces the number of comparisons, improves comparison efficiency, eliminates the need for manually writing matching rules, and can adapt to various data types and new data sources.

Description

Technical Field

[0001] This application relates to the field of data integration technology, and in particular to a master data integration method, apparatus, electronic device and storage medium. Background Technology

[0002] Master data is the core foundational data for enterprise digital transformation, used to describe core business entities such as customers, suppliers, materials, and organizational structures. It has high value, high sharing, and high consistency requirements. As enterprise information systems continue to be built and expanded, master data is generally scattered across multiple heterogeneous systems such as Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), Supplier Relationship Management (SRM), and Quality Management System (QMS). This results in problems such as inconsistent data standards, inconsistent field naming, non-standard formats, duplicate entries, and asynchronous updates, leading to a large number of data silos and redundant records, severely impacting business collaboration, data analysis, and decision-making accuracy.

[0003] Entity parsing and master data integration are key technologies for solving the above problems. Their core objective is to identify duplicate records belonging to the same real-world entity in multi-source data, complete deduplication, merging and integration, and finally generate unified, standardized, and high-quality master data.

[0004] However, current mainstream master data integration solutions generally suffer from the following technical shortcomings: Existing methods mostly use rule engines to achieve record matching, requiring business and technical personnel to formulate matching rules one by one based on field types, data characteristics, and business scenarios, such as name similarity rules, address rules, and ID number rules. The number of rules grows exponentially with the increase of data sources. When data contains abbreviations, aliases, acronyms, misspellings, or format variations, rules are prone to missed or misjudgments. When adding new data sources or changing data formats, rules need to be redeveloped, debugged, and maintained, which is time-consuming, costly, and difficult to scale. Traditional entity parsing technology solutions usually use a full Cartesian product pairwise comparison method, with a time complexity as high as O(n²). In enterprise master data scenarios with millions or even tens of millions of records, the number of comparisons grows explosively, resulting in extremely long computation time and huge resource consumption, which cannot meet the needs of real-time or near real-time master data integration. Summary of the Invention

[0005] This application provides a master data integration method, apparatus, electronic device, and storage medium, which effectively reduces the amount of master data entity parsing and comparison, eliminates reliance on manual rules, improves matching accuracy and stability, and can automatically generate high-quality master data.

[0006] In a first aspect, this application provides a master data integration method, the method comprising: acquiring multi-source heterogeneous master data; standardizing and cleaning the multi-source heterogeneous master data to generate multiple block keys and inverted index lists, and constructing a candidate alignment set containing the intersection of the inverted index lists corresponding to the multiple block keys; extracting and normalizing the multi-level similarity of each pair of candidate records in the candidate alignment set, and constructing a standardized feature vector; inputting the standardized feature vector into a learning model to obtain the matching probability of each pair of candidate records, and making a hierarchical decision based on the matching probability and at least one matching threshold; constructing an undirected matching graph with the candidate records as vertices and each pair of candidate records whose matching probability reaches the matching threshold corresponding to the merging decision as edges, and generating entity clusters from the undirected matching graph using a disjoint-set data structure algorithm; determining the data quality score of each field of each candidate record within the entity cluster, determining the optimal value of each field based on the data quality score, and generating integrated master data.

[0007] In conjunction with the first aspect, in one possible implementation, the extraction and normalization of the multi-level similarity of each pair of candidate records in the candidate comparison set includes: calculating the edit distance, Jaro-Winkler distance, and cosine similarity of the text fields to determine the string similarity of each pair of candidate records; using a word vector model to semantically encode the name-class fields to obtain semantic embedding vectors and calculating the cosine similarity of the semantic embedding vectors; calculating the time difference of the time fields of each pair of candidate records and converting the time difference into a normalized time similarity using an exponential decay function; calculating the geographical distance between the geographical entities corresponding to the address fields of each pair of candidate records and normalizing the geographical distance to obtain geographical similarity; predefining credibility weights for each data source of each pair of candidate records, encoding the data verification status, and weightedly fusing the credibility weights with the encoded data verification status to obtain source feature similarity.

[0008] In conjunction with the first aspect, in one possible implementation, the learning model comprises a three-layer structure; the first layer is a random forest for handling non-linear text feature interactions; the second layer is a gradient boosting tree for optimizing boundary samples, which include qualification certificate conflicts and addresses that are similar but have large name differences; and the third layer is logistic regression for calibrating probabilities.

[0009] In conjunction with the first aspect, in one possible implementation, the at least one matching threshold includes a first matching threshold and a second matching threshold, wherein the first matching threshold is greater than the second matching threshold; the hierarchical decision-making based on the matching probability and the at least one matching threshold includes: merging candidate records whose matching probability reaches the first matching threshold, manually reviewing candidate records whose matching probability is less than the first matching threshold but reaches the second matching threshold, and not merging candidate records whose matching probability is less than the second matching threshold.

[0010] In conjunction with the first aspect, in one possible implementation, before determining the data quality score of each field of each candidate record within the entity cluster, the method further includes: calculating the cohesion of the entity cluster; and submitting the entity clusters with cohesion less than a preset cohesion threshold to manual review.

[0011] In conjunction with the first aspect, in one possible implementation, the method further includes: obtaining the results of manual review and inputting the results of manual review into the training set of the learning model; updating the increment of the learning model based on the training set, and identifying repetitive patterns in the manual review and converting them into automatic rules that can be embedded in the learning model.

[0012] In conjunction with the first aspect, in one possible implementation, determining the data quality score of each field of each candidate record within the entity cluster includes: performing a weighted summation of each field of each candidate record within the entity cluster according to completeness, timeliness, source credibility, and verification status to obtain the data quality score.

[0013] Secondly, this application provides a master data integration device, which includes a data access module, a segmentation module, a feature engineering module, a matching decision module, a clustering module, and a master data generation module. The data access module is used to acquire multi-source heterogeneous master data; the segmentation module is used to standardize and clean the multi-source heterogeneous master data, generate multiple segmentation keys and inverted index lists, and construct a candidate alignment set containing the intersection of the inverted index lists corresponding to the multiple segmentation keys; the feature engineering module is used to extract and normalize the multi-level similarity of each pair of candidate records in the candidate alignment set, and construct a standardized feature vector; the matching module... The decision module is used to input the standardized feature vector into the learning model to obtain the matching probability of each pair of candidate records, and to make hierarchical decisions based on the matching probability and at least one matching threshold; the clustering module is used to construct an undirected matching graph with each pair of candidate records as vertices and the matching probability reaching the matching threshold corresponding to the merging decision as edges, and to generate entity clusters from the undirected matching graph using a disjoint-set data structure algorithm; the master data generation module is used to determine the data quality score of each field of each candidate record in the entity cluster, determine the optimal value of each field based on the data quality score, and generate integrated master data.

[0014] In conjunction with the second aspect, in one possible implementation, when the feature engineering module extracts and normalizes the multi-level similarity of each pair of candidate records in the candidate comparison set: it calculates the edit distance, Jaro-Winkler distance, and cosine similarity of the text fields to determine the string similarity of each pair of candidate records; it uses a word vector model to semantically encode the name-class fields to obtain semantic embedding vectors and calculates the cosine similarity of the semantic embedding vectors; it calculates the time difference of the time fields of each pair of candidate records and converts the time difference into a normalized time similarity using an exponential decay function; it calculates the geographical distance between the geographical entities corresponding to the address fields of each pair of candidate records and normalizes the geographical distance to obtain geographical similarity; it predefines credibility weights for each data source of each pair of candidate records, encodes the data verification status, and performs weighted fusion of the credibility weights and the encoded data verification status to obtain source feature similarity.

[0015] In conjunction with the second aspect, in one possible implementation, the learning model includes a three-layer structure; the first layer is a random forest, used to handle non-linear text feature interactions; the second layer is a gradient boosting tree, used to optimize boundary samples, which include qualification certificate conflicts and addresses that are similar but have large name differences; the third layer is logistic regression, used to calibrate probabilities.

[0016] In conjunction with the second aspect, in one possible implementation, the at least one matching threshold includes a first matching threshold and a second matching threshold, wherein the first matching threshold is greater than the second matching threshold; when the matching decision module makes a hierarchical decision based on the matching probability and the at least one matching threshold: the candidate records whose matching probability reaches the first matching threshold are merged; the candidate records whose matching probability is less than the first matching threshold but reaches the second matching threshold are manually reviewed; and the candidate records whose matching probability is less than the second matching threshold are not merged.

[0017] In conjunction with the second aspect, in one possible implementation, the master data integration device further includes a cohesion module, which calculates the cohesion of the entity cluster before determining the data quality score of each field of each candidate record within the entity cluster; entity clusters with a cohesion less than a preset cohesion threshold are submitted for manual review.

[0018] In conjunction with the second aspect, in one possible implementation, the master data integration device further includes an update module, which is used to obtain the manual review results and input the manual review results into the training set of the learning model; update the increment of the learning model based on the training set; and identify repetitive patterns in the manual review and convert them into automatic rules that can be embedded in the learning model.

[0019] In conjunction with the second aspect, in one possible implementation, when the master data generation module determines the data quality score of each field of each candidate record within the entity cluster, it performs a weighted summation of each field of each candidate record within the entity cluster according to completeness, timeliness, source credibility, and verification status to obtain the data quality score.

[0020] Thirdly, this application provides an electronic device including one or more processors and a memory storing computer-executable instructions; the computer-executable instructions, when executed by the one or more processors, cause the one or more processors to perform the master data integration method as described in the first aspect or any possible implementation of the first aspect.

[0021] Fourthly, this application provides a computer-readable storage medium storing computer-readable instructions that, when executed by a computer, implement the master data integration method as described in the first aspect or any possible implementation thereof.

[0022] The master data integration method provided in this application standardizes and cleans multi-source heterogeneous master data, generating multiple block keys and inverted index lists. It then constructs a candidate comparison set containing the intersection of the inverted index lists corresponding to multiple block keys, significantly reducing the number of comparisons, improving comparison efficiency, and meeting implementation requirements. This method extracts and normalizes the multi-level similarity of each pair of candidate records in the candidate comparison set, then constructs standardized feature vectors. A learning model is used to determine the matching probability of each pair of candidate records based on the standardized feature vectors, eliminating the need for manually written matching rules and adapting to various data types and new data sources. Furthermore, this method constructs an undirected matching graph and generates entity clusters using a disjoint-set data structure algorithm. Within each entity cluster, the optimal data quality score for each field of each candidate record is used to generate the integrated master data, making the integrated master data more accurate and reliable. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram illustrating a master data integration method provided for some embodiments of this application; Figure 2 A schematic diagram of the extraction and normalization process provided for some embodiments of this application; Figure 3 A flowchart illustrating the process of calculating the cohesion of entity clusters and determining entity clusters requiring manual review, provided for some embodiments of this application; Figure 4 A flowchart illustrating the process of updating the learning model based on the results of manual review, provided for some embodiments of this application; Figure 5 A schematic diagram of the structure of a master data integration device provided for some embodiments of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Unless otherwise defined, all terms used in this application (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, unless expressly defined herein.

[0027] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is to say, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all information and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0029] The parameter values ​​or models mentioned in the embodiments of this application are merely examples. The technical solutions of this application are not limited to these parameters or models and can be replaced with equivalent parameters or models.

[0030] This application provides a master data integration method, such as Figure 1 As shown, the method includes steps S101 to S106.

[0031] S101. Obtain multi-source heterogeneous master data.

[0032] Multi-source heterogeneous master data refers to master data from different sources, which may have different data structures, storage formats, and access interfaces. For example, multi-source heterogeneous master data includes 450,000 supplier records from an ERP system, 380,000 supplier records from an SRM system, and 290,000 supplier records from a QMS system, totaling 1.12 million supplier records. Each supplier record includes at least one of the following: supplier name, unified social credit code, registered address, production address, contact person, contact number, bank, qualification certificate number, product category, cooperation start time, quality rating, and data source identifier.

[0033] S102. Standardize and clean the multi-source heterogeneous master data to generate multiple block keys and inverted index lists, and construct a candidate alignment set containing the intersection of the inverted index lists corresponding to multiple block keys.

[0034] Standardized cleaning refers to tasks such as removing spaces, removing erroneous records, and standardizing font formats. For example, when performing standardized cleaning on supplier records, the following steps are taken: Unified Social Credit Code: Remove spaces and check digits, and standardize to an 18-digit format; Supplier Name: Remove common suffixes such as "Limited Company" and "Joint-Stock Company" and use only full-width and half-width characters; Address: Parse province, city, district, and county information and convert it into standard administrative division codes; Contact number: Remove area code and extension number identifiers, and uniformly use 11-digit mobile phone number or 10-digit landline number; Remove null and outlier values: Filter records with empty or incorrectly formatted unified social credit codes.

[0035] Different partitioning keys are used depending on the different sources of heterogeneous master data. For example, when the heterogeneous master data is supplier data, multiple partitioning keys can include precise partitioning keys, fuzzy partitioning keys, geographic partitioning keys, and time partitioning keys. Among them, precise partitioning keys can include unified social credit code keys, telephone keys, and qualification keys. Module partitioning keys can be fuzzy keys for supplier names. The specific generation method is shown in Table 1.

[0036] Table 1

[0037] Inverted indexes originate from the practical application of finding records based on attribute values. Each item in the inverted index list includes an attribute value and the address of the record that has that attribute value. The attribute value is not determined by the record itself, but rather the location of the record is determined by the attribute value.

[0038] The candidate comparison set is generated by taking the intersection of the inverted index list. For the 1.12 million supplier records in the example above, the candidate comparison set has 890,000 comparison pairs. Compared with the 627.2 billion comparison pairs corresponding to the pairwise comparison method, the number of comparisons is reduced by 99.9986%, which greatly improves the comparison efficiency.

[0039] S103. Extract and normalize the multi-level similarity of each pair of candidate records in the candidate comparison set, and construct a standardized feature vector.

[0040] Step S103 compares each pair of candidate records from multiple perspectives to obtain multiple levels of similarity, providing a basis for subsequent determination of whether each pair of candidate records is the same record and whether they can be merged. For example, in the case of multi-source heterogeneous master data being supplier data, similarity can be extracted and normalized based on aspects such as supplier name, cooperation start time, geographical location, and information on supplied products or services to form multi-level similarity.

[0041] Standardized feature vectors are constructed based on multi-level similarity to facilitate processing by subsequent learning models.

[0042] S104. Input the standardized feature vector into the learning model to obtain the matching probability of each pair of candidate records, and make hierarchical decisions based on the matching probability and at least one matching threshold.

[0043] The learning model can output the matching probability of each pair of candidate records based on the standardized feature vector of each pair of input candidate records. The matching probability represents the probability that the pair of candidate records are the same record. It does not rely on manually written matching rules and can adapt to various data and new data sources. When making hierarchical decisions based on the matching probability and at least one matching threshold, the matching probability corresponding to each pair of candidate records is compared with the matching threshold, and then hierarchical decisions such as merging or not merging are made.

[0044] For example, with only one matching threshold (e.g., 0.85), the matching probability of each pair of candidate records is compared with the matching threshold. When the matching probability is greater than the matching threshold, the corresponding pair of candidate records is merged. When the matching probability is less than or equal to the matching threshold, the non-corresponding pair of candidate records is merged.

[0045] S105. Construct an undirected matching graph with candidate records as vertices and each pair of candidate records whose matching probability reaches the matching threshold corresponding to the merging decision as edges, and generate entity clusters from the undirected matching graph using the disjoint-set data structure algorithm.

[0046] Step S104 determines the matching probability between each pair of candidate records. However, there is a lack of correlation between multiple pairs of candidate records. Step S105 establishes a relationship network between multiple pairs of candidate records, thereby ensuring that no matching relationships are missed. For example, step S104 determines the matching probability between supplier A and supplier B, and the matching probability between supplier B and supplier C also reaches the matching threshold for the merger decision. However, the matching probability between supplier A and supplier C does not reach the matching threshold for the merger decision. Step S105 establishes a connection between supplier A, supplier B, and supplier C through an undirected matching graph, forming an entity cluster.

[0047] S106. Determine the data quality score of each field of each candidate record within the entity cluster, determine the optimal value of each field based on the data quality score, and generate the integrated master data.

[0048] Step S106 identifies the highest quality fields in each master data entry within the entity cluster, thereby making the integrated master data more accurate and reliable.

[0049] In some embodiments of this application, step S103, which involves extracting and normalizing the multi-level similarity of each pair of candidate records in the candidate comparison set, includes steps S201 to S205, such as... Figure 2 As shown.

[0050] S201. Calculate the edit distance, Jaro-Winkler distance, and cosine similarity of the text fields to determine the string similarity of each pair of candidate records.

[0051] Edit distance is a quantitative metric that measures the degree of difference between two strings. It is quantified by calculating the minimum number of operations required to transform one string into another. Jaro-Winkler distance is an improvement on Jaro distance by giving extra weight to common prefixes, resulting in higher similarity scores for strings with identical prefixes. Haversine distance is the shortest spherical distance between two points on the Earth's surface (assuming an ideal sphere), calculated using their latitude and longitude coordinates. This distance is the arc length along the Earth's surface, not the Euclidean distance through the Earth's center.

[0052] For example, Jaro-Winkler distance is used to calculate the similarity of supplier name strings, such as handling abbreviation variations like "Huabei Precision Machinery" and "Huabei Jinggong Machinery"; edit distance is used to calculate the similarity of registered address strings, such as handling differences in administrative division expressions like "Beijing Changping District" and "Beijing Changping".

[0053] S202. Use a word vector model to semantically encode the name class field, obtain semantic embedding vectors, and calculate the cosine similarity of the semantic embedding vectors.

[0054] Word vector models are used to map words to a real vector space, capturing the semantic information of words so that words with similar meanings are closer together in the vector space.

[0055] For example, step S202 is used to perform semantic similarity analysis on supplier names, such as identifying synonyms like "gear reducer" and "speed reducer", "hydraulic pump" and "oil pump".

[0056] S203. Calculate the time difference of the time field of each pair of candidate records and convert the time difference into a normalized time similarity using an exponential decay function.

[0057] The greater the time difference in the time field of each pair of candidate records, the lower the similarity obtained in step S203; the smaller the time difference in the time field of each pair of candidate records, the higher the similarity obtained in step S203.

[0058] For example, by calculating the time difference between the start times of cooperation for each pair of candidate suppliers and converting the time difference into a normalized time similarity using an exponential decay function, it is possible to identify the historical cooperation records of the same supplier in different systems.

[0059] S204. Calculate the geographical distance between the geographical entities corresponding to the address fields of each pair of candidate records, and normalize the geographical distance to obtain the geographical similarity. For example, Haversine distance can be used to calculate the geographical similarity of production addresses, such as to assess the geographical concentration of supplier production facilities and support logistics cost analysis.

[0060] S205. Predefine credibility weights for each data source of each pair of candidate records, encode the data verification status, and perform weighted fusion of the credibility weights and the encoded data verification status to obtain the source feature similarity.

[0061] For example, the predefined credibility weights for each data source are as follows: the credibility weight for the ERP system is 0.9, the credibility weight for the SRM system is 0.85, and the credibility weight for the QMS system is 0.8.

[0062] In some embodiments of this application, the learning model comprises a three-layer structure. The first layer is a random forest, used to handle non-linear text feature interactions. The second layer is a gradient boosting tree, used to optimize boundary samples, including qualification certificate conflicts and addresses that are similar but have significantly different names. The third layer is logistic regression, used to calibrate probabilities.

[0063] For example, when the multi-source heterogeneous master data consists of supplier records, the random forest package uses 200 decision trees to handle non-linear text feature interactions such as supplier name and address; the gradient boosting tree iterates for 500 rounds to optimize boundary samples such as qualification certificate conflicts and similar addresses but large name differences; the regularization parameter of the logistic regression is set to 0.05, and the matching probability is output.

[0064] In some embodiments of this application, at least one matching threshold includes a first matching threshold and a second matching threshold, wherein the first matching threshold is greater than the second matching threshold.

[0065] The hierarchical decision-making based on matching probability, first matching threshold and second matching threshold in step S104 includes: merging candidate records whose matching probability reaches the first matching threshold, manually reviewing candidate records whose matching probability is less than the first matching threshold but reaches the second matching threshold, and not merging candidate records whose matching probability is less than the second matching threshold.

[0066] For example, if the first matching threshold is 0.85, a pair of candidate records with a matching probability ≥ 0.85 will be merged; if the second matching threshold is 0.65, a pair of candidate records with a matching probability ≥ 0.65 and a matching probability > 0.85 will be manually reviewed; and a pair of candidate records with a matching probability < 0.65 will not be merged.

[0067] In some embodiments of this application, before step S106, the master data integration method further includes steps S301 and S302, such as... Figure 3 As shown.

[0068] S301. Calculate the cohesion of an entity cluster. Cohesion represents the degree of closeness among multiple candidate records in an entity cluster; the higher the cohesion, the closer the multiple candidate records in the entity cluster are, and the greater the probability that the multiple candidate records in the entity cluster represent the same entity; similarly, the lower the cohesion, the more distant the multiple candidate records in the entity cluster are, and the less likely that the multiple candidate records in the entity cluster represent the same entity.

[0069] S302. Submit entity clusters with cohesion less than the preset cohesion threshold to manual review.

[0070] For example, with a cohesion threshold of 0.85, multiple candidate records within an entity cluster with a cohesion ≥ 0.85 are treated as the same entity and merged; multiple candidate records within an entity cluster with a cohesion < 0.85 are temporarily not treated as the same entity, and are manually reviewed to determine whether the entity cluster is the same entity; when the multi-source heterogeneous master data is supplier records, entity clusters with a cohesion less than the preset cohesion threshold mainly include the following types: group enterprises with the head office and branch offices registered separately, suppliers that have changed their names but whose unified social credit codes have not changed, and production bases and sales companies that are registered separately.

[0071] In some embodiments of this application, step S106, which determines the data quality score of each field of each candidate record within the entity cluster, includes: weighting and summing each field of each candidate record within the entity cluster according to completeness, timeliness, source credibility, and verification status to obtain the data quality score.

[0072] For example, the weight of integrity is 0.25, the weight of timeliness is 0.25, the weight of source credibility is 0.35, and the weight of verification status is 0.15.

[0073] In addition, when the multi-source heterogeneous master data is supplier data, the following scenarios allow for multi-value retention: if the supplier has multiple production bases (such as Beijing, Shanghai, and Guangzhou) and the geographical coordinate difference is ≤0.05, retain the primary address and two backup addresses; if the supplier simultaneously possesses multiple certificates such as ISO9001, ISO14001, and IATF16949, all of them should be retained in the extended field.

[0074] After step S106, as Figure 4 As shown, the master data integration method also includes steps S401 and S402.

[0075] S401. Obtain the results of manual review and input them into the training set of the learning model.

[0076] S402. Update the incremental learning model based on the training set, identify repetitive patterns in manual review, and transform them into automatic rules that can be embedded in the learning model.

[0077] For example, when the multi-source heterogeneous master data is supplier data, typical audit scenarios and rule transformations are shown in Table 2.

[0078] Table 2

[0079] To verify the technical effectiveness of this application, the following test uses 1.12 million pieces of supplier master data as a unified test set. The master data integration method provided in this application is compared with the current mainstream traditional rule-driven master data management method in the industry in multiple dimensions. The test indicators cover core dimensions such as matching accuracy, data processing time, and manual sorting workload. The industry-standard workload for manual sorting of similar data is 300 person-days. The specific comparison results are shown in Table 3.

[0080] Table 3

[0081] As shown in Table 3, the comparison results indicate that the master data integration method provided in this application improves the matching accuracy by 24.5% and the full data processing efficiency by 41.6 times compared to the traditional rule-driven master data management method. It also reduces the workload of manual sorting by 95%, significantly reduces operation and maintenance costs, and shortens the adaptation cycle of new data sources. It effectively solves the core pain points of traditional master data management methods, such as high rule maintenance costs, poor real-time performance, and inefficient manual review. It can quickly adapt to multiple heterogeneous systems in the manufacturing industry (such as ERP, SRM, QMS), efficiently generate high-quality master data, significantly reduce the cost of supply chain data governance, and provide strong support for enterprise intelligent manufacturing and supply chain digital transformation.

[0082] like Figure 5As shown, this application also provides a master data integration device 500, which includes a data access module 501, a segmentation module 502, a feature engineering module 503, a matching decision module 504, a clustering module 505, and a master data generation module 506. The data access module 501 is used to acquire multi-source heterogeneous master data. The segmentation module 502 is used to standardize and clean the multi-source heterogeneous master data, generate multiple segmentation keys and inverted index lists, and construct a candidate alignment set containing the intersection of the inverted index lists corresponding to multiple segmentation keys. The feature engineering module 503 is used to extract and normalize the multi-level similarity of each pair of candidate records in the candidate alignment set, and construct a standardized feature vector. The matching decision module 504 is used to input the standardized feature vector into a learning model to obtain the matching probability of each pair of candidate records, and make a hierarchical decision based on the matching probability and at least one matching threshold. The clustering module constructs an undirected matching graph with candidate records as vertices and each pair of candidate records whose matching probability reaches the matching threshold corresponding to the merging decision as edges. It then generates entity clusters from the undirected matching graph using a disjoint-set data structure algorithm. The master data generation module 506 determines the data quality score for each field of each candidate record within the entity cluster, determines the optimal value for each field based on the data quality score, and generates the integrated master data.

[0083] When feature engineering module 503 extracts and normalizes the multi-level similarity of each pair of candidate records in the candidate comparison set: it calculates the edit distance, Jaro-Winkler distance, and cosine similarity of the text fields to determine the string similarity of each pair of candidate records. It uses a word vector model to semantically encode name fields, obtaining semantic embedding vectors and calculating the cosine similarity of these vectors. It calculates the time difference of the time field for each pair of candidate records and converts it into a normalized time similarity using an exponential decay function. It calculates the geographic distance between the geographic entities corresponding to the address fields of each pair of candidate records, normalizes the geographic distance, and obtains the geographic similarity. It predefines credibility weights for each data source for each pair of candidate records, encodes the data verification status, and weights and fuses the credibility weights with the encoded data verification status to obtain the source feature similarity.

[0084] The learning model consists of a three-layer structure. The first layer is a random forest, used to handle non-linear text feature interactions. The second layer is a gradient boosting tree, used to optimize boundary samples, including conflicting qualification certificates and addresses that are similar but have significantly different names. The third layer is logistic regression, used to calibrate probabilities.

[0085] At least one matching threshold includes a first matching threshold and a second matching threshold, wherein the first matching threshold is greater than the second matching threshold. When the matching decision module 504 makes a hierarchical decision based on the matching probability and at least one matching threshold: candidate records with a matching probability reaching the first matching threshold are merged; candidate records with a matching probability less than the first matching threshold but reaching the second matching threshold are manually reviewed; and candidate records with a matching probability less than the second matching threshold are not merged.

[0086] The master data integration device 500 also includes a cohesion module, which calculates the cohesion of the entity cluster before determining the data quality score for each field of each candidate record within the entity cluster. Entity clusters with a cohesion less than a preset cohesion threshold are submitted for manual review.

[0087] The master data integration device 500 also includes an update module, which obtains the results of manual review and inputs them into the training set of the learning model. The update module updates the incremental changes of the learning model based on the training set, identifies recurring patterns in the manual reviews, and transforms them into automatic rules that can be embedded into the learning model.

[0088] When determining the data quality score of each field of each candidate record within an entity cluster, the master data generation module 506 performs a weighted summation of each field of each candidate record within the entity cluster based on completeness, timeliness, source credibility, and verification status to obtain the data quality score.

[0089] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0090] This application also provides an electronic device including one or more processors and a memory storing computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to perform the master data integration method described above.

[0091] In this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0092] The memory may include read-only memory and random access memory (RAM), and provides instructions and data to the processor. The memory may also include non-volatile random access memory. Optionally, the random access memory may be, for example, high-bandwidth memory.

[0093] This application also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a computer, implement the master data integration method described above.

[0094] The aforementioned storage media include, but are not limited to, random access memory, read-only memory, cache, hard disk, or memory card.

[0095] The various embodiments in this specification are described in a progressive manner. For the same or similar parts between the various embodiments, please refer to each other. Each embodiment focuses on describing the differences from other embodiments.

[0096] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. A master data integration method, characterized by, include: Acquire multi-source heterogeneous master data; The multi-source heterogeneous master data is standardized and cleaned to generate multiple block keys and inverted index lists, and a candidate alignment set containing the intersection of the inverted index lists corresponding to the multiple block keys is constructed. Extract and normalize the multi-level similarity of each pair of candidate records in the candidate comparison set, and construct a standardized feature vector; The standardized feature vector is input into the learning model to obtain the matching probability of each pair of candidate records, and a hierarchical decision is made based on the matching probability and at least one matching threshold. An undirected matching graph is constructed using the candidate records as vertices and each pair of candidate records whose matching probability reaches the matching threshold corresponding to the merging decision as edges. Entity clusters are then generated from the undirected matching graph using a disjoint-set data structure algorithm. Determine the data quality score for each field of each candidate record within the entity cluster, determine the optimal value for each field based on the data quality score, and generate integrated master data.

2. The method of claim 1, wherein, The extraction and normalization of the multi-level similarity of each pair of candidate records in the candidate comparison set includes: Calculate the edit distance, Jaro-Winkler distance, and cosine similarity of the text fields to determine the string similarity of each pair of candidate records; The name-class field is semantically encoded using a word vector model to obtain semantic embedding vectors, and the cosine similarity of the semantic embedding vectors is calculated. Calculate the time difference of the time field for each pair of candidate records and convert the time difference into a normalized time similarity using an exponential decay function; Calculate the geographic distance between geographic entities corresponding to the address fields of each pair of candidate records, and normalize the geographic distance to obtain geographic similarity; For each pair of candidate records, a confidence weight is predefined for each data source, and the data verification status is encoded. The confidence weight and the encoded data verification status are then weighted and fused to obtain the source feature similarity.

3. The method of claim 1, wherein, The learning model comprises a three-layer structure; The first layer structure is a random forest, used to handle non-linear text feature interactions; The second layer structure is a gradient boosting tree, used to optimize boundary samples, which include qualification certificate conflicts and addresses that are similar but have large name differences. The third layer is logistic regression, used to calibrate probabilities.

4. The method of claim 1, wherein, The at least one matching threshold includes a first matching threshold and a second matching threshold, wherein the first matching threshold is greater than the second matching threshold; The hierarchical decision-making based on the matching probability and at least one matching threshold includes: Candidate records with a matching probability reaching the first matching threshold are merged; candidate records with a matching probability less than the first matching threshold but reaching the second matching threshold are manually reviewed; and candidate records with a matching probability less than the second matching threshold are not merged.

5. The method of claim 1, wherein, Before determining the data quality score for each field of each candidate record within the entity cluster, the method further includes: Calculate the cohesion of the entity cluster; Entity clusters with a cohesion less than a preset cohesion threshold are submitted for manual review.

6. The method of claim 5, wherein, The method further includes: Obtain the results of manual review and input the results of manual review into the training set of the learning model; The learning model is updated incrementally based on the training set, and repetitive patterns in manual review are identified and transformed into automatic rules that can be embedded in the learning model.

7. The method of claim 1, wherein, The process of determining the data quality score for each field of each candidate record within the entity cluster includes: The data quality score is obtained by weighting and summing each field of each candidate record within the entity cluster according to completeness, timeliness, source credibility, and verification status.

8. A master data integration apparatus, characterized by comprising: include: The data access module is used to acquire multi-source heterogeneous master data; The block segmentation module is used to standardize and clean the multi-source heterogeneous master data, generate multiple block keys and inverted index lists, and construct a candidate alignment set containing the intersection of the inverted index lists corresponding to the multiple block keys. The feature engineering module is used to extract and normalize the multi-level similarity of each pair of candidate records in the candidate comparison set and construct a standardized feature vector. The matching decision module is used to input the standardized feature vector into the learning model, obtain the matching probability of each pair of candidate records, and make a hierarchical decision based on the matching probability and at least one matching threshold. The clustering module is used to construct an undirected matching graph with each pair of candidate records as vertices and each pair of candidate records whose matching probability reaches the matching threshold corresponding to the merging decision as edges, and to generate entity clusters from the undirected matching graph using a disjoint-set data structure algorithm. The master data generation module is used to determine the data quality score of each field of each candidate record in the entity cluster, determine the optimal value of each field based on the data quality score, and generate integrated master data.

9. An electronic device, comprising: include: One or more processors; as well as A memory storing computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to perform the master data integration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions that, when executed by a computer, implement the master data integration method as described in any one of claims 1 to 7.

Images