A Lexical-Level Entity Matching Method and System for Heterogeneous Attributes

Abstract

This invention provides a word-level entity matching method for heterogeneous attributes, comprising: S1: obtaining a set of entity pairs and dividing the set into a training set and a test set; S2: constructing a cross-word matching matrix using the training set, and reconstructing word vectors using the cross-word matching matrix to obtain word-level matching vectors; S3: training a matching model using the word-level matching vectors to obtain a trained matching model; S4: matching the test set using the trained matching model to obtain entity matching results. This invention converts words in attributes into vectors, and constructs a cross-word matching matrix by comparing these vectors with the vectors of each word in the entity to be matched. The cross-word matching matrix contains the vector information of the entire entity, and can adaptively obtain suitable matching objects for each attribute. It has the advantages of high accuracy and robustness in entity matching between data, and can effectively handle the dirty data problem that occurs in entity matching.

Description

Technical Field

[0001] This invention relates to the field of entity matching, and in particular to a word-level entity matching method and system for heterogeneous attributes. Background Technology

[0002] With the advancement of my country's informatization process, the databases of various enterprises and government departments have gradually improved. However, the data often fails to realize its full value due to isolation. To respond to the national requirements of strengthening data aggregation, sharing, and open development, enhancing overall authorization and management, promoting interconnectivity, and breaking down "data silos," the correlation between data is crucial, thus requiring entity matching technology. Entity matching aims to identify data records that point to the same real-world entity. Most existing entity matching methods rely on the assumption that the entity records to be parsed are homogeneous, i.e., their attributes are aligned. Examples include the DeepMatcher and Magellan methods. The former encodes entities as a whole for comparison, failing to capture word-level information. The latter requires manually building merge lists and candidate sets and specifying features for parsing, making it sensitive to missing values ​​and dirty data. In reality, entities in real-world datasets are often heterogeneous, coming from different sources and represented with different attributes. Furthermore, entity attribute values ​​may be redundant, noisy, missing, misaligned, or misspelled. Therefore, existing entity matching methods cannot effectively handle this type of data. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a lexical-level entity matching method for heterogeneous attributes, comprising:

[0004] S1: Obtain the set of entity pairs and divide the set of entity pairs into a training set and a test set;

[0005] S2: Construct a cross-word matching matrix using the training set, and reconstruct the word vectors using the cross-word matching matrix to obtain word-level matching vectors;

[0006] S3: Train the matching model using word-level matching vectors to obtain a trained matching model;

[0007] S4: Match the test set using the trained matching model to obtain entity matching results.

[0008] Preferably, step S2 specifically includes:

[0009] S21: Obtain the t-th word of the i-th attribute of the first entity e1 in the training set. 1it Where i is the attribute number and t is the lexical number; lexical w 1it Inputting into a bidirectional LSTM network yields word vectors, where w is the word. 1itThe vector is Where u is the number of output units in the bidirectional LSTM network, R 2u The dimension of the vector is 2u×1;

[0010] S22: Obtain the vector matrix constructed from the vectors of all words of the second entity e2 in the training set. Where Q is the total number of all lexical units in the second entity e2;

[0011] S23: Transfer vector h 1it The cross-word matching matrix is ​​obtained by comparing it with all vectors in the vector matrix H2.

[0012] S24: The cross-word matching matrix C 1it Inputting the HighwayNet network yields the output matrix. Output matrix G 1it Attention vectors are obtained after inputting a linear layer and a softmax function.

[0013] S25: Applying an element-wise function to the attention vector v 1it Convert to selection vector

[0014] S26: Through cross-lexical matching matrix C 1it and selection vector s 1it Calculate and obtain the word w 1it The word-level matching vector r 1it The expression is:

[0015]

[0016] S27: Repeat steps S21-S26 to obtain all word-level matching vectors for the first entity;

[0017] S28: Following steps S21-S27, obtain all word-level matching vectors for the second entity.

[0018] Preferably, the cross-term matching matrix C 1it The calculation formula is:

[0019]

[0020] Among them, e Q It is a one-dimensional vector with each element equal to 1 and a length of Q, therefore the outer product result is... It is a vector with Q column vectors, each of which is h. 1it The matrix.

[0021] Preferably, in step S24, the output matrix G 1itThe expression is:

[0022] G 1it =HighwayNet(C 1it )

[0023] Attention vector v 1it The expression is

[0024] v 1it =softmax(wG) 1it +b)

[0025] in, and To match the model's learning parameters, HighwayNet() represents the HighwayNet network, and softmax represents the softmax function.

[0026] Preferably, step S25 specifically includes:

[0027] If V 1it [l] = max(V) 1it ), then s 1it [1] = 1, otherwise 0; where l is V 1it The element number in the file.

[0028] Preferably, step S4 specifically includes:

[0029] S41: Match the test set using the trained matching model to obtain the word-level matching vectors r of the first entity in the test set. li1 r 1i2 ...r lin , where n is the total number of lexical units for the i-th attribute of the first entity in the test set;

[0030] S42: Summing the matching vectors of each word level yields the attribute vector r of the i-th attribute of the first entity. 1i The calculation formula is:

[0031]

[0032] S43: Following steps S41-S42, calculate and obtain the attribute vector r of the i-th attribute of the second entity. 2i ;

[0033] S44: Transfer attribute vector r 1i and attribute vector r 2i Perform a one-dimensional convolution to obtain vectors r1 and r2 with the same dimension. Summing vectors r1 and r2 yields the final vector R.

[0034] S45: Input the final vector R into the multilayer perceptron of the trained matching model to obtain the entity matching result.

[0035] A lexical-level entity matching system for heterogeneous attributes, comprising:

[0036] The entity pair acquisition module is used to acquire a set of entity pairs and divide the set of entity pairs into a training set and a test set;

[0037] The vector matching module is used to construct a cross-word matching matrix through the training set, and reconstruct the word vectors through the cross-word matching matrix to obtain word-level matching vectors;

[0038] The matching model training module is used to train the matching model using word-level matching vectors to obtain a trained matching model.

[0039] The entity matching module is used to match the test set using a trained matching model to obtain entity matching results.

[0040] The present invention has the following beneficial effects:

[0041] By converting the tokens in the attributes into vectors, and comparing these vectors with the vectors of each token of the entity to be matched, a cross-token matching matrix is ​​constructed. This cross-token matching matrix contains the vector information of the entire entity and can adaptively obtain the appropriate matching object for each attribute. It has the advantages of high accuracy and robustness in entity matching between data and can effectively handle the dirty data problem that occurs in entity matching. Attached Figure Description

[0042] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0043] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0044] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0045] Reference Figure 1 This invention provides a word-level entity matching method for heterogeneous attributes, comprising:

[0046] S1: Obtain the set of entity pairs and divide the set of entity pairs into a training set and a test set;

[0047] S2: Construct a cross-word matching matrix using the training set, and reconstruct the word vectors using the cross-word matching matrix to obtain word-level matching vectors;

[0048] S3: Train the matching model using word-level matching vectors to obtain a trained matching model;

[0049] S4: Match the test set using the trained matching model to obtain entity matching results.

[0050] Furthermore, in heterogeneous and dirty entity data, the information corresponding to one attribute may be scattered across multiple attributes of another entity. This means that in order to obtain an accurate comparison object for an attribute value, we must consider the entire entity as a whole, not just a specific attribute. To address this, this invention uses a global selection mechanism to select the most similar lexical term in another entity for each lexical term.

[0051] Step S2 is as follows:

[0052] S21: Obtain the t-th word of the i-th attribute of the first entity e1 in the training set. 1it Where i is the attribute number and t is the lexical number; lexical w 1it Inputting into a bidirectional LSTM network yields word vectors, where w is the word. 1it The vector is Where u is the number of output units in the bidirectional LSTM network, R 2u The dimension of the vector is 2u×1;

[0053] Specifically, in word vectors, each word is represented as a distribution of weights (positive or negative); typically, these dimensions can vary independently; each word is represented by setting appropriate weights on multiple dimensions, and each vector dimension contributes to the representation of many words; word vectors learned from data have good semantic properties, so semantic relationships can be reflected by the vector differences between words.

[0054] This invention allows you to directly query the GloVe word vector dictionary to obtain the required word vectors.

[0055] This invention treats each attribute as a sequence of lexical units and uses the aforementioned word vectors to train the vectors of the lexical units; formally, given a sequence of lexical units w... t Let t∈[1,k] be the attribute a, where k is the number of lexical units of a. First, each lexical unit is represented as a pre-trained word vector. Then, a bidirectional LSTM is used to fuse their contextual information; and the result is used as a word w. t ;

[0056] S22: Obtain the vector matrix constructed from the vectors of all words of the second entity e2 in the training set. Where Q is the total number of all lexical units in the second entity e2;

[0057] S23: Transfer vector h 1it The cross-word matching matrix is ​​obtained by comparing it with all vectors in the vector matrix H2.

[0058] S24: The cross-word matching matrix C 1it Inputting the HighwayNet network yields the output matrix. Output matrix G 1it Attention vectors are obtained after inputting a linear layer and a softmax function.

[0059] S25: Applying an element-wise function to the attention vector v 1it Convert to selection vector

[0060] S26: Through cross-lexical matching matrix C 1it and selection vector s 1it Calculate and obtain the word w 1it The word-level matching vector r 1it The expression is:

[0061]

[0062] Specifically, no matter how complex the relationship between attributes is, or whether the values ​​are misaligned, by using the cross-terminal matching matrix mentioned above, a suitable matching object can be adaptively obtained for each attribute, thus effectively solving the problems of pattern heterogeneity and misaligned dirty data.

[0063] S27: Repeat steps S21-S26 to obtain all word-level matching vectors for the first entity;

[0064] S28: Following steps S21-S27, obtain all word-level matching vectors for the second entity.

[0065] Furthermore, the cross-word matching matrix C 1it The calculation formula is:

[0066]

[0067] Among them, e Q It is a one-dimensional vector with each element equal to 1 and a length of Q, therefore the outer product result is... It is a vector with Q column vectors, each of which is h. 1it The matrix.

[0068] Furthermore, in step S24, the output matrix G 1it The expression is:

[0069] G 1it =HighwayNet(C 1it )

[0070] Attention vector v 1it The expression is

[0071] v 1it =softmax(wG) 1it +b)

[0072] in, and To match the model's learning parameters, HighwayNet() represents the HighwayNet network, and softmax represents the softmax function.

[0073] Furthermore, step S25 specifically involves:

[0074] If V 1it [1] = max(V 1it ), then s 1it [l] = 1, otherwise 0; where l is V 1it The element number in the file.

[0075] Furthermore, since heterogeneous entity records are composed of different attributes, it is necessary to aggregate attribute-level comparison evidence from two entity records to make the final entity matching decision.

[0076] Step S4 is as follows:

[0077] S41: Match the test set using the trained matching model to obtain the word-level matching vectors r of the first entity in the test set. li1 r 1i2 ...r lin , where n is the total number of lexical units for the i-th attribute of the first entity in the test set;

[0078] S42: Summing the matching vectors of each word level yields the attribute vector r of the i-th attribute of the first entity. 1i The calculation formula is:

[0079]

[0080] S43: Following steps S41-S42, calculate and obtain the attribute vector r of the i-th attribute of the second entity. 2i ;

[0081] S44: Transfer attribute vector r 1i and attribute vector r 2i Perform a one-dimensional convolution to obtain vectors r1 and r2 with the same dimension. Summing vectors r1 and r2 yields the final vector R.

[0082] S45: Input the final vector R into the multilayer perceptron of the trained matching model to obtain the entity matching result.

[0083] Experimental Evaluation

[0084] Evaluation metrics include: accuracy, recall, precision, f1-score, etc.

[0085]

[0086]

[0087]

[0088]

[0089] TP, TN, FP, and FN represent instances of true positive, true negative, false positive, and false negative, respectively.

[0090] 1. Import word vectors

[0091] Import the 300-dimensional word embeddings from the pre-trained GloVe word vector dictionary and continuously refine them during training.

[0092] 2. Data Preparation

[0093] Experiments were conducted using a homogeneous dataset from the real world (Walmart-Amazon), a dirty dataset (from which 50% of the attributes of all tuples from the same data source were randomly discarded), and a heterogeneous dataset (where certain attributes of the homogeneous dataset were merged, for example, merging "brand" and "category" to obtain the "brand-category" attribute to generate heterogeneous data).

[0094] 3. Data partitioning

[0095] The dataset is divided in a 6:2:2 ratio, with 60% of the data used as the training set, 20% as the validation set, and the remaining data as the test set.

[0096] 4. Model Training

[0097] The matching model is used to classify and train the training set. Then, the trained model is used to predict the test set to obtain the f1-score.

[0098] 5 Experimental Results

[0099] Data types Magellan DeepMarcher This model Homogeneous data 70.9 67.6 76.0 Dirty data 23.1 55.1 69.7 Heterogeneous data Null 66.0 67.5

[0100] A lexical-level entity matching system for heterogeneous attributes, comprising:

[0101] The entity pair acquisition module is used to acquire a set of entity pairs and divide the set of entity pairs into a training set and a test set;

[0102] The vector matching module is used to construct a cross-word matching matrix through the training set, and reconstruct the word vectors through the cross-word matching matrix to obtain word-level matching vectors;

[0103] The matching model training module is used to train the matching model using word-level matching vectors to obtain a trained matching model.

[0104] The entity matching module is used to match the test set using a trained matching model to obtain entity matching results.

[0105] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0106] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the unit claims listing several devices, several of these devices may be embodied by the same hardware item. The use of the terms first, second, and third, etc., does not indicate any order and can be interpreted as identifiers.

[0107] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for heterogeneous attribute oriented word-level entity matching, the method comprising: The method comprises the following steps: S1: obtaining a set of entity pairs, and dividing the set of entity pairs into a training set and a test set; S2: constructing a cross-wordpiece matching matrix through the training set, reconstructing a vector of a wordpiece through the cross-wordpiece matching matrix, and obtaining a wordpiece-level matching vector; Step S2 is specifically: S21: obtaining a first token of a first attribute of a first entity in a training set ; wherein i is the number of the attribute, and t is the number of the token; inputting the token into a bidirectional LSTM network to obtain a vector of the token, and the vector of the token is ; wherein u is the number of output units in the bidirectional LSTM network, and R 2u represents that the dimension of the vector is 2u x 1;​​​ S22: obtaining a second entity in the training set a vector matrix of all word vectors ; wherein, the second entity the total number of all word vectors in the second entity S23: compare the vector with all vectors in the vector matrix and obtain a cross token match matrix ; S24: obtaining an output matrix from the HighwayNet network obtaining an output matrix from the HighwayNet network obtaining an output matrix from the HighwayNet network obtaining an output matrix from the HighwayNet network obtaining an output matrix from the HighwayNet network S25: convert the attention vector to a selection vector by an element-wise function ; and ; and S26: Compute the token-level match vector for the token by cross-token match matrix and selection vector Compute the token-level match vector for the token of the token , expressed as: = S27: repeating steps S21-S26 to obtain all wordpiece-level matching vectors of the first entity; S28: obtaining all wordpiece-level matching vectors of the second entity according to steps S21-S27; Cross-token matching matrix The formula for calculating is: wherein, is a 1-by-Q matrix of ones, and is a Q-by-1 vector of ones. a vector, so the outer product results in ) is a matrix with Q column vectors and each column vector is ​ S3: training a matching model through the wordpiece-level matching vector, and obtaining a trained matching model; S4: matching the test set through the trained matching model, and obtaining an entity matching result.

2. The method for heterogeneous attribute oriented word-level entity matching according to claim 1, characterized in that, The output matrix in step S24 The expression is: Attention vector The expression for wherein, and are learning parameters of the matching model, HighwayNet() represents a HighwayNet network, and softmax represents a softmax function.

3. The method for heterogeneous attribute oriented word-level entity matching according to claim 1, wherein, Step S25 is specifically: like ,but Otherwise, it is 0; where for The element number in the file.

4. The method for heterogeneous attribute oriented word-level entity matching according to claim 1, characterized in that, Step S4 is specifically: S41: Match the test set by the trained matching model to obtain all word-level matching vectors r of the first entity in the test set 1i1 ,r 1i2 ……r 1in Wherein n is the total number of word units of the i-th attribute of the first entity in the test set; S42: add each word-level matching vector to obtain an attribute vector of the i-th attribute of the first entity The calculation formula is: S43: According to steps S41-S42, the attribute vector of the i-th attribute of the second entity is calculated ; S44: obtaining a property vector and a property vector performing one-dimensional convolution to obtain vectors r1 and r2 with the same dimension, and summing the vectors r1 and r2 to obtain a final vector R; S45: inputting the final vector R into a multi-layer perception machine of the trained matching model to obtain an entity matching result.

5. A heterogeneous attribute oriented wordpiece level entity matching system, comprising: The method comprises the following steps: An entity pair acquisition module is configured to obtain a set of entity pairs, and divide the set of entity pairs into a training set and a test set; A vector matching module is configured to construct a cross-wordpiece matching matrix through the training set, reconstruct a vector of a wordpiece through the cross-wordpiece matching matrix, and obtain a wordpiece-level matching vector, and the steps are specifically: S21: obtain the i-th token of the j-th attribute of the first entity in the training set ; wherein i is the number of the attribute, and t is the number of the token; input the token into a bidirectional LSTM network to obtain a vector of the token, and the vector of the token is ; wherein u is the number of output units in the bidirectional LSTM network, and R 2u represents that the dimension of the vector is 2u×1;​​​ S22: obtaining a second entity in the training set a vector matrix of all word vectors ; wherein, the second entity the total number of all word vectors in the second entity S23: compare the vector with all vectors in the vector matrix and obtain a cross token match matrix ; S24: obtaining an output matrix from the HighwayNet network obtaining an output matrix from the HighwayNet network obtaining an output matrix from the HighwayNet network obtaining an output matrix from the HighwayNet network obtaining an output matrix from the HighwayNet network S25: convert the attention vector to a selection vector by an element-wise function ; and ; and S26: Compute the token-level match vector for the token by cross-token match matrix and selection vector Compute the token-level match vector for the token of the token , expressed as: = S27: repeating steps S21-S26 to obtain all wordpiece-level matching vectors of the first entity; S28: obtaining all wordpiece-level matching vectors of the second entity according to steps S21-S27; Cross-token matching matrix The formula for calculating is: wherein, is a Q-length vector of ones a vector, so the outer product results in ) is a matrix with Q column vectors and each column vector is ​ A matching model training module is configured to train a matching model through the wordpiece-level matching vector, and obtain a trained matching model; An entity matching module is configured to match the test set through the trained matching model, and obtain an entity matching result.

Images