A method and apparatus for synthesizing an entity resolution dataset

Abstract

The present invention discloses a method and device for synthesizing an entity resolution dataset, wherein the method includes: obtaining an entity resolution dataset and calculating a similarity vector distribution of a real dataset, generating a false entity based on the entity resolution dataset, and initiating the generation of the entire entity resolution dataset based on the false entity; performing similarity vector calculation on entity pairs without labels in the generated entity resolution dataset, calculating a first posterior probability distribution of the similarity vector and a second posterior probability distribution of the similarity vector based on the similarity vector distribution of the real dataset, and if the first posterior probability distribution is greater than or equal to the second posterior probability distribution, marking the entity pairs without labels as a match; otherwise, marking them as a mismatch. The present invention can synthesize a false entity resolution dataset with the same similarity vector distribution as the real entity resolution dataset, and the entities in the synthesized entity resolution dataset are similar to the real entities.

Description

Technical Field

[0001] The present invention relates to the field of data integration technology, and in particular to a method and device for synthesizing entity resolution data sets. Background Art

[0002] In the big data landscape, the internet is teeming with fragmented information, yet this information holds immense commercial value. For example, different e-commerce platforms contain diverse product information, pricing records, and purchasing channels, as well as user purchase histories, preferences, and profiles. Social networking sites aggregate personal information, social profiles, and real-time statuses of diverse users. Effectively integrating this fragmented information can facilitate data integration and sharing, improve data quality, and reduce data analysis costs. Entity resolution (ER), which involves finding entities that reference the same real-world object across two different datasets, is a core problem in data integration. In reality, many companies possess large datasets that require entity resolution for data integration. Designing entity resolution solutions is challenging for non-entity resolution experts within a company, so publishing entity resolution datasets to identify effective solutions is a natural approach. However, most companies are reluctant to release their real datasets for various reasons, such as privacy concerns. Therefore, synthesizing entity resolution datasets is a crucial issue. A typical solution in the machine learning and statistics community is to create proxy datasets based on real datasets and release these proxy datasets to the public to train machine learning models, so that these models trained on the proxy datasets can be applied by companies to real datasets.

[0003] Similarly, if we can synthesize entity resolution datasets that are completely fake and look like real entity resolution datasets, companies can release these generated entity resolution datasets. This will help entity resolution algorithm researchers obtain more data to study entity resolution models, and also help entity resolution data owners, who can obtain more effective models to perform entity resolution tasks. Summary of the Invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the present invention aims to propose a method for synthesizing entity resolution datasets that can generate entities that are very similar to real entities. The generated entity resolution dataset can maintain the performance of entity resolution in the original entity resolution dataset, that is, the entity resolution model trained using the synthesized entity resolution dataset and the entity resolution model trained using the real entity resolution dataset have similar performance on the same test set. The generated entity resolution dataset can protect the privacy of the real entity resolution dataset, so the synthesized entity resolution dataset can be released to seek entity resolution solutions, and the entity resolution dataset can be quickly synthesized, thereby achieving the purpose of data integration and sharing.

[0006] Another object of the present invention is to provide a device for synthesizing entity resolution datasets.

[0007] To achieve the above objectives, the present invention proposes, on one hand, a method for synthesizing an entity resolution dataset, comprising: obtaining an entity resolution dataset, and calculating a similarity vector distribution of a real dataset based on the entity resolution dataset; wherein the entity resolution dataset includes two real datasets that require entity resolution, and labels indicating whether entities in the two datasets match; generating a false entity based on the entity resolution dataset, and initiating the generation of the entire entity resolution dataset based on the false entity; the entire entity resolution dataset is a false entity resolution dataset; calculating similarity vectors for unlabeled entity pairs in the generated entity resolution dataset, calculating a first posterior probability distribution of the similarity vector and a second posterior probability distribution of the similarity vector based on the similarity vector distribution of the real dataset, and if the first posterior probability distribution is greater than or equal to the second posterior probability distribution, marking the unlabeled entity pair as a match; otherwise, marking it as a mismatch. The method for synthesizing an entity resolution dataset according to an embodiment of the present invention can synthesize a false entity resolution dataset with the same similarity vector distribution as the real entity resolution dataset, and the entities in the synthesized entity resolution dataset are similar to the real entities. The entity resolution dataset can be quickly synthesized, thereby achieving the purpose of data integration and sharing.

[0008] To achieve the above-mentioned object, the present invention further provides a device for synthesizing an entity resolution dataset, comprising:

[0009] An acquisition module is configured to acquire an entity resolution dataset and calculate a similarity vector distribution for a real dataset based on the entity resolution dataset. The entity resolution dataset includes two real datasets requiring entity resolution and labels indicating whether entities in the two datasets match. A generation module is configured to generate a false entity based on the entity resolution dataset and initiate the generation of the entire entity resolution dataset based on the false entity. The entire entity resolution dataset is a false entity resolution dataset. A judgment module is configured to calculate similarity vectors for unlabeled entity pairs in the generated entity resolution dataset, calculate a first a posteriori probability distribution of the similarity vectors and a second a posteriori probability distribution of the similarity vectors based on the similarity vector distribution of the real dataset, and label the unlabeled entity pairs as matching if the first a posteriori probability distribution is greater than or equal to the second a posteriori probability distribution; otherwise, label them as mismatching. The apparatus for synthesizing an entity resolution dataset according to an embodiment of the present invention can synthesize a false entity resolution dataset with the same similarity vector distribution as the real entity resolution dataset, and the entities in the synthesized entity resolution dataset are similar to the real entities. This allows for rapid synthesis of entity resolution datasets, thereby achieving the goals of data integration and sharing.

[0010] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0012] Figure 1 A flowchart of a method for synthesizing entity parsing datasets according to an embodiment of the present invention;

[0013] Figure 2 A schematic diagram of similarity vectors generated from an entity resolution dataset according to an embodiment of the present invention;

[0014] Figure 3 A schematic diagram of an overview of a synthetic entity resolution dataset according to an embodiment of the present invention;

[0015] Figure 4 A schematic diagram showing an overview of training and generating similar character strings according to an embodiment of the present invention;

[0016] Figure 5 This is a diagram illustrating an operation example of generating string attributes in an entity matching dataset according to an embodiment of the present invention;

[0017] Figure 6 Schematic diagram of the structure of an apparatus for synthesizing an entity parsing dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0020] The following describes a method and apparatus for synthesizing an entity resolution dataset according to an embodiment of the present invention with reference to the accompanying drawings.

[0021] Figure 1 It is a flowchart of a method for synthesizing an entity resolution dataset according to an embodiment of the present invention.

[0022] like Figure 1 As shown, the method for synthesizing an entity resolution dataset includes the following steps:

[0023] S1, obtain the entity resolution dataset, and calculate the similarity vector distribution of the real dataset based on the entity resolution dataset; wherein the entity resolution dataset contains two real datasets that need to be resolved, and labels indicating whether the entities in the two datasets match.

[0024] It is understandable that the present invention obtains the entity resolution dataset E real : Includes two data sets A that need to be matched real and B real , and the set of matching entity pairs M in the two datasets real and the set of unmatched entity pairs N real . According to the matching entity pair set M real Calculate the matching similarity vector According to the set of unmatched entity pairs N real Calculate the mismatch similarity vector The schematic diagram of the similarity vector generated by the present invention is as follows: Figure 2 shown.

[0025] Specifically, the specific process of step S1 of the present invention is as follows:

[0026] (S1-1) According to the matching entity pair set M real Calculate the matching similarity vector According to the set of unmatched entity pairs Nreal Calculate the mismatch similarity vector

[0027] (S1-2) Calculate the distribution of similarity vectors of real entity resolution dataset: Model the distribution of real entity resolution similarity vectors as a mixed Gaussian distribution The probability of entity pair matching is The matching similarity vector distribution is a mixed Gaussian distribution M real , the distribution of unmatched similarity vectors is a mixed Gaussian distribution According to the matching similarity vector Calculate M real , according to the unmatched similarity vector calculate Calculate M real or The same method is used to real For example, first minimize The AIC (Akaike Information Criterion) value is used to obtain M real The optimal number of normal distributions g is obtained, and then the parameters of each normal distribution are obtained through the EM (Expectation-Maximization) algorithm.

[0028] by For example, the EM algorithm calculates The parameters of the normal distribution with the maximum probability of each vector appearing in :

[0029]

[0030] where Θ is M real The parameter of the distribution, M real Contains g normal distributions p1,p2,...,p g For 1≤i≤g, p i ~N(μ i ,Σ i ),π i is the i-th normal distribution (i.e. p i ) and π1+2+…+ g =1.

[0031] The EM algorithm is an iterative process. First, the parameter μ k ,Σ k ,π k Initialized to Then the iteration begins. In the tth round of iteration, the following steps are performed in sequence:

[0032] (S1-2-1) Given the parameters of the previous iteration (i.e., the t-1 iteration), calculate x i The probability of belonging to the kth normal distribution:

[0033]

[0034] (S1-2-2) Recalculate the parameters of the t-th iteration by maximizing the likelihood function:

[0035]

[0036]

[0037]

[0038] Repeat the above steps until the likelihood function converges.

[0039] (S1-3) Initialize the synthetic entity resolution dataset E stn , including two data sets A to be generated syn and B stn , and the set of matching entity pairs M in the two datasets syn and the set of unmatched entity pairs N syn .

[0040] S2, generates a fake entity based on the entity resolution dataset, and starts the generation of the entire entity resolution dataset based on the fake entity; the entire entity resolution dataset is a fake entity resolution dataset.

[0041] It is understandable that, based on the real entity parsing dataset, manually or automatically syn The dataset generates a fake entity e, and then generates the remaining fake entity parsing datasets based on the fake entity e to start the generation of the entire fake entity parsing dataset.

[0042] Furthermore, the method of manually generating a fake entity includes: generating a value for each attribute in the entity resolution dataset. For numeric and categorical attributes, a value is randomly sampled from the attribute's range; for text attributes, a string that does not exist in the real data is selected from the domain knowledge of the attribute.

[0043] Furthermore, the method of automatically generating a fake entity includes: using a GAN (Generative Adversarial Network) model to automatically synthesize a new entity.

[0044] It can be known that the GAN model consists of two components: the generator and the discriminator Generator Can generate fake entities. The generator receives a random noise, which can generate a fake entity after passing through a fully connected neural network. The output of different neurons can be used as different attributes of the entity. It is a binary classification model that is trained to determine whether an entity is real or generated by the generator. The training data of the discriminator includes real entities and synthetic entities, where the label of real entities is 1 and the label of synthetic entities is 0. and During the training process, an adversarial minimax game is performed. After the GAN model is trained, the generator can be used Synthesize a new fake entity that is similar to a real entity. After the entity is generated, it can be added to A syn , at this time A syn ={e},B syn ={}.

[0045] Next, iteratively generate A syn and B syn Entities in: Sample an entity e from the generated entity resolution dataset, sample a similarity vector x from the calculated similarity vector distribution, and generate a new entity e′ based on entity e and similarity vector x, so that the similarity vectors of e and e′ are close to x. Add the generated entities to A respectively syn and B syn At the same time, according to whether the entity pairs match, the entity pairs are added to M syn or N syn .

[0046] Specifically, the specific process of step S2 of the present invention is as follows:

[0047] (S2-1) From the generated entity parsing dataset (i.e. A syn UB syn ) Randomly sample an entity e.

[0048] (S2-2) Similarity vector distribution obtained from calculation Sampling to get a similarity vector x: The probability of Sampling from the distribution, The probability of Sampling from a distribution.

[0049] (S2-3) Generate a new entity e' based on the entity e and the similarity vector x, so that the similarity vectors of e and e' are close to x. The method of generating a new entity e' based on the entity e and the similarity vector x, so that the similarity vectors of e and e' are close to x, includes:

[0050] For numerical attributes, the generation is based on the similarity function, similarity value and the value of entity e on the attribute;

[0051] For a categorical attribute, the categorical attribute with the closest similarity value to the value of entity e on the attribute is selected as the generated value;

[0052] For text attributes, according to the similarity value, the corresponding Transformer is selected, the value of the entity e on the attribute is input into the Transformer as a sequence, and the output result of the Transformer is used as the generated text attribute value. Specifically, the problem of the method is defined as: based on a string s, a similarity function f, and a similarity value sim, a new string s′ is generated so that sim=f(s,s′). The content of the method is: for different similarity bins, multiple Transformer models are trained. For sim∈[0,1], the interval [0,1] is divided into k non-overlapping and continuous intervals I1,I2,...,I k . Train k Transformer models M i ,0≤i≤k, where M i The training data is the domain knowledge with similarity in the interval I i Using string pairs in domain knowledge as training data can protect the privacy of real data. For a text attribute, first crawl some strings from the domain knowledge of the text attribute, then enumerate these strings in pairs, calculate the similarity of each string pair, and divide them into different bins according to the similarity of the string pairs. Finally, train a Transformer model for different similarity intervals (bins). After training multiple Transformer models, for a given string s and similarity value sim, first check which interval sim belongs to. Assume sim∈I i , input string s into model M i , multiple sampling is performed during the decoding process of the model, and multiple output results s′1,s′2,s′3,... of the model are obtained as candidate strings. The similarity between these candidate strings and s is calculated, and the string with the closest similarity to s to sim is returned as s′.

[0053] (S2-4) For the generated entity e′, check whether it is similar to the real entity and whether it satisfies the distribution of the learned real similarity vector. If not, reject the entity and regenerate it until an entity that satisfies the above two conditions is generated.

[0054] (S2-4-1) The method for checking whether the generated entity e′ is similar to the real entity includes: using the discriminator of the GAN model To determine whether the generated entity e′ looks like a real entity. The probability that e′ is predicted to be a real entity can be output: if the probability is less than β, e′ is rejected; if the probability is greater than or equal to β, e′ is accepted. Where β is a parameter, β∈[0,1]. By adjusting the value of β, the strictness of rejection can be set: when β is infinitely close to 0, e′ will be accepted.

[0055] (S2-4-2) The method for checking whether the generated entity e′ satisfies the distribution of the learned real similarity vector includes: calculating the distribution of the similarity vector of the currently generated entity resolution data set after adding e′ to the currently generated entity resolution data set. if Become distant If it is farther, reject e′, otherwise accept e′. Let T e Let T′ be the entity dataset to which entity e belongs. e is the entity dataset to which entity e′ belongs. Because x is So the similarity vector of entity pair (e,e′) obeys distribution, but when e′ is added to T′ e When , more entity pairs will be generated. The similarity vectors of these entity pairs may not satisfy distribution, we should reject e′.

[0056] (S2-4-2-1) Calculate the similarity vector distribution of the synthetic entity resolution dataset When we first start to synthesize data, A syn and B syn Small, we can directly calculate the similarity vector X of the synthetic dataset syn By calculating the entity pairs in the synthetic entity resolution dataset and The probability of , the entity pair can be annotated (that is, whether the entity pair matches):

[0057]

[0058]

[0059] in P n (x) = 1-P m (x). and After that, the similarity vector distribution of the synthetic entity resolution dataset can be calculated according to the method in (S1-2)

[0060] (S2-4-2-2) Update the similarity vector distribution of the synthetic entity resolution dataset Calculate when e′ is added to T′ e After that, the newly added similarity vectors, namely e′ and T e Similarity vector of entities in :

[0061] ΔX syn ={x (a,) |a=e′,b∈T e}

[0062] By calculating ΔX syn The entity pair belongs to and The probability of , the entity pair can be annotated (that is, whether the entity pair matches):

[0063]

[0064]

[0065] Update π:

[0066]

[0067] renew

[0068]

[0069] renew

[0070]

[0071]

[0072]

[0073] in

[0074] (S2-4-2-3) If ΔX syn Make Become distant Further, that is and The JSD distance is greater than and JSD distance:

[0075]

[0076] Then reject e′; otherwise, accept e′.

[0077] (S2-5) If e is from A syn Sampling obtained from B, add e′ to B syn ; If e is from B syn Sampling obtained from the above, add e′ to A syn If x is from The entity pair (e, e′) is added to M syn ; If x is from The entity pair (e, e′) is added to U syn .

[0078] Iterate the above steps (S2-1, S2-2, S2-3, S2-4, S2-5) until all entities are synthesized, i.e. |A syn |=|A real |,|B syn |=|B real |.

[0079] S3, calculate the similarity vector for the unlabeled entity pairs in the generated entity resolution dataset, calculate the first posterior probability distribution of the similarity vector based on the similarity vector distribution of the real dataset, and the second posterior probability distribution of the similarity vector. If the first posterior probability distribution is greater than or equal to the second posterior probability distribution, the unlabeled entity pair is marked as a match; otherwise, it is marked as a mismatch.

[0080] Specifically, for the entity pairs without labels in the generated entity resolution dataset, calculate their similarity vector x, and according to the similarity vector distribution of the real dataset Calculate x belongs to The posterior probability and x belongs to The posterior probability distribution P n (x) = 1-P m (x). If P m (x)≥P n (x), the entity pair is marked as a match; otherwise, it is marked as a mismatch.

[0081] In summary, Figure 3 FIG. 1 is a schematic diagram showing an overview of a synthetic entity resolution dataset according to an embodiment of the present invention. Figure 3 As shown, get the entity resolution dataset E real : Includes two data sets A that need to be matched real and B real , and the set of matching entity pairs M in the two datasets realand the set of unmatched entity pairs N real . According to the matching entity pair set M real Calculate the matching similarity vector According to the set of unmatched entity pairs N real Calculate the mismatch similarity vector Calculate the distribution of similarity vectors: Based on the matching similarity vectors Calculate the distribution of matching similarity vectors, based on the unmatched similarity vectors Calculate the distribution of unmatched similarity vectors and the probability of entity pair matching. Generate a fake entity manually or automatically to start the generation of the entire entity resolution dataset. Sample an entity e from the generated entity resolution dataset, sample a similarity vector x from the calculated similarity vector distribution, and generate a new entity e′ based on entity e and similarity vector x, so that the similarity vectors of e and e′ are close to x. Add the generated entities to A respectively. syn and B syn At the same time, according to whether the entity pairs match, the entity pairs are added to M syn or N syn . Iterate the above steps until all entities are synthesized. For the entities obtained in the generation process, check whether they are similar to the real entities and whether they meet the distribution of the learned real similarity vectors. If not, reject the entity and regenerate it until an entity that meets the above two conditions is generated. For the entity pairs without labels in the generated entity resolution dataset, calculate their posterior probabilities of matching and mismatching based on the similarity vector distribution and label them.

[0082] In summary, the present invention obtains two data sets that need to be matched for entity matching and labels indicating whether the entities in the two data sets match, calculates the matched similarity vectors and the unmatched similarity vectors obtained from the two data sets; based on the matched and unmatched similarity vectors, respectively learns the distribution of the matched similarity vectors and the distribution of the unmatched similarity vectors, and simultaneously learns the probability of matching and unmatching of entity pairs, thereby obtaining the distribution of the similarity vectors of the entire entity matching data set; samples are taken from the learned similarity vector distribution, and two entities are generated based on the sampled similarity vectors, so that the similarity vectors of the two entities are close to the sampled similarity vectors. Specifically, a Transformer model is used to generate textual data in the entity, while a GAN model is trained using real data, and a discriminator of the GAN model is used to determine whether the generated entity resembles the real entity. At the same time, the embodiment of the present application proposes a method for entity rejection to ensure that the similarity vectors of all synthesized entity pairs meet the distribution of the learned real similarity vectors. Through this embodiment, a fake entity resolution data set with the same similarity vector distribution as the real entity resolution data set can be synthesized, and the entities in the synthesized entity resolution data set are similar to the real entities.

[0083] According to an embodiment of the present invention, a method for synthesizing an entity resolution dataset is provided, wherein an entity resolution dataset is obtained and a similarity vector distribution of a real dataset is calculated based on the entity resolution dataset; wherein the entity resolution dataset includes two real datasets that need to be entity resolved and labels indicating whether the entities in the two datasets match; a false entity is generated based on the entity resolution dataset, and the generation of the entire entity resolution dataset is initiated based on the false entity; the entire entity resolution dataset is a false entity resolution dataset; similarity vectors are calculated for entity pairs without labels in the generated entity resolution dataset, and a first posterior probability distribution of the similarity vector and a second posterior probability distribution of the similarity vector are calculated based on the similarity vector distribution of the real dataset; if the first posterior probability distribution is greater than or equal to the second posterior probability distribution, the entity pairs without labels are marked as matched; otherwise, they are marked as mismatched. The present invention can synthesize a false entity resolution dataset with the same similarity vector distribution as the real entity resolution dataset, and the entities in the synthesized entity resolution dataset are similar to the real entities. Thus, the entity resolution dataset can be quickly synthesized, thereby achieving the purpose of data integration and sharing.

[0084] In order to implement the above embodiment, Figure 6 As shown, this embodiment also provides an apparatus 10 for synthesizing an entity resolution dataset. The apparatus 10 includes: an acquisition module 100 , a generation module 200 and a judgment module 300 .

[0085] An acquisition module 100 is configured to acquire an entity resolution dataset and calculate a similarity vector distribution of a real dataset based on the entity resolution dataset; wherein the entity resolution dataset includes two real datasets requiring entity resolution and labels indicating whether entities in the two datasets match;

[0086] A generating module 200 is configured to generate a false entity based on the entity resolution data set, and to initiate generation of the entire entity resolution data set based on the false entity; the entire entity resolution data set is the false entity resolution data set;

[0087] The judgment module 300 is used to calculate the similarity vector of the entity pairs without labels in the generated entity resolution data set, calculate the first posterior probability distribution of the similarity vector based on the similarity vector distribution of the real data set, and the second posterior probability distribution of the similarity vector. If the first posterior probability distribution is greater than or equal to the second posterior probability distribution, the entity pairs without labels are marked as matched; otherwise, they are marked as mismatched.

[0088] The apparatus for synthesizing entity resolution datasets according to an embodiment of the present invention can synthesize a fake entity resolution dataset with the same similarity vector distribution as a real entity resolution dataset, and the entities in the synthesized entity resolution dataset are similar to the real entities. This allows for rapid synthesis of entity resolution datasets, thereby achieving the goal of data integration and sharing.

[0089] It should be noted that the above explanation of the embodiment of the method for synthesizing an entity resolution dataset is also applicable to the apparatus for synthesizing an entity resolution dataset of this embodiment, and will not be repeated here.

[0090] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0091] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0092] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for synthesizing an entity resolution dataset, characterized in that: The following steps are involved: S1. Obtain an entity resolution dataset and calculate a similarity vector distribution of a real dataset based on the entity resolution dataset; wherein the entity resolution dataset includes two real datasets that require entity resolution and labels indicating whether entities in the two datasets match; wherein the entity resolution dataset includes basic information, price records, and purchase channels of different products on different e-commerce platforms, as well as purchase records, purchase preferences, and user profiles of different users; and personal information, social tracks, and real-time status of different users aggregated on different social networking sites; S2, generating a false entity based on the entity resolution data set, and starting generation of the entire entity resolution data set according to the false entity; the entire entity resolution data set is the false entity resolution data set; S3, performing similarity vector calculation on the unlabeled entity pairs in the generated entity resolution dataset, calculating a first posterior probability distribution of the similarity vector and a second posterior probability distribution of the similarity vector based on the similarity vector distribution of the real dataset, and marking the unlabeled entity pair as a match if the first posterior probability distribution is greater than or equal to the second posterior probability distribution; otherwise, marking it as a mismatch; It is characterized by the entity resolution dataset Contains two real datasets that require entity resolution and , and the set of matching entity pairs in the two datasets and the set of unmatched entity pairs , said S1, comprises: S1.1, according to the matching entity pair set Calculate the matching similarity vector , according to the set of unmatched entity pairs Calculate the mismatch similarity vector ; S1.2, model the distribution of real entity resolution similarity vectors as a mixed Gaussian distribution , where the probability of entity pair matching is , the matching similarity vector distribution is a mixed Gaussian distribution , the distribution of unmatched similarity vectors is a mixed Gaussian distribution , according to the similarity vector of the matching Calculate the , according to the mismatch similarity vector Calculate the ; S1.3, Initialize the synthetic entity resolution dataset , including the two datasets to be generated and , and the set of matching entity pairs in the two datasets and the set of unmatched entity pairs ; Generating a fake entity comprises: Generate a value for each attribute in the generated entity resolution dataset. For numeric or categorical attributes, randomly sample a value from the value range of the numeric or categorical attribute; for text attributes, select a string that does not exist in the real data from the domain knowledge of the text attribute; and Use the GAN model to automatically synthesize a new entity; The S2 includes: S2.1, randomly sample entities from the generated entity resolution dataset ; S2.2, as described The probability of Sampling from the distribution, with 1- The probability of Sampling from a distribution; S2.3, according to the entity and the similarity vector Generate new entities , making and The similarity vector is close to ; S2.4, for the new entity , check whether it is similar to the real entity and whether it meets the two conditions of the distribution of the learned real similarity vector. If not, reject the new entity , and regenerate until an entity that meets the two conditions is generated; S2.5, if It is from The sample obtained from Join ;if It is from The sample obtained from Join ;if It is from The entity pairs are sampled from the distribution Join ;if It is from The entity pairs are sampled from the distribution Join .

2. The method for synthesizing an entity resolution dataset according to claim 1, wherein: Said S2.4 includes: S2.41, for checking the new entity generated Whether it is similar to the real entity includes: using the discriminator of the GAN model Determine the new entity generated Does it seem like a real entity? S2.42, for checking the new entity generated Whether the distribution of the learned real similarity vector is satisfied includes: calculating the new entity After being added to the currently generated entity resolution dataset, the distribution of the similarity vector of the currently generated entity resolution dataset ,if and If the distance between them is greater than the preset distance, it will be rejected. , otherwise accept ;make For Entity The entity dataset to which it belongs, For Entity The entity dataset to which it belongs.

3. The method for synthesizing an entity resolution dataset according to claim 2, wherein: Said S2.42 includes: By calculating the entity pairs in the synthetic entity resolution dataset and The probability of labeling the entity pairs is: in, .

4. The method for synthesizing an entity resolution dataset according to claim 3, wherein: Said S2.42 further includes: Calculate when Join After that, the newly added similarity vector, and Similarity vector of entities in : By calculation The entity pair belongs to and The probability of labeling the entity pairs is: renew : renew : renew : in, .

5. The method for synthesizing an entity resolution dataset according to claim 4, wherein: Said S2.42 further includes: if Make and If the distance between them is greater than the preset distance, and The JSD distance is greater than and JSD distance: Reject Otherwise, accept .

6. The method for synthesizing an entity resolution dataset according to claim 5, wherein: The S3 includes: For the unlabeled entity pairs in the generated entity resolution dataset, calculate the similarity vector of the unlabeled entity pairs , according to the similarity vector distribution of the real data set Calculate the Belong to the The posterior probability , and the Belong to the The posterior probability distribution of ,if , then the entity pair without a label is marked as a match; otherwise, it is marked as a mismatch.

7. A device for synthesizing an entity resolution dataset using the method of claim 1, characterized in that: include: An acquisition module is used to acquire an entity resolution dataset and calculate a similarity vector distribution of a real dataset based on the entity resolution dataset; wherein the entity resolution dataset includes two real datasets that require entity resolution and labels indicating whether entities in the two datasets match; A generation module, configured to generate a false entity according to the entity resolution data set, and to start generation of the entire entity resolution data set according to the false entity; the entire entity resolution data set is the false entity resolution data set; A judgment module is used to calculate the similarity vector of the unlabeled entity pairs in the generated entity resolution data set, calculate the first posterior probability distribution of the similarity vector and the second posterior probability distribution of the similarity vector based on the similarity vector distribution of the real data set, and mark the unlabeled entity pair as a match if the first posterior probability distribution is greater than or equal to the second posterior probability distribution; otherwise, mark it as a mismatch.

Images