A method for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm

Abstract

The invention relates to the technical field of data integration, and discloses a method for automatic fusion of heterogeneous data in data space based on an unsupervised learning clustering algorithm, comprising the following steps: S1. preprocessing heterogeneous data; S2. pattern extraction, obtaining attribute names and attribute values ​​of heterogeneous data; S3. pretraining a word embedding module through corpus; S4. inputting the attribute name into the pretrained word embedding module, and obtaining an attribute name vector; S5. judging whether a source pattern is consistent with a target pattern, if so, proceeding to the next step; if not, inputting the attribute value into a hybrid matcher, and calculating the similarity between the source pattern and the target pattern of the attribute value, judging whether the source pattern matches the target pattern, if so, proceeding to the next step; S6. clustering integration is performed according to the attribute name vector by an unsupervised learning clustering algorithm. The invention solves the problem that the prior art does not consider the heterogeneous synonymy of attribute names, the accuracy of matching cannot be guaranteed, and the problem of value data conflict cannot be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data integration, and more specifically, to an automatic fusion method for heterogeneous data in data space based on an unsupervised learning clustering algorithm. Background Art

[0002] With the transition from IT (Internet technology) to DT (data technology), data is becoming increasingly large in scale, complex in source, and diverse in type and structure, making enterprises face the challenge of data asset management. The "data islands" formed by independent business applications lack relevance due to their closed and isolated nature, making it difficult for data to flow in business applications.

[0003] The core task of data integration is to integrate multi-source heterogeneous data that are interrelated and physically isolated from each other, so that users can access the data in a transparent way. The essence of data integration is pattern matching. In the process of data fusion, it is necessary to find the corresponding relationship between the source pattern and the target pattern, that is, matching, and then convert the data from the source pattern to the target pattern.

[0004] Initially, the task of data conversion between schemas was completed manually by domain experts. This manual solution was time-consuming and labor-intensive. Therefore, people began to study automatic matching technology to improve the efficiency of data conversion. The automated integration of multi-source heterogeneous data has uncertainty in schema mapping. For example, the address field of the source schema may correspond to the home address of the target schema, or it may correspond to the email address. Figure 1As shown in Figure 1, in order to eliminate the uncertainty of similar pattern mapping, a semi-automatic single intermediary pattern is generated in XML or relational database management system. The COMA system improves the semantic mapping between the input patterns of two data sources by combining multiple matchers. Another system, SF (Similarity Flooding), converts the data source model to be matched into a directed labeled graph as the input of the system, generates the mapping between the response nodes of the graph as the output, and then selects the matching subset from the mapping set using appropriate filters according to the specific matching target. The above pattern matching method uses semi-automatic technology to map two patterns at a time, which is not applicable in the scenario of large-scale data sharing. Saleem et al. proposed the PORSCHE scheme, which first converts the data source pattern into a tree structure, and uses tree mining technology and a set of language matchers to complete the process of creating an intermediary pattern tree from a large number of input pattern trees. Inevitably, the goal of this scheme is to generate an intermediary pattern with feedback from domain experts, and it has not achieved true automation. Ding Guohui et al. proposed a multi-pattern matching method based on the DBSCAN algorithm, introduced the vector space model, converted the attributes in the source pattern into points in the vector space, and divided the similar attributes of different patterns into different clusters through clustering. This method can achieve multi-mode matching, but it does not take into account the situation where attribute names are heterogeneous and synonymous, the accuracy of the matching cannot be guaranteed, and it cannot solve the problem of value data conflicts.

[0005] However, the existing technology does not take into account the situation where attribute names are heterogeneous and synonymous, the matching accuracy cannot be guaranteed, and the problem of value data conflicts cannot be solved. Therefore, how to invent an automated fusion method that takes into account attribute name heterogeneity and synonymy, ensures matching accuracy, and solves value data conflicts is a problem that urgently needs to be solved in this technical field. Summary of the invention

[0006] In order to solve the problem that the prior art does not take into account the heterogeneous synonyms of attribute names, the matching accuracy cannot be guaranteed, and the problem of value data conflicts cannot be solved, the present invention provides a method for automatic fusion of heterogeneous data in data space based on an unsupervised learning clustering algorithm, which has the characteristics of accurate matching and can solve the problem of accurate matching.

[0007] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows:

[0008] The method for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm includes the following steps:

[0009] S1. Obtain heterogeneous data and preprocess the heterogeneous data;

[0010] S2. Perform pattern extraction on the preprocessed heterogeneous data to obtain the attribute names and attribute values ​​of the heterogeneous data;

[0011] S3. Obtain corpus and pre-train the word embedding module through the corpus;

[0012] S4. Input the attribute name into the pre-trained word embedding module and obtain the attribute name vector;

[0013] S5. Preselect a target pattern, determine whether the source pattern and the target pattern are consistent, if they are consistent, proceed to the next step, if they are inconsistent, input the attribute value into the hybrid matcher, and calculate the similarity between the source pattern and the target pattern of the attribute value, determine whether the source pattern and the target pattern match, if they match, proceed to the next step;

[0014] S6. Based on the attribute name vector, clustering integration is performed using an unsupervised learning clustering algorithm.

[0015] Preferably, the attribute value includes a vector representation of each value in the attribute, the total number of all values ​​of the attribute, and the total number of each value of the attribute.

[0016] Furthermore, a corpus is obtained and the word embedding module is pre-trained through the corpus, specifically: whether the corpus used for word embedding training includes attribute names, if so, the context word vector of the target vocabulary w(t) in the corpus is represented as {w(tn),…,w(t-1),w(t+1),…,w(t+n)}, and word embedding training is performed through the continuous bag-of-words model; if not, the words are split into continuous string subsets n-grams of length n, and word embedding training is performed through the continuous bag-of-words model based on n-grams.

[0017] Furthermore, the hybrid matcher includes a grammar matcher, a semantic matcher, an instance matcher, and a structure matcher; the grammar matcher performs similarity matching by identifying literally similar character strings; the semantic matcher performs similarity matching through semantic information; the instance matcher performs similarity matching through the statistical distribution of attribute values ​​and semantic information; and the structure matcher performs similarity matching by matching the parent class name of the field and its path.

[0018] Furthermore, the statistical distribution of attribute values ​​is matched with semantic information, specifically: through formula (1):

[0019]

[0020] Calculate the similarity of attribute values ​​and determine the mapping relationship between the source mode and the target mode; where v a is the attribute value vector, v i Represents the vector representation of a value in the attribute, N a Represents the total number of all attribute values, K i Indicates the total number of values ​​for a property.

[0021] Furthermore, the attribute value is input into the hybrid matcher and the similarity between the source mode and the target mode of the attribute value is calculated. The specific steps are as follows:

[0022] A1. Obtain the initial weight σ corresponding to each matcher through formula (2) i (p):

[0023]

[0024] Where x represents the similarity value calculated by each matcher, p is less than 1 and greater than 0, and α is the center point of the sigmoid function;

[0025] A2. The weight of the similarity results of each matcher compared with the similarity results of the total mixed matcher is obtained by formula (3): i :

[0026]

[0027] A3. Through formula (4):

[0028] Sim(x1,x2)=ω1*Sim syn (x1,x2)+ω2*Sim sem (x1,x2)+ω3*Sim val (x1,x2)+ω4*Sim str (x1,x2) (4)

[0029] Calculate the final similarity Sim(x1,x2) of the corresponding attributes of the source pattern x1 and the target pattern x2, where ω1 is the weight of the grammatical matcher, ω2 is the weight of the semantic matcher, ω3 is the weight of the instance matcher, ω4 is the weight of the structure matcher, and ω1+ω2+ω3+ω4=1, Sim syn (x1,x2) is the similarity of the grammar obtained by the grammar matcher, Sim sem (x1, x2) is the semantic similarity obtained by the semantic matcher, Sim val (x1, x2) is the similarity based on attribute values ​​obtained by the instance matcher, Sim str (x1,x2) is the structure-based similarity obtained by the structure matcher.

[0030] Furthermore, clustering integration is performed based on the attribute name vector through an unsupervised learning clustering algorithm, specifically:

[0031] The vector space model is introduced, and the attribute name vectors in the source mode and the target mode are reduced to data points in the two-dimensional vector space through the PCA algorithm; according to the integration scale of the preprocessed data, the affinity propagation algorithm or the DBSCAN algorithm is used for clustering integration.

[0032] Furthermore, an affinity propagation algorithm is used, which is as follows: all data points are regarded as potential cluster centers, and the lines between data points are connected to form a network. The cluster center of each sample is calculated through message passing on each edge in the network. The cluster center is the label of the global pattern finally stored in the data space.

[0033] Furthermore, the DBSCAN algorithm is used for clustering integration. The specific steps are as follows:

[0034] K1. Select a core object without a category as the basis, the core object is a point in any space, and its domain contains at least m points;

[0035] K2. Find a sample set with density reachable to the basic data point to form a cluster;

[0036] K3. Find the next core object and repeat step K1 until all core objects of heterogeneous data have categories.

[0037] The system for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm comprises a preprocessing module, a pattern extraction module, a word embedding module, a hybrid matcher and a pattern clustering module; the preprocessing module preprocesses the acquired heterogeneous data; the pattern extraction module performs pattern extraction on the preprocessed heterogeneous data and obtains the attribute names and attribute values ​​of the heterogeneous data; the word embedding module performs word embedding on the obtained attribute names and obtains attribute name vectors; the hybrid matcher calculates the similarity between the source pattern and the target pattern of the heterogeneous data through the attribute values ​​and determines whether the source pattern and the target pattern match; the pattern clustering module clusters and integrates the attribute names using unsupervised learning clustering technology.

[0038] The beneficial effects of the present invention are as follows:

[0039] The present invention extracts patterns from pre-processed heterogeneous data to obtain attribute names and attribute values ​​of the heterogeneous data; inputs attribute names into a pre-trained word embedding module to obtain attribute name vectors; calculates the similarity between a source pattern and a target pattern of the attribute value through a hybrid matcher to determine whether the source pattern and the target pattern match; after confirming the match, the present invention clusters and integrates the attribute names through an unsupervised learning clustering technology, thereby achieving accurate mapping between the source pattern and the target pattern, and aggregates multiple patterns through a clustering algorithm, thereby solving the problem that the prior art does not consider heterogeneous synonymy of attribute names, the matching accuracy cannot be guaranteed, and the problem of value data conflict cannot be solved, and the present invention has the characteristics of accurate matching and can solve the problem of accurate matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a data integration diagram of the mediation mode.

[0041] Figure 2 It is a flowchart of a method for automatic fusion of heterogeneous data in data space based on an unsupervised learning clustering algorithm.

[0042] Figure 3 It is a block diagram of data integration in data space in a method for automatic fusion of heterogeneous data in data space based on an unsupervised learning clustering algorithm.

[0043] Figure 4 It is a schematic diagram of the continuous bag-of-words model in Example 2.

[0044] Figure 5 It is a schematic diagram of the continuous bag-of-words model based on n-gram in Example 2.

[0045] Figure 6 This is a schematic diagram of the difference between the similarity measurement methods of the semantic matcher and the clustering algorithm in Example 2. DETAILED DESCRIPTION

[0046] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Example 1

[0048] like Figure 2 As shown, the method for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm includes the following steps:

[0049] S1. Obtain heterogeneous data and preprocess the heterogeneous data;

[0050] S2. Perform pattern extraction on the preprocessed heterogeneous data to obtain the attribute names and attribute values ​​of the heterogeneous data;

[0051] S3. Obtain corpus and pre-train the word embedding module through the corpus;

[0052] S4. Input the attribute name into the pre-trained word embedding module and obtain the attribute name vector;

[0053] S5. Preselect a target pattern, determine whether the source pattern and the target pattern are consistent, if they are consistent, proceed to the next step, if they are inconsistent, input the attribute value into the hybrid matcher, and calculate the similarity between the source pattern and the target pattern of the attribute value, determine whether the source pattern and the target pattern match, if they match, proceed to the next step;

[0054] S6. Based on the attribute name vector, clustering integration is performed using an unsupervised learning clustering algorithm.

[0055] Example 2

[0056] like Figure 2 , Figure 3 As shown, the method for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm includes the following steps:

[0057] S1. Obtain heterogeneous data and preprocess the heterogeneous data;

[0058] S2. Perform pattern extraction on the preprocessed heterogeneous data to obtain the attribute names and attribute values ​​of the heterogeneous data;

[0059] S3. Obtain corpus and pre-train the word embedding module through the corpus;

[0060] S4. Input the attribute name into the pre-trained word embedding module and obtain the attribute name vector;

[0061] S5. Preselect a target pattern, determine whether the source pattern and the target pattern are consistent, if they are consistent, proceed to the next step, if they are inconsistent, input the attribute value into the hybrid matcher, and calculate the similarity between the source pattern and the target pattern of the attribute value, determine whether the source pattern and the target pattern match, if they match, proceed to the next step; in step S2, the preprocessed heterogeneous data is subjected to pattern extraction, the corresponding pattern metadata is extracted, and a csv file is generated; the metadata is the attribute name and attribute value;

[0062] S6. Based on the attribute name vector, clustering integration is performed using an unsupervised learning clustering algorithm.

[0063] In this embodiment, the source mode of heterogeneous data is diverse, including two parts: value data and metadata. Value data is the value of data, that is, the content, and metadata is the description data of data, which identifies the context of the value. In a relational database, metadata is the library name, table name, and field name; in a non-relational database, metadata is the key or the label name of each layer; in semi-structured data such as text, metadata is the abstract and summarized keyword information;

[0064] In a specific embodiment, the attribute value includes a vector representation of each value in the attribute, the total number of all values ​​of the attribute, and the total number of each value of the attribute.

[0065] In a specific embodiment, in the process of data integration, matching is required through semantic similarity; in the process of calculating the semantic similarity of pattern attributes, previous work usually uses WordNet, which is an English dictionary based on cognitive linguistics designed by the Cognitive Science Laboratory of Princeton University, but this method has limited coverage and is not flexible enough; in recent years, word embeddings trained by deep learning neural networks have demonstrated their powerful capabilities in traditional NLP tasks, among which common word embedding methods such as word2vec require input corpus for model training.

[0066] In a specific embodiment, a corpus is obtained and a word embedding module is pre-trained using the corpus, specifically: whether the corpus used for word embedding training includes attribute names is determined; if so, the context word vector of the target vocabulary w(t) in the corpus is represented as {w(tn),…,w(t-1),w(t+1),…,w(t+n)}, and word embedding training is performed using a continuous bag-of-words model; if not, the words are split into continuous string subsets n-grams of length n, and word embedding training is performed using a continuous bag-of-words model based on n-grams.

[0067] like Figure 4 As shown, in a specific embodiment, the context word vector of the target vocabulary w(t) in the corpus is represented as {w(tn), ..., w(t-1), w(t+1), ..., w(t+n)}, and the extracted metadata is trained for word embedding through the continuous bag-of-words model, specifically:

[0068] A1. Input the context word vector into the continuous bag-of-words model of the context prediction target vocabulary, wherein the context word vector is a one-hot encoded V-dimensional vector;

[0069] A2. Connect the context word vector to the hidden layer of the continuous bag-of-words model through a V*N dimensional weight matrix, and connect it from the hidden layer to the output layer of the continuous bag-of-words model through an N*V dimensional weight matrix to obtain the vector of the target word.

[0070] like Figure 5As shown, in a specific embodiment, the words in the corpus are split into continuous string subsets n-grams of length n, and word embedding training is performed through a continuous bag-of-words model based on n-grams, specifically: the training granularity of the context word vector is reduced from the word level to the character level, the training of the vocabulary word vector is represented as the superposition of the character-level n-gram vector of the word, the attribute semantic similarity of the pattern is obtained, and matching is performed according to the attribute semantic similarity.

[0071] In a specific embodiment, the hybrid matcher includes a grammar matcher, a semantic matcher, an instance matcher, and a structure matcher; the grammar matcher performs similarity matching by identifying literally similar character strings; the semantic matcher performs similarity matching through semantic information; the instance matcher performs similarity matching through the statistical distribution of attribute values ​​and semantic information; the structure matcher performs similarity matching by matching the parent class name of the field and its path.

[0072] In a specific embodiment, the four basic matchers, namely, syntax matcher, semantic matcher, instance matcher, and structure matcher, have poor effects when used alone. For example, the syntax matcher can only recognize two strings that are literally similar, the value type of the attribute of the instance matcher needs to be a string type and the distribution of the values ​​cannot be too sparse. The structure matcher mainly solves problems such as the incorrect matching caused by the fact that the table names of the source schema and the target schema do not correspond but the attributes in the table match in a relational database. Therefore, this solution comprehensively considers the degree of influence of the above matchers on the matching results.

[0073] In a specific embodiment, the statistical distribution of attribute values ​​is matched with the semantic information, specifically: by formula (1):

[0074]

[0075] Calculate the similarity of attribute values ​​and determine the mapping relationship between the source mode and the target mode; where v a is the attribute value vector, v i Represents the vector representation of a value in the attribute, N a Represents the total number of all attribute values, K i Indicates the total number of values ​​for a property.

[0076] In a specific embodiment, the attribute value is input into the hybrid matcher, and the similarity between the source mode and the target mode of the attribute value is calculated, and the specific steps are as follows:

[0077] A1. Through formula (2):

[0078]

[0079] Get the initial weight σ corresponding to each matcher i (p), where x represents the similarity value calculated by each matcher, p is less than 1 and greater than 0, and α is the center point of the sigmoid function;

[0080] A2. The weight of the similarity results of each matcher compared with the similarity results of the total mixed matcher is obtained by formula (3): i :

[0081]

[0082] A3. Through formula (4):

[0083] Sim(x1,x2)=ω1*Sim syn (x1,x2)+ω2*Sim sem (x1,x2)+ω3*Sim val (x1,x2)+ω4*Sim str (x1,x2) (4)

[0084] Calculate the final similarity Sim(x1,x2) of the corresponding attributes of the source pattern x1 and the target pattern x2, where ω1 is the weight of the grammatical matcher, ω2 is the weight of the semantic matcher, ω3 is the weight of the instance matcher, ω4 is the weight of the structure matcher, and ω1+ω2+ω3+ω4=1, Sim syn (x1,x2) is the similarity of the grammar obtained by the grammar matcher, Sim sem (x1, x2) is the semantic similarity obtained by the semantic matcher, Sim val (x1, x2) is the similarity based on attribute values ​​obtained by the instance matcher, Sim str (x1,x2) is the structure-based similarity obtained by the structure matcher.

[0085] In a specific embodiment, this embodiment proposes to solve the problem of discovering semantic correspondences between multi-modal attributes through unsupervised learning clustering technology, and select the mode attribute name closest to the cluster center as the label of the global mode, so that the heterogeneous data sources added subsequently can be transformed into the global mode.

[0086] In a specific embodiment, clustering integration is performed according to the attribute name vector by using an unsupervised learning clustering algorithm, specifically:

[0087] The vector space model is introduced, and the attribute name vectors in the source and target patterns are reduced to data points in the two-dimensional vector space through the PCA algorithm; according to the integration scale of the preprocessed data, the affinity propagation algorithm or the DBSCAN algorithm is used for clustering integration. When the integration scale of the data is small, the affinity propagation algorithm is used for clustering integration, and when the integration scale of the data is large, the DBSCAN algorithm is used for clustering integration.

[0088] In a specific embodiment, an affinity propagation algorithm is used, specifically: all data points are regarded as potential cluster centers, the lines between the data points are connected to form a network, and the cluster center of each sample is calculated through message passing on each edge in the network. The cluster center is the label of the global pattern finally stored in the data space.

[0089] In a specific embodiment, the DBSCAN algorithm uses density as a measure of the differences between clusters and can mine clusters of any shape. At the same time, the clustering process is independent of the input order of data and has the ability to handle outliers.

[0090] In a specific embodiment, the DBSCAN algorithm is used for clustering integration, and the specific steps are:

[0091] K1. Select a core object without a category as the basis, the core object is a point in any space, and its domain contains at least m points;

[0092] K2. Find a sample set with density reachable to the basic data point to form a cluster;

[0093] K3. Find the next core object and repeat step K1 until all core objects of heterogeneous data have categories.

[0094] like Figure 6 As shown, in a specific embodiment, semantic information is involved in both the semantic matcher and the clustering algorithm, but the similarity metrics of the two are slightly different:

[0095] The commonly used distance metric in clustering algorithms is Euclidean distance, and the specific formula is:

[0096]

[0097] Where x and y represent two points in the vector space. Euclidean distance is interpreted as the length of the line segment connecting two points. It reflects the absolute difference in value. The closer the distance, the more similar they are. The clustering task is to find a set of similar points. It is necessary to reflect the absolute difference between clusters. After testing, it is found that the Euclidean distance is more efficient as a measure of similarity.

[0098] The semantic matcher, for the task of judging the matching between the source pattern and the target pattern, needs to reflect the relative difference between them. It uses cosine similarity as the metric, and its calculation formula is:.

[0099]

[0100] Cosine similarity reflects the cosine of the angle between two vectors. The smaller the angle, the greater the similarity between the two vectors.

[0101] Example 3

[0102] The method for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm includes the following steps:

[0103] S1. Obtain heterogeneous data and pre-process the heterogeneous data. In this embodiment, the data to be matched is loaded into the system through the open API interface of the source data, and each mode is checked in turn to see whether there are null values ​​or abnormal data. When it is found that the data is incomplete, it is necessary to fill in the data with values ​​or discard it according to the actual storage requirements.

[0104] S2. Perform pattern extraction on the preprocessed heterogeneous data to obtain the attribute names and attribute values ​​of the heterogeneous data; in this embodiment, it is necessary to extract complete data attribute names from each data source and store them in a file, which can be in txt or csv format.

[0105] S3. Obtain corpus and pre-train the word embedding module through the corpus; in this embodiment, a corpus package related to the storage data field is downloaded from Wikipedia, and the fast text method is selected from the open source gensim library to input the corpus for training the word embedding module. If the corpus package is large enough, the model training time will be relatively long; obtain value data from the data source, obtain the distribution of data values ​​by traversing data statistics, and generate a corresponding statistical table.

[0106] S4. Input the attribute name into the pre-trained word embedding module and obtain the attribute name vector; in this embodiment, the word embedding module imports the above-mentioned data attribute name file, converts the attribute name into a high-dimensional vector space, and reduces the dimension through principal component analysis to obtain points in two-dimensional space. The distance between points implies the similarity between attributes.

[0107] S5. Preselect a target pattern, determine whether the source pattern and the target pattern are consistent, if they are consistent, proceed to the next step, if they are inconsistent, input the attribute value into the hybrid matcher, and calculate the similarity between the source pattern and the target pattern of the attribute value, determine whether the source pattern and the target pattern match, if they match, proceed to the next step;

[0108] S6. Based on the attribute name vector, clustering integration is performed using an unsupervised learning clustering algorithm.

[0109] In a specific embodiment, the attribute value includes a vector representation of each value in the attribute, the total number of all values ​​of the attribute, and the total number of each value of the attribute.

[0110] In a specific embodiment, a corpus is obtained and a word embedding module is pre-trained using the corpus, specifically: whether the corpus used for word embedding training includes attribute names is determined; if so, the context word vector of the target vocabulary w(t) in the corpus is represented as {w(tn),…,w(t-1),w(t+1),…,w(t+n)}, and word embedding training is performed using a continuous bag-of-words model; if not, the words are split into continuous string subsets n-grams of length n, and word embedding training is performed using a continuous bag-of-words model based on n-grams.

[0111] In a specific embodiment, the hybrid matcher includes a grammar matcher, a semantic matcher, an instance matcher, and a structure matcher; the grammar matcher performs similarity matching by identifying literally similar character strings; the semantic matcher performs similarity matching through semantic information; the instance matcher performs similarity matching through the statistical distribution of attribute values ​​and semantic information; the structure matcher performs similarity matching by matching the parent class name of the field and its path.

[0112] In this embodiment, if the attribute names in the source pattern and the target pattern are inconsistent, a matching pair of the source pattern and the target pattern is formed, and the matching pairs are respectively input into the syntax matcher, the semantic matcher, the numerical matcher, and the structural matcher to obtain corresponding similarity values.

[0113] In a specific embodiment, the statistical distribution of attribute values ​​is matched with the semantic information, specifically: by formula (1):

[0114]

[0115] Calculate the similarity of attribute values ​​and determine the mapping relationship between the source mode and the target mode; where v a is the attribute value vector, v i Represents the vector representation of a value in the attribute, N a Represents the total number of all attribute values, K i Indicates the total number of values ​​for a property.

[0116] In a specific embodiment, the attribute value is input into the hybrid matcher, and the similarity between the source mode and the target mode of the attribute value is calculated, and the specific steps are as follows:

[0117] A1. Obtain the initial weight σ corresponding to each matcher through formula (2) i (p):

[0118]

[0119] Where x represents the similarity value calculated by each matcher, p is less than 1 and greater than 0, and α is the center point of the sigmoid function;

[0120] A2. The weight of the similarity results of each matcher compared with the similarity results of the total mixed matcher is obtained by formula (3): i :

[0121]

[0122] A3. Through formula (4):

[0123] Sim(x1,x2)=ω1*Sim syn (x1,x2)+ω2*Sim sem (x1,x2)+ω3*Sim val (x1,x2)+ω4*Sim str (x1,x2) (4))

[0124] Calculate the final similarity Sim(x1,x2) of the corresponding attributes of the source pattern x1 and the target pattern x2, where ω1 is the weight of the grammatical matcher, ω2 is the weight of the semantic matcher, ω3 is the weight of the instance matcher, ω4 is the weight of the structure matcher, and ω1+ω2+ω3+ω4=1, Sim syn (x1,x2) is the similarity of the grammar obtained by the grammar matcher, Sim sem (x1, x2) is the semantic similarity obtained by the semantic matcher, Sim val (x1, x2) is the similarity based on attribute values ​​obtained by the instance matcher, Sim str (x1,x2) is the structure-based similarity obtained by the structure matcher.

[0125] In this embodiment, when the final similarity is lower than the set threshold, it indicates that the matching pair is invalid.

[0126] In this embodiment, during the mutual mapping process, valid matching pairs also need to detect whether the values ​​conflict, to ensure the accuracy and reliability of data fusion. The conflict resolution strategy mainly determines valid data through multi-mode voting and data source confidence.

[0127] In a specific embodiment, clustering integration is performed according to the attribute name vector by using an unsupervised learning clustering algorithm, specifically:

[0128] The vector space model is introduced, and the attribute name vectors in the source mode and the target mode are reduced to data points in the two-dimensional vector space through the PCA algorithm; according to the integration scale of the preprocessed data, the affinity propagation algorithm or the DBSCAN algorithm is used for clustering integration.

[0129] In a specific embodiment, an affinity propagation algorithm is used, specifically: all data points are regarded as potential cluster centers, the lines between the data points are connected to form a network, and the cluster center of each sample is calculated through message passing on each edge in the network. The cluster center is the label of the global pattern finally stored in the data space.

[0130] In a specific embodiment, the DBSCAN algorithm is used for clustering integration, and the specific steps are:

[0131] K1. Select a core object without a category as the basis, the core object is a point in any space, and its domain contains at least m points;

[0132] K2. Find a sample set with density reachable to the basic data point to form a cluster;

[0133] K3. Find the next core object and repeat step K1 until all core objects of heterogeneous data have categories.

[0134] In this embodiment, vectorized attribute names, assuming the number is M, are combined into an M*2-dimensional matrix and input into the clustering module. The appropriate clustering algorithm is automatically selected according to the size of M to generate a global pattern of the data and store it in the data space.

[0135] The present invention extracts patterns from pre-processed heterogeneous data to obtain attribute names and attribute values ​​of the heterogeneous data; inputs attribute names into a pre-trained word embedding module to obtain attribute name vectors; calculates the similarity between a source pattern and a target pattern of the attribute value through a hybrid matcher to determine whether the source pattern and the target pattern match; after confirming the match, the present invention clusters and integrates the attribute names through an unsupervised learning clustering technology, thereby achieving accurate mapping between the source pattern and the target pattern, and aggregates multiple patterns through a clustering algorithm, thereby solving the problem that the prior art does not consider heterogeneous synonymy of attribute names, the matching accuracy cannot be guaranteed, and the problem of value data conflict cannot be solved, and the present invention has the characteristics of accurate matching and can solve the problem of accurate matching.

[0136] Example 4

[0137] The system for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm comprises a preprocessing module, a pattern extraction module, a word embedding module, a hybrid matcher and a pattern clustering module; the preprocessing module preprocesses the acquired heterogeneous data; the pattern extraction module performs pattern extraction on the preprocessed heterogeneous data and obtains the attribute names and attribute values ​​of the heterogeneous data; the word embedding module performs word embedding on the obtained attribute names and obtains attribute name vectors; the hybrid matcher calculates the similarity between the source pattern and the target pattern of the heterogeneous data through the attribute values ​​and determines whether the source pattern and the target pattern match; the pattern clustering module clusters and integrates the attribute names using unsupervised learning clustering technology.

[0138] Obviously, the above embodiments of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm, characterized by: The following steps are involved: S1. Obtain heterogeneous data and preprocess the heterogeneous data; S2. Perform pattern extraction on the preprocessed heterogeneous data to obtain the attribute names and attribute values ​​of the heterogeneous data; S3. Obtain corpus and pre-train the word embedding module through the corpus; S4. Input the attribute name into the pre-trained word embedding module and obtain the attribute name vector; S5. Preselect a target pattern, determine whether the source pattern and the target pattern are consistent, if they are consistent, proceed to the next step, if they are inconsistent, input the attribute value into the hybrid matcher, and calculate the similarity between the source pattern and the target pattern of the attribute value, determine whether the source pattern and the target pattern match, if they match, proceed to the next step; The hybrid matcher includes a grammar matcher, a semantic matcher, an instance matcher, and a structure matcher; the grammar matcher performs similarity matching by identifying strings with similar literals; the semantic matcher performs similarity matching by semantic information; the instance matcher performs similarity matching by statistical distribution of attribute values ​​and semantic information; the structure matcher performs similarity matching by matching the parent class name of the field and its path; Matching is done through the statistical distribution of attribute values ​​and semantic information, specifically: through formula (1): Calculate the similarity of attribute values ​​and determine the mapping relationship between the source pattern and the target pattern; where v a is the attribute value vector, v i Represents the vector representation of a value in the attribute, N a Represents the total number of all attribute values, K i Indicates the total number of values ​​of a certain attribute; Input the attribute value into the hybrid matcher and calculate the similarity between the source mode and the target mode of the attribute value. The specific steps are as follows: A1. Obtain the initial weight σ corresponding to each matcher through formula (2) i (p): Where x represents the similarity value calculated by each matcher, p is less than 1 and greater than 0, and α is the center point of the sigmoid function; A2. The weight of the similarity results of each matcher compared with the similarity results of the total mixed matcher is obtained by formula (3): i : A3. Through formula (4): Sim(x1,x2)=ω1*Sim syn (x1,x2)+ω2·Sim sem (x1,x2)+ω3*Sim val (x1,x2)+ω4*Sim str (x1,x2) (4) Calculate the final similarity Sim(x1,x2) of the corresponding attributes of the source pattern x1 and the target pattern x2; where ω1 is the weight of the grammatical matcher, ω2 is the weight of the semantic matcher, ω3 is the weight of the instance matcher, ω4 is the weight of the structure matcher, and ω1+ω2+ω3+ω4=1, Sim syn (x1,x2) is the similarity of the grammar obtained by the grammar matcher, Sim sem (x1, x2) is the semantic similarity obtained by the semantic matcher, Sim val (x1, x2) is the similarity based on attribute values ​​obtained by the instance matcher, Sim str (x1, x2) is the structure-based similarity obtained by the structure matcher; S6. Based on the attribute name vector, clustering integration is performed using an unsupervised learning clustering algorithm.

2. The method for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm according to claim 1, characterized in that: The attribute value includes a vector representation of each value in the attribute, the total number of all attribute values, and the total number of each attribute value.

3. The method for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm according to claim 1, characterized in that: Obtain corpus and use the corpus to pre-train the word embedding module, specifically: determine whether the corpus used for word embedding training includes attribute names. If so, represent the context word vector of the target vocabulary w(t) in the corpus as {w(tn),…,w(t-1),w(t+1),…,w(t+n)}, and perform word embedding training through the continuous bag-of-words model; if not, split the word into continuous string subsets n-grams of length n, and perform word embedding training through the continuous bag-of-words model based on n-gram.

4. The method for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm according to claim 2, characterized in that: According to the attribute name vector, clustering integration is performed through unsupervised learning clustering algorithm, specifically: The vector space model is introduced, and the attribute name vectors in the source mode and the target mode are reduced to data points in the two-dimensional vector space through the PCA algorithm; according to the integration scale of the preprocessed data, the affinity propagation algorithm or the DBSCAN algorithm is used for clustering integration.

5. The method for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm according to claim 4 is characterized by: The affinity propagation algorithm is used, specifically: all data points are regarded as potential cluster centers, the lines between data points are connected to form a network, and the cluster center of each sample is calculated through message passing on each edge in the network. The cluster center is the label of the global pattern finally stored in the data space.

6. The method for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm according to claim 5, characterized in that: The DBSCAN algorithm is used for clustering integration. The specific steps are: K1. Select a core object without a category as the basis, the core object is a point in any space, and its domain contains at least m points; K2. Find a sample set with density reachable to the basic data point to form a cluster; K3. Find the next core object and repeat step K1 until all core objects of heterogeneous data have categories.

7. A system for automatic fusion of heterogeneous data in data space based on unsupervised learning clustering algorithm, characterized by: A method for automatically fusing heterogeneous data in a data space as described in any one of claims 1 to 6, comprising a preprocessing module, a pattern extraction module, a word embedding module, a hybrid matcher, and a pattern clustering module; The preprocessing module preprocesses the acquired heterogeneous data; the pattern extraction module extracts patterns from the preprocessed heterogeneous data and obtains attribute names and attribute values ​​of the heterogeneous data; the word embedding module embeds words on the obtained attribute names and obtains attribute name vectors; the hybrid matcher calculates the similarity between the source pattern and the target pattern of the heterogeneous data through the attribute values ​​and determines whether the source pattern and the target pattern match; the pattern clustering module clusters and integrates the attribute names using unsupervised learning clustering technology.

Images