Dark data semantic association method and system based on structured representation

Abstract

This invention relates to the fields of data governance and cross-institutional data fusion technology, and discloses a method and system for semantic association of dark data based on structured representation. The method includes: extracting column feature vectors from multi-source data, introducing a sliding window between columns to construct a value domain map, and obtaining column embedding vectors through singular value decomposition; constructing a constraint relationship matrix by combining the reference embedding vectors, generating a manifold projection vector through dimensionality reduction using a projection objective function, and performing unsupervised clustering to output labeled embedding vectors; calculating the migration cost to construct a cost matrix and solving the probability transition matrix; extracting column matching pairs to construct a feature alignment dictionary and outputting a reconstructed data table. This invention eliminates local feature confusion caused by high-dimensional overlapping distributions, effectively mines the contextual topological relationships between columns, and constructs a globally optimal mapping.

Description

Technical Field

[0001] This invention relates to the fields of data governance and cross-institutional data fusion technology, specifically to a method and system for semantic association of dark data based on structured representation. Background Technology

[0002] With the development of big data and cross-domain joint computing technologies, heterogeneous system data governance and cross-institutional data fusion systems typically adopt semantic matching methods based on data table headers, field names, or metadata dictionaries to achieve feature alignment and reconstruction of multi-source structured data.

[0003] In the process of aligning features of multi-source structured data, the data governance system needs to use natural language processing technology to calculate the semantic similarity of the header metadata of each heterogeneous data table, or manually formulate static field mapping rules for specific data interfaces, so as to complete the construction of alignment semantics and the reconstruction of heterogeneous data tables.

[0004] However, when faced with dark data scenarios that lack table header definitions, have meaningless field names, or are deliberately obfuscated, the traditional static semantic parsing logic based on table headers becomes completely ineffective because the system has completely lost explicit metadata anchors. In this environment, relying solely on the underlying value distribution of data columns for independent reverse derivation can easily lead to severe confusion of local features in the high-dimensional feature space due to the varying value distribution patterns and significant overlap across different business scenarios. Furthermore, independent column value comparisons erase the multi-dimensional contextual topological relationships between columns within the data table. When transferring local features to the global alignment stage, the loss of multi-dimensional structural information during dimensionality reduction results in combinatorial explosion-style mismatches. This prevents the system from constructing a uniquely deterministic globally optimal mapping in an unsupervised environment without prior manual annotation. Moreover, after completing the mapping and alignment of heterogeneous features, there is a lack of effective means to overcome the barriers of inconsistent field numbers and orders to quickly filter data tables with similar structures, making it even more difficult to accurately mine cross-table similar data entities pointing to the same physical object at the row level. Therefore, how to eliminate local confusion caused by high-dimensional distribution overlap, effectively mine and utilize the inter-column contextual topological relationships within the data table, construct a globally topologically isomorphic optimal mapping, and then accurately identify and output cross-table similar data sets in massive heterogeneous tables are key technical problems that urgently need to be solved in this field. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a dark data semantic association method based on structured representation. This method involves extracting data value sets, concatenating column feature vectors, and introducing a sliding window between columns to construct a value domain map to generate column embedding vectors. A constraint relationship matrix is ​​then constructed using a reference embedding vector, and a projection objective function is employed to reduce dimensionality and cluster the output labeled embedding vectors. Furthermore, optimal transfer theory is used to calculate migration costs, construct a cost matrix, and derive a probability transition matrix. This achieves the effects of eliminating local feature confusion caused by overlapping high-dimensional distributions, fully exploring the contextual topological relationships between columns within the data table, and deterministically reconstructing the data table based on the extracted column matching.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a method for semantic association of dark data based on structured representation, including:

[0008] The system acquires multi-source data and extracts data value sets from numerical and character columns, concatenates them to obtain column feature vectors, introduces a sliding window between columns based on column feature vectors to calculate the co-occurrence probability between columns, constructs and outputs a value range map, performs random walk sampling based on the value range map to construct a node co-occurrence matrix, and performs singular value decomposition on the node co-occurrence matrix to obtain column embedding vectors.

[0009] Obtain the reference embedding vector and construct a joint feature matrix by combining the column embedding vectors. Construct a constraint relation matrix using the joint feature matrix. Set the projection objective function based on the constraint relation matrix. Perform dimensionality reduction projection on the joint feature matrix based on the projection objective function to obtain the manifold projection vector. Perform unsupervised clustering on the manifold projection vector to obtain the connected clusters and noise point sets. Assign cluster labels to the connected clusters and output the labeled embedding vectors.

[0010] The migration cost is calculated based on the labeled embedding vectors. The cost matrix is ​​constructed and output using the migration cost. A kernel matrix is ​​constructed based on the cost matrix. The kernel matrix is ​​alternately updated and element-level reconstructed to obtain the probability transition matrix. Column matching pairs are extracted from the probability transition matrix. A feature alignment dictionary is constructed based on the column matching pairs and an alignment reconstruction operation is performed to output the reconstructed data table.

[0011] Furthermore, the method for obtaining the column feature vector includes:

[0012] The proportion of numerical elements in a set of statistical data values ​​that can be losslessly parsed into numerical values ​​is obtained.

[0013] When the proportion of numerical elements exceeds the preset type determination threshold, the corresponding numerical or character column is determined as the target numerical column, and the skewness, kurtosis, and Gaussian mixture model feature vector of the target numerical column are calculated.

[0014] When the proportion of the numerical elements is less than or equal to the preset type determination threshold, the corresponding numerical column or character column is determined as the target character column, and the regular contour code and character-level frequency vector are extracted from the target character column.

[0015] The skewness, kurtosis, Gaussian mixture model feature vectors, regular contour codes, and character-level frequency vectors are concatenated in a fixed order, and zeros are padded at the positions of missing features to output column feature vectors.

[0016] Furthermore, the method for calculating the eigenvectors of the Gaussian mixture model includes:

[0017] All elements in the target numerical column that can be losslessly parsed into numerical values ​​are used as fitting samples. A Gaussian mixture model consisting of multiple Gaussian components is preset, and each Gaussian component consists of mixture weights, component mean, and component variance.

[0018] The expectation step and the maximization step are executed alternately to iteratively solve the parameters of the Gaussian mixture model. In the expectation step, the posterior probability of each fitted sample belonging to each Gaussian component is calculated based on the current parameters of each Gaussian component. In the maximization step, the mixture weight, component mean and component variance of each Gaussian component are updated in a weighted manner based on the posterior probability.

[0019] Once the iteration converges, the updated mixture weights, component means, and component variances of all Gaussian components are extracted and concatenated in order to output the Gaussian mixture model feature vector.

[0020] Furthermore, the method for obtaining the regular contour encoding and character-level frequency vector includes:

[0021] The numbers, uppercase letters, and lowercase letters in each string element of the target character column are uniformly mapped and replaced character by character to generate morphological template strings. The top eight morphological template strings with the highest frequency are counted, and the normalized occurrence frequency of each morphological template string is calculated. The morphological template strings and their normalized occurrence frequencies are combined to form a regular contour code.

[0022] The string elements are segmented sequentially using a sliding window to extract and summarize consecutive two-character substrings. The top 32 consecutive two-character substrings with the highest frequency are counted, and the normalized frequency of each consecutive two-character substring is calculated. The consecutive two-character substrings and their corresponding normalized frequencies are combined to form a character-level frequency vector.

[0023] Furthermore, the method for obtaining the value range map includes:

[0024] By using a sliding window between columns to slide across multi-source data, the co-occurrence count and chance count are calculated when the values ​​of any two numeric or character columns are not empty placeholders. The co-occurrence count is divided by the chance count to obtain the co-occurrence probability between columns.

[0025] Instantiate a data column node for each numeric and character column, and use the corresponding column feature vector as the node feature vector. Extract the original data value from the data value set and instantiate it as a data value node.

[0026] The frequency percentage is obtained by dividing the frequency of the original data value in the corresponding data value set by the total number of non-empty data rows in the corresponding numeric or character column.

[0027] Using the co-occurrence probability between columns as weights, undirected connecting lines are established between data column nodes as column association edges. Using the frequency proportion as weights, undirected connecting lines are established between data column nodes and data value nodes as value attribution edges, and the value domain map is output.

[0028] Furthermore, the method for obtaining the column embedding vector includes:

[0029] Using each data column node and data value node in the value domain graph as the starting point of the walk, and based on the state transition probability obtained by normalizing the weight of the column association edge or value belonging edge connected to the current node, a random walk sampling with a preset walk step size and a preset number of samplings is performed to generate a walk sequence set containing the node visit sequence.

[0030] Set the sequence context window to slide in the node access sequence, count the original frequency of any two nodes that co-occur, and fill it into the node co-occurrence matrix;

[0031] Calculate the pointwise mutual information value of each node pair to replace the original frequency and generate a pointwise mutual information weighted matrix;

[0032] Perform singular value decomposition on the pointwise mutual information weighted matrix, extract the left singular vector column and scale it proportionally to obtain a low-dimensional dense vector as the column embedding vector.

[0033] Furthermore, the method for constructing the constraint relation matrix includes:

[0034] The reference embedding vector and the column embedding vector are stacked and concatenated row by row to construct a joint feature matrix;

[0035] Two column embedding vectors or reference embedding vectors are randomly extracted from the joint feature matrix to form an embedding vector pair. The set of data values ​​of the corresponding numeric or character columns in each data row is extracted from the multi-source data.

[0036] Map the original data values ​​in the data value set to discrete symbols;

[0037] The joint and independent occurrence probabilities of discrete symbols on the same data row are statistically analyzed to calculate the mutual information entropy, and the Euclidean distance between embedded vector pairs is also calculated.

[0038] Fill all mutual information entropy into a preset first symmetric matrix to form a mutual information entropy submatrix, fill all Euclidean distances into a preset second symmetric matrix to form an Euclidean distance submatrix, concatenate the mutual information entropy submatrix and the Euclidean distance submatrix, and output the constraint relation matrix.

[0039] Furthermore, the method for obtaining the manifold projection vector includes:

[0040] Extract the mutual information entropy and Euclidean distance recorded in the constraint relationship matrix, set the projection objective function, and calculate the median of the mutual information entropy as the mutual information entropy judgment threshold;

[0041] An initial nearest neighbor graph is constructed using each column embedding vector in the joint feature matrix and the reference embedding vector as graph nodes, and the connection probability between graph nodes is determined by Euclidean distance. The connection probability of graph nodes with mutual information entropy greater than the mutual information entropy judgment threshold is set as the highest probability value, and the connection probability of graph nodes with mutual information entropy less than or equal to the mutual information entropy judgment threshold is set as the lowest probability value to complete the adjacency topology transformation.

[0042] Assign initial continuous low-dimensional spatial coordinates to each graph node in the nearest neighbor graph, calculate the gradient of the projection objective function with respect to the low-dimensional spatial coordinates of each graph node, continuously update the low-dimensional spatial coordinates along the gradient descent direction until the preset convergence condition is met, and use the converged low-dimensional spatial coordinates as the manifold projection vector.

[0043] Furthermore, the method for obtaining the labeled embedding vector includes:

[0044] Calculate the Euclidean distances of all manifold projection vectors and set the neighborhood radius;

[0045] Count the number of other manifold projection vectors whose Euclidean distance to any manifold projection vector is less than or equal to the neighborhood radius. If the number is greater than or equal to the minimum neighborhood number, then the manifold projection vector is determined to be the core point.

[0046] When there is a continuous transmission path between two manifold projection vectors consisting of core points, and the Euclidean distance between adjacent vectors on the path is less than or equal to the neighborhood radius, the two are determined to be mutually density-reachable manifold projection vectors.

[0047] Manifold projection vectors that are density-reachable with each other are grouped into the same predefined connected cluster, while isolated manifold projection vectors are assigned to a predefined set of noise points.

[0048] If a connected cluster contains the manifold projection vector corresponding to the reference embedding vector, the reference cluster label with the highest frequency is used as the cluster label; otherwise, an anonymous cluster number is assigned as the cluster label, the determined cluster label is assigned to the corresponding manifold projection vector, and the labeled embedding vector is output.

[0049] Furthermore, the method for constructing the cost matrix includes:

[0050] The source data table and target data table in the multi-source data are counted, as well as the total number of unknown data columns contained therein. The labeled embedding vector corresponding to each unknown data column is used as a support point. The reciprocal of the total number is used as the probability mass assigned to each support point, thus forming the source discrete distribution manifold and the target discrete distribution manifold respectively.

[0051] Calculate the bulldozer distance between the labeled embedding vectors corresponding to each unknown data column pair in the source discrete manifold and the target discrete manifold;

[0052] A cost penalty is imposed on cross-cluster mapping of unknown data columns carrying inconsistent cluster labels, and the bulldozer distance after the penalty is applied is used as the migration cost for feature matching.

[0053] The migration costs of all unknown data pairs are filled into the corresponding intersection positions of a preset two-dimensional real matrix to construct the output cost matrix.

[0054] Furthermore, the method for outputting the reconstructed data table includes:

[0055] The kernel matrix is ​​obtained by dividing the migration cost in the cost matrix by the preset entropy regularization coefficient and taking the negative number, and then performing an exponential operation.

[0056] Construct source quality vector and target quality vector based on probability quality, and initialize left scaling vector and right scaling vector respectively;

[0057] The left scaling vector is updated by dividing the source mass vector by the row sum of the product of the kernel matrix and the right scaling vector, and the right scaling vector is updated by dividing the target mass vector by the column sum of the product of the kernel matrix and the updated left scaling vector. This process is repeated alternately until convergence.

[0058] The probability transition matrix is ​​obtained by performing element-wise multiplication between the converged left and right scaling vectors and the kernel matrix.

[0059] Extract the maximum mapping probability of each row in the probability transition matrix, and combine the corresponding unknown data columns into column matching pairs or mark them as isolated data columns;

[0060] Construct a feature alignment dictionary using column matching pairs, concatenate unknown data columns of the source and target data tables based on the feature alignment dictionary, fill in null values ​​for isolated data columns, and output the reconstructed data table.

[0061] Furthermore, the method for obtaining the similar data set includes:

[0062] Extract the total number of all column matching pairs from the probability transition matrix and divide it by the total number of unknown data columns in the source data table and the target data table respectively to obtain two percentage values. The larger percentage value is taken as the table-level structure similarity.

[0063] If the table-level structural similarity is greater than or equal to the preset table-level similarity threshold, then the source data table and the target data table are determined to be similar structured data tables.

[0064] Get and iterate through the key-value pairs in the feature alignment dictionary, and define the unknown data columns with deterministic mapping relationships as the intersection data columns;

[0065] Extract the column value sequence of the intersection data column from the source data table and the target data table, extract the original data value at the same index position in the same row and perform format standardization processing to obtain the standardized data value;

[0066] A comprehensive similarity metric is calculated using standardized data values. If the comprehensive similarity metric is greater than or equal to a preset entity determination threshold, the corresponding data behavior is determined to be similar data, and all similar data in the approximate structure data table is aggregated into a similar data set.

[0067] Secondly, the present invention provides a dark data semantic association system based on structured representation, which is used to implement the above-mentioned dark data semantic association method based on structured representation, the system comprising:

[0068] The graph embedding module is used to acquire multi-source data and extract the data value sets of numerical and character columns, concatenate them to obtain column feature vectors, introduce a sliding window between columns based on the column feature vectors to count the co-occurrence probability between columns, construct and output the value range graph, perform random walk sampling based on the value range graph to construct the node co-occurrence matrix, and perform singular value decomposition on the node co-occurrence matrix to obtain the column embedding vectors.

[0069] Clustering and labeling module: used to obtain reference embedding vectors, construct a joint feature matrix by combining column embedding vectors, construct a constraint relation matrix using the joint feature matrix, set a projection objective function based on the constraint relation matrix, perform dimensionality reduction projection on the joint feature matrix based on the projection objective function to obtain manifold projection vectors, perform unsupervised clustering on the manifold projection vectors to obtain a set of connected clusters and noise points, assign cluster labels to connected clusters, and output labeled embedding vectors;

[0070] Cost Reconstruction Module: This module calculates the migration cost based on the labeled embedding vectors, constructs and outputs a cost matrix using the migration cost, constructs a kernel matrix based on the cost matrix, performs alternating updates and element-level reconstruction on the kernel matrix to obtain a probability transition matrix, extracts column matching pairs from the probability transition matrix, constructs a feature alignment dictionary based on the column matching pairs and performs alignment reconstruction operations, outputs a reconstructed data table, extracts intersection data columns based on the feature alignment dictionary and performs similarity metric calculations to obtain a similar data set.

[0071] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0072] This invention generates column embedding vectors by extracting column feature vectors and constructing value domain maps, achieving feature construction independent of table header text. This addresses the pain points of static semantic parsing logic failure and severe confusion of local features caused by the loss of explicit metadata anchors. The constraint relation matrix and projection objective function reduce the dimensionality of continuous features and cluster them to output labeled embedding vectors, achieving accurate anchoring from continuous low-dimensional spatial coordinates to discrete business semantic units. This solves the problem of matching ambiguity caused by the lack of discrete semantic anchors during global alignment. Based on the bulldozer distance, a cost matrix is ​​constructed and the probability transition matrix is ​​solved, avoiding the loss of multi-dimensional context structure topology relationship and matching conflicts caused by independent comparison of a single column, thus completing deterministic data table reconstruction. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 A flowchart illustrating the method for semantic association of dark data based on structured representation provided in this embodiment of the invention;

[0075] Figure 2 This is a timing diagram of node access sequence sampling provided in an embodiment of the present invention;

[0076] Figure 3 This is a schematic diagram of a manifold dimensionality reduction projection space based on mutual information constraints provided in an embodiment of the present invention.

[0077] Figure 4 This is a schematic diagram of optimal transmission global bipartite graph alignment provided in an embodiment of the present invention;

[0078] Figure 5 This is a flowchart of feature alignment dictionary generation and structured reconstruction provided in an embodiment of the present invention;

[0079] Figure 6 A functional block diagram of a dark data semantic association system based on structured representation provided in an embodiment of the present invention. Detailed Implementation

[0080] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0081] Example 1

[0082] Please see Figure 1 As shown, this embodiment provides a method for semantic association of dark data based on structured representation, including:

[0083] Step S10: Obtain multi-source data and extract the data value sets of numerical and character columns, concatenate them to obtain column feature vectors, introduce a sliding window between columns based on the column feature vectors to count the co-occurrence probability between columns, construct and output the value range map, perform random walk sampling based on the value range map to construct the node co-occurrence matrix, perform singular value decomposition on the node co-occurrence matrix to obtain the column embedding vector.

[0084] Further, step S10 includes:

[0085] Step S11: Obtain multi-source data, extract the data value set of numeric columns and the data value set of character columns from the multi-source data and concatenate them to obtain the column feature vector.

[0086] In scenarios involving heterogeneous system data governance and cross-organizational data fusion, particularly in dark data processing scenarios requiring cross-source field alignment of data tables from multiple independent business systems, the data tables from each business system are unified in encoding and filled with missing values ​​through a standard data interface adaptation layer before converging into a unified processing pipeline. However, traditional alignment methods generally rely on data table headers, field names, or metadata dictionaries for semantic similarity calculation. Once the header of a data column is encrypted, randomized, or deliberately obfuscated with meaningless characters, the static semantic parsing logic based on the header completely fails because there is no matching literal text on the input side, causing the entire alignment pipeline to break at the source. To construct feature anchors from the data values ​​themselves without reading any header text, multidimensional statistical morphological quantization is performed on the underlying data values ​​of each unknown data column. The aim is to transform dark data tables, which retain only the data value matrix and stripped of all field definitions, into column-level feature representations carrying distribution morphological fingerprints.

[0087] Multi-source data is acquired through a standard data interface adaptation layer. This standard data interface adaptation layer refers to a data preprocessing module deployed at the data exit points of each independent business system, responsible for unifying the encoding of the multi-source raw data and filling in missing values. It masks the heterogeneous differences in character encoding standards and null value representation methods among different business systems, ensuring that subsequent processing links receive data carriers with consistent formats. The multi-source data refers to a two-dimensional data table set exported from multiple independent business systems and aggregated after processing by the standard data interface adaptation layer. Its header rows have been replaced with semantically meaningless placeholder sequences, thus forming a dark data table that retains only the data value matrix while stripping away all field definitions. The multi-source data includes several data tables, each composed of interwoven data rows and columns. Data rows are stacked vertically to record individual data records. Each data record refers to a single, complete business event from the same business system, represented as a row in the data table. Its internal payload is the specific values ​​of that business event across all data fields. Data columns are arranged horizontally to carry the sequence of column values ​​for the same data field across all data rows. A data field is a concept describing a specific dimension of a business event; in this multi-source data, the table header has been replaced with a semantically meaningless placeholder, and it is implicitly identified only by column indexes. The sequence of column values ​​refers to the original set of data values ​​for the same data column arranged in row order across all data rows. Its internal element types include strings, integers, and floating-point numbers, reflecting the distribution of that data field across all data records. By traversing all data columns of each data table in the multi-source data, and extracting the non-null values ​​of any column across all data rows, a set of data values ​​is formed.

[0088] The dominant data type of the elements in a data value set is determined. Specifically, the percentage of elements in the data value set that can be losslessly parsed into numeric values ​​is counted. When the percentage of elements that can be losslessly parsed into numeric values ​​is greater than the type determination threshold, the data column is determined to be a numeric column; otherwise, the data column is determined to be a character column. Elements that can be losslessly parsed into numeric values ​​refer to the original values ​​in the data value set that can be directly converted into integer or floating-point numbers through numeric parsing rules without precision loss or format truncation during the conversion process. This includes pure numeric strings, signed numeric strings, and floating-point format strings containing decimal points, but excludes values ​​containing non-numeric characters, unit symbols, or mixed text. The type determination threshold is a standard industry threshold value, set based on the empirical lower bound of the percentage of legitimate elements in pure numeric columns mixed with a small number of contaminated characters in engineering practice. It can be adjusted according to data quality requirements to avoid a very small number of contaminated characters causing numeric columns to be misjudged as character columns. For example, it is set to 0.9. For data value sets determined to be numeric columns, their skewness (S) and kurtosis (C) are calculated using standard mathematical statistics methods. The skewness measures the degree of asymmetry of the column value sequence relative to the mean, and the kurtosis measures the thickness of the tail of the column value sequence relative to a normal distribution. Its analytical expression is: ; Where i is the traversal index of the numerical element, ranging from 1 to N, and N is the total number of elements in the data set that can be losslessly parsed into numerical values. This indicates that the numeric element at index i is being iterated over. The skewness and kurtosis are the arithmetic mean of all numerical elements in the data column. The construction of these values ​​is based on the probability distribution moment theory, which states that the shape of any one-dimensional distribution can be uniquely characterized by its central moments. By using the skewness obtained by normalizing the third moment and the kurtosis obtained by normalizing the fourth moment, it is possible to capture the two highly discriminative morphological fingerprints of asymmetry and tailing of the data column value sequence without relying on any field semantics.

[0089] The data column is further fitted with a Gaussian mixture model to capture the multimodal distribution structure of the column value sequence at multiple cluster centers. Specifically, all N numerical elements in the data set that can be losslessly parsed into numerical values ​​are used as fitting samples. The Gaussian mixture model is preset to consist of F Gaussian components. The number of components F is set based on the empirical upper bound of the number of main cluster centers presented by the structured data field in most cases, aiming to achieve a controllable balance between fitting ability and overfitting risk. For example, it is set to 3. Each Gaussian component corresponds to an independent local cluster center, used to describe the probability density distribution of the fitted sample near that cluster center. The Gaussian components are identified by component index j, ranging from 1 to F, and each Gaussian component is composed of mixing weights. , component mean Component variance The composition, the hybrid weight This refers to the proportion of samples in the fitted sample that are explained by the j-th Gaussian component, reflecting the relative weight of the j-th cluster center in the overall distribution, and satisfying the following conditions: and ; Component mean This refers to the location of the cluster center characterized by the j-th Gaussian component, i.e., the expected value of the normal distribution corresponding to that component, and the component variance. This refers to the dispersion of the cluster center characterized by the j-th Gaussian component, i.e., the variance of the normal distribution corresponding to this component, reflecting the numerical fluctuation range of the fitted sample near this cluster center. The expectation-maximization algorithm is called to iteratively solve for the parameters of the F Gaussian components. Specifically, before the iteration begins, the parameters of the F Gaussian components are initialized: the initial mixing weights of each Gaussian component are set... All assigned values This indicates that, in the initial state, each cluster center has an equal explanatory weight for the fitted sample. F unique values ​​are randomly selected from the N fitted samples. When N < F, F is adjusted to N, meaning the number of Gaussian components does not exceed the total number of fitted samples. The j-th Gaussian component is assigned its initial mean value according to its component index in ascending order, representing the initial position of each cluster center on the numerical axis. The initial variance of each Gaussian component is... The sample variance is assigned to all N fitted samples. The sample variance is calculated by dividing the sum of the squares of the differences between all fitted samples and the arithmetic mean by the number of fitted samples N. This means that in the initial state, each cluster center covers all fitted samples with the same degree of dispersion.

[0090] In each iteration, the expectation step and the maximization step are executed alternately. The expectation step calculates the posterior probability of each fitted sample belonging to the j-th Gaussian component based on the mixture weights, component mean, and component variance of the current j-th Gaussian component. The posterior probability refers to the conditional probability that the fitted sample is generated by the j-th Gaussian component under the current parameter conditions. It is calculated as follows: The probability density value of the j-th Gaussian component at the current fitted sample is calculated using the mean and variance of the current component; this probability density value is multiplied by the mixing weights; and then divided by the sum of the products of the probability density values ​​of all F Gaussian components at the current fitted sample and their respective mixing weights. The analytical expression is: Where k represents the Gaussian component traversal index of the summation term in the denominator, ranging from 1 to F. It has the same meaning as the component index j in the numerator but is used to traverse the probability density contribution of all F Gaussian components to the fitted sample. The mixed weights for traversing the Gaussian components at index k, and... The same definition applies, representing the relative weight of the k-th Gaussian component in the overall distribution; To iterate over the Gaussian component at index k, the mean of the component is... The definition is the same, representing the position of the k-th cluster center on the numerical axis. To iterate over the variance of the Gaussian component at index k, and... The same definition applies, representing the numerical fluctuation range of the fitted samples near the k-th cluster center, where π is the constant pi. The function is an exponential function with the natural constant e as its base. The maximization step, based on all posterior probabilities obtained from the expectation step, sequentially performs weighted updates on the mixture weights, component mean, and component variance of the j-th Gaussian component. First, the soft allocation of the j-th Gaussian component is calculated. The soft allocation quantity refers to the sum of the posterior probabilities of all N fitted samples for the j-th Gaussian component, reflecting the equivalent number of samples that the j-th Gaussian component actually interprets under the current parameter conditions. Its analytical expression is: Based on the soft allocation, the mean of the j-th Gaussian component is updated to the weighted average of all fitted samples, with each sample having a posterior probability of the j-th Gaussian component as its weight. That is, fitted samples with higher posterior probabilities contribute more to the cluster center position of that component. The analytical expression is: The variance of the j-th Gaussian component is updated to the weighted average of the squares of the differences between all fitted samples and the updated component mean, with the weights being the a posteriori probabilities. This means that fitted samples with higher a posteriori probabilities contribute more to the dispersion of that component. The analytical expression is: The mixing weight of the j-th Gaussian component is updated to the proportion of the soft allocation of that component to the total number of fitted samples, reflecting the relative weight of the j-th cluster center in the overall distribution after the update. The analytical expression is: After updating the weighted average of the mixture weights, component means, and component variances of all F Gaussian components sequentially from component index 1 to F, the maximum absolute value of the changes in the three parameters (mixture weights, component means, and component variances) of each of the F Gaussian components before and after this iteration is calculated. If this maximum value is less than the convergence threshold, the iteration stops; otherwise, the next iteration proceeds with the updated parameters, alternating between the expected and maximize steps. The convergence threshold is set based on an empirical lower bound of numerical accuracy where the parameter update amount tends to stabilize. For example, the convergence threshold is set to... After iterative convergence, the mixture weights, component means, and component variances of all F Gaussian components are extracted and concatenated sequentially according to component indices to form the feature vector of the nine-dimensional Gaussian mixture model.

[0091] For a set of data values ​​identified as character columns, its regular contour encoding and character-level frequency are extracted. Specifically, the regular contour encoding refers to uniformly mapping Arabic numerals to the character 'd', uppercase letters to 'A', lowercase letters to 'a', and other symbol characters to their original form, after which a morphological template string corresponding to the string element is obtained. The morphological template string refers to the equal-length replacement result of abstracting the specific numerical and alphabetical information of each character in the string element, retaining only the character category skeleton, used to characterize the structural arrangement pattern of the string element while masking its specific content differences. The frequency of morphological template strings corresponding to all string elements in the column value sequence of this data column is obtained by counting the number of times they appear. The frequency of morphological template strings refers to the statistical result of the number of times each type of morphological template string appears in all column value sequence elements of this data column, reflecting the degree of distribution concentration of this data column at the character structure skeleton level. The top eight most frequent morphological template strings and their normalized frequencies are used to construct an eight-dimensional regular contour encoding vector. When the total number of different morphological template strings is less than eight, zero-value padding is performed on the insufficient dimensions to ensure that the vector dimension is fixed at eight dimensions. This vector is used to characterize the structural skeleton pattern of the data column value sequence. The character-level frequency extraction process is as follows: all string elements in the data column value sequence are sequentially segmented with a sliding step size of one and a window length of two. Each consecutive two-character substring is extracted. All the binary character substrings obtained from the segmentation of all string elements are summarized and their respective frequencies are counted. The top thirty-two binary character substrings with the highest frequencies and their normalized frequencies are used to construct a thirty-two-dimensional character-level frequency vector. When the total number of different binary character substrings is less than 32, zero-value padding is performed on the insufficient dimensions to ensure that the vector dimension is fixed at 32 dimensions. This vector is used to characterize the local character combination habits of the data column value sequence. The skewness, kurtosis, nine-dimensional Gaussian mixture model feature vector, eight-dimensional regular contour encoding vector, and character-level frequency vector are concatenated in a fixed order. Zeros are padded at character-type feature positions for numerical columns and at numerical feature positions for character-type columns, resulting in the column feature vector corresponding to that data column. The column feature vector is a one-dimensional real number vector with a fixed dimension. The fixed dimension refers to the total dimension after concatenating skewness and kurtosis (each occupying one dimension), the nine-dimensional Gaussian mixture model feature vector (nine dimensions), the eight-dimensional regular contour encoding vector (eight dimensions), and the character-level frequency vector (thirty-two dimensions). This dimension is set based on the sum of the dimensions of all the aforementioned numerical and character-type distribution morphology indicators; for example, it is set to 51.

[0092] Step S12: Based on the column feature vector, introduce a sliding window between columns in the multi-source data to count the co-occurrence probability between columns, and construct and output the value range map.

[0093] After obtaining the column feature vectors, in order to capture the multi-dimensional contextual structure topological relationship between columns within the data table, multi-source data is reused, and a sliding window between columns is introduced in the data row direction of the data table to statistically analyze the inter-column relationships.

[0094] Specifically, the column sliding window refers to a continuous data column interval with a preset window width, measured in units of data columns within the same data row, i.e., a single data record. The window width is set based on empirical values ​​of the local correlation range of adjacent data fields in the data table, and can be appropriately adjusted according to the number of columns in the data table; for example, it is set to 5. The column sliding window slides along the column axis of the data table from the first column to the last column with a step size of one. Each slide covers the current column and the adjacent data columns to the right of the current column, with a number equal to the window width variable minus one. For each data table in the multi-source data, the total number of unknown data columns in the data table is obtained. A co-occurrence count matrix with a dimension equal to the total number multiplied by the total number is initialized, as well as a chance count matrix of the same dimension. The rows and columns of the co-occurrence count matrix and the chance count matrix correspond one-to-one with all the unknown data columns of the data table, and the initial values ​​of the matrix elements are all zero. Traverse all data rows of the data table. For the currently traversed data row, slide the column sliding window from the first column to the last column. Within each window position, define any two unknown data columns covered by the current window as the first unknown data column and the second unknown data column, and assign them the first column index u and the second column index v, respectively. The values ​​of the first column index u and the second column index v are both in the range of 1 to the total number of unknown data columns. Extract the specific values ​​of the first unknown data column corresponding to the first column index u and the second unknown data column corresponding to the second column index v in the data row: If and only if the values ​​of both columns are not empty placeholders, increment the count values ​​of row index u and column index v in both the co-occurrence count matrix and the chance count matrix by one to ensure that both matrices are symmetric; if the value of either column is an empty placeholder, increment only the count value at the same coordinate position in the chance count matrix by one, without increasing the count value of the co-occurrence count matrix; after traversing all data rows of the data table, perform element-wise division on the co-occurrence count matrix and the chance count matrix: for each pair of data columns identified by the first column index and the second column index respectively, divide the corresponding value in the co-occurrence count matrix by the corresponding value in the chance count matrix to obtain the co-occurrence probability matrix. The off-diagonal elements in the co-occurrence probability matrix represent the co-occurrence probabilities between columns, with values ​​between 0 and 1. These probabilities characterize the statistical closeness of the co-observation of valid data for any two data fields within the same data record. Based on the column feature vectors and the co-occurrence probability matrix, a value range graph is constructed. This value range graph is an undirected weighted graph structure model composed of a set of nodes and a set of edges. The node set includes both data column nodes and data value nodes, while the edge set includes both column-related edges and value-attributed edges.The specific construction process is as follows: For each unknown data column in the data table, a data column node is instantiated. A column feature vector containing skewness, kurtosis, a nine-dimensional Gaussian mixture model feature vector, regular contour encoding, and character-level frequency with fixed dimensions is used as the node feature vector for that data column node. Next, the column value sequence of each unknown data column is sorted in descending order of frequency. The most frequent values ​​are extracted, with the number of extracted high-frequency values ​​represented by the high-frequency extraction quantity variable H. A data value node is instantiated for each extracted original data value. Subsequently, an edge set is constructed. An undirected connection line is established between any two data column nodes as a column association edge, and the co-occurrence probability between columns in the co-occurrence probability matrix is ​​extracted as the weight of the column association edge. An undirected connection line is established between each data column node and its H extracted data value nodes as a value attribution edge. The frequency of the original data value in the corresponding column value sequence is divided by the total number of non-empty data rows in the column to obtain the frequency ratio, which is used as the weight of the value attribution edge. Finally, through node instantiation and edge weight assignment operations, the column feature vector and discrete data values ​​are uniformly woven into a value domain map that reflects the internal distribution pattern of the data column and the context topology between columns.

[0095] Step S13: Construct a node co-occurrence matrix by performing random walk sampling based on the value range spectrum, and perform singular value decomposition based on the node co-occurrence matrix to obtain the column embedding vector.

[0096] After obtaining the fixed node feature vectors and the weights of column association edges and value-attribution edges in the value domain graph, in order to compress them into equal-length dense vectors that can be directly consumed, avoid the computational redundancy and dimensionality curse caused by directly inputting high-dimensional sparse graph structures downstream, and at the same time make the final column embedding vector carry both the intra-column distribution pattern and the inter-column topological relationship, graph embedding mapping is performed with all nodes of the graph as the sampling starting point.

[0097] Specifically, each data column node and each data value node in the value domain graph is used as the starting point for the walk, and a random walk sampling is performed with a preset walk step size and a preset number of samplings. The walk step size is set based on the empirical order of the network topology diameter of the value domain graph, for example, 40; the number of samplings is set based on the empirical lower bound to ensure sufficient node sampling, for example, 10. During the random walk sampling process, the weights of each column association edge or value-attribute edge connected to the current node are normalized. Specifically, the weight of the column association edge or value-attribute edge connected to the current node is divided by the sum of the weights of all column association edges and value-attribute edges connected to that node to obtain the state transition probability corresponding to that column association edge or value-attribute edge. The walk with a preset number of samplings and a preset walk step size is repeated from each starting point of the value domain graph, and the nodes visited sequentially along the state transition probability during each walk are recorded to form a node visit sequence. After all walk sampling is completed, the generated node visit sequences constitute a walk sequence set. A node co-occurrence matrix is ​​constructed based on the walk sequence set. A sequence context window is introduced, which has a sequence window width. The sequence window width is set based on empirical values ​​for capturing high-order local proximity relationships between nodes in the walk sequence; for example, it is set to 5. It should be noted that the sequence context window here is different from the column sliding window concept; the former operates on node access sequences, while the latter operates on the column axis direction of the data table. All node access sequences in the walk sequence set are traversed. In each node access sequence, the sequence context window is slid to count any two nodes that co-occur within the sequence window width. These two co-occurring nodes are defined as node pairs, and the co-occurrence frequency of each node pair is counted. The co-occurrence frequency is defined as the original frequency of the node pair and is used as the value of the matrix element to fill the row and column intersection position corresponding to the node pair in the node co-occurrence matrix. Subsequently, the node co-occurrence matrix is ​​transformed into a point-by-point mutual information weighted matrix. Specifically, based on the node co-occurrence matrix, the joint co-occurrence probability and the independent occurrence probability of each node pair are calculated. The joint co-occurrence probability is calculated by dividing the original frequency of the node pair by the sum of the original frequencies of all node pairs in the node co-occurrence matrix. The independent occurrence probability is calculated by summing the original frequencies of all node pairs containing the node and then dividing by the sum of the original frequencies of all node pairs in the node co-occurrence matrix. The joint co-occurrence probability is divided by the product of the two independent occurrence probabilities, and the logarithm of the division is taken. If the logarithm is less than zero, it is set to zero to obtain the pointwise mutual information value of each node pair. The pointwise mutual information value is used to replace the original frequencies in the node co-occurrence matrix to obtain the pointwise mutual information weighted matrix. The singular value decomposition algorithm is called to perform singular value decomposition on the pointwise mutual information weighted matrix to obtain the left singular matrix, the singular value diagonal matrix, and the right singular matrix.The target dimension for dimensionality reduction is set based on an empirical value that balances the sufficiency of node feature representation with computational overhead; for example, it is set to 64. From the left singular matrix, the left singular vector columns corresponding to the largest singular values ​​of the target dimension are extracted. Each left singular vector column is then scaled proportionally to the square root of the corresponding singular value in the singular value diagonal matrix, resulting in low-dimensional dense vectors corresponding to each data column node and each data value node in the value domain graph. From the low-dimensional dense vectors corresponding to each data column node and each data value node obtained from the graph embedding mapping above, only the low-dimensional dense vectors corresponding to all data column nodes are extracted and used as the column embedding vectors for the corresponding unknown data columns. The column embedding vector is a one-dimensional real vector whose numerical load integrates the distribution pattern of the column feature vectors and the inter-column context topology information of the value domain graph. See also. Figure 2 This is a timing diagram of node access sequence sampling provided in an embodiment of the present invention. Figure 2 This example illustrates the process of generating a node visit sequence by performing a random walk sampling in the value domain graph, starting from the initial node, within a two-dimensional coordinate system with the time step as the horizontal axis and node identifiers as the vertical axis. Each solid circle in the graph represents a data column node or data value node visited within the current time step, and the broken line connecting two adjacent solid circles represents the transition path between nodes. By recording the time series generated from multiple samplings, the method demonstrates how the walk path transitions between graph nodes, thus fully capturing the local high-order proximity topological relationships between nodes in the graph.

[0098] Step S10 addresses the technical problems in dark data scenarios lacking table header definitions. These problems include the failure of traditional static semantic parsing logic due to the loss of explicit semantic anchors, and the potential for severe local feature confusion arising from relying solely on the underlying value distribution of a single data column. Specifically, multi-source data and column feature vectors construct a basic distributional feature foundation independent of the table header text; the value domain graph elevates single column features to a panoramic network containing multi-dimensional contextual topological relationships; and the column embedding vectors effectively compress the sparse graph topology.

[0099] Step S20: Obtain the reference embedding vector and construct a joint feature matrix by combining the column embedding vectors. Construct a constraint relation matrix using the joint feature matrix. Set the projection objective function based on the constraint relation matrix. Perform dimensionality reduction projection on the joint feature matrix based on the projection objective function to obtain the manifold projection vector. Perform unsupervised clustering on the manifold projection vector to obtain the connected clusters and noise point sets. Assign cluster labels to the connected clusters and output the labeled embedding vectors.

[0100] Further, step S20 includes:

[0101] Step S21: Obtain the reference embedding vector and construct a joint feature matrix by combining the column embedding vectors. Use the joint feature matrix to construct the constraint relationship matrix.

[0102] Obtain reference embedding vectors, and stack and concatenate the column embedding vectors and reference embedding vectors row by row to construct a joint feature matrix. Specifically, the reference embedding vectors refer to the column embedding vectors corresponding to the benchmark business system data tables with known field semantics; each reference embedding vector carries a reference cluster label for its corresponding known data field. The reference cluster label refers to the standard business meaning identifier corresponding to the known data field in the business system. Determine the total number of column embedding vectors corresponding to each unknown data column and the total number of reference embedding vectors. The constructed joint feature matrix is ​​a two-dimensional real number matrix, the number of which is the sum of the total number of column embedding vectors and the total number of reference embedding vectors. Each row of the joint feature matrix contains only one column embedding vector or one reference embedding vector.

[0103] Iterate through all column embedding vectors and reference embedding vectors contained in the joint feature matrix. Randomly extract two column embedding vectors or reference embedding vectors from the joint feature matrix, defining them as the first embedding vector and the second embedding vector, respectively. The first and second embedding vectors correspond to the first and second data columns in the original data table, respectively. Calculate the mutual information entropy and Euclidean distance between the first and second embedding vectors. The mutual information entropy measures the statistical dependence of the data columns corresponding to the two embedding vectors at the column value sequence level. Specifically, extract the column value sequences of the first data column corresponding to the first embedding vector and the second data column corresponding to the second embedding vector, and perform equal-frequency binning discretization on both sets of column value sequences. The equal-frequency binning discretization process is as follows: All original data values ​​in the column value sequence are arranged in ascending order of size, and the arranged original data values ​​are divided into several intervals containing the same number of original data values ​​according to a preset number of discretization bins. The number of discretization bins is set based on an empirical value that balances the preservation of data distribution characteristics with the reduction of computational complexity; for example, it is set to 10. Subsequently, a discrete symbol is assigned to each interval to replace the original data value within the corresponding interval. The discrete symbol refers to a classification label used to uniquely identify its interval, mapping continuous original data values ​​to discrete category states. All discrete symbols of the first data column are extracted to form a first set of value symbols; and all discrete symbols of the column value sequence of the second data column are extracted to form a second set of value symbols. Let the first set of value symbols contain P unique first discrete symbols, denoted by T, with a value range of 1 to P; and the second set of value symbols contain Q unique second discrete symbols, denoted by Y, with a value range of 1 to Q. The joint occurrence probability L1 of the values ​​in the first and second data columns, where each value represents the T-th first discrete symbol and the Y-th second discrete symbol respectively, is calculated on the same data row. Simultaneously, the independent occurrence probability L2 of the first data column value representing the T-th first discrete symbol and the independent occurrence probability L3 of the second data column value representing the Y-th second discrete symbol are calculated. Based on these joint and independent occurrence probabilities, the mutual information entropy L is calculated, with the analytical expression as follows: Where log represents the logarithmic function to base 2. The Euclidean distance is used to measure the geometric separation between the first and second embedding vectors in the vector space. The specific calculation process is as follows: calculate the Euclidean distance between the first and second embedding vectors to obtain the distance value representing the geometric separation. The column embedding vectors and reference embedding vectors are paired to form embedding vector pairs. After traversing all embedding vector pairs formed by column embedding vectors and reference embedding vectors in the joint feature matrix, all mutual information entropies are filled into the first symmetric matrix to form a mutual information entropy submatrix, and all Euclidean distances are filled into the second symmetric matrix to form an Euclidean distance submatrix. The mutual information entropy submatrix and the Euclidean distance submatrix are concatenated to construct and output the constraint relation matrix. The elements in the constraint relation matrix record the statistical dependency strength and geometric distance of all embedding vector pairs.

[0104] Step S22: Set the projection objective function based on the constraint relation matrix, and perform dimensionality reduction projection on the joint feature matrix based on the projection objective function to obtain the manifold projection vector.

[0105] After obtaining the constraint relation matrix, to integrate the business semantic dependencies implied by mutual information entropy into the spatial geometric structure after dimensionality reduction projection, and to avoid the scattering of similar features and mis-clustering of different features caused by directly using the original Euclidean distance as the sole basis for dimensionality reduction, a projection mechanism is established that uses mutual information entropy and Euclidean distance as constraints. The mutual information entropy of the mutual information entropy submatrix and the Euclidean distance of the Euclidean distance submatrix recorded in the constraint relation matrix are extracted, and the projection objective function is set.

[0106] The projection objective function constrains the evolution direction of the relative distance between each column embedding vector and the reference embedding vector in the projected target low-dimensional space during the dimensionality reduction process. This ensures that column embedding vectors or reference embedding vectors with mutual information entropy greater than a preset mutual information entropy threshold approximate each other in the target low-dimensional space, while those with mutual information entropy less than or equal to the mutual information entropy threshold move away from each other. The total number Z of all column embedding vectors and reference embedding vectors in the joint feature matrix is ​​determined, where Z equals the sum of the total number of column embedding vectors and the total number of reference embedding vectors. The joint feature matrix is ​​double-traversed using the first matrix row index O1 and the second matrix row index O2, both ranging from 1 to Z. The analytical expression of the projection objective function is: Where G represents the projected objective function value; This indicates that the record in the constraint relation matrix corresponds to the first element in the joint characteristic matrix. The column embedding vector or reference embedding vector of the row, and the first row. The value of the mutual information entropy between the column embedding vectors or reference embedding vectors of the row; The mutual information entropy threshold is used to classify any two column embedding vectors or reference embedding vectors in the joint feature matrix into semantically related and semantically unrelated categories. All off-diagonal elements of the mutual information entropy submatrix in the constraint relation matrix are extracted, and the median of all off-diagonal elements is calculated. This median is used as the dynamic value of the mutual information entropy threshold, with the midpoint of the current data's mutual information entropy distribution serving as the adaptive boundary. The repulsion term weight coefficient is a penalty factor used to control the mutual repulsion between semantically unrelated column embedding vectors or reference embedding vectors during dimensionality reduction. It is set based on empirical values ​​of the relative strength of the attraction and repulsion terms in the dimensionality of the projected objective function. For example, it is set to 0.5. The joint characteristic matrix is ​​represented by the first... The column embedding vector or reference embedding vector of the row, and the first row. The Euclidean distance between the column embedding vectors or reference embedding vectors of the row in the projected target low-dimensional space.

[0107] Based on the projection objective function, a uniform manifold approximation and projection algorithm are used to perform a dimensionality reduction projection operation on the joint feature matrix. Specifically, an initial nearest neighbor graph is constructed to represent the local manifold structure in the high-dimensional space. This initial nearest neighbor graph is an undirected weighted graph where each column embedding vector and each reference embedding vector in the joint feature matrix are graph nodes, and the connection probability between graph nodes is determined by the Euclidean distance recorded in the constraint relation matrix. Subsequently, the connection probability of the initial nearest neighbor graph is topologically modified according to the mutual information entropy judgment threshold: two graph nodes with mutual information entropy greater than the mutual information entropy judgment threshold are extracted, and their connection probability in the initial nearest neighbor graph is forcibly set to the highest probability value, i.e., set to 1; two graph nodes with mutual information entropy less than or equal to the mutual information entropy judgment threshold are extracted, and their connection probability in the initial nearest neighbor graph is forcibly set to the lowest probability value, i.e., set to 0, thereby completing the adjacency topology modification of the initial nearest neighbor graph based on mutual information entropy. Finally, a target low-dimensional space is constructed. The target low-dimensional space refers to a continuous mathematical space used to carry the projected feature vectors, which is generated after the adjacency topology modification of the initial nearest neighbor graph is completed. The dimension of the target low-dimensional space is represented by a projection target dimension variable, which refers to the number of coordinate axes in the target low-dimensional space. This number is set based on an empirical boundary value for achieving maximum feature dimension compression while fully preserving the mutual information entropy and Euclidean distance constraints of each embedding vector in the joint feature matrix; for example, it is set to 8. In the target low-dimensional space, each graph node in the nearest neighbor graph is assigned a continuous low-dimensional space coordinate using a randomly distributed initialization method. This continuous low-dimensional space coordinate is a one-dimensional real vector with a dimension length equal to the projection target dimension variable. This one-dimensional real vector corresponds one-to-one with a graph node in the nearest neighbor graph and is used to quantify the spatial geometric position of the corresponding column embedding vector or reference embedding vector in the target low-dimensional space. Then, using the projection objective function as the optimization objective, the gradient of the projection objective function relative to the low-dimensional space coordinates of each graph node is calculated. The stochastic gradient descent algorithm is used to continuously update the low-dimensional space coordinates of each graph node along the gradient descent direction until the convergence condition is met. The convergence condition refers to the absolute value of the change in the projected objective function between two adjacent iterations being less than a preset accuracy threshold, or the number of iterations reaching a preset maximum number of iterations. The accuracy threshold is set based on an empirical lower bound for the coordinate changes required to determine if the relative positions of each graph node in the target low-dimensional space reach a topologically stable state. For example, it is set as follows: The maximum number of iterations is set based on an empirical upper bound on computational cost to prevent the stochastic gradient descent algorithm from getting stuck in infinite oscillations; for example, it is set to 500 iterations. The low-dimensional continuous spatial coordinates of the projection target dimension variables corresponding to all column embedding vectors obtained after the stochastic gradient descent algorithm converges are extracted and used as manifold projection vectors. The manifold projection vector is a low-dimensional real vector that integrates mutual information entropy and Euclidean distance semantic dependencies. See also...Figure 3 This is a schematic diagram of the manifold dimensionality reduction projection space based on mutual information constraints provided in an embodiment of the present invention. Figure 3 The diagram illustrates the joint feature matrix space on the left and the target low-dimensional space on the right. In the left space, scatter points represent embedded vectors in the joint feature matrix. The solid black lines connecting the scatter points represent strong constraints where mutual information entropy is greater than a decision threshold, and dashed lines between the scatter points represent relationships where mutual information entropy is less than or equal to the decision threshold. The arrow in the middle represents the projection mapping process. In the right space, after the dimensionality reduction projection operation, features with strong constraints are forcibly clustered, forming similar feature clusters. Semantically unrelated feature clusters are separated by geometric distance due to the repulsion term in the projection objective function, achieving differential feature repulsion. The similar feature clustering refers to the geometric clustering region formed by multiple manifold projection vectors with mutual information entropy greater than the mutual information entropy decision threshold, driven by the projection objective function, as they approach each other in the target low-dimensional space. The differential feature repulsion refers to the geometric separation state exhibited by semantically unrelated manifold projection vectors being separated by geometric distance in the target low-dimensional space due to the repulsion term in the projection objective function.

[0108] Step S23: Perform unsupervised clustering on the manifold projection vector to obtain a set of connected clusters and noise points. Assign cluster labels to the connected clusters and output labeled embedding vectors.

[0109] After obtaining the manifold projection vector, in order to divide the manifold projection vector in the target low-dimensional space into discrete business semantic units and avoid the lack of discrete semantic anchors in the global alignment matching of continuous feature vectors, which would cause matching ambiguity, density clustering and cluster label propagation mechanism are invoked to assign a definite cluster label to the manifold projection vector corresponding to each unknown data column.

[0110] Density clustering is invoked to perform cluster partitioning on all manifold projection vectors, resulting in several connected clusters and sets of noise points. The total manifold projection vectors include those corresponding to unknown data columns and those corresponding to reference embedding vectors. Density clustering uses neighborhood radius and minimum neighborhood number as control parameters. The neighborhood radius is dynamically calculated: by calculating the pairwise Euclidean distances between all manifold projection vectors, arranging all Euclidean distances in ascending order, and extracting the Euclidean distance value corresponding to the fifth percentile as the neighborhood radius, its setting is based on adaptively determining the connectivity determination range using the nearest neighbor scale of the current sample density distribution. The minimum neighborhood number is set based on an empirical lower bound of the minimum number of data columns required to form an effective business semantic connected cluster, aiming to suppress the formation of false connected clusters by occasional noise features; for example, it is set to 3. Based on the neighborhood radius and the minimum number of neighborhood points, mutually density-reachable manifold projection vectors are determined: For any manifold projection vector among all manifold projection vectors, the number of other manifold projection vectors whose Euclidean distance to it is less than or equal to the neighborhood radius is counted. If this number is greater than or equal to the minimum number of neighborhood points, the manifold projection vector is determined to be a core point. For any two manifold projection vectors, if there is a continuous propagation path between them consisting of core points, and the Euclidean distance between any two adjacent manifold projection vectors on the continuous propagation path is less than or equal to the neighborhood radius, then the two manifold projection vectors are determined to be mutually density-reachable manifold projection vectors. All mutually density-reachable manifold projection vectors are grouped into the same connected cluster, and isolated manifold projection vectors that do not meet the minimum number of neighborhood points requirement and do not belong to any connected cluster are assigned to the noise point set.

[0111] For each connected cluster obtained from the partitioning, if the connected cluster contains a manifold projection vector corresponding to a reference embedding vector, then the reference cluster label carried by the reference embedding vector is extracted; if all manifold projection vectors corresponding to reference embedding vectors contained in the connected cluster carry the same reference cluster label, then the same reference cluster label is used as the cluster label of the connected cluster; if the manifold projection vectors corresponding to reference embedding vectors contained in the connected cluster carry multiple different reference cluster labels, then the frequency of each reference cluster label appearing in the connected cluster is counted, and the reference cluster label with the highest frequency is used as the cluster label of the connected cluster; if the connected cluster does not contain any manifold projection vector corresponding to any reference embedding vector, then a globally increasing anonymous cluster number is assigned to the connected cluster, and the anonymous cluster number is used as the cluster label of the connected cluster. The globally increasing anonymous cluster number refers to a unique numerical identifier generated by sequentially incrementing from an initial integer during the clustering process, according to the order in which connected clusters without reference labels are discovered, and is used to provide an independent classification identifier for feature sets lacking known business semantic references.

[0112] After determining the cluster label for each connected cluster, the cluster label is propagated and assigned to the manifold projection vector corresponding to each unknown data column contained within that connected cluster. For manifold projection vectors assigned to the noise point set, a reserved label is assigned; this reserved label is used to identify the corresponding data column as a low-confidence sample during global alignment matching. After completing the cluster label propagation, the manifold projection vectors corresponding to the unknown data columns with the appended cluster labels are defined as labeled embedding vectors, and these labeled embedding vectors are output. The labeled embedding vectors are low-dimensional real-valued vectors with appended discrete business semantic identifiers.

[0113] Step S20 addresses the technical problems of severe local feature confusion due to high-dimensional distribution overlap and matching ambiguity caused by the lack of discrete semantic anchors when performing global alignment matching. This is achieved by using reference embedding vectors, column embedding vectors, joint feature matrices, constraint relation matrices, projection objective functions, manifold projection vectors, connected clusters and noise point sets, cluster labels, and annotation embedding vectors. Specifically, the joint feature matrix formed by fusing the reference embedding vectors and column embedding vectors, along with the constraint relation matrix derived from it, jointly construct paired semantic supervision signals. The projection objective function drives dimensionality reduction projection to generate geometrically separable manifold projection vectors. The connected clusters and noise point sets obtained by clustering the manifold projection vectors, combined with the propagation of cluster labels, are ultimately transformed into annotation embedding vectors, completing the anchoring from continuous low-dimensional spatial coordinates to discrete business semantic units, serving as the core semantic driving credential for subsequent operations.

[0114] Step S30: Calculate the migration cost based on the labeled embedding vectors, construct and output the cost matrix using the migration cost, construct the kernel matrix based on the cost matrix, perform alternating updates and element-level reconstruction on the kernel matrix to obtain the probability transition matrix, extract column matching pairs from the probability transition matrix, construct a feature alignment dictionary based on the column matching pairs and perform alignment reconstruction operation, output the reconstructed data table, extract the intersection data columns based on the feature alignment dictionary and perform similarity metric calculation to obtain the similar data set.

[0115] Further, step S30 includes:

[0116] Step S31: Calculate the migration cost based on the labeled embedding vector, and construct and output the cost matrix using the migration cost.

[0117] After obtaining the labeled embedding vectors, in order to align unknown data columns at the global level of the entire data table and avoid losing the inter-column context topology information contained in the value domain map by relying solely on the labeled embedding vectors of a single data column for independent comparison, a cost construction mechanism is established to abstract the entire data table into a discrete distributed manifold and measure the migration cost between data tables.

[0118] Specifically, two data tables are arbitrarily extracted from the multi-source data, defined as the source data table and the target data table, respectively. The labeled embedding vectors corresponding to all unknown data columns in the source data table and the target data table are obtained. The source data table is abstracted into a source discrete distribution manifold, and the target data table is abstracted into a target discrete distribution manifold. The discrete distribution manifold refers to an empirical probability distribution model composed of the labeled embedding vectors of all unknown data columns in a data table and their corresponding probability masses. The probability mass is used as the weight constraint boundary for each feature support point to participate in migration in optimal transmission matching. Specifically: for the source data table, the total number of unknown data columns it contains is counted, and the labeled embedding vector corresponding to each unknown data column is taken as a support point. The reciprocal of the total number is calculated, and this reciprocal is used as the probability mass assigned to each support point. All support points and their probability masses together constitute the source discrete distribution manifold. Similarly, for the target data table, the total number of unknown data columns it contains is counted, and the labeled embedding vector corresponding to each unknown data column is taken as a support point. The reciprocal of the total number is calculated as the probability mass assigned to each support point. All support points and their probability masses together constitute the target discrete distribution manifold. Suppose the source data table contains E1 unknown data columns, and the first source column index C1 is used for traversal, with values ​​ranging from 1 to E1; the target data table contains E2 unknown columns, and the first target column index C2 is used for traversal, with values ​​ranging from 1 to E2. For the C1th unknown data column in the source discrete manifold and the C2th unknown data column in the target discrete manifold, the bulldozer distance, or Wasserstein distance, between their corresponding labeled embedding vectors is calculated. This distance serves as the migration cost for feature matching from the unknown data column in the source data table to the same unknown data column in the target data table. The bulldozer distance is a well-known distance metric based on optimal transport theory, used to quantify the cost function required to transfer the probability mass corresponding to the source data column to the target data column in the target low-dimensional space. Its analytical expression is: Where W represents the bulldozer distance, which will be defined subsequently as the relocation cost. For dimension traversal indexes, ranging from 1 to... , This represents the dimension of the labeled embedding vector, and its value is equal to the dimension variable of the projection target. This indicates that the labeled embedding vector corresponding to the C1th unknown data column in the source data table is in the [missing information - likely a specific position or column name]. Component values ​​on the dimension; This indicates that the labeled embedding vector corresponding to the C2th unknown data column in the target data table is at the [missing information]. Component values ​​on the dimension; This represents the cluster label carried by the C1th unknown data column in the source data table. This represents the cluster label carried by the C2th unknown data column in the target data table. This is an indicator function; it executes when the condition within the parentheses is true. and The value is 1 when they are not equal, and 0 when the condition is not met. The cluster label inconsistency penalty coefficient is used to impose an additional cost penalty on matches that cross different cluster labels. It is set based on an empirical value that makes cross-cluster mappings with inconsistent cluster labels have a migration cost that is on the order of geometric Euclidean distance. For example, it is set to 10.

[0119] Initialize a two-dimensional real matrix, setting the number of rows to the total number of unknown data columns in the source data table and the number of columns to the total number of unknown data columns in the target data table. Set the initial value of each element in the two-dimensional real matrix to zero. Traverse all pairs of unknown data columns in both the source and target discrete manifolds. For pairs of unknown data columns formed by the unknown data columns of the source data table identified by the first source column index and the unknown data columns of the target data table identified by the first target column index, fill the calculated migration cost of this unknown data column pair into the intersection of the row index (i.e., the first source column index) and the column index (i.e., the first target column index) in the two-dimensional real matrix, replacing the initial value at that position. After completing the traversal and filling operations for all unknown data column pairs, define the two-dimensional real matrix as a cost matrix and output it. The cost matrix records the migration cost between all pairs of unknown data columns in the source and target data tables.

[0120] Step S32: Construct a kernel matrix based on the cost matrix, and perform alternating updates and element-level reconstruction on the kernel matrix to obtain the probability transition matrix.

[0121] After obtaining the cost matrix, in order to obtain the feature matching scheme with the minimum cumulative migration cost between the source data table and the target data table from the global topological isomorphism level, and to avoid the loss of context structure topological relationship and matching conflict caused by relying solely on the independent comparison of a single data column, an entropy regularization optimal transmission solution mechanism based on alternating updates of scaling vectors is established to obtain the globally optimal matching scheme under probability quality constraints.

[0122] Specifically, an entropy regularization coefficient is set, which is based on an empirical value that achieves a balance between accelerating iterative convergence and suppressing excessive diffusion in feature matching schemes; for example, it is set to 0.05. A kernel matrix construction operation is performed on the cost matrix: each migration cost in the cost matrix is ​​divided by the entropy regularization coefficient and the result is negative. Using the natural constant e as the base, the negative result is used as an exponent for exponentialization, resulting in a kernel matrix with the same dimensions as the cost matrix.

[0123] Extract the probabilistic mass assigned to each discrete support point in the source discrete manifold, and construct a one-dimensional column vector with a dimension equal to the total number of unknown data columns in the source data table as the source mass vector. The value of each component in the source mass vector is equal to the probabilistic mass corresponding to each unknown data column in the source data table. Extract the probabilistic mass assigned to each discrete support point in the target discrete manifold, and construct a one-dimensional row vector with a dimension equal to the total number of unknown data columns in the target data table as the target mass vector. The value of each component in the target mass vector is equal to the probabilistic mass corresponding to each unknown data column in the target data table. Initialize a left scaling vector of dimension E1 and a right scaling vector of dimension E2, assigning each element in the left and right scaling vectors an initial value of 1. The alternating update algorithm is invoked to perform iterative scaling vector operations: In each iteration, each row element of the kernel matrix is ​​multiplied by the corresponding element of the current right scaling vector and summed to obtain the current row sum vector. Each component of the source quality vector is divided by the corresponding component of the current row sum vector, and the result is used as the updated left scaling vector. Subsequently, each column element of the kernel matrix is ​​multiplied by the corresponding element of the updated left scaling vector and summed to obtain the current column sum vector. Each component of the target quality vector is divided by the corresponding component of the current column sum vector, and the result is used as the updated right scaling vector. A convergence criterion and a maximum iteration limit are set. The convergence criterion is set based on an empirical lower bound for determining the accuracy at which the scaling vector values ​​tend to stabilize; for example, it is set to... The maximum iteration count is set based on an empirical upper bound for constraining the computational cost budget of a single optimal transmission solution; for example, it is set to 1000 iterations. After each alternating update, the maximum absolute value of the difference between each component of the left scaling vector after the current iteration and the left scaling vector of the previous iteration is calculated. If this maximum value is less than the convergence criterion, the iteration is considered converged and the loop is terminated; if the maximum value is greater than or equal to the convergence criterion, the next iteration continues until the iteration count reaches the maximum iteration count limit, at which point the loop is forcibly terminated. If the convergence criterion is not met and the loop is forcibly terminated when the maximum iteration count limit is reached, a convergence failure flag is generated. After the iteration terminates, the kernel matrix is ​​reconstructed element-wise based on the left scaling vector and right scaling vector obtained from the last iteration update: the kernel matrix is ​​traversed using the first source column index and the first target column index. For elements in the kernel matrix whose row index is the first source column index and column index is the first target column index, they are multiplied by the C1th component of the left scaling vector and then multiplied by the C2th component of the right scaling vector to obtain the reconstructed probability weight, which is defined as the transmission probability. A two-dimensional real matrix consisting of all transmission probabilities is defined as the probability transition matrix and output. The row and column dimensions of the probability transition matrix are consistent with the cost matrix. Internally, it records the probability weights of each unknown data column in the source data table being mapped to each unknown data column in the target data table, representing the optimal matching distribution between the source and target data tables under global topological isomorphism. See also... Figure 4 This is a schematic diagram of optimal transmission global bipartite graph alignment provided in an embodiment of the present invention. Figure 4 The diagram illustrates the optimal transport mapping logic for migrating from a source discrete manifold to a target discrete manifold. The nodes on the left represent unknown data columns in the source discrete manifold, which are abstracted from the source data table (e.g., source data columns 1 to 3). The nodes on the right represent unknown data columns in the target discrete manifold, which are abstracted from the target data table (e.g., target data columns 1 to 3). The rectangular bar markers above the nodes represent the probability mass carried by each discrete support point in the feature transfer operation. The lines connecting the nodes represent the feature matching paths after the optimal transport solution is obtained. The final thickness of the lines intuitively reflects the weight of the corresponding transport probability in the probability transition matrix; thicker, darker solid lines represent optimal matching relationships with high transport probabilities.

[0124] Step S33: Extract column matching pairs from the probability transition matrix, construct a feature alignment dictionary based on the column matching pairs, perform alignment reconstruction operation, and output the reconstructed data table.

[0125] After obtaining the probability transition matrix, in order to transform the probabilistic transmission probability into deterministic mapping matching rules and avoid data misalignment caused by ambiguous matching relationships during cross-table fusion, a mapping relationship extraction and data table reconstruction mechanism based on maximum probability decoding is established to complete the final transformation from feature space distribution alignment to physical data table entity merging.

[0126] Specifically, the number of rows in the probability transition matrix corresponds to the total number of unknown data columns in the source data table, and the number of columns corresponds to the total number of unknown data columns in the target data table. The probability transition matrix is ​​traversed row by row using the first source column index. For a specific row identified by the first source column index, the maximum value among all transmission probabilities contained in that row is found. This maximum value is defined as the maximum mapping probability. If multiple source data columns have maximum mapping probabilities pointing to the same target data column, only the mapping relationship with the largest absolute value of the transmission probability is retained as a column matching pair, and the remaining source data columns are marked as isolated data columns. The first target column index of the column containing the maximum mapping probability is extracted. A mapping confidence threshold is set, which is based on an empirical lower bound for determining the true validity of feature matching relationships; for example, it is set to 0.8. The feature matching result is determined based on the extracted maximum mapping probability and convergence failure flag. Specifically, if no convergence failure flag is obtained, and the maximum mapping probability is greater than or equal to the mapping confidence threshold, then the C1th unknown data column in the source data table is determined to be a successful match with the C2th unknown data column in the target data table, and the C1th unknown data column in the source data table and the C2th unknown data column in the target data table are combined into a column matching pair. If a convergence failure flag is obtained, or the maximum mapping probability is less than the mapping confidence threshold, then the C1th unknown data column in the source data table has no valid match in the target data table, and the C1th unknown data column in the source data table is marked as an isolated data column. After traversing all rows of the probability transition matrix, all column matching pairs are summarized. For each column matching pair, the first source column index of the unknown data column in the source data table is used as the key, and the first target column index of the corresponding unknown data column in the target data table is used as the value to construct a feature alignment dictionary. The feature alignment dictionary refers to a set of key-value pairs that record the deterministic field mapping relationship between the source data table and the target data table. Based on the key-value pairs recorded in the feature alignment dictionary, an alignment and reconstruction operation is performed on the source and target data tables. Specifically, an initial empty data table is created. All key-value pairs in the feature alignment dictionary are traversed, and the column value sequences of the unknown data columns in the source data table corresponding to the key and the unknown data columns in the target data table corresponding to the value are extracted. The two column value sequences are vertically concatenated along the vertical dimension of the data rows and written as an aligned column in the initial empty data table. For unknown data columns marked as isolated data columns, the column value sequences of these isolated columns are extracted and written as a separate column in the initial empty data table. Simultaneously, the columns of the probability transition matrix are traversed, and data columns in the target data table that are not matched by any source column are also marked as isolated columns and written to the reconstructed data table in the same way. Null values ​​are then filled in the data rows in the initial empty data table where the corresponding data for these columns is missing. After completing all column value sequence concatenation and placeholder filling, the initial empty data table is defined as the reconstructed data table and output.The reconstructed data table refers to a standardized two-dimensional data carrier that eliminates spatial misalignment of features in multi-source structured data and achieves precise alignment between global semantics and underlying data values. See also. Figure 5 This is a flowchart illustrating the feature alignment dictionary generation and structured reconstruction process provided in an embodiment of the present invention. Figure 5 The diagram illustrates the data flow evolution from extracting deterministic matching rules to generating a physical data table. The top matrix block represents the input probability transition matrix; the middle nodes show the logical branches that extract the maximum mapping probability and combine it with convergence failure flags to determine the feature matching result. If a match is successful, column matching pairs are generated and a feature alignment dictionary is constructed with the first source column index as the key and the first target column index as the value; if no valid match is found, it is marked as an isolated data column. In the bottom reconstruction stage, the system uses an initialized empty data table as a carrier, performs vertical concatenation of the column value sequences of the source and target data tables based on the dictionary records, fills empty spaces in isolated data columns, and finally completes the alignment and concatenation and outputs the reconstructed data table.

[0127] Step S34: Extract the intersection data columns based on the feature alignment dictionary and perform similarity metric calculation to obtain a similar data set.

[0128] After outputting the reconstructed data tables, in order to further identify data tables with similar structures in massive multi-source data and filter cross-table data records pointing to the same physical object, a table-level filtering and row-level entity recognition mechanism based on feature alignment dictionary is established.

[0129] Specifically, the total number of column matching pairs extracted and summarized from the probability transition matrix is ​​counted. This total number of column matching pairs is divided by the total number of unknown data columns in both the source and target data tables, yielding two percentage values. The larger of these two percentage values ​​is defined as the table-level structural similarity. This table-level structural similarity quantifies the overlap ratio of the two data tables in their overall column structure. A table-level similarity threshold is set, based on an empirical lower bound for determining the column overlap ratio of the two data tables to have sufficient analytical value; for example, it is set to 0.5. The table-level structural similarity is compared with the table-level similarity threshold. Only when the table-level structural similarity is greater than or equal to the threshold is the corresponding source and target data tables considered to have approximate structural data tables. An approximate structural data table refers to a combination of source and target data tables whose table-level structural similarity reaches a preset ratio requirement, without needing to consider whether the total number and arrangement order of the unknown data columns are exactly the same. For the source and target data tables that are determined to be approximately structurally similar, unknown data columns with matching relationships are extracted based on the feature alignment dictionary. Specifically, all key-value pairs in the feature alignment dictionary are traversed, and the unknown data columns in the source data table corresponding to the key and the unknown data columns in the target data table corresponding to the value are extracted. Since the keys and values ​​in the dictionary are determined to be matched successfully based on the maximum mapping probability and the mapping confidence threshold, this pair of unknown data columns with a deterministic mapping relationship is defined as the intersection data column.

[0130] After determining all intersecting data columns, to compare specific content at the row level, for each pair of intersecting data columns, corresponding column value sequences are extracted from the source and target data tables respectively. Subsequently, a row traversal mechanism is introduced, sliding along the same row index positions in the source and target data tables. For the currently traversed data row, the original data values ​​of the source and target data tables in the current row of the intersecting data column are extracted from the extracted column value sequences, thus completing the transition from the column to the row dimension. Due to differences in the input specifications of multi-source data, format standardization processing is performed on the two sets of extracted original data values. This format standardization processing includes: removing leading and trailing spaces from the original data values, uniformly mapping full-width characters to half-width characters, and removing preset special delimiters, such as forward slashes, commas, or hyphens, to eliminate literal format differences caused solely by different writing habits, resulting in standardized data values. A comprehensive similarity metric is calculated based on the standardized data values. The comprehensive similarity metric is used to quantify the overall similarity between the source and target data tables across all intersecting columns in the current data row. Specifically, it is calculated as follows: for each pair of intersecting columns, if determined to be character columns, the normalized Levenstein distance between the standardized data values ​​of the source and target data tables is calculated as the single-column similarity; if determined to be numerical columns, the complementary value obtained by dividing the absolute value of the difference between the two standardized data values ​​by their maximum value is calculated as the single-column similarity. After obtaining the single-column similarities of the current data row across all intersecting columns, the arithmetic mean of all single-column similarities is calculated and defined as the comprehensive similarity metric.

[0131] An entity determination threshold is set, which is based on the empirical lower bound of the comprehensive similarity between two data records from different sources that point to the same object at the physical level. For example, it is set to 0.85. The comprehensive similarity metric of the current data row is compared with the entity determination threshold. Only if the comprehensive similarity metric is greater than or equal to the threshold is the data in the current data row of the source and target data tables determined to be the same object at the physical entity level, and these two rows are marked as similar data. Similar data refers to cross-table data rows in the source and target data tables that highly match in the extracted intersection data columns, essentially recording the same physical entity. Finally, all data rows determined to be the same object are summarized, thereby identifying and outputting the set of similar data that coexist in the approximately structured data tables.

[0132] Step S30 addresses the matching conflicts caused by relying solely on independent comparisons of a single data column, and the difficulty in identifying cross-table associations of entity objects in massive heterogeneous data tables, by using labeled embedding vectors, cost matrices, probability transition matrices, column matching pairs, feature alignment dictionaries, approximate structure data tables, and similar data sets. Specifically, labeled embedding vectors abstract the source and target data tables into discrete manifolds; the cost matrix quantifies feature transfer costs based on bulldozer distance; the probability transition matrix represents the optimal matching distribution under global topological isomorphism; the feature alignment dictionary, combined with extracted column matching pairs, completes data alignment and reconstruction; the approximate structure data table overcomes the barrier of inconsistent field numbers and order, achieving rapid table-level structural clustering; and the similar data set further mines entity data pointing to the same physical object across tables based on intersecting data columns.

[0133] Example 2

[0134] This embodiment, based on Embodiment 1, provides a dark data semantic association system based on structured representation, such as... Figure 6 As shown, it includes:

[0135] The graph embedding module is used to acquire multi-source data and extract the data value sets of numerical and character columns, concatenate them to obtain column feature vectors, introduce a sliding window between columns based on the column feature vectors to count the co-occurrence probability between columns, construct and output the value range graph, perform random walk sampling based on the value range graph to construct the node co-occurrence matrix, and perform singular value decomposition on the node co-occurrence matrix to obtain the column embedding vectors.

[0136] Clustering and labeling module: used to obtain reference embedding vectors, construct a joint feature matrix by combining column embedding vectors, construct a constraint relation matrix using the joint feature matrix, set a projection objective function based on the constraint relation matrix, perform dimensionality reduction projection on the joint feature matrix based on the projection objective function to obtain manifold projection vectors, perform unsupervised clustering on the manifold projection vectors to obtain a set of connected clusters and noise points, assign cluster labels to connected clusters, and output labeled embedding vectors;

[0137] Cost Reconstruction Module: This module calculates the migration cost based on the labeled embedding vectors, constructs and outputs a cost matrix using the migration cost, constructs a kernel matrix based on the cost matrix, performs alternating updates and element-level reconstruction on the kernel matrix to obtain a probability transition matrix, extracts column matching pairs from the probability transition matrix, constructs a feature alignment dictionary based on the column matching pairs and performs alignment reconstruction operations, outputs a reconstructed data table, extracts intersection data columns based on the feature alignment dictionary and performs similarity metric calculations to obtain a similar data set.

[0138] In the graph embedding module, the process involves acquiring multi-source data and extracting and concatenating the data value sets of numerical and character columns to obtain column feature vectors. Based on these feature vectors, a sliding window is introduced between columns to calculate co-occurrence probabilities, constructing and outputting a value domain graph. Random walk sampling is then performed on the value domain graph to construct a node co-occurrence matrix. Singular value decomposition is then performed on the node co-occurrence matrix to obtain column embedding vectors, including:

[0139] Step S11: Obtain multi-source data, extract the data value set of numeric columns and the data value set of character columns from the multi-source data and concatenate them to obtain the column feature vector;

[0140] Step S12: Based on the column feature vector, introduce a sliding window between columns in the multi-source data to count the co-occurrence probability between columns, construct and output the value range map;

[0141] Step S13: Construct a node co-occurrence matrix by performing random walk sampling based on the value range spectrum, and perform singular value decomposition based on the node co-occurrence matrix to obtain the column embedding vector.

[0142] In the clustering annotation module, the steps include: obtaining reference embedding vectors, constructing a joint feature matrix by combining column embedding vectors, constructing a constraint relation matrix using the joint feature matrix, setting a projection objective function based on the constraint relation matrix, performing dimensionality reduction projection on the joint feature matrix based on the projection objective function to obtain manifold projection vectors, performing unsupervised clustering on the manifold projection vectors to obtain a set of connected clusters and noise points, assigning cluster labels to the connected clusters, and outputting annotation embedding vectors, including:

[0143] Step S21: Obtain the reference embedding vector and construct a joint feature matrix by combining it with the column embedding vector; then construct the constraint relationship matrix using the joint feature matrix.

[0144] Step S22: Set the projection objective function according to the constraint relation matrix, and perform dimensionality reduction projection on the joint feature matrix based on the projection objective function to obtain the manifold projection vector;

[0145] Step S23: Perform unsupervised clustering on the manifold projection vector to obtain a set of connected clusters and noise points. Assign cluster labels to the connected clusters and output labeled embedding vectors.

[0146] In the cost reconstruction module, the migration cost is calculated based on the labeled embedding vectors, a cost matrix is ​​constructed and output using the migration cost, a kernel matrix is ​​constructed based on the cost matrix, and the kernel matrix is ​​alternately updated and reconstructed element-wise to obtain a probability transition matrix. Column matching pairs are extracted from the probability transition matrix, a feature alignment dictionary is constructed based on the column matching pairs, and an alignment reconstruction operation is performed to output a reconstructed data table. Intersection data columns are extracted based on the feature alignment dictionary, and a similarity metric is calculated to obtain a similar data set, including:

[0147] Step S31: Calculate the migration cost based on the labeled embedding vectors, and construct and output the cost matrix using the migration cost;

[0148] Step S32: Construct a kernel matrix based on the cost matrix, and perform alternating updates and element-level reconstruction on the kernel matrix to obtain the probability transition matrix;

[0149] Step S33: Extract column matching pairs from the probability transition matrix, construct a feature alignment dictionary based on the column matching pairs, perform alignment reconstruction operation, and output the reconstructed data table;

[0150] Step S34: Extract the intersection data columns based on the feature alignment dictionary and perform similarity metric calculation to obtain a similar data set.

[0151] In addition, the parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of the corresponding technical solutions in the prior art have not been described in detail, so as to avoid excessive elaboration.

[0152] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely preferred embodiments of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for semantic association of dark data based on structured representation, characterized in that, The method includes: The system acquires multi-source data and extracts data value sets from numerical and character columns, concatenates them to obtain column feature vectors, introduces a sliding window between columns based on column feature vectors to calculate the co-occurrence probability between columns, constructs and outputs a value range map, performs random walk sampling based on the value range map to construct a node co-occurrence matrix, and performs singular value decomposition on the node co-occurrence matrix to obtain column embedding vectors. Obtain the reference embedding vector and construct a joint feature matrix by combining the column embedding vectors. Construct a constraint relation matrix using the joint feature matrix. Set the projection objective function based on the constraint relation matrix. Perform dimensionality reduction projection on the joint feature matrix based on the projection objective function to obtain the manifold projection vector. Perform unsupervised clustering on the manifold projection vector to obtain the connected clusters and noise point sets. Assign cluster labels to the connected clusters and output the labeled embedding vectors. The migration cost is calculated based on the labeled embedding vectors. The cost matrix is ​​constructed and output using the migration cost. A kernel matrix is ​​constructed based on the cost matrix. The kernel matrix is ​​alternately updated and reconstructed element-wise to obtain the probability transition matrix. Column matching pairs are extracted from the probability transition matrix. A feature alignment dictionary is constructed based on the column matching pairs and an alignment reconstruction operation is performed to output the reconstructed data table. The intersection data columns are extracted based on the feature alignment dictionary and a similarity metric is calculated to obtain a similar data set.

2. The method for semantic association of dark data based on structured representation according to claim 1, characterized in that, The method for obtaining the column feature vector includes: The proportion of numerical elements in a set of statistical data values ​​that can be losslessly parsed into numerical values ​​is obtained. When the proportion of numerical elements exceeds the preset type determination threshold, the corresponding numerical or character column is determined as the target numerical column, and the skewness, kurtosis, and Gaussian mixture model feature vector of the target numerical column are calculated. When the proportion of the numerical elements is less than or equal to the preset type determination threshold, the corresponding numerical column or character column is determined as the target character column, and the regular contour code and character-level frequency vector are extracted from the target character column. The skewness, kurtosis, Gaussian mixture model feature vectors, regular contour codes, and character-level frequency vectors are concatenated in a fixed order, and zeros are padded at the positions of missing features to output column feature vectors.

3. The method for semantic association of dark data based on structured representation according to claim 2, characterized in that, The method for calculating the eigenvectors of the Gaussian mixture model includes: All elements in the target numerical column that can be losslessly parsed into numerical values ​​are used as fitting samples. A Gaussian mixture model consisting of multiple Gaussian components is preset, and each Gaussian component consists of mixture weights, component mean, and component variance. The expectation step and the maximization step are executed alternately to iteratively solve the parameters of the Gaussian mixture model. In the expectation step, the posterior probability of each fitted sample belonging to each Gaussian component is calculated based on the current parameters of each Gaussian component. In the maximization step, the mixture weight, component mean and component variance of each Gaussian component are updated in a weighted manner based on the posterior probability. Once the iteration converges, the updated mixture weights, component means, and component variances of all Gaussian components are extracted and concatenated in order to output the Gaussian mixture model feature vector.

4. The method for semantic association of dark data based on structured representation according to claim 2, characterized in that, The method for obtaining the regular contour encoding and character-level frequency vector includes: The numbers, uppercase letters, and lowercase letters in each string element of the target character column are uniformly mapped and replaced character by character to generate morphological template strings. The top eight morphological template strings with the highest frequency are counted, and the normalized occurrence frequency of each morphological template string is calculated. The morphological template strings and their normalized occurrence frequencies are combined to form a regular contour code. The string elements are segmented sequentially using a sliding window to extract and summarize consecutive two-character substrings. The top 32 consecutive two-character substrings with the highest frequency are counted, and the normalized frequency of each consecutive two-character substring is calculated. The consecutive two-character substrings and their corresponding normalized frequencies are combined to form a character-level frequency vector.

5. The method for semantic association of dark data based on structured representation according to claim 1, characterized in that, The method for obtaining the value range map includes: By using a sliding window between columns to slide across multi-source data, the co-occurrence count and chance count are calculated when the values ​​of any two numeric or character columns are not empty placeholders. The co-occurrence count is divided by the chance count to obtain the co-occurrence probability between columns. Instantiate a data column node for each numeric and character column, and use the corresponding column feature vector as the node feature vector. Extract the original data value from the data value set and instantiate it as a data value node. The frequency percentage is obtained by dividing the frequency of the original data value in the corresponding data value set by the total number of non-empty data rows in the corresponding numeric or character column. Using the co-occurrence probability between columns as weights, undirected connecting lines are established between data column nodes as column association edges. Using the frequency proportion as weights, undirected connecting lines are established between data column nodes and data value nodes as value attribution edges, and the value domain map is output.

6. The method for semantic association of dark data based on structured representation according to claim 5, characterized in that, The method for obtaining the column embedding vector includes: Using each data column node and data value node in the value domain graph as the starting point of the walk, and based on the state transition probability obtained by normalizing the weight of the column association edge or value belonging edge connected to the current node, a random walk sampling with a preset walk step size and a preset number of samplings is performed to generate a set of walk sequences containing node visit sequences. Set the sequence context window to slide in the node access sequence, count the original frequency of any two nodes that co-occur, and fill it into the node co-occurrence matrix; Calculate the pointwise mutual information value of each node pair to replace the original frequency and generate a pointwise mutual information weighted matrix; Perform singular value decomposition on the pointwise mutual information weighted matrix, extract the left singular vector column and scale it proportionally to obtain a low-dimensional dense vector as the column embedding vector.

7. The method for semantic association of dark data based on structured representation according to claim 1, characterized in that, The method for constructing the constraint relation matrix includes: The reference embedding vector and the column embedding vector are stacked and concatenated row by row to construct a joint feature matrix; Two column embedding vectors or reference embedding vectors are randomly extracted from the joint feature matrix to form an embedding vector pair. The set of data values ​​of the corresponding numeric or character columns in each data row is extracted from the multi-source data. Map the original data values ​​in the data value set to discrete symbols; The joint and independent occurrence probabilities of discrete symbols on the same data row are statistically analyzed to calculate the mutual information entropy, and the Euclidean distance between embedded vector pairs is also calculated. Fill all mutual information entropy into a preset first symmetric matrix to form a mutual information entropy submatrix, fill all Euclidean distances into a preset second symmetric matrix to form an Euclidean distance submatrix, concatenate the mutual information entropy submatrix and the Euclidean distance submatrix, and output the constraint relation matrix.

8. The method for semantic association of dark data based on structured representation according to claim 7, characterized in that, The method for obtaining the manifold projection vector includes: Extract the mutual information entropy and Euclidean distance recorded in the constraint relationship matrix, set the projection objective function, and calculate the median of the mutual information entropy as the mutual information entropy judgment threshold; An initial nearest neighbor graph is constructed using each column embedding vector in the joint feature matrix and the reference embedding vector as graph nodes, and the connection probability between graph nodes is determined by Euclidean distance. The connection probability of graph nodes with mutual information entropy greater than the mutual information entropy judgment threshold is set as the highest probability value, and the connection probability of graph nodes with mutual information entropy less than or equal to the mutual information entropy judgment threshold is set as the lowest probability value to complete the adjacency topology transformation. Assign initial continuous low-dimensional spatial coordinates to each graph node in the nearest neighbor graph, calculate the gradient of the projection objective function with respect to the low-dimensional spatial coordinates of each graph node, continuously update the low-dimensional spatial coordinates along the gradient descent direction until the preset convergence condition is met, and use the converged low-dimensional spatial coordinates as the manifold projection vector.

9. The method for semantic association of dark data based on structured representation according to claim 1, characterized in that, The method for obtaining the labeled embedding vector includes: Calculate the Euclidean distances of all manifold projection vectors and set the neighborhood radius; Count the number of other manifold projection vectors whose Euclidean distance to any manifold projection vector is less than or equal to the neighborhood radius. If the number is greater than or equal to the minimum neighborhood number, then the manifold projection vector is determined to be the core point. When there is a continuous transmission path between two manifold projection vectors consisting of core points, and the Euclidean distance between adjacent vectors on the path is less than or equal to the neighborhood radius, the two are determined to be mutually density-reachable manifold projection vectors. Manifold projection vectors that are density-reachable with each other are grouped into the same predefined connected cluster, while isolated manifold projection vectors are assigned to a predefined set of noise points. If a connected cluster contains the manifold projection vector corresponding to the reference embedding vector, the reference cluster label with the highest frequency is used as the cluster label; otherwise, an anonymous cluster number is assigned as the cluster label, the determined cluster label is assigned to the corresponding manifold projection vector, and the labeled embedding vector is output.

10. The method for semantic association of dark data based on structured representation according to claim 1, characterized in that, The method for constructing the cost matrix includes: The source data table and target data table in the multi-source data are counted, as well as the total number of unknown data columns contained therein. The labeled embedding vector corresponding to each unknown data column is used as a support point. The reciprocal of the total number is used as the probability mass assigned to each support point, thus forming the source discrete distribution manifold and the target discrete distribution manifold respectively. Calculate the bulldozer distance between the labeled embedding vectors corresponding to each unknown data column pair in the source discrete manifold and the target discrete manifold; A cost penalty is imposed on cross-cluster mapping of unknown data columns carrying inconsistent cluster labels, and the bulldozer distance after the penalty is applied is used as the migration cost for feature matching. The migration costs of all unknown data pairs are filled into the corresponding intersection positions of a preset two-dimensional real matrix to construct the output cost matrix.

11. The method for semantic association of dark data based on structured representation according to claim 10, characterized in that, The method for outputting the reconstructed data table includes: The kernel matrix is ​​obtained by dividing the migration cost in the cost matrix by the preset entropy regularization coefficient and taking the negative number, and then performing an exponential operation. Construct source quality vector and target quality vector based on probability quality, and initialize left scaling vector and right scaling vector respectively; The left scaling vector is updated by dividing the source mass vector by the row sum of the product of the kernel matrix and the right scaling vector, and the right scaling vector is updated by dividing the target mass vector by the column sum of the product of the kernel matrix and the updated left scaling vector. This process is repeated alternately until convergence. The probability transition matrix is ​​obtained by performing element-wise multiplication between the converged left and right scaling vectors and the kernel matrix. Extract the maximum mapping probability of each row in the probability transition matrix, and combine the corresponding unknown data columns into column matching pairs or mark them as isolated data columns; Construct a feature alignment dictionary using column matching pairs, concatenate unknown data columns of the source and target data tables based on the feature alignment dictionary, fill in null values ​​for isolated data columns, and output the reconstructed data table.

12. The method for semantic association of dark data based on structured representation according to claim 11, characterized in that, The method for obtaining the similar data set includes: Extract the total number of all column matching pairs from the probability transition matrix and divide it by the total number of unknown data columns in the source and target data tables respectively to obtain two percentage values. The larger percentage value is taken as the table-level structure similarity. If the table-level structural similarity is greater than or equal to the preset table-level similarity threshold, then the source data table and the target data table are determined to be similar structured data tables. Get and iterate through the key-value pairs in the feature alignment dictionary, and define the unknown data columns with deterministic mapping relationships as the intersection data columns; Extract the column value sequence of the intersection data column from the source data table and the target data table, extract the original data value at the same index position in the same row and perform format standardization processing to obtain the standardized data value; A comprehensive similarity metric is calculated using standardized data values. If the comprehensive similarity metric is greater than or equal to a preset entity determination threshold, the corresponding data behavior is determined to be similar data, and all similar data in the approximate structure data table is aggregated into a similar data set.

13. A dark data semantic association system based on structured representation, used to implement the dark data semantic association method based on structured representation as described in any one of claims 1-12, characterized in that, The system includes: The graph embedding module is used to acquire multi-source data and extract the data value sets of numerical and character columns, concatenate them to obtain column feature vectors, introduce a sliding window between columns based on the column feature vectors to count the co-occurrence probability between columns, construct and output the value range graph, perform random walk sampling based on the value range graph to construct the node co-occurrence matrix, and perform singular value decomposition on the node co-occurrence matrix to obtain the column embedding vectors. Clustering and labeling module: used to obtain reference embedding vectors, construct a joint feature matrix by combining column embedding vectors, construct a constraint relation matrix using the joint feature matrix, set a projection objective function based on the constraint relation matrix, perform dimensionality reduction projection on the joint feature matrix based on the projection objective function to obtain manifold projection vectors, perform unsupervised clustering on the manifold projection vectors to obtain a set of connected clusters and noise points, assign cluster labels to connected clusters, and output labeled embedding vectors; Cost Reconstruction Module: This module calculates the migration cost based on the labeled embedding vectors, constructs and outputs a cost matrix using the migration cost, constructs a kernel matrix based on the cost matrix, performs alternating updates and element-level reconstruction on the kernel matrix to obtain a probability transition matrix, extracts column matching pairs from the probability transition matrix, constructs a feature alignment dictionary based on the column matching pairs and performs alignment reconstruction operations, outputs a reconstructed data table, extracts intersection data columns based on the feature alignment dictionary and performs similarity metric calculations to obtain a similar data set.

Images