A method and system for automatic standardization processing based on multi-source heterogeneous data

Abstract

The application relates to the technical field of data governance and information processing, and discloses an automatic standardization processing method and system based on multi-source heterogeneous data. The method comprises the following steps: obtaining a characteristic fragment set by preprocessing source system data, and calculating and constructing a characteristic matching matrix; extracting an associated node pair from the characteristic matching matrix and obtaining a preliminary grouping set by traversal; performing statistical extraction on the preliminary grouping set to obtain a context mode set; extracting a core term combination based on the context mode set and searching a group semantic image in a business ontology library; calculating the spatial geometric distance between a candidate triple and the group semantic image by using a word embedding model to determine the real semantic type of a new inflow field; and reconstructing an updated matching matrix by extracting semantic drift features according to source system change data, and recombining the updated semantic image based on local aggregation density. The method can realize high-precision semantic recognition and self-adaptive evolution of multi-source heterogeneous data.

Description

Technical Field

[0001] This invention relates to the field of data governance and information processing technology, and in particular to an automatic standardization processing method and system based on multi-source heterogeneous data. Background Technology

[0002] In today's context of accelerated digital transformation, enterprises and institutions face the need to aggregate and integrate massive amounts of heterogeneous data from multiple sources. In big data processing scenarios, this data often originates from underlying databases, external interfaces, or legacy systems. Only by standardizing and aligning this complex data with high quality and building a unified data foundation can we effectively support upper-level analysis, decision-making, and intelligent applications.

[0003] Currently, most mainstream data standardization methods rely on manually defined rules or preset mapping templates. In one existing technology, data governance systems typically hard-code heterogeneous data fields from external sources to corresponding categories in a standard data dictionary based on regular expressions, keyword matching of field names, or simple data type format features. However, through derivation and business practice analysis of the above-mentioned existing technologies, it has been found that they have insurmountable limitations. On the one hand, the true meaning of field semantics is highly dependent on the context and business flow logic. Different source systems have a large number of implicit differences in the expression of the same business concept (i.e., the widespread phenomenon of homonyms or synonyms with different names). Relying solely on surface names or format features for static template comparison is prone to matching failures and misjudgments. On the other hand, as the source system table structure iterates or new data sources are continuously integrated, the original static rule system cannot adaptively adjust, which can easily lead to a large number of related fields having chain classification errors. This forces technical personnel to frequently intervene manually and rewrite rules, resulting in extremely high maintenance costs and serious delays.

[0004] Existing technologies suffer from low accuracy in semantic recognition of multi-source heterogeneous data and are unable to adapt to dynamic changes in the source system. Summary of the Invention

[0005] This invention provides an automatic standardization processing method and system based on multi-source heterogeneous data to solve the technical problems of low semantic recognition accuracy and inability to adapt to dynamic changes in source systems in existing technologies.

[0006] In a first aspect, to address the aforementioned technical problems, this invention provides an automatic standardization processing method based on multi-source heterogeneous data, comprising: Acquire source system data and preprocess it to obtain a set of characterization fragments; The set of representation fragments is subjected to feature comparison to obtain a comparison feature set, and a comprehensive similarity is calculated. A feature matching matrix is ​​constructed based on the comprehensive similarity. The feature matching matrix is ​​numerically compared with the calculated judgment threshold to obtain the associated node pairs. The depth-first search algorithm is then used to traverse and calculate the associated node pairs to obtain a preliminary group set. Data mapping is performed on the preliminary group set to obtain the underlying data samples. Feature distribution statistics are performed on the underlying data samples to obtain the co-occurrence intensity sequence. Topological extraction is performed based on the co-occurrence intensity sequence to obtain the context pattern set. The core term combination is obtained by spatial distribution calculation based on the context pattern set, and the core term combination is input into the pre-built business ontology library for mapping retrieval to obtain the group semantic profile. A new inflow field is obtained, and semantic extraction is performed on the new inflow field to obtain candidate triples. A pre-trained word embedding model is used to calculate the spatial geometric distance between the candidate triples and the semantic profile of the group, and the true semantic type is determined based on the spatial geometric distance. Obtain source system change data, perform cross-analysis between the source system change data and the group semantic profile to obtain semantic drift features, reconstruct the feature matching matrix based on the semantic drift features to obtain an updated matching matrix, perform topological calculation on the updated matching matrix to obtain local aggregation density, and reorganize nodes based on the local aggregation density to obtain an updated semantic profile.

[0007] Secondly, the present invention provides an automatic standardization processing system based on multi-source heterogeneous data, comprising: The data preprocessing module is used to acquire source system data and preprocess it to obtain a set of characterization fragments; The feature comparison module is used to perform feature comparison on the set of representation fragments to obtain a comparison feature set, calculate the comprehensive similarity, and construct a feature matching matrix based on the comprehensive similarity. The graph network partitioning module is used to compare the feature matching matrix with the calculated judgment threshold to obtain the associated node pairs, and to use the depth-first search algorithm to traverse and calculate the associated node pairs to obtain the preliminary grouping set. The feature distribution statistics module is used to perform data mapping on the preliminary group set to obtain the underlying data samples, perform feature distribution statistics on the underlying data samples to obtain the co-occurrence intensity sequence, and perform topological extraction based on the co-occurrence intensity sequence to obtain the context pattern set. The semantic anchoring module is used to calculate the spatial distribution of the context pattern set to obtain the core term combination, and input the core term combination into a pre-built business ontology library for mapping retrieval to obtain the group semantic profile. The dynamic mapping module is used to obtain new inflow fields, perform semantic extraction on the new inflow fields to obtain candidate triples, use a pre-trained word embedding model to calculate the spatial geometric distance between the candidate triples and the semantic profile of the group, and determine the true semantic type based on the spatial geometric distance. The adaptive evolution module is used to acquire source system change data, perform cross-analysis between the source system change data and the group semantic profile to obtain semantic drift features, reconstruct the feature matching matrix based on the semantic drift features to obtain an updated matching matrix, perform topological calculation on the updated matching matrix to obtain local aggregation density, and perform node reorganization based on the local aggregation density to obtain an updated semantic profile.

[0008] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention constructs a feature matching matrix by comparing features of multi-source heterogeneous data and weighted summation, and uses a depth-first search algorithm to traverse the associated node pairs to obtain a preliminary group set. This transforms the originally isolated heterogeneous fields into a global topological graph structure with connectivity, breaking the limitations of traditional hard-coded matching of rules based on surface names or single formats. It effectively mines and extracts the hidden graph structure association features between massive fields, thereby significantly improving the accuracy of field aggregation and the quality of automated initial screening in complex business scenarios such as homonyms or synonyms.

[0009] (2) This invention locks the core terms by performing feature distribution statistical calculations on the underlying data samples, constructs a global group semantic profile, and uses a pre-trained word embedding model to perform spatial mapping and feature pooling on the candidate triples of the newly inflowing fields and the benchmark anchor set to calculate the spatial geometric distance. It accurately quantifies the highly complex contextual dependencies into a high-dimensional spatial vector comparison process under a unified dimension, completely getting rid of the dependence of traditional static templates on a large number of manual prior annotation rules, thereby achieving non-intervention and high-precision recognition of the true semantic types of various newly inflowing unknown fields.

[0010] (3) This invention extracts semantic drift features by cross-analyzing source system change data and global group semantic profiles, dynamically reconstructs feature matching matrix, and automatically triggers node redistribution and profile update based on the fluctuation of local aggregation density. It constructs an adaptive evolution closed-loop mechanism from the perception of underlying data anomalies to the correction of global topology weights, which gives the system the ability to continuously learn and correct itself in dynamic and complex environments. It effectively solves the industry pain point that static rule system gradually fails as data source iterates and is prone to error cascading propagation in the prior art, greatly reduces the maintenance cost of the system in the later stage, and ensures the robustness and semantic consistency of the long-term operation of the enterprise-level unified data foundation. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the automatic standardization processing method based on multi-source heterogeneous data provided in the first embodiment of the present invention; Figure 2 This is a schematic diagram of the automatic standardization processing system based on multi-source heterogeneous data provided in the second embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Reference Figure 1 The first embodiment of the present invention provides an automatic standardization processing method based on multi-source heterogeneous data, including the following steps: S11, acquire source system data and preprocess it to obtain a set of characterization fragments; S12, perform feature comparison on the set of representation fragments to obtain a comparison feature set, calculate the comprehensive similarity, and construct a feature matching matrix based on the comprehensive similarity; S13, compare the feature matching matrix with the calculated judgment threshold to obtain the associated node pairs, and use the depth-first search algorithm to traverse and calculate the associated node pairs to obtain the preliminary grouping set. S14, perform data mapping on the preliminary group set to obtain bottom-level data samples, perform feature distribution statistics on the bottom-level data samples to obtain co-occurrence intensity sequences, and perform topological extraction based on the co-occurrence intensity sequences to obtain context pattern sets; S15, calculate the spatial distribution of the core term combination based on the context pattern set, and input the core term combination into the pre-built business ontology library for mapping retrieval to obtain the group semantic profile. S16, obtain a new inflow field, perform semantic extraction on the new inflow field to obtain candidate triples, use a pre-trained word embedding model to calculate the spatial geometric distance between the candidate triples and the group semantic profile, and determine the true semantic type based on the spatial geometric distance; S17, obtain source system change data, perform cross-analysis between the source system change data and the group semantic profile to obtain semantic drift features, reconstruct the feature matching matrix based on the semantic drift features to obtain an updated matching matrix, perform topological calculation on the updated matching matrix to obtain local aggregation density, and perform node reorganization based on the local aggregation density to obtain an updated semantic profile.

[0014] In step S11, source system data is acquired and preprocessed to obtain a set of characterization fragments, including: The source system data is parsed to obtain an initial record list and field type identifiers, and the initial record list is checked for non-emptiness to obtain a sample value list. Based on the field type identifier, the sample value list is subjected to split feature extraction to obtain morphological feature data; The morphological feature data is structured and encapsulated to obtain the set of representation fragments.

[0015] In one implementation, this embodiment connects to multiple heterogeneous source systems via a data access interface to pull data from the source systems. It performs table structure parsing on the table data definition language files in the source system data, extracts table structure definition information, and maps the extracted field names and data type configurations to the field type identifiers. Simultaneously, it reads all data rows in the table to generate the initial record list. It should be noted that this embodiment limits the field type identifiers to only three fixed categories: text, numeric, and date. The specific process of performing non-empty validation on the initial record list to obtain a sample value list is as follows: This embodiment iterates through each field value in the initial record list, removing data rows containing empty values, consecutive whitespace characters, and data rows containing preset invalid placeholder characters. The preset invalid placeholder characters only include strings consisting of the digit 99999 and strings consisting of N / A.

[0016] It is worth noting that this embodiment sets a preset sampling buffer threshold. This preset sampling buffer threshold is determined using a statistical analysis method based on the confidence interval of a normal distribution. Specifically, the process involves calculating the minimum sample size that satisfies a 95% confidence level and an error margin of 5%, and then rounding this minimum sample size up to determine the preset sampling buffer threshold. For example, this calculation yields a value of 385. In this embodiment, the field values ​​that pass the non-empty check are sequentially and continuously filled into the sampling buffer. When the number of data entries in the sampling buffer reaches the preset sampling buffer threshold, the reading operation stops, and the data set in the sampling buffer is output as the sample value list.

[0017] In one implementation, this embodiment extracts morphological feature data from the sample value list based on the field type identifier. Specifically, the sample value list is distributed to three corresponding processing logic branches according to the field type identifier. For the sample value list determined to be text, this embodiment uses a pre-established business dictionary to perform word boundary segmentation on the contained string data to extract business phrase fragments.

[0018] It should be noted that the construction process of the pre-established business dictionary is as follows: Plain text data of historical operation manuals within the corresponding business domain is collected; a Hidden Markov Model (HMM) is used to segment and count word frequencies in the plain text data; the probability value of each segmented word is calculated; and noun combinations and verb combinations with probability values ​​exceeding a preset probability threshold are stored in a relational database to constitute the business dictionary. The HMM uses business domain text with labeled segmentation boundaries as training corpus, and estimates the model's state transition probability and emission probability using the maximum likelihood estimation method. The training corpus consists of no less than 10,000 standard business statements. The preset probability threshold is determined by Zip's law combined with a long-tail word frequency truncation threshold. In this embodiment, the value is specifically 0.05, meaning that the top 5% of high-frequency words are sorted from highest to lowest frequency, and the proportion of each word frequency to the total word frequency is accumulated until this proportion reaches 95%. All words prior to this point constitute the high-frequency word set. For the list of sample values ​​identified as numerical, this embodiment sorts the specific values ​​in the list in ascending order, calculates and extracts the values ​​at five fixed positions: minimum, 25th percentile, median, 75th percentile, and maximum. The sequence formed by these five fixed positions is then extracted as the distribution quantile feature. For the list of sample values ​​identified as date, this embodiment uses regular expression matching to extract the corresponding format pattern string. The extracted results are limited to four-digit year, two-digit month, and two-digit date formats without separators, formats with hyphen separators, and formats with slash separators. This embodiment summarizes the business phrase fragments, distribution quantile features, and format pattern strings output from the above processing and merges them to generate the morphological feature data.

[0019] It should be noted that the specific process of obtaining the representation fragment set by structurally encapsulating the morphological feature data in this embodiment is as follows: This embodiment extracts the source system unique identifier and the timestamp sequence number when the current acquisition operation occurred from the source system data. The field type identifier, the morphological feature data, the source system unique identifier, and the timestamp sequence number are then merged into an independent data block according to a preset key-value pair mapping relationship. For example, this embodiment uses JavaScript Object Notation (JSON) format to physically encapsulate this data block. Specifically, the Field_ID key maps the source system unique identifier to the field type identifier, the Morph_Feat key maps to the array structure containing the morphological feature data, and the Timestamp key maps to the timestamp sequence number. This embodiment performs serialization encoding on the encapsulated data block to generate a binary data stream, defines this binary data stream as the representation fragment set, and outputs it to the metadata repository.

[0020] In step S12, feature comparison is performed on the set of representation fragments to obtain a comparison feature set, and a comprehensive similarity is calculated. A feature matching matrix is ​​then constructed based on the comprehensive similarity.

[0021] The process includes performing feature comparison on the set of representation fragments to obtain a comparison feature set, and calculating a comprehensive similarity, including: The candidate field pairs are obtained by combining the set of representation fragments in pairs; The candidate field pairs are classified and compared to obtain the differences in type dimension, and the candidate field pairs are frequency-statistically analyzed to obtain the term co-occurrence frequency; The comparison feature set is obtained by merging and concatenating the type dimension differences and the term co-occurrence frequencies. The comprehensive similarity is obtained by weighting and summing the comparison feature sets using pre-configured weight coefficients.

[0022] In one implementation, this embodiment combines the representation fragment set in pairs to obtain candidate field pairs. Specifically, the Cartesian product operation is used to pair all elements in the representation fragment set and extract the field type identifier combination for each pair of elements.

[0023] It should be noted that the specific process of classifying and comparing the candidate field pairs to obtain the differences in type dimensions is as follows: In this embodiment, the combination of field type identifiers contained in the candidate field pairs is determined. If both candidate field pairs are text types, this embodiment uses a character n-gram algorithm to segment the two business phrase fragments into character substring sets of length N, where the value of parameter N is determined by engineering calibration. In this embodiment, the shortest business abbreviation in the historical mapping samples is extracted, its character length is calculated and rounded down, and the specific value of N is 2. The ratio of the number of intersection elements to the number of union elements of the two character substring sets is calculated to obtain the character overlap ratio.

[0024] It should be noted that if both candidate field pairs are numeric types, this embodiment extracts the distribution quantile features of both, calculates the reciprocal of the Euclidean distance between the corresponding percentile values ​​in the spatial coordinate system, and obtains the numeric quantile difference. If the candidate field pairs are heterogeneous types or both are date types, this embodiment performs a hash value comparison on the format pattern strings of both; if the hash values ​​match, a value of 1 is assigned; otherwise, a value of 0 is assigned, obtaining the format matching degree. This embodiment uniformly defines the character overlap ratio, numeric quantile difference, or format matching degree calculated above as the type dimension difference.

[0025] For example, the calculation logic for obtaining the term co-occurrence frequency by performing frequency statistics on the candidate field pairs is as follows: In this embodiment, the absolute number of times the business phrase fragments contained in the candidate field pairs appear simultaneously in the annotation text of the historical data dictionary table is retrieved, and this absolute number is divided by the total number of times the single business phrase fragment with the most occurrences to obtain the normalized term co-occurrence frequency. In this embodiment, a vector concatenation operation is performed on the type dimension difference and the term co-occurrence frequency to generate a one-dimensional numerical vector, which is then extracted as the comparison feature set.

[0026] It is worth noting that in this embodiment, the comprehensive similarity is obtained by weighted summation of the comparison feature set using pre-configured weight coefficients. The pre-configured weight coefficients are determined using a performance-driven method. An initial training set is constructed by collecting positive samples of synonymous fields and negative samples of non-synonymous fields from historical business systems that have been manually labeled. To address the extreme imbalance in the number of positive and negative samples in the initial training set, this embodiment uses the Synthetic Minority Oversampling Technique (SMOTE algorithm) to interpolate positive samples of synonymous fields in the feature space, or introduces an inverse proportional penalty weight based on the sample class frequency in the subsequent loss function to construct a balanced training set. The logistic regression algorithm is used as the classifier model, with the goal of maximizing the log-likelihood function. The classifier model is iteratively trained on the balanced training set using a gradient-based optimization algorithm until the loss function converges. The linear weight parameters corresponding to each feature dimension in the trained classifier model are extracted, and the absolute values ​​of the linear weight parameters are normalized before being output as the pre-configured weight coefficients.

[0027] In one implementation, this embodiment uses a maximum-minimum normalization algorithm to linearly map the continuous numerical features in the comparison feature set, uniformly scaling them to a numerical range of 0 to 1. Then, this embodiment multiplies each normalized value in the comparison feature set with its corresponding pre-configured weight coefficient and sums the results to obtain the comprehensive similarity between 0 and 1. Finally, this embodiment constructs a two-dimensional matrix using all fields of the source system data as row and column indices, and fills the calculated comprehensive similarity into the corresponding row and column intersection positions of the two-dimensional matrix to generate the feature matching matrix.

[0028] In step S13, the feature matching matrix is ​​numerically compared with the calculated judgment threshold to obtain associated node pairs. A depth-first search algorithm is then used to traverse and calculate the associated node pairs to obtain a preliminary grouping set, including: The matrix elements in the feature matching matrix are compared with the judgment threshold to obtain strongly correlated feature values; The associated node pairs are obtained by establishing undirected connecting edges based on the matrix index of the feature matching matrix according to the strongly correlated feature values; An undirected graph structure is constructed using the associated node pairs, and the connected components are extracted by traversing the undirected graph structure using the depth-first search algorithm. The preliminary grouping set is obtained by performing aggregation calculations on the connected components and isolated nodes other than the undirected connected edges.

[0029] In one implementation, this embodiment compares the matrix elements in the feature matching matrix with the calculated judgment threshold to obtain strongly correlated feature values. It should be noted that the judgment threshold is determined using the maximum inter-class variance method in statistical analysis. In this embodiment, all off-diagonal matrix elements in the feature matching matrix are extracted to form a one-dimensional numerical sequence; the numerical space between 0 and 1 is divided into M equal intervals, with the parameter M specifically set to 100; all equal intervals are traversed as candidate thresholds, and the one-dimensional numerical sequence is divided into high-similarity classes (above the candidate threshold) and low-similarity classes (below the candidate threshold); the inter-class variance of the two classes is calculated respectively, and the candidate threshold corresponding to the maximum inter-class variance is selected and determined as the judgment threshold. In this embodiment, the feature matching matrix is ​​traversed, and matrix elements with values ​​greater than the judgment threshold are retained and extracted as the strongly correlated feature values.

[0030] For example, the specific implementation of establishing an undirected connection edge based on the matrix index of the feature matching matrix according to the strong correlation feature value to obtain the associated node pair is as follows: In this embodiment, the row index field and column index field corresponding to the generation of the strong correlation feature value are extracted, the row index field and column index field are combined to define the associated node pair, and an undirected connection edge connecting the two is generated in the corresponding data structure.

[0031] It is worth noting that each graph node in the undirected graph structure is physically and logically uniquely mapped to and corresponds to a set of representation fragments extracted in step S11 and its underlying original fields. In this embodiment, all mapped graph nodes are treated as a node set, and all undirected connecting edges are treated as an edge set to construct and generate the undirected graph structure.

[0032] It should be noted that the specific implementation logic of this embodiment for traversing the undirected graph structure and extracting connected components using the depth-first search algorithm is as follows: First, an empty list of visited nodes is initialized. Then, a graph node not present in the visited node list is randomly selected as the starting traversal point. Next, adjacent graph nodes are visited along the undirected edges connected to the starting traversal point, and the visited nodes are simultaneously added to the current connected component set and the visited node list. This edge-traversal action is repeated until there are no unvisited adjacent graph nodes on the current path. At this point, a backtracking operation is performed until all graph nodes directly or indirectly connected to the starting traversal point are added to the current connected component set, and this is output as a complete connected component. The above process of randomly selecting the starting traversal point and performing depth-first traversal is repeated until all graph nodes containing connected edges in the undirected graph structure are added to the visited node list, thus extracting all the connected components.

[0033] In one implementation, this embodiment performs aggregation calculations on the connected components and isolated nodes other than the undirected connecting edges to obtain the preliminary grouping set. This embodiment extracts graph nodes with a degree of zero from the undirected graph structure, defines them as isolated nodes, and encapsulates each isolated node into a single-element group; the connected components extracted in the previous steps are encapsulated into multi-element groups; all the single-element groups and all the multi-element groups are merged and added to the same list structure to generate the preliminary grouping set.

[0034] In step S14, the preliminary group set is mapped to obtain the underlying data samples, the underlying data samples are statistically analyzed for feature distribution to obtain the co-occurrence intensity sequence, and the context pattern set is obtained by topological extraction based on the co-occurrence intensity sequence.

[0035] Specifically, the co-occurrence intensity sequence is obtained by performing feature distribution statistics on the underlying data samples, and a context pattern set is obtained by topological extraction based on the co-occurrence intensity sequence, including: The common occurrence frequency and conditional probability values ​​are obtained by performing feature distribution statistics on the underlying data samples; Contextual features are constructed based on the co-occurrence frequency, and effective dependency units are obtained by weighting the contextual features using the conditional probability values. The co-occurrence intensity sequence is obtained by arranging the effective dependency units in descending order according to their weighted values; The context pattern set is obtained by extracting structured connection features from the co-occurrence intensity sequence.

[0036] In one implementation, this embodiment performs data mapping on the preliminary grouping set to obtain the underlying data sample. Specifically, this embodiment extracts all unique identifiers for each field within the preliminary grouping set, and uses these unique identifiers to perform a structured query language retrieval in the source system database to extract all business transaction records for the corresponding field within a fixed time window. This embodiment sorts the all business transaction records by timestamp and removes null rows, then outputs the underlying data sample. The length of the fixed time window is determined using the law of large numbers in statistics, specifically calculated as the minimum business statistical period required to satisfy the coverage of business data distribution characteristics, and its specific value is set to 30 days.

[0037] It should be noted that the specific implementation logic of obtaining the common occurrence frequency and conditional probability values ​​by performing feature distribution statistics on the underlying data samples in this embodiment is as follows: This embodiment uses a single business transaction record as the context boundary, and uses a pre-established business dictionary to perform word segmentation processing on the underlying data samples to extract business phrases.

[0038] It should be noted that the construction process of the pre-established business dictionary involves collecting plain text data of historical operation manuals within the corresponding business domain, using a Hidden Markov Model to segment and count word frequencies in the plain text data, calculating the probability value of each segmented word, and storing noun combinations and verb combinations with probability values ​​exceeding a preset probability threshold into a relational database. The preset probability threshold is determined by Zipf's law combined with a long-tail word frequency truncation threshold; in this embodiment, it is specifically set to 0.05 (i.e., the top 5% of high-frequency words). This embodiment counts the absolute number of times any two business phrases appear simultaneously in the same business transaction record and extracts this as the common occurrence frequency. Simultaneously, it counts the total number of times the prerequisite business phrase, serving as a condition, appears independently in all the underlying data samples. The common occurrence frequency is divided by the total number of occurrences, and the resulting value is extracted as the conditional probability value.

[0039] For example, the specific process of constructing contextual association features based on the co-occurrence frequency and using the conditional probability value to weight the contextual association features to obtain effective dependency units is as follows: In this embodiment, two business phrases with a co-occurrence frequency greater than zero and their positional order are combined and encapsulated into a directed triple structure containing a preceding node, an action node, and a following node. This directed triple structure is defined as the contextual association feature. In this embodiment, the conditional probability value is multiplied by the geometric mean of the basic feature weights of the preceding and following nodes in the contextual association feature to calculate a weighted value. The contextual association feature is then bound and encapsulated with the corresponding weighted value, and the output is the effective dependency unit. Assuming that the preceding phrase "overdue" occurs independently 1000 times, and it co-occurs with the following phrase "penalty interest" 950 times, the calculated conditional probability value is 0.95.

[0040] The basic feature weight is obtained by taking the logarithm of the frequency of each business phrase in the global data dictionary and dividing it by the sum of the logarithmic frequencies of all business phrases, and this normalized weight is used as the basic feature weight of the node. If the node is appearing for the first time, the basic feature weight is the default value of one divided by the total number of nodes in the current group. The global data dictionary is generated by aggregating all entity words in the pre-built business ontology library and standardized field annotation texts in the historical business system, and after deduplication and word frequency statistics, it serves as the statistical basis for the basic feature weight.

[0041] It is worth noting that in this embodiment, the effective dependency units are sorted in descending order according to their weighted values ​​to obtain the co-occurrence intensity sequence, and the co-occurrence intensity sequence is subjected to structured connection feature extraction to obtain the context pattern set. Specifically, in this embodiment, the effective dependency units with weighted values ​​exceeding a preset dependency threshold are extracted after traversing the descending order of the co-occurrence intensity sequence. The preset dependency threshold is determined using a performance-driven method, by plotting the receiver operation feature curves of precision and recall in the offline test set, finding the probability critical point value corresponding to the maximization of the Youden exponent, and setting it to 0.85. In this embodiment, effective dependency units with overlapping first and last nodes are directionally concatenated to construct a fixed evolution path combination containing three or more business phrases, and the set composed of all constructed fixed evolution path combinations is extracted as the context pattern set; wherein, the offline test set consists of business phrase pairs with confirmed dependency relationships in the historical business system as positive samples, and randomly selected business phrase pairs without dependency relationships as negative samples, with a positive-to-negative sample ratio of 1:1, and is constructed after manual verification.

[0042] In step S15, core term combinations are obtained by spatial distribution calculation based on the context pattern set. These core term combinations are then input into a pre-built business ontology library for mapping and retrieval to obtain a group semantic profile, including: Phrase extraction is performed on the context pattern set to obtain candidate phrase combinations, and position offset calculation is performed on the candidate phrase combinations to obtain the position offset variance; The candidate phrase combinations are sorted in ascending order based on the position offset variance to obtain high-confidence combinations, and the core term combinations are obtained by calculating the metric value of the high-confidence combinations. The core terms are combined and input into the pre-built business ontology library for mapping and retrieval to obtain a business meaning description. The dependency relationship of the business meaning description is parsed to obtain a semantic anchor table. The group semantic profile is obtained by structurally encapsulating the business meaning description and the semantic anchor table.

[0043] In one implementation, this embodiment extracts candidate phrase combinations from the context pattern set and calculates the positional offset variance from the candidate phrase combinations. This embodiment traverses the fixed evolution path contained in the context pattern set, extracting each pair of spatially adjacent business phrases as candidate phrase combinations. For each candidate phrase combination, this embodiment retrieves all occurrence positions in the entire set of underlying data samples, records the character index distance between the two phrases in the same text sequence, and uses the absolute value of this character index distance as the positional offset. This embodiment calculates the arithmetic mean of all recorded positional offsets, then calculates the sum of squares of the differences between each positional offset and the arithmetic mean, divides the sum of squares by the total number of occurrences, and outputs the calculated numerical result as the positional offset variance.

[0044] It should be noted that in this embodiment, the candidate phrase combinations are sorted in ascending order based on the positional offset variance to obtain high-confidence combinations. The core term combination is then calculated using a metric. In this embodiment, the candidate phrase combination with the smallest positional offset variance is placed at the beginning of the sequence. The top K candidate phrase combinations are selected as the high-confidence combinations. The value of parameter K is determined by calculating the information entropy gain rate of the combination under different K values, selecting the inflection point value where the gain rate tends to level off; specifically, it is set to 3. In this embodiment, the metric of the high-confidence combinations is calculated by dividing the common frequency of the high-confidence combination by its corresponding 1, adding the sum of the positional offset variances, and then normalizing the metric to the zero-to-one interval. The normalization factor is the maximum metric of all candidate phrase combinations within the current group to ensure comparability of the metric within the same group. In this embodiment, the high-confidence combination with the highest calculated metric is selected and output as the core term combination.

[0045] For example, assuming that the candidate phrase combination "principal" and "deduction" appear together 5000 times in the sample and their position offset is constant at 1, the calculated position offset variance is 0. Then the calculated metric value is 5000 divided by 1, and the result is 5000.

[0046] It is worth noting that in this embodiment, the core terminology combination is input into a pre-built business ontology library for mapping and retrieval to obtain a business meaning description. Dependency relation parsing is then performed on the business meaning description to obtain a semantic anchor table. The pre-built business ontology library is constructed by collecting national standards and industry regulatory norms texts, extracting entity words using a named entity recognition algorithm, and using a relation extraction algorithm based on a preset grammar rule template. For example, a regular expression matching template is applied, consisting of a first entity word, a state relation word, and a second entity word concatenated sequentially, where the state relation word is 'belongs to', 'contains', or 'corresponds to'; or a regular expression matching template is applied, consisting of an action entity word, an action relation word, and an object entity word concatenated sequentially, where the action relation word is 'acts on'. The hierarchical and attribute relationships between entity words are then extracted, and the extracted entity words and their associated hierarchical and attribute relationships are persistently stored using a graph database data structure.

[0047] In one implementation, this embodiment uses the core terminology combination as the query key and inputs it into the pre-built business ontology library for precise matching and retrieval. It extracts the standard definition text data bound to the matching graph nodes and outputs it as the business meaning description. This embodiment calls a dependency parsing algorithm to construct a syntax tree for the business meaning description, extracts dependency relationship paths containing subject-verb-object grammatical structures, and maps the subject, verb, and object components to corresponding column fields in a three-dimensional table structure, generating the semantic anchor table.

[0048] In one implementation, the business meaning description and the semantic anchor table can be structurally encapsulated to obtain the group semantic profile. Specifically, this embodiment creates an empty document object based on Extensible Markup Language (EXPLAIN) format, writes the business meaning description into the description attribute node of the document object, and simultaneously converts the three-dimensional table data in the semantic anchor table into nested text elements row by row, writing them into the structure attribute node of the document object. This embodiment then performs hash verification and binary compression encoding operations on the document object after the data writing is complete, generating an immutable structured file object, which is then output as the group semantic profile.

[0049] In step S16, a new inflow field is obtained, and semantic extraction is performed on the new inflow field to obtain candidate triples. A pre-trained word embedding model is used to calculate the spatial geometric distance between the candidate triples and the group semantic profile, and the true semantic type is determined based on the spatial geometric distance.

[0050] Specifically, semantic extraction is performed on the newly inflowing field to obtain candidate triples, and a pre-trained word embedding model is used to calculate the spatial geometric distance between the candidate triples and the semantic profile of the group, including: The context logic of the new inflow field is used to extract the associated nodes, and the candidate triplet is constructed based on the associated nodes. The logical structure of the semantic profiles of the groups is extracted to obtain a set of baseline anchor points; The candidate triples and the baseline anchor set are spatially mapped using the pre-trained word embedding model to obtain an initial spatial matrix. The initial spatial matrix is ​​then subjected to feature pooling to obtain candidate spatial vectors and baseline spatial vectors. The spatial geometric distance is obtained by calculating the angle between the candidate spatial vector and the reference spatial vector using the cosine similarity algorithm.

[0051] In one implementation, this embodiment obtains a new inflow field and extracts verbs and nouns adjacent to it within a set character window by scanning the business operation log text containing the new inflow field, defining them as associated nodes. It should be noted that the size of the set character window is determined based on the average topological span between the action subject and object in historical business logs; in this embodiment, it is specifically 5 to 10 characters. This embodiment arranges the predicate noun, the operation verb, and the new inflow field in syntactic dependency order to construct the candidate triplet with entity and action logical structure. The specific process of parsing the semantic profile of the group is as follows: extracting the standard logical dependency structure table data stored in its structural attribute nodes; serializing the standard business entity and action combinations recorded in the table data to extract a text set; and outputting this text set as the baseline anchor set. The business operation log text is collected from the application server log file of the source system, with a time range of thirty days prior to the current time. The collected content includes the field names and values ​​in the user operation interface, input parameters, and returned results.

[0052] It is worth noting that the pre-trained word embedding model is constructed by building a multi-layer bidirectional self-attention encoder network model, setting the number of network layers to 12, the hidden layer feature dimension to 768, and the number of heads in the multi-head attention mechanism to 12. Preprocessing is performed on a large-scale general corpus and a business domain text library. In this embodiment, word segmentation filtering is performed in conjunction with a pre-configured business-reserved vocabulary list. This business-reserved vocabulary list is automatically constructed by parsing standard English abbreviations (such as ID, SN, AMT, etc.) from the national data standard specification dictionary and the enterprise-level basic data element directory library. Only words not in the specified vocabulary are removed. The system retains meaningless stop words within the business vocabulary with a length less than 3 to ensure the feature integrity of key business abbreviations, and uses regular expressions to filter special business symbols. A Transformer-based encoder architecture is adopted, using masked language modeling as the fitting principle. Partial word blocks in the input sequence are replaced with masked tags, and their true words are predicted to construct a loss function. An adaptive moment estimation optimization algorithm is used to perform iterative training. The network weights are adjusted through backpropagation to minimize the loss function. After the loss function converges, fixed network parameters are extracted to obtain the pre-trained word embedding model.

[0053] In one implementation, this embodiment inputs the candidate triples and the baseline anchor set as text sequences into the pre-trained word embedding model, extracts the numerical matrix output from the last hidden layer, and defines it as the initial space matrix. The specific logic for feature pooling of the initial space matrix is ​​as follows: this embodiment introduces an attention mask matrix, forces the weights at the filling positions to a value of 0, and performs an arithmetic mean along the sequence length dimension only on the hidden layer vectors of valid business phrase fragments, thereby compressing and outputting a one-dimensional tensor of the same dimension, namely the candidate space vector corresponding to the candidate triples and the baseline space vector corresponding to the baseline anchor set.

[0054] In one implementation, this embodiment calculates the dot product between the candidate spatial vector and the reference spatial vector, divides the dot product by the product of the magnitudes of the two vectors, calculates the inverse cosine of the quotient, and extracts the absolute value of the inverse cosine as the spatial geometric distance.

[0055] The determination of the true semantic type based on the spatial geometric distance includes: The spatial geometric distance is compared with the dynamically determined attribution threshold to obtain the comparison and verification results; If the comparison and verification result indicates that the spatial geometric distance is less than the attribution determination threshold, then the new inflow field is mapped to the underlying semantic type corresponding to the group semantic profile, and the underlying semantic type is determined as the true semantic type; If the comparison and verification result indicates that the spatial geometric distance is greater than or equal to the attribution determination threshold, then an independent group is established based on the new inflow field, a classification label is generated based on the independent group, and the classification label is determined as the true semantic type.

[0056] In one implementation, this embodiment compares the spatial geometric distance with a dynamically determined attribution threshold to obtain a comparison verification result. The attribution threshold is determined using statistical analysis. A positive sample dataset is constructed from synonym pairs confirmed to have the same business meaning in a historical database. The spatial geometric distances of all synonym pairs in this positive sample dataset are calculated after mapping using the pre-trained word embedding model, and the numerical distribution of all calculated distances is statistically analyzed. This embodiment performs a skewness test on the numerical distribution of spatial distances. If the distribution exhibits significant long-tail characteristics, i.e., the calculated Fisher-Pearson skewness coefficient is greater than a preset coefficient (typically 1.0), the cumulative probability quantile is dynamically adjusted to the 90% to 95% range based on the long-tail decay rate. If there are no significant long-tail characteristics, the quantile value corresponding to the 95% cumulative probability distribution position is extracted by default, and this determined quantile value is set as the attribution threshold. This embodiment compares whether the current spatial geometric distance is less than the attribution threshold and outputs a Boolean state value as the comparison verification result.

[0057] It should be noted that if the Boolean value of the comparison verification result is true, indicating that the spatial geometric distance is less than the attribution determination threshold, this embodiment reads the classification attribute recorded in the metadata configuration table of the semantic profile of the group currently participating in the comparison, adds the data dictionary mapping relationship of the new inflow field to the underlying semantic type list corresponding to the classification attribute, extracts the underlying semantic type and assigns it as the real semantic type.

[0058] For example, if the Boolean value of the comparison verification result is false, indicating that the spatial geometric distance is greater than or equal to the attribution determination threshold, this embodiment allocates a new primary key identifier in the global relational database to establish the independent grouping. This embodiment extracts the original string of the newly inflowing field, uses a word segmentation tool to extract its core noun components, and concatenates them with a preset unknown group prefix to generate a string sequence. This string sequence is output as a classification label, and the classification label is written into the independent grouping as the true semantic type. The preset unknown group prefix is ​​obtained by concatenating a random string with the eight-digit date of the current system timestamp, serving as the classification label prefix for the independent grouping.

[0059] In step S17, source system change data is obtained; semantic drift features are obtained by cross-analysis of the source system change data and the group semantic profile; the feature matching matrix is ​​reconstructed based on the semantic drift features to obtain an updated matching matrix; topological calculation is performed on the updated matching matrix to obtain local aggregation density; and node reorganization is performed based on the local aggregation density to obtain an updated semantic profile, including: The source system change data is parsed for dependency relationships to obtain the affected field set, and the affected field set is sampled to obtain the underlying change sample; The semantic profiles of the groups are used to extract basic semantic features, and the semantic drift features are obtained by cross-analysis of the underlying change samples and the basic semantic features. The updated matching matrix is ​​obtained by reconstructing the underlying matching values ​​corresponding to the feature matching matrix based on the semantic drift features. The number of connecting edges and the number of nodes are obtained by extracting topological features from the updated matching matrix, and the local aggregation density is obtained by calculating the ratio of the number of connecting edges to the number of nodes. The fluctuation of the local aggregation density is measured to obtain the density change range. If the density change range is greater than the statistically determined density fluctuation threshold, the nodes of the affected field set are reassigned to obtain the recombined semantic cluster. The features of the recombined semantic cluster are summarized to obtain the updated semantic profile.

[0060] In one implementation, this embodiment obtains source system change data, performs abstract syntax tree parsing on the structured query language script contained within the source system change data, extracts all column fields associated with table structure modifications or operational logic adjustments in the script, merges and removes duplicates, and outputs the affected field set. In this embodiment, the most recently generated non-empty data rows for each field in the affected field set are read from the data storage system, and a fixed number of rows are extracted in reverse chronological order to form the underlying change sample.

[0061] It should be noted that in this embodiment, node attribute parsing is performed on the group semantic profile to extract the expected frequency of the recorded business terms distribution, and the output is the basic semantic feature. The specific calculation logic for cross-analysis of the underlying change sample and the basic semantic feature is as follows: the actual occurrence frequency of the core business terms extracted from the underlying change sample is counted, the relative entropy value between the actual occurrence frequency and the expected frequency contained in the basic semantic feature is calculated, and the relative entropy value is extracted as the semantic drift feature.

[0062] It is worth noting that in this embodiment, the row and column indices corresponding to the affected field set in the feature matching matrix are extracted. The attenuation factor generated by the above relative entropy value after nonlinear inverse mapping (the specific calculation logic is to use the constant value 1 as the numerator and the sum of the constant value 1 and the relative entropy value as the denominator to calculate the quotient of the two) is multiplied by the original underlying matching value at the intersection of the row and column. The calculated new product value is used to cover the original value to perform the reconstruction calculation operation. The matrix data modified after full field coverage is output as the updated matching matrix.

[0063] In one implementation, this embodiment locates the locally connected subgraph affected by the modification operation in the updated matching matrix, extracts the total number of unique graph nodes with a greater than zero associated node count in the locally connected subgraph, and defines it as the node count; it also extracts the total number of undirected connected edges with non-zero weight values ​​in the locally connected subgraph, and defines it as the connected edge count. This embodiment multiplies the connected edge count by a constant value of 2 to obtain a first product, and multiplies the node count by the difference between the node count and itself minus 1 to obtain a second product. It should be noted that, to avoid calculation errors, this embodiment pre-determines whether the node count is less than or equal to 1. If this condition is met, resulting in a second product of 0, the local aggregation density is directly assigned the minimum value of 0; otherwise, this embodiment divides the first product by the second product, and the calculated graph density value is extracted as the local aggregation density. The constant 2 is derived from the standard formula of twice the number of edges in the definition of graph density, used to convert the number of edges in the undirected graph into a sum of degrees. After cutting connected edges with weights below the average, the local aggregation density is recalculated based on the cut subgraph and used for comparison of subsequent density change magnitudes.

[0064] For example, this embodiment calculates the absolute difference between the local aggregation density of the current update cycle and the local aggregation density of the immediately preceding historical update cycle, and outputs it as the density change amplitude. If the density change amplitude is greater than a statistically determined density fluctuation threshold, the process for determining the density fluctuation threshold is as follows: The system statistically analyzes the historical local aggregation density sequence during a steady-state operation period without external structural changes or data inflow, calculates the arithmetic standard deviation of the time series, and extracts three times the arithmetic standard deviation as the density fluctuation threshold. In this embodiment, the system cuts off the connecting edges in the locally connected subgraph whose weight values ​​are lower than the average weight of the current subgraph. A density-based noise-applied spatial clustering algorithm is used to perform connectivity clustering on the graph topology after the connecting edges are cut. The neighborhood search radius of this algorithm is set to 0.5 times the average edge weight of the current subgraph, and the minimum number of nodes is set to 2, to generate multiple independent new connected components. The new connected components are output as the reconstructed semantic cluster. In this embodiment, all nodes in the reconstructed semantic cluster are traversed, all phrase components are re-statistically analyzed to construct a new dependency anchor table set, and the updated semantic profile is generated and output. After obtaining the updated semantic profile, this embodiment triggers a reverse synchronization mechanism to overwrite or append the new classification labels and reconstructed dependency anchor table in the updated semantic profile to the aforementioned pre-built business ontology library according to the graph database writing protocol, replacing the original semantic profiles of the corresponding groups. In subsequent steps, when obtaining new inflow fields, the updated business ontology library is used as the retrieval basis, thereby completing the global closed-loop iteration and adaptive evolution of the underlying business semantic dictionary.

[0065] In summary, this invention breaks the rigid dependence of traditional static mapping rules on physical surface features by performing multi-dimensional feature comparison and depth-first graph structure partitioning on multi-source heterogeneous data. By integrating contextual co-occurrence statistics of underlying data with feature pooling mapping of high-dimensional semantic space, it constructs accurate group semantic profiles, completely solving the matching failure and error cascading problems caused by implicit semantic differences such as homonyms or synonyms. Furthermore, by introducing a closed-loop feedback mechanism based on cross-entropy semantic drift and local aggregation density awareness, the system is endowed with the ability to reorganize topology nodes and reverse synchronize the ontology library under dynamic changes in the source system structure and business data. This invention ultimately achieves non-interventional, high-precision automatic identification and standardized alignment of the true semantics of massive multi-source heterogeneous fields, and realizes continuous adaptive evolution and autonomous correction of a unified data foundation and system standardized processing capabilities in complex and ever-changing digital business scenarios.

[0066] Reference Figure 2 The second embodiment of the present invention provides an automatic standardization processing system based on multi-source heterogeneous data, comprising: The data preprocessing module is used to acquire source system data and preprocess it to obtain a set of characterization fragments; The feature comparison module is used to perform feature comparison on the set of representation fragments to obtain a comparison feature set, calculate the comprehensive similarity, and construct a feature matching matrix based on the comprehensive similarity. The graph network partitioning module is used to compare the feature matching matrix with the calculated judgment threshold to obtain the associated node pairs, and to use the depth-first search algorithm to traverse and calculate the associated node pairs to obtain the preliminary grouping set. The feature distribution statistics module is used to perform data mapping on the preliminary group set to obtain the underlying data samples, perform feature distribution statistics on the underlying data samples to obtain the co-occurrence intensity sequence, and perform topological extraction based on the co-occurrence intensity sequence to obtain the context pattern set. The semantic anchoring module is used to calculate the spatial distribution of the context pattern set to obtain the core term combination, and input the core term combination into a pre-built business ontology library for mapping retrieval to obtain the group semantic profile. The dynamic mapping module is used to obtain new inflow fields, perform semantic extraction on the new inflow fields to obtain candidate triples, use a pre-trained word embedding model to calculate the spatial geometric distance between the candidate triples and the semantic profile of the group, and determine the true semantic type based on the spatial geometric distance. The adaptive evolution module is used to acquire source system change data, perform cross-analysis between the source system change data and the group semantic profile to obtain semantic drift features, reconstruct the feature matching matrix based on the semantic drift features to obtain an updated matching matrix, perform topological calculation on the updated matching matrix to obtain local aggregation density, and perform node reorganization based on the local aggregation density to obtain an updated semantic profile.

[0067] It should be noted that the automatic standardization processing system based on multi-source heterogeneous data provided in this embodiment of the invention is used to execute all the process steps of the automatic standardization processing method based on multi-source heterogeneous data in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0068] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0069] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. An automatic standardization processing method based on multi-source heterogeneous data, characterized in that, include: The source system data is acquired and preprocessed to obtain a set of representation fragments; wherein, the table structure is parsed on the table data definition language file in the source system data, the table structure definition information is extracted, and the extracted field names and field data types are mapped to field type identifiers, wherein the field type identifiers only include three fixed categories: text type, numeric type and date type. The set of representation fragments is subjected to feature comparison to obtain a comparison feature set, and a comprehensive similarity is calculated. A feature matching matrix is ​​constructed based on the comprehensive similarity. The feature matching matrix is ​​numerically compared with the calculated judgment threshold to obtain the associated node pairs. The depth-first search algorithm is then used to traverse and calculate the associated node pairs to obtain a preliminary group set. Data mapping is performed on the preliminary group set to obtain the underlying data samples. Feature distribution statistics are performed on the underlying data samples to obtain the co-occurrence intensity sequence. Topological extraction is performed based on the co-occurrence intensity sequence to obtain the context pattern set. The core term combination is obtained by spatial distribution calculation based on the context pattern set, and the core term combination is input into the pre-built business ontology library for mapping retrieval to obtain the group semantic profile. A new inflow field is obtained, and semantic extraction is performed on the new inflow field to obtain candidate triples. A pre-trained word embedding model is used to calculate the spatial geometric distance between the candidate triples and the semantic profile of the group, and the true semantic type is determined based on the spatial geometric distance. Obtain source system change data, perform cross-analysis between the source system change data and the group semantic profile to obtain semantic drift features, reconstruct the feature matching matrix based on the semantic drift features to obtain an updated matching matrix, perform topological calculation on the updated matching matrix to obtain local aggregation density, and reorganize nodes based on the local aggregation density to obtain an updated semantic profile.

2. The automatic standardization processing method based on multi-source heterogeneous data according to claim 1, characterized in that, The process of acquiring source system data and preprocessing it to obtain a set of characterization fragments includes: The source system data is parsed to obtain an initial record list and field type identifiers, and the initial record list is checked for non-emptiness to obtain a sample value list. Based on the field type identifier, the sample value list is subjected to split feature extraction to obtain morphological feature data; The morphological feature data is structured and encapsulated to obtain the set of representation fragments.

3. The automatic standardization processing method based on multi-source heterogeneous data according to claim 1, characterized in that, The step of performing feature comparison on the set of representation fragments to obtain a comparison feature set and calculating the comprehensive similarity includes: The candidate field pairs are obtained by combining the set of representation fragments in pairs; The candidate field pairs are classified and compared to obtain the differences in type dimension, and the candidate field pairs are frequency-statistically analyzed to obtain the term co-occurrence frequency; The comparison feature set is obtained by merging and concatenating the type dimension differences and the term co-occurrence frequencies. The comprehensive similarity is obtained by weighting and summing the comparison feature sets using pre-configured weight coefficients.

4. The automatic standardization processing method based on multi-source heterogeneous data according to claim 1, characterized in that, The process involves numerically comparing the feature matching matrix with the calculated judgment threshold to obtain associated node pairs, and then using a depth-first search algorithm to traverse and calculate the associated node pairs to obtain a preliminary grouping set, including: The matrix elements in the feature matching matrix are compared with the judgment threshold to obtain strongly correlated feature values; The associated node pairs are obtained by establishing undirected connecting edges based on the matrix index of the feature matching matrix according to the strongly correlated feature values; An undirected graph structure is constructed using the associated node pairs, and the connected components are extracted by traversing the undirected graph structure using the depth-first search algorithm. The preliminary grouping set is obtained by performing aggregation calculations on the connected components and isolated nodes other than the undirected connected edges.

5. The automatic standardization processing method based on multi-source heterogeneous data according to claim 1, characterized in that, The step of performing feature distribution statistics on the underlying data samples to obtain a co-occurrence intensity sequence, and then performing topological extraction based on the co-occurrence intensity sequence to obtain a contextual pattern set, includes: The common occurrence frequency and conditional probability values ​​are obtained by performing feature distribution statistics on the underlying data samples; Contextual features are constructed based on the co-occurrence frequency, and effective dependency units are obtained by weighting the contextual features using the conditional probability values. The co-occurrence intensity sequence is obtained by arranging the effective dependency units in descending order according to their weighted values; The context pattern set is obtained by extracting structured connection features from the co-occurrence intensity sequence.

6. The automatic standardization processing method based on multi-source heterogeneous data according to claim 1, characterized in that, The process of calculating the spatial distribution of core term combinations based on the context pattern set, and then inputting these core term combinations into a pre-built business ontology library for mapping and retrieval to obtain a group semantic profile, includes: Phrase extraction is performed on the context pattern set to obtain candidate phrase combinations, and position offset calculation is performed on the candidate phrase combinations to obtain the position offset variance; The candidate phrase combinations are sorted in ascending order based on the position offset variance to obtain high-confidence combinations, and the core term combinations are obtained by calculating the metric value of the high-confidence combinations. The core terms are combined and input into the pre-built business ontology library for mapping and retrieval to obtain a business meaning description. The dependency relationship of the business meaning description is parsed to obtain a semantic anchor table. The group semantic profile is obtained by structurally encapsulating the business meaning description and the semantic anchor table.

7. The automatic standardization processing method based on multi-source heterogeneous data according to claim 1, characterized in that, The process of semantically extracting candidate triples from the newly inflowing field and calculating the spatial geometric distance between the candidate triples and the group semantic profile using a pre-trained word embedding model includes: The context logic of the new inflow field is used to extract the associated nodes, and the candidate triplet is constructed based on the associated nodes. The logical structure of the semantic profiles of the groups is extracted to obtain a set of baseline anchor points; The candidate triples and the baseline anchor set are spatially mapped using the pre-trained word embedding model to obtain an initial spatial matrix. The initial spatial matrix is ​​then subjected to feature pooling to obtain candidate spatial vectors and baseline spatial vectors. The spatial geometric distance is obtained by calculating the angle between the candidate spatial vector and the reference spatial vector using the cosine similarity algorithm.

8. The automatic standardization processing method based on multi-source heterogeneous data according to claim 1, characterized in that, The process of determining the true semantic type based on the spatial geometric distance includes: The spatial geometric distance is compared with the dynamically determined attribution threshold to obtain the comparison and verification results; If the comparison and verification result indicates that the spatial geometric distance is less than the attribution determination threshold, then the new inflow field is mapped to the underlying semantic type corresponding to the group semantic profile, and the underlying semantic type is determined as the true semantic type; If the comparison and verification result indicates that the spatial geometric distance is greater than or equal to the attribution determination threshold, then an independent group is established based on the new inflow field, a classification label is generated based on the independent group, and the classification label is determined as the true semantic type.

9. The automatic standardization processing method based on multi-source heterogeneous data according to claim 1, characterized in that, The process of acquiring source system change data, performing cross-analysis between the source system change data and the group semantic profile to obtain semantic drift features, reconstructing the feature matching matrix based on the semantic drift features to obtain an updated matching matrix, performing topological calculation on the updated matching matrix to obtain local aggregation density, and reorganizing nodes based on the local aggregation density to obtain an updated semantic profile includes: The source system change data is parsed for dependency relationships to obtain the affected field set, and the affected field set is sampled to obtain the underlying change sample; The semantic profiles of the groups are used to extract basic semantic features, and the semantic drift features are obtained by cross-analysis of the underlying change samples and the basic semantic features. The updated matching matrix is ​​obtained by reconstructing the underlying matching values ​​corresponding to the feature matching matrix based on the semantic drift features. The number of connecting edges and the number of nodes are obtained by extracting topological features from the updated matching matrix, and the local aggregation density is obtained by calculating the ratio of the number of connecting edges to the number of nodes. The fluctuation of the local aggregation density is measured to obtain the density change range. If the density change range is greater than the statistically determined density fluctuation threshold, the nodes of the affected field set are reassigned to obtain the recombined semantic cluster. The features of the recombined semantic cluster are summarized to obtain the updated semantic profile.

10. An automatic standardization processing system based on multi-source heterogeneous data, characterized in that, include: The data preprocessing module is used to acquire source system data and preprocess it to obtain a set of representation fragments. Specifically, it performs table structure parsing on the table data definition language file in the source system data, extracts table structure definition information, and maps the extracted field names and field data types to field type identifiers. The field type identifiers only include three fixed categories: text type, numeric type, and date type. The feature comparison module is used to perform feature comparison on the set of representation fragments to obtain a comparison feature set, calculate the comprehensive similarity, and construct a feature matching matrix based on the comprehensive similarity. The graph network partitioning module is used to compare the feature matching matrix with the calculated judgment threshold to obtain the associated node pairs, and to use the depth-first search algorithm to traverse and calculate the associated node pairs to obtain the preliminary grouping set. The feature distribution statistics module is used to perform data mapping on the preliminary group set to obtain the underlying data samples, perform feature distribution statistics on the underlying data samples to obtain the co-occurrence intensity sequence, and perform topological extraction based on the co-occurrence intensity sequence to obtain the context pattern set. The semantic anchoring module is used to calculate the spatial distribution of the context pattern set to obtain the core term combination, and input the core term combination into a pre-built business ontology library for mapping retrieval to obtain the group semantic profile. The dynamic mapping module is used to obtain new inflow fields, perform semantic extraction on the new inflow fields to obtain candidate triples, use a pre-trained word embedding model to calculate the spatial geometric distance between the candidate triples and the semantic profile of the group, and determine the true semantic type based on the spatial geometric distance. The adaptive evolution module is used to acquire source system change data, perform cross-analysis between the source system change data and the group semantic profile to obtain semantic drift features, reconstruct the feature matching matrix based on the semantic drift features to obtain an updated matching matrix, perform topological calculation on the updated matching matrix to obtain local aggregation density, and perform node reorganization based on the local aggregation density to obtain an updated semantic profile.

Images