Data processing method and system for enterprise digital transformation platform

Abstract

The embodiment of the invention provides a data processing method and system for an enterprise digital transformation platform, and belongs to the field of data processing. The method comprises the steps that structured field information from all heterogeneous data sources is acquired, and preprocessing operation is executed on the structured field information; constructing the processed structured field information into an embedded input sequence, splicing the embedded input sequence into a natural language fragment according to a preset template, and inputting the natural language fragment into a fine-tuned semantic coding model to obtain a corresponding semantic embedded vector; identifying similar field groups by adopting a clustering algorithm based on density or a hierarchical structure, and classifying each group of structured field information into a semantic cluster; and generating a corresponding standard field identifier for each semantic clustering cluster, and storing the generated standard field identifier in a standard field index database of the platform after digital transformation. According to the scheme, the field unified management and cross-system data alignment capability of the enterprise digital platform is remarkably enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing method for an enterprise digital transformation platform and a data processing system for an enterprise digital transformation platform. Background Art

[0002] With the deep promotion of enterprise digital transformation, enterprise-level platforms need to uniformly manage and integrate heterogeneous structured data from multiple business systems, external interfaces, and historical data tables. There are often significant differences in field naming specifications, semantic expressions, data types, and structural formats among these data sources, resulting in serious obstacles in data fusion, cross-system querying, metric normalization, and data governance. Therefore, how to perform semantic alignment and standardization processing on the fields in structured data has become one of the key technical issues in the design of the digital platform's middle layer.

[0003] In the prior art, for field standardization, methods such as manually presetting mapping rules, field name matching dictionaries, regular parsing, or static templates are mainly used. Such methods rely on expert experience to build a rule library, which not only has high costs and difficult maintenance, but also is difficult to cover new data sources that change rapidly or have strong domain professionalism. In cases where there are semantic differences but similar names between fields, or semantic proximity but diverse naming methods, traditional rule matching methods cannot accurately identify the potential semantic consistency between fields, resulting in unsatisfactory field merging and standard field generation effects.

[0004] In addition, some solutions attempt to introduce pre-trained language models to improve the field semantic understanding ability, but do not combine the "field name + field description + example value" composite feature structure inherent in structured fields. Only using the field name unit as the model input, the semantic information is extremely sparse, affecting the clustering effect, and ultimately unable to construct a high-quality standard field index system, restricting the improvement of the platform-level data governance and intelligent query capabilities.

[0005] In summary, the existing solutions still have problems such as single field understanding, insufficient model adaptation, and rough semantic clustering in the semantic standardization of heterogeneous structured fields. There is an urgent need for a more intelligent, refined, and structure-aware field semantic standardization method to improve the field alignment efficiency and data availability of enterprise digital transformation platforms. Summary of the Invention

[0006] The purpose of the embodiments of the present invention is to provide a data processing method and system for an enterprise digital transformation platform, so as to at least solve the problems of weak semantic understanding ability of heterogeneous structured fields, dependence on manual rules in the field standardization process, and inaccurate generation of standard fields in the prior art.

[0007] To achieve the above object, a first aspect of the present invention provides a data processing method for an enterprise digital transformation platform, the method comprising: when performing data aggregation of various heterogeneous data sources, obtaining structured field information from various heterogeneous data sources, and performing a preprocessing operation on the structured field information; constructing the processed structured field information into an embedded input sequence, splicing the embedded input sequence into a natural language segment according to a preset template, and inputting the natural language segment into a fine-tuned semantic encoding model to obtain a corresponding semantic embedding vector; identifying similar field groups by using a density-based or hierarchical clustering algorithm, and classifying each group of structured field information into a semantic clustering cluster; generating a corresponding standard field identifier for each semantic clustering cluster, and storing the generated standard field identifier in the standard field index library of the platform after digital transformation.

[0008] Optionally, performing a preprocessing operation on the structured field information includes: performing unified character encoding conversion, full lowercase conversion, and punctuation cleaning on the field name, field description, and field example value in the structured field information respectively; using a built-in tokenizer to perform language segmentation on the field description and field example value, and removing stop words; based on a preset domain term library, performing a mapping replacement operation on the terms in the field description, the preset domain database including the corresponding relationship between synonymous terms and standard terms existing in multiple heterogeneous systems; performing a format recognition operation on the field example value, recognizing date format, numerical unit, and / or currency symbol, and generating a structured type tag according to the recognition result to obtain the preprocessed structured field information.

[0009] Optionally, constructing the processed structured field information into an embedded input sequence includes: constructing a natural language template containing the field name, field description, and field example value for each piece of structured field information, and constructing the corresponding structured field information into a unified text segment according to a predefined format and the constructed template; encoding the obtained text segment into a standard Token sequence, and processing low-frequency characters or unrecognized words in the Token sequence by using a subword segmentation mechanism to obtain the embedded input sequence of each piece of structured field information; after obtaining the embedded input sequence of each piece of structured field information, the method further includes: performing batch normalization processing on all embedded input sequences, the normalization processing including length padding, Mask processing, and field boundary annotation.

[0010] Optionally, splice the embedded input sequence into natural language fragments according to a preset template and input them into the fine-tuned semantic encoding model to obtain corresponding semantic embedding vectors, including: inputting the natural language fragments in the embedded input sequence into a language model based on the Transformer structure; introducing three types of marker vectors at the model input end, which are respectively used to mark the field name segment, the field description segment, and the example value segment; extracting sequence-level semantic features through the hidden layer of the semantic encoding model, and performing average pooling on the sequence-level semantic features to generate fixed-length semantic embedding vectors; performing layer normalization and Dropout operations on the fixed-length semantic embedding vectors, and using the fixed-length semantic embedding vectors as the semantic embedding vectors that uniquely represent the structured field information.

[0011] Optionally, use a density-based or hierarchical clustering algorithm to identify similar field groups and classify each group of structured field information into a semantic clustering cluster, including: constructing a vector space index structure based on all semantic embedding vectors, the index structure uses an inverted index and combines local sensitive hashing; performing a preliminary cluster division based on the density clustering algorithm and the spatial density of the semantic embedding vectors to obtain initial clustering clusters; performing principal component analysis on each initial clustering cluster, identifying the corresponding distribution main axis, and judging whether there are polysemous fields in the corresponding initial clustering cluster, and taking the initial clustering cluster with polysemous fields as a polysemous cluster; using a sub-clustering algorithm to perform a secondary division on the polysemous cluster to obtain multiple sub-clusters of each polysemous cluster, and using the central vector within the cluster as the representative vector of each sub-cluster; marking the structured field information that cannot be assigned to any cluster as semantic isolated fields and recording them in the candidate field table.

[0012] Optionally, generate corresponding standard field identifiers for each semantic clustering cluster, including: in each semantic clustering cluster, select the structured field information with the semantic embedding vector closest to the cluster center as the representative field; extract the field name from the representative field and perform character regular cleaning to remove the naming redundancy structure to obtain the input field name; use the fine-tuned language model to regenerate the standard field name for the input field name, and perform a semantic difference comparison with the original input field name, and retain the input field name with a similarity higher than the similarity threshold; summarize the field type information of all structured field information in the corresponding semantic clustering cluster, and determine the unified field type based on the majority rule; write the generated input field name, field type, the semantic embedding vector of the representative field, and the source system to which it belongs into the standard field identifier document, and generate a unique identifier code as the standard field identifier of the corresponding semantic clustering cluster.

[0013] Optionally, after obtaining the standard field identifiers of each semantic clustering cluster, the method further includes establishing a mapping record for the standard field identifiers; establishing a mapping record for the standard field identifiers includes: generating a mapping entry for the structured field information associated with each standard field identifier, the mapping entry including a field name, a field description, a field example value, a source system identifier, and a clustering cluster ID; organizing the mapping entries according to the standard field identifiers to form a mapping relation table; numbering the version of the mapping relation table, marking the generation time, the version of the clustering algorithm used, and the semantic model identifier; establishing a one-to-one binding relationship between the mapping relation table and the standard field identifiers to obtain a corresponding mapping record; and writing the mapping record into the standard field index library.

[0014] Optionally, before storing the generated standard field identifiers into the standard field index library of the platform after digital transformation, the method further includes performing a uniqueness check on the standard field identifiers; the uniqueness check on the standard field identifiers includes: calculating the pairwise cosine similarity between the semantic embedding vectors of the standard field identifiers to be stored and the embedding vectors of all existing standard field identifiers in the index library; performing aggregation analysis on the standard field identifiers with similarity exceeding a set threshold to determine whether there is a duplicate definition; if there is a duplicate definition, calling an artificial review interface and providing an embedding vector distribution map and the original field example content as auxiliary information, and in response to a review instruction, performing operations such as retaining, merging, or replacing the existing standard field identifiers, and recording the version number of the operation behavior and the operator ID; if there is no duplicate situation, directly storing it into the standard field index library and recording it as a newly added field.

[0015] A second aspect of the present invention provides a data processing system for an enterprise digital transformation platform, the system including: a collection unit, configured to obtain structured field information from each heterogeneous data source and perform a preprocessing operation on the structured field information when performing data aggregation of each heterogeneous data source; a vector generation unit, configured to construct the processed structured field information into an embedding input sequence, splice the embedding input sequence into a natural language segment according to a preset template, and input it into a fine-tuned semantic encoding model to obtain a corresponding semantic embedding vector; a clustering unit, configured to identify similar field groups by using a density-based or hierarchical clustering algorithm and classify each group of structured field information into a semantic clustering cluster; and a storage unit, configured to generate a corresponding standard field identifier for each semantic clustering cluster and store the generated standard field identifier into the standard field index library of the platform after digital transformation.

[0016] On the other hand, the present invention provides a computer-readable storage medium, on which instructions are stored, and when the instructions run on a computer, the computer is caused to execute the above-mentioned data processing method for an enterprise digital transformation platform.

[0017] Through the above technical solution, the solution of the present invention enhances the expression of structured field information by introducing a semantic encoding model, improves the modeling ability of potential semantic relationships between fields in heterogeneous data sources, and overcomes the limitations of traditional field standardization methods that rely on manual rules or surface name matching. By constructing natural language fragments and inputting them into the fine-tuned semantic model to obtain semantic embedding vectors, the field clustering process becomes more accurate and has stronger context awareness. Combining density-based or hierarchical clustering algorithms can effectively identify groups of fields with similar semantics and improve the accuracy of field normalization and standard field generation. The finally generated standard field identifiers are uniformly stored in the standard field index library, which helps to build a highly consistent and highly scalable field management system, significantly enhancing the field unified management and cross-system data alignment capabilities of the enterprise digital platform.

[0018] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent specific embodiments section. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. They are used together with the following specific embodiments to explain the embodiments of the present invention, but do not limit the embodiments of the present invention. In the drawings: Figure 1 is a flowchart of the steps of a data processing method for an enterprise digital transformation platform provided by an embodiment of the present invention; Figure 2 is a system structure diagram of a data processing system for an enterprise digital transformation platform provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The following will describe in detail the specific embodiments of the present invention with reference to the drawings. It should be understood that the specific embodiments described herein are only for explaining and illustrating the present invention and are not used to limit the present invention.

[0021] Figure 1 is a method flowchart of a data processing method for an enterprise digital transformation platform provided by an embodiment of the present invention. As Figure 1 shown, an embodiment of the present invention provides a data processing method for an enterprise digital transformation platform, and the method includes: Step S10: When performing data aggregation of various heterogeneous data sources, obtain structured field information from various heterogeneous data sources and perform preprocessing operations on the structured field information.

[0022] Specifically, perform preprocessing operations on the structured field information, including: performing unified character encoding conversion, all-lowercase conversion, and punctuation cleaning on the field name, field description, and field example value in the structured field information; using a built-in tokenizer to perform language segmentation on the field description and field example value, and removing stop words; based on a preset domain term library, performing mapping replacement operations on the terms in the field description, where the preset domain database contains the synonymous terms existing in multiple heterogeneous systems and their corresponding standard terms; performing format recognition operations on the field example value, identifying date formats, numerical units, and / or currency symbols, and generating structured type tags according to the recognition results to obtain the preprocessed structured field information.

[0023] In the embodiment of the present invention, when performing data aggregation of various heterogeneous data sources, first obtain structured field information from multiple business systems, external interfaces, or historical databases. The structured field information includes field-level metadata such as field names, field descriptions, and field example values. Since there are significant differences in field naming conventions, language expressions, data formats, etc. among different data sources, directly fusing and using them will cause difficulties in field alignment. Therefore, it is necessary to perform unified preprocessing operations on the above-mentioned structured field information to improve its semantic resolvability and input consistency.

[0024] Specifically, perform character standardization processing on the field name, field description, and field example value uniformly. This includes converting all characters to the UTF-8 encoding format to eliminate differences in encoding methods between different systems; converting all English letters to lowercase to avoid affecting semantic recognition due to mixed use of uppercase and lowercase; at the same time, cleaning the punctuation marks in the field text, removing symbols without actual semantics, such as commas, quotation marks, parentheses, etc., to ensure the simplicity and consistency of semantic representation.

[0025] Furthermore, for the content of the field description and field example value, use a built-in tokenizer (such as a composite tokenizer based on a dictionary + statistical rules), which is especially suitable for processing semantic block division in a Chinese-English mixed environment. After tokenization, further remove stop words, such as words with no actual semantic contribution like "de", "wei", "he", "in", "of", etc., to improve the density and semantic focusing ability of the subsequent input information of the model.

[0026] Further, based on a preset domain term library, perform term standardization replacement on the professional terms in the field description. The domain term library is constructed by summarizing the common term variants and their standard expressions in each business domain within the enterprise, supporting many-to-one mapping. For example, "customer number", "user ID", and "customer code" are all mapped to "customer identifier". Through this operation, the risk of semantic splitting caused by expression differences is significantly reduced, laying a foundation for field clustering and semantic modeling. In addition, perform type recognition processing on the field example values, and adopt a hybrid strategy of regular rules and dictionaries to identify their data types and context meanings. For example, date formats (such as yyyy-mm-dd, dd / mm / yyyy, etc.), numerical units (such as kg, m², %), and currency symbols (such as ¥, $, €) can be automatically recognized, and a structured type tag vector is generated and attached to the field information. This tag not only improves the model's understanding ability of the field usage but also provides interpretable feature support for clustering or downstream field naming generation.

[0027] Through the above multi-dimensional preprocessing operations, unified standard processing of structured field information in character format, semantic expression, and type tags is achieved. In terms of technical effects, on the one hand, the standardization and expression integrity of the input of the subsequent semantic encoding model are significantly improved, reducing semantic ambiguity and redundancy, and enhancing the model's ability to distinguish and aggregate field semantics; on the other hand, this structured preprocessing framework is scalable, facilitating adaptation to the semantic standard rules of different enterprises or fields, contributing to the construction of a reusable and highly accurate standard field recognition and management system, and providing underlying support for heterogeneous field alignment, standardized indexing, and cross-system integration in the enterprise digital platform.

[0028] Step S20: Construct the processed structured field information into an embedded input sequence, splice the embedded input sequence into a natural language fragment according to a preset template, and input it into the fine-tuned semantic encoding model to obtain the corresponding semantic embedding vector.

[0029] Specifically, constructing the processed structured field information into an embedded input sequence includes: constructing a natural language template containing the field name, field description, and field example value for each piece of structured field information, and constructing the corresponding structured field information into a unified text fragment using a predefined format and the constructed template; encoding the obtained text fragment into a standard Token sequence, and processing the low-frequency characters or unrecognized words in the Token sequence using a subword segmentation mechanism to obtain the embedded input sequence of each piece of structured field information; after obtaining the embedded input sequence of each piece of structured field information, the method further includes: performing batch normalization processing on all embedded input sequences, and the normalization processing includes length padding, Mask processing, and field boundary annotation.

[0030] Further, splicing the embedded input sequence into natural language fragments according to a preset template and inputting the fragments into a fine-tuned semantic encoding model to obtain corresponding semantic embedding vectors, including: inputting the natural language fragments in the embedded input sequence into a language model based on the Transformer structure; introducing three types of marker vectors at the input end of the model, which are respectively used to mark the field name segment, the field description segment, and the example value segment; extracting sequence-level semantic features through the hidden layer of the semantic encoding model, and performing average pooling on the sequence-level semantic features to generate a fixed-length semantic embedding vector; performing layer normalization and Dropout operations on the fixed-length semantic embedding vector, and using the fixed-length semantic embedding vector as the semantic embedding vector that uniquely represents the structured field information.

[0031] In the embodiments of the present invention, after completing the preprocessing operations of the structured field information of various heterogeneous data sources, in order to further achieve semantic-level field aggregation and standard field generation, it is necessary to construct an embedding representation and perform semantic encoding on the preprocessed structured field information. The goal of this step is to transform traditional structured field information into an input form that can be processed by a language model, and then obtain a vector representation with unified semantic expression ability through deep semantic modeling, which serves as the basis for subsequent semantic clustering and standard field generation.

[0032] Specifically, first, each processed structured field information is constructed into an embedded input sequence. Structured field information usually includes three types of content: field name, field description, and field example value. To ensure the consistency and integrity of information input, it is necessary to construct a unified natural language template for organizing these three types of content into text segments that can be understood by the language model. For example, the following natural language template structure can be defined: "The field name is [X], its description is [Y], and for example, the value is [Z]." Where [X], [Y], and [Z] correspond to the field name, the field description, and the field example value respectively. This template not only retains the structural information of the field but also strengthens the semantic relationship between different components within the field through language connection, enabling the language model to better capture context information during processing.

[0033] Next, encode the above natural language template text fragments into a standard Token sequence. Tokenization is a prerequisite for a language model to understand text. Usually, the built-in vocabulary of the model is used for encoding to break the string into recognizable tokens. During this process, if there are low-frequency words or domain-specific terms that do not exist in the model vocabulary, to avoid information loss or encoding errors, a subword segmentation mechanism (such as Byte Pair Encoding, SentencePiece, etc.) can be introduced to further decompose the unknown word into several common subword units, so that even if the model is not familiar with the overall vocabulary, it can obtain its semantic features from the components.

[0034] To improve the inference efficiency and alignment consistency of the subsequent model, it is also necessary to perform unified batch normalization on the Token sequence. Normalization includes three aspects: First is length padding. To make each input sequence have a consistent length when entering the language model, sequences with insufficient length are padded with special tokens (such as [PAD]) to align to the maximum length. Second is masking. By generating a Mask vector to mask the padded part, preventing it from participating in semantic calculations. Finally is field boundary annotation. By inserting or appending position identifiers (such as segment ids) in the Token sequence, the start and end positions of the field name segment, description segment, and example value segment are clearly defined, providing guidance for the model to extract the semantics within the structure later.

[0035] After completing the construction of the embedded input sequence, it is input into the fine-tuned semantic encoding model in a preset format. This semantic encoding model uses a language model based on the Transformer architecture, which has the ability to model context semantics and can effectively capture the deep semantic relationships in the input text. At the model input stage, to further enhance the model's perception ability of the field structure, three types of auxiliary marker vectors are introduced, corresponding to the field name segment, field description segment, and field example value segment respectively. Each type of marker vector serves as a "structural guidance signal" and is concatenated in the Embedding layer, enabling the model to distinguish the differences in semantic roles of different segments when performing the multi-head attention mechanism, and thus learning more discriminative field structure semantic features in the hidden layer.

[0036] During the forward propagation of the model, the semantic representations in the input sequence are extracted layer by layer through multiple encoding layers of the Transformer. In the last layer of the hidden layer, the context representation of each Token position is extracted, and the sequence-level features are compressed into a fixed-length vector through the MeanPooling strategy, serving as the overall semantic embedding vector of this piece of structured field information. Compared with directly using a vector at a specific position (such as the [CLS] position), MeanPooling can more fully integrate the information of each part in the entire field text, improving the stability and generality of the semantic representation.

[0037] To further improve the usability of the representation and the robustness of the training process, layer normalization and dropout operations are performed on the generated fixed-length embedding vectors. On the one hand, layer normalization helps to stabilize the vector distribution among different input batches, avoiding the impact of input data distribution differences on the subsequent model performance. On the other hand, the dropout operation effectively prevents the model from overfitting during fine-tuning by randomly masking the outputs of some neurons, enhancing the generalization ability of the embedding representation. Finally, the obtained fixed-length semantic embedding vectors, as the unique semantic expressions of each piece of structured field information, can be directly used for similarity measurement, clustering analysis, and semantic alignment operations among fields in the semantic space. This vector not only integrates the naming information and semantic descriptions of the fields but also provides supplementary context scenarios through example values, so it has stronger semantic expression ability than traditional field name-based matching methods.

[0038] Based on the solution of the present invention, through natural language templates and standardized Token processing, the structures of different source fields are made consistent at the model input level, reducing the sensitivity of the model to input differences. The field names, descriptions, and example values are unified and integrated into the embedding sequence to achieve semantic complementation and enhance the accuracy of semantic modeling. Paragraph identifiers and boundary annotations are introduced to enable the model to recognize the internal structure of the fields and improve the ability to distinguish the semantic roles of the fields. Through sub-word segmentation, normalization processing, and the dropout mechanism, the adaptability of the model to new domain terms and edge cases is improved. The generated semantic embedding vectors have good context awareness and field expression integrity, and are suitable as the input basis for field clustering and standard field generation.

[0039] Step S30: Use a density-based or hierarchical clustering algorithm to identify groups of similar fields and classify each piece of structured field information into a semantic clustering cluster.

[0040] Specifically, a vector space index structure is constructed based on all semantic embedding vectors. The index structure uses an inverted index and combines local sensitive hashing. Based on the density clustering algorithm and the spatial density of the semantic embedding vectors, a preliminary cluster division is performed to obtain initial clustering clusters. Principal component analysis is performed on each initial clustering cluster to identify the corresponding main distribution axis and determine whether there are polysemous fields in the corresponding initial clustering cluster. The initial clustering clusters with polysemous fields are regarded as polysemous clusters. The polysemous clusters are further divided using a sub-clustering algorithm to obtain multiple sub-clusters of each polysemous cluster, and the central vector within the cluster is used as the representative vector of each sub-cluster. The structured field information that cannot be assigned to any cluster is marked as a semantic isolated field and recorded in the candidate field table.

[0041] In an embodiment of the present invention, to achieve the classification and standardization processing of structured field information from heterogeneous data sources at the semantic level, after obtaining the semantic embedding vectors of each piece of structured field information, the present invention uses a density-based or hierarchical clustering algorithm to perform semantic clustering analysis on all semantic embedding vectors to identify field groups with similar expressed meanings or consistent functions, and classify them into unified semantic clustering clusters, providing a basis for the subsequent generation of standard field identifiers.

[0042] Specifically, first, before the clustering operation, a vector space index structure is constructed based on all semantic embedding vectors. Considering that the number of fields may reach tens of thousands or even a higher order of magnitude, the efficiency of traditional linear similarity search is relatively low. Therefore, in this embodiment, an inverted index structure combined with a locality-sensitive hashing (LSH) mechanism is used to construct a semantic index. The inverted index buckets and classifies field vectors according to hash signatures, enabling vector lookup to no longer traverse all fields but only perform similarity retrieval in candidate buckets, greatly improving the similarity query efficiency before clustering.

[0043] Subsequently, based on the constructed index structure, a density-based clustering algorithm (such as DBSCAN, HDBSCAN, etc.) is executed on all semantic embedding vectors, using the distribution density of semantic vectors in space as the clustering basis. During this process, for each field vector, calculate its neighbor distribution density in the semantic space and accordingly determine whether it belongs to a high-density core point, a boundary point, or a discrete point. Density clustering does not require presetting the number of clusters and is suitable for field distributions with unknown class numbers, which is conducive to dynamically identifying field clusters at different semantic granularities. After clustering, a set of initial clustering clusters is obtained, and each clustering cluster represents a group of structured field information with similar semantics.

[0044] To further improve the semantic consistency of the clustering results, in this embodiment, principal component analysis (PCA) is performed on each initial clustering cluster to identify the main distribution direction of the field embedding vectors in the cluster. By analyzing the variance distribution of the main axis direction, it is possible to determine whether there is semantic polysemy in the clustering cluster. If there are multiple obvious distribution peaks in the main axis direction of a certain initial clustering cluster, it indicates that the cluster may have mixed in multiple fields with similar semantics but different meanings, and it is determined as a polysemous cluster.

[0045] In the case of being judged as a polysemous cluster, to improve the clustering granularity and semantic purity, a sub-clustering algorithm (such as hierarchical clustering or K-Means) is further used to perform a secondary division inside it. The specific approach is as follows: taking the principal component axis as the distribution reference, performing sub-clustering operations such as K-Means or Agglomerative Clustering, and subdividing the field vectors in the polysemous cluster into multiple sub-clusters. Each sub-cluster represents a smaller and semantically purer field set. After sub-clustering is completed, the central vector in each sub-cluster (i.e., the field vector that is closest to the average of all field vectors in the sub-cluster in the embedding space) is extracted as the representative vector of the sub-cluster, providing an aggregation basis for subsequent standard field identification.

[0046] In addition, for the field information that is not assigned to any clustering cluster during the initial density clustering process (usually isolated points or fields in low-density regions), it is marked as semantically isolated fields. Semantically isolated fields represent field information that cannot form a clustering group with any other fields under the current clustering conditions, and may be newly emerged fields, fields with special names, or incorrect fields. The system records these fields in the candidate field table for subsequent manual review, manual clustering, or re-clustering strategies.

[0047] This embodiment realizes high-precision semantic clustering of structured field information by introducing an index structure to optimize vector access efficiency, adopting a clustering method combining density and hierarchy, performing principal component semantic deconstruction and polysemy recognition, and dealing with isolated fields outside the clustering. Compared with the traditional method of field classification only based on field names or literal similarity, the solution of the present invention can handle the semantic approximation and subtle differences between fields more comprehensively and meticulously, especially showing good aggregation effect and intelligent feature normalization ability when facing a large number of heterogeneous, redundant, and ambiguously named fields in the enterprise digital transformation platform.

[0048] Step S40: Generate corresponding standard field identifiers for each semantic clustering cluster, and store the generated standard field identifiers in the standard field index library of the platform after digital transformation.

[0049] Specifically, corresponding standard field identifiers are generated for each semantic clustering cluster, including: in each semantic clustering cluster, selecting the structured field information with the semantic embedding vector closest to the clustering center as the representative field; extracting the field name from the representative field and performing character regular cleaning to remove the naming redundancy structure to obtain the input field name; using the fine-tuned language model to regenerate the standard field name for the input field name and performing a semantic difference comparison with the original input field name, and retaining the input field names with a similarity higher than the similarity threshold; summarizing the field type information of all structured field information in the corresponding semantic clustering cluster and determining the unified field type based on the majority rule; writing the generated input field name, field type, semantic embedding vector of the representative field, and the source system to which it belongs into the standard field identifier document, and generating a unique identification code as the standard field identifier for the corresponding semantic clustering cluster.

[0050] In the embodiment of the present invention, for each identified semantic clustering cluster, to achieve field semantic unification and platform-level standardized management, the present invention provides a method for generating standard field identifiers and stores the generation results in the standard field index library of the digital transformation platform to support subsequent operations such as data integration, field mapping, and query.

[0051] Specifically, in the process of generating the standard field identifier, the representative field is first identified from each semantic clustering cluster. The selection basis of the representative field is: calculating the Euclidean distance or cosine similarity between the semantic embedding vectors of all structured field information in the cluster and the center vector of the clustering cluster, and selecting a piece of structured field information with the smallest distance from the center vector of the clustering cluster as the representative field. This representative field is regarded as the typical field that can best express the common semantic meaning in this clustering cluster. Subsequently, the field name is extracted from the representative field as the candidate source of the standard field name. Perform a character-level regular cleaning operation on this field name to remove non-semantic naming redundancy structures, such as automatically generated prefix numbers, system identifiers, underscore separators, version numbers, and other irrelevant components, to obtain a normalized input field name.

[0052] Furthermore, the input field name is input into a language generation model that has been fine-tuned in the target business domain, such as a pre-trained language model based on the Transformer architecture (such as T5, BART), to generate a standard field name with more accurate semantics and unified style. The generation process not only considers the field word meaning but also implicitly introduces the language style and naming norms in the target business domain, which helps to improve the accuracy and consistency of the standardized output.

[0053] After the standard field names are generated, they are compared with the original input field names for semantic differences. The comparison method can use the cosine similarity in the pre-trained embedding space to evaluate the semantic proximity between the two, or use the aggregate similarity calculation method based on sentence vectors. When the similarity exceeds the set similarity threshold (for example, 0.8), the input field name is retained and the generated result is accepted; otherwise, the generated result will be marked as suspicious and submitted to the manual proofreading queue to ensure the accuracy and controllability of the standard field identification.

[0054] In addition, in order to enhance the information integrity of standard field identifiers, it is also necessary to count and integrate the field type information of all structured field information in the current semantic clustering cluster. Field types can include basic data types such as text, integer, floating point, Boolean, date, currency, etc., and can also be extended to composite types such as enumeration and JSON structure. The majority rule principle is used to unify field types, that is, the type with the highest frequency of occurrence among all field types is used as the final field type of the standard field to ensure that the standard field has the greatest compatibility in actual data applications. Finally, the generated standard field name, field type, semantic embedding vector representing the field, and system identification information of the referenced source field are integrated and encapsulated into a standard field identifier document. At the same time, a set of globally unique identification codes (UUID or hash fingerprint) are generated for each standard field identifier to ensure consistent reference and efficient retrieval in various components of the platform. The standard field identifier document will be written into the standard field index library of the enterprise digital transformation platform.

[0055] Preferably, after obtaining the standard field identifiers of each semantic clustering cluster, the method also includes establishing a mapping record for the standard field identifier; establishing a mapping record for the standard field identifier includes: generating a mapping entry for the structured field information associated with each standard field identifier, the mapping entry including the field name, field description, field example value, source system identifier and clustering cluster ID; organizing the mapping entries according to the standard field identifier to form a mapping relationship table; version numbering the mapping relationship table to mark the generation time, the clustering algorithm version used and the semantic model identifier; establishing a one-to-one binding relationship between the mapping relationship table and the standard field identifier to obtain the corresponding mapping record; and writing the mapping record into a standard field index library.

[0056] In the implementation of the present invention, for each standard field identifier, all structured field information in the semantic cluster to which it belongs is traversed. For each structured field information, a corresponding mapping entry is generated. The mapping entry contains the following fields: original field name, field description information, field example value, source system identifier (such as system code, system name) to which the field belongs, and cluster ID, which is used to locate the attribution of the field in the cluster structure. Among them, the field description information and the field example value are both pre-processed versions to ensure format consistency.

[0057] Furthermore, using the standard field identifier as the primary key, aggregate and organize the above mapping entries to construct a structured mapping relationship table. Each standard field identifier corresponds to a unique mapping relationship table, which records all the original field sets represented by the standard field and their key information. To ensure the data change management ability and auditability of the table, assign an independent version number to each mapping relationship table, and record its generation timestamp, the semantic encoding model identifier currently adopted (such as model training batch, parameter configuration), as well as the name and version number of the clustering algorithm used in the table, to ensure the traceability and comparability of each field clustering and standardization operation.

[0058] Furthermore, establish a one-to-one binding relationship between the mapping relationship table and the corresponding standard field identifier. This binding relationship is implemented through reference in the platform, so that each standard field identifier has a queryable and updatable field source linked list, with good interface usability. Finally, write the generated standard field mapping record as an independent record into the standard field index library of the platform, and store and maintain it synchronously with the standard field identifier, supporting functions such as on-demand query, batch export, and API retrieval.

[0059] Preferably, before storing the generated standard field identifier into the standard field index library of the platform after digital transformation, the method further includes performing a uniqueness check on the standard field identifier; the uniqueness check of the standard field identifier includes: calculating the pairwise cosine similarity between the semantic embedding vector of the standard field identifier to be stored and the embedding vectors of all existing standard field identifiers in the index library; performing aggregation analysis on the standard field identifiers with similarity exceeding the set threshold to determine whether there is a duplicate definition; if there is a duplicate definition, call the manual review interface, and provide the embedding vector distribution map and the original field example content as auxiliary information, and in response to the review instruction, perform operations such as retaining, merging, or replacing the existing standard field identifier, and record the version number of the operation behavior and the operator ID; if there is no duplicate situation, directly store it in the standard field index library and record it as a new field.

[0060] In the embodiment of the present invention, obtain the semantic embedding vector corresponding to the standard field identifier to be stored as the semantic representation basis of the current field; at the same time, call the embedding vectors of all existing standard field identifiers from the standard field index library to construct a comparison vector set. All semantic embedding vectors are fixed-length vectors, with unified dimensions and vector space normalization attributes, to support efficient and comparable vector similarity calculations. Use the vector cosine similarity algorithm to perform pairwise similarity calculations on the semantic embedding vector of the current standard field identifier and all existing embedding vectors in the index library to obtain a similarity score list. To optimize efficiency, this process can be implemented based on a pre-constructed vector index structure (such as FAISS or HNSW) for fast nearest neighbor retrieval.

[0061] After obtaining all similarity results, filter out the standard field identifiers with similarity higher than the set threshold (e.g., 0.92) as potential duplicate candidates and proceed to the next step of aggregation analysis. Aggregation analysis includes visualizing the semantic distribution of these highly similar field identifiers, generating a two-dimensional distribution map of embedding vectors based on principal component analysis (PCA) or t-SNE algorithm to assist users in identifying the semantic overlap regions between fields.

[0062] If there are multiple field identifiers with abnormally close similarities, that is, the system determines them as potentially duplicate defined fields, then call the manual review interface. This interface will display in a visual form: 1) the names, descriptions, and example values of the fields to be stored and candidate duplicate fields; 2) the two-dimensional distribution map of the semantic embedding vectors of each field; 3) the information of their source systems and affiliated clustering clusters. Based on the visual information and experience judgment, the reviewers can execute the following operation instructions: retain the current standard field identifier, merge the current field into an existing field identifier, or replace the existing field identifier with the current field.

[0063] Respond to the review instructions, update the standard field index library, and record the operation log. This log includes meta-information such as change type (retain / merge / replace), target field ID, field ID being operated on, reviewer identity ID, operation timestamp, field version number, etc., for subsequent auditing and version tracking.

[0064] If no existing field identifier higher than the similarity threshold is found during the above comparison process, directly store the current standard field identifier in the index library and register it as a "newly added field". The system also generates a unique identification code and version record for this field, marking metadata such as its creation time, generation process, and semantic coding model version.

[0065] Through the above uniqueness verification process of standard field identifiers, the global consistency and naming standardization of standard fields can be significantly improved, effectively avoiding redundant semantic definitions of fields and the ineffective expansion of platform field resources. At the same time, introducing a manual review mechanism and visual semantic comparison technology enhances the controllability and interpretability of the system, adapts to real challenges such as fuzzy semantic boundaries and multiple meanings and duplicate names of fields in complex enterprise scenarios, and improves the standard unification ability of the enterprise digital transformation platform at the data governance level.

[0066] Figure 2 It is the system structure diagram of the data processing system for the enterprise digital transformation platform provided by an embodiment of the present invention. As Figure 2As shown in the figure, an embodiment of the present invention provides a data processing system for an enterprise digital transformation platform. The system includes: a collection unit, which is used to obtain structured field information from various heterogeneous data sources and perform preprocessing operations on the structured field information when aggregating data from various heterogeneous data sources; a vector generation unit, which is used to construct the processed structured field information into an embedded input sequence, splice the embedded input sequence into a natural language segment according to a preset template, and input it into a fine-tuned semantic encoding model to obtain a corresponding semantic embedding vector; a clustering unit, which is used to identify similar field groups by using a density-based or hierarchical clustering algorithm and classify each group of structured field information into a semantic clustering cluster; an entry unit, which is used to generate a corresponding standard field identifier for each semantic clustering cluster and store the generated standard field identifier in the standard field index library of the platform after digital transformation.

[0067] An embodiment of the present invention also provides a computer-readable storage medium, on which instructions are stored, and when the instructions run on a computer, the computer is made to execute the above-mentioned data processing method for an enterprise digital transformation platform.

[0068] Those skilled in the art can understand that all or part of the steps in the method for implementing the above embodiments can be completed by instructing relevant hardware through a program. The program is stored in a storage medium and includes several instructions to make a single-chip microcomputer, a chip or a processor execute all or part of the steps of the method described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disc that can store program codes.

[0069] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the technical concept scope of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention. In addition, it should be noted that, among the various specific technical features described in the above specific embodiments, they can be combined in any appropriate manner without conflict. To avoid unnecessary repetition, the embodiments of the present invention will not separately describe various possible combination methods.

[0070] In addition, any combination can be made between various different embodiments of the present invention, as long as it does not violate the idea of the embodiments of the present invention, and it should also be regarded as the content disclosed in the embodiments of the present invention.

Claims

1. A data processing method for an enterprise digital transformation platform, characterized in that, The method includes: When performing data aggregation of various heterogeneous data sources, obtaining structured field information from various heterogeneous data sources and performing preprocessing operations on the structured field information; Constructing the processed structured field information into an embedded input sequence, splicing the embedded input sequence into a natural language segment according to a preset template, and inputting it into a fine-tuned semantic encoding model to obtain a corresponding semantic embedding vector; Using a density-based or hierarchical clustering algorithm to identify similar field groups and classifying each group of structured field information into a semantic clustering cluster; Generating a corresponding standard field identifier for each semantic clustering cluster and storing the generated standard field identifier in the standard field index library of the platform after digital transformation.

2. The method according to claim 1, wherein Performing preprocessing operations on the structured field information includes: Performing unified character encoding conversion, all-lowercase conversion, and punctuation cleaning on the field name, field description, and field example value in the structured field information respectively; Using a built-in tokenizer to perform language segmentation on the field description and field example value and removing stop words; Based on a preset domain term library, performing mapping replacement operations on the terms in the field description, and the preset domain database contains the corresponding relationships between synonymous terms and standard terms existing in multiple heterogeneous systems; Performing format recognition operations on the field example value, recognizing date formats, numerical units, and / or currency symbols, and generating structured type tags according to the recognition results to obtain preprocessed structured field information.

3. The method according to claim 1, wherein Constructing the processed structured field information into an embedded input sequence includes: Constructing a natural language template containing the field name, field description, and field example value for each piece of structured field information, and constructing the corresponding structured field information into a unified text segment using a predefined format and the constructed template; Encoding the obtained text segment into a standard Token sequence and processing the low-frequency characters or unrecognized words in the Token sequence using a sub-word segmentation mechanism to obtain the embedded input sequence of each structured field information; After obtaining the embedded input sequence of each structured field information, the method further includes: Performing batch normalization processing on all embedded input sequences, and the normalization processing includes length padding, Mask processing, and field boundary annotation.

4. The method according to claim 1, characterized in that, Splicing the embedded input sequence into a natural language segment according to a preset template and inputting it into a fine-tuned semantic encoding model to obtain a corresponding semantic embedding vector, including: Inputting the natural language segment in the embedded input sequence into a language model based on the Transformer structure; Introducing three types of marker vectors at the model input end, which are used to mark the field name segment, field description segment, and example value segment respectively; Extracting sequence-level semantic features through the hidden layer of the semantic encoding model and performing average pooling on the sequence-level semantic features to generate a fixed-length semantic embedding vector; Performing layer normalization and Dropout operations on the fixed-length semantic embedding vector, and using the fixed-length semantic embedding vector as the semantic embedding vector that uniquely represents the structured field information.

5. The method according to claim 1, characterized in that Identify groups of similar fields using density - based or hierarchical clustering algorithms, and classify the structured field information of each group into a semantic clustering cluster, including: Construct a vector space index structure based on all semantic embedding vectors. The index structure uses an inverted index and incorporates locality - sensitive hashing; Perform preliminary cluster division based on the density clustering algorithm and the spatial density of semantic embedding vectors to obtain initial clustering clusters; Perform principal component analysis on each initial clustering cluster, identify the corresponding main distribution axis, and determine whether there are polysemous fields within the corresponding initial clustering cluster. Mark the initial clustering clusters with polysemous fields as polysemous clusters; Use a sub - clustering algorithm to perform secondary division on the polysemous clusters to obtain multiple sub - clusters of each polysemous cluster, and use the central vector within the cluster as the representative vector of each sub - cluster; Mark the structured field information that cannot be assigned to any cluster as semantic isolated fields and record them in the candidate field table; 6. The method according to claim 1, characterized in that, Generate corresponding standard field identifiers for each semantic clustering cluster, including: In each semantic clustering cluster, select the structured field information with the semantic embedding vector closest to the cluster center as the representative field; Extract the field name from the representative field and perform character regular cleaning to remove naming redundancy structures to obtain the input field name; Use the fine - tuned language model to regenerate the standard field name for the input field name and perform a semantic difference comparison with the original input field name, retaining the input field names with a similarity higher than the similarity threshold; Summarize the field type information of all structured field information in the corresponding semantic clustering cluster and determine the unified field type based on the majority rule; Write the generated input field name, field type, semantic embedding vector of the representative field, and the source system to which it belongs into the standard field identifier document, and generate a unique identification code as the standard field identifier for the corresponding semantic clustering cluster; 7. The method according to claim 1, characterized in that, After obtaining the standard field identifiers of each semantic clustering cluster, the method further includes establishing mapping records for the standard field identifiers; Establishing mapping records for the standard field identifiers includes: Generate mapping entries for the structured field information associated with each standard field identifier. The mapping entries include field name, field description, field example value, source system identifier, and cluster ID; Organize the mapping entries according to the standard field identifiers to form a mapping relationship table; Assign a version number to the mapping relationship table, marking the generation time, the version of the clustering algorithm used, and the semantic model identifier; Establish a one - to - one binding relationship between the mapping relationship table and the standard field identifier to obtain the corresponding mapping record; Write the mapping record into the standard field index library; 8. The method according to claim 1, characterized in that Before storing the generated standard field identifiers into the standard field index library of the platform after digital transformation, the method further includes performing a uniqueness check on the standard field identifiers; The uniqueness check of the standard field identifiers includes: Perform pairwise cosine similarity calculations on the semantic embedding vectors of the standard field identifiers to be stored and the embedding vectors of all existing standard field identifiers in the index library; Perform aggregation analysis on the standard field identifiers with similarity exceeding the set threshold to determine whether there are duplicate definitions; If it is a repeated definition, call the manual review interface and provide the embedded vector distribution diagram and the original field example content as auxiliary information. In response to the review instruction, retain, merge, or replace the existing standard field identifiers, and record the version number of the operation behavior and the operator ID; If there is no repetition, directly store it in the standard field index library and record it as a new field.

9. A data processing system for an enterprise digital transformation platform, characterized in that, The system includes: A collection unit, configured to obtain structured field information from various heterogeneous data sources and perform preprocessing operations on the structured field information when performing data aggregation of various heterogeneous data sources; A vector generation unit, configured to construct the processed structured field information into an embedded input sequence, splice the embedded input sequence into a natural language segment according to a preset template, and input it into a fine-tuned semantic encoding model to obtain a corresponding semantic embedded vector; A clustering unit, configured to identify similar field groups by using a density-based or hierarchical clustering algorithm and classify each group of structured field information into a semantic clustering cluster; A storage unit, configured to generate a corresponding standard field identifier for each semantic clustering cluster and store the generated standard field identifier in the standard field index library of the digitized transformation platform.

10. A computer-readable storage medium, characterized in that, Instructions are stored on the computer-readable storage medium, and when running on a computer, the computer is caused to execute the data processing method for an enterprise digital transformation platform according to any one of claims 1-8.

Images