A data processing method and system for enterprise digital transformation platforms

Abstract

This invention provides a data processing method and system for enterprise digital transformation platforms, belonging to the field of data processing. The method includes: acquiring structured field information from various heterogeneous data sources and performing preprocessing operations on the structured field information; constructing an embedded input sequence from the processed structured field information; concatenating the embedded input sequence into natural language fragments according to a preset template and inputting it into a fine-tuned semantic coding model to obtain corresponding semantic embedding vectors; identifying similar field groups using density-based or hierarchical clustering algorithms and grouping each group of structured field information into a semantic cluster; generating a corresponding standard field identifier for each semantic cluster and storing the generated standard field identifier in the standard field index library of the platform after digital transformation. This invention significantly enhances the unified field management and cross-system data alignment capabilities of enterprise digital platforms.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a data processing method and a data processing system for an enterprise digital transformation platform. Background Technology

[0002] As enterprises deepen their digital transformation, enterprise-level platforms need to manage and integrate heterogeneous structured data from multiple business systems, external interfaces, and historical data tables in a unified manner. These data sources often differ significantly in terms of field naming conventions, semantic expression, data types, and structural formats, leading to serious obstacles in data fusion, cross-system queries, metric normalization, and data governance. Therefore, how to semantically align and standardize fields in structured data has become one of the key technical issues in the design of a digital platform's middleware.

[0003] In existing technologies, field standardization mainly employs methods such as manually pre-defined mapping rules, field name matching dictionaries, regular expression parsing, or static templates. These methods rely on expert experience to build rule bases, which are not only costly and difficult to maintain, but also struggle to cover rapidly changing or highly specialized new data sources. When fields have semantic differences but similar names, or semantic similarities but diverse naming conventions, traditional rule matching methods cannot accurately identify the potential semantic consistency between fields, resulting in unsatisfactory field merging and standard field generation effects.

[0004] In addition, some solutions attempt to introduce pre-trained language models to improve the semantic understanding of fields, but they do not combine the composite feature structure of "field name + field description + example value" inherent in structured fields. Instead, they only use field name units as model input, resulting in extremely sparse semantic information, which affects the clustering effect. Ultimately, they cannot build a high-quality standard field indexing system, which restricts the improvement of platform-level data governance and intelligent query capabilities.

[0005] In summary, existing solutions still suffer from problems such as limited understanding of fields, insufficient model adaptation, and coarse semantic clustering in handling the semantic standardization of heterogeneous structured fields. There is an urgent need for a more intelligent, refined, and structure-aware method for field semantic standardization to improve the field alignment efficiency and data availability of enterprise digital transformation platforms. Summary of the Invention

[0006] The purpose of this invention is to provide a data processing method and system for enterprise digital transformation platforms, so as to at least solve the problems of weak semantic understanding ability of heterogeneous structured fields, reliance on manual rules in the field standardization process, and inaccurate generation of standard fields in the prior art.

[0007] To achieve the above objectives, the first aspect of the present invention provides a data processing method for an enterprise digital transformation platform. The method includes: when performing data aggregation from various heterogeneous data sources, obtaining structured field information from each heterogeneous data source and performing preprocessing operations on the structured field information; constructing the processed structured field information into an embedded input sequence, concatenating the embedded input sequence into a natural language fragment according to a preset template, and inputting it into a fine-tuned semantic coding model to obtain a corresponding semantic embedding vector; using a density-based or hierarchical clustering algorithm to identify similar field groups and classifying each group of structured field information into a semantic cluster; generating a corresponding standard field identifier for each semantic cluster and storing the generated standard field identifier in the standard field index library of the platform after digital transformation.

[0008] Optionally, preprocessing operations are performed on the structured field information, including: performing unified character encoding conversion, all-lowercase conversion, and punctuation cleanup on the field names, field descriptions, and field example values ​​in the structured field information, respectively; using a built-in word segmenter to perform language segmentation on the field descriptions and field example values, and removing stop words; performing mapping and replacement operations on the terms in the field descriptions based on a preset domain terminology database, wherein the preset domain database contains synonyms and their standard terminology correspondences existing in multiple heterogeneous systems; performing format recognition operations on the field example values, recognizing date formats, numerical units, and / or currency symbols, and generating structured type tags based on the recognition results to obtain preprocessed structured field information.

[0009] Optionally, the processed structured field information is constructed into an embedded input sequence, including: constructing a natural language template containing a field name, field description, and field example value for each structured field information, and constructing a unified text fragment using a predefined format and the constructed template; encoding the obtained text fragment into a standard token sequence, and processing low-frequency characters or unrecognized words appearing in the token sequence using a sub-word segmentation mechanism to obtain the embedded input sequence of each structured field information; after obtaining the embedded input sequence of each structured field information, the method further includes: performing batch normalization processing on all embedded input sequences, the normalization processing including length padding, mask processing, and field boundary labeling.

[0010] Optionally, the embedded input sequence is concatenated into natural language fragments according to a preset template and input into a fine-tuned semantic coding model to obtain the corresponding semantic embedding vector. This includes: inputting the natural language fragments in the embedded input sequence into a language model based on a Transformer structure; introducing three types of labeled vectors at the model input end, which are used to label field name segments, field description segments, and example value segments, respectively; extracting sequence-level semantic features through the hidden layer of the semantic coding model, and performing average pooling on the sequence-level semantic features to generate a fixed-length semantic embedding vector; performing layer normalization and Dropout operations on the fixed-length semantic embedding vector, and using the fixed-length semantic embedding vector as the semantic embedding vector that uniquely represents the structured field information.

[0011] Optionally, a density-based or hierarchical clustering algorithm is used to identify similar field groups, and each group of structured field information is assigned to a semantic cluster. This includes: constructing a vector space index structure based on all semantic embedding vectors, wherein the index structure uses an inverted index and incorporates locality-sensitive hashing; performing preliminary clustering based on density clustering algorithm and spatial density of semantic embedding vectors to obtain initial clusters; performing principal component analysis on each initial cluster to identify the corresponding distribution principal axis and determine whether there are ambiguous fields within the corresponding initial cluster, and designating the initial clusters with ambiguous fields as ambiguous clusters; performing secondary partitioning on the ambiguous clusters using a sub-clustering algorithm to obtain multiple sub-clusters for each ambiguous cluster, and using the center vector within the cluster as the representative vector of each sub-cluster; marking structured field information that fails to be assigned to any cluster as semantically isolated fields and recording them in a candidate field table.

[0012] Optionally, for each semantic cluster, a corresponding standard field identifier is generated, including: in each semantic cluster, selecting the structured field information whose semantic embedding vector is closest to the cluster center as the representative field; extracting the field name from the representative field and performing character regular expression cleaning to remove redundant naming structures to obtain the input field name; using a fine-tuned language model to regenerate the standard field name from the input field name, and comparing it with the original input field name semantically, retaining the input field name with a similarity higher than the similarity threshold; summarizing the field type information of all structured field information in the corresponding semantic cluster, and determining a unified field type based on majority rules; writing the generated input field name, field type, semantic embedding vector of the representative field, and the source system to which it belongs into the standard field identifier document, and generating a unique identifier code as the standard field identifier for the corresponding semantic cluster.

[0013] Optionally, after obtaining the standard field identifiers for each semantic cluster, the method further includes establishing mapping records for the standard field identifiers; establishing mapping records for the standard field identifiers includes: generating mapping entries for the structured field information associated with each standard field identifier, wherein the mapping entries include field name, field description, field example value, source system identifier, and cluster ID; organizing the mapping entries according to the standard field identifiers to form a mapping relationship table; assigning version numbers to the mapping relationship table, marking the generation time, the version of the clustering algorithm used, and the semantic model identifier; establishing a one-to-one binding relationship between the mapping relationship table and the standard field identifiers to obtain the corresponding mapping records; and writing the mapping records into a standard field index library.

[0014] Optionally, before storing the generated standard field identifiers in the standard field index library of the platform after digital transformation, the method further includes performing a uniqueness check on the standard field identifiers. The uniqueness check on the standard field identifiers includes: calculating the pairwise cosine similarity between the semantic embedding vector of the standard field identifier to be stored and the embedding vectors of all existing standard field identifiers in the index library; performing aggregation analysis on standard field identifiers with similarity exceeding a set threshold to determine whether they are duplicate definitions; if they are duplicate definitions, calling the manual review interface and providing an embedding vector distribution map and original field example content as auxiliary information, responding to the review instruction, performing retention, merging, or replacement of existing standard field identifiers, and recording the version number and operator ID for the operation; if there are no duplicates, directly storing them in the standard field index library and recording them as new fields.

[0015] A second aspect of the present invention provides a data processing system for an enterprise digital transformation platform. The system includes: a data acquisition unit, configured to acquire structured field information from various heterogeneous data sources when performing data aggregation from these sources, and to perform preprocessing operations on the structured field information; a vector generation unit, configured to construct an embedded input sequence from the processed structured field information, concatenate the embedded input sequence into a natural language fragment according to a preset template, and input it into a finely tuned semantic coding model to obtain a corresponding semantic embedding vector; a clustering unit, configured to identify similar field groups using a density-based or hierarchical clustering algorithm, and to group each group of structured field information into a semantic cluster; and a database storage unit, configured to generate a corresponding standard field identifier for each semantic cluster and store the generated standard field identifier in the standard field index database of the platform after digital transformation.

[0016] On the other hand, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned data processing method for an enterprise digital transformation platform.

[0017] Through the above technical solution, this invention enhances the expression of structured field information by introducing a semantic coding model, thereby improving the modeling ability of potential semantic relationships between fields from heterogeneous data sources and overcoming the limitations of traditional field standardization methods that rely on manual rules or surface-level name matching. By constructing natural language fragments and inputting them into a finely tuned semantic model to obtain semantic embedding vectors, the field clustering process becomes more accurate and context-aware. Combined with density-based or hierarchical clustering algorithms, semantically similar field groups can be effectively identified, improving the accuracy of field normalization and standard field generation. The final generated standard field identifiers are uniformly stored in a standard field index library, which helps to build a highly consistent and scalable field management system, significantly enhancing the unified field management and cross-system data alignment capabilities of enterprise digital platforms.

[0018] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0020] Figure 1 This is a flowchart of the steps of a data processing method for an enterprise digital transformation platform provided by one embodiment of the present invention;

[0021] Figure 2 This is a system architecture diagram of a data processing system for an enterprise digital transformation platform provided by one embodiment of the present invention. Detailed Implementation

[0022] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0023] Figure 1 This is a flowchart of a data processing method for an enterprise digital transformation platform provided by one embodiment of the present invention. Figure 1 As shown, this invention provides a data processing method for an enterprise digital transformation platform, the method comprising:

[0024] Step S10: When performing data aggregation from various heterogeneous data sources, obtain structured field information from each heterogeneous data source and perform preprocessing operations on the structured field information.

[0025] Specifically, perform preprocessing operations on the structured field information, including: performing unified character encoding conversion, all-lowercase conversion, and punctuation cleaning on the field name, field description, and field example value in the structured field information; using a built-in tokenizer to perform language segmentation on the field description and field example value, and removing stop words; based on a preset domain term library, performing mapping replacement operations on the terms in the field description, where the preset domain database contains the synonymous terms existing in multiple heterogeneous systems and their corresponding relationships with standard terms; performing format recognition operations on the field example value, recognizing date formats, numerical units, and / or currency symbols, and generating structured type tags according to the recognition results to obtain the preprocessed structured field information.

[0026] In the embodiments of the present invention, when performing data aggregation of various heterogeneous data sources, first obtain structured field information from multiple business systems, external interfaces, or historical databases. The structured field information includes field-level metadata such as field names, field descriptions, and field example values. Since there are significant differences in field naming conventions, language expressions, data formats, etc. among different data sources, directly integrating and using them will cause difficulties in field alignment. Therefore, it is necessary to perform unified preprocessing operations on the above structured field information to improve its semantic resolvability and input consistency.

[0027] Specifically, perform character standardization processing on the field name, field description, and field example value uniformly. This includes converting all characters to the UTF-8 encoding format to eliminate differences in encoding methods between different systems; converting all English letters to lowercase to avoid affecting semantic recognition due to mixed use of uppercase and lowercase; at the same time, cleaning the punctuation marks in the field text, removing symbols without actual semantics, such as commas, quotation marks, parentheses, etc., to ensure the simplicity and consistency of semantic representation.

[0028] Furthermore, for the content of the field description and field example value, use a built-in tokenizer (such as a composite tokenizer based on a dictionary + statistical rules), which is especially suitable for processing semantic block division in a Chinese-English mixed environment. After tokenization, further remove stop words, such as words without actual semantic contributions like "de", "wei", "he", "in", "of", etc., to improve the density and semantic focusing ability of the subsequent input information of the model.

[0029] Furthermore, based on a pre-defined domain terminology library, standardized replacements are performed on the technical terms in the field descriptions. This domain terminology library is constructed by aggregating common term variations from various business domains within the enterprise and their standard expressions, supporting many-to-one mappings. For example, "customer number," "user ID," and "customer ID number" are all mapped to "customer identifier." This operation significantly reduces the risk of semantic fragmentation caused by differences in expression, laying the foundation for field clustering and semantic modeling. In addition, type recognition processing is performed on field example values, employing a hybrid strategy of regular expressions and dictionaries to identify their data type and contextual meaning. For example, it can automatically recognize date formats (such as yyyy-mm-dd, dd / mm / yyyy, etc.), numerical units (such as kg, m², %, etc.), and currency symbols (such as ¥, $, €), generating structured type tag vectors that are appended to the field information. This tagging not only enhances the model's understanding of the field's purpose but also provides interpretable features for clustering or downstream field naming generation.

[0030] Through the aforementioned multi-dimensional preprocessing operations, standardized processing of structured field information in terms of character format, semantic expression, and type labels is achieved. Technically, this significantly improves the standardization and completeness of subsequent semantic encoding model inputs, reduces semantic ambiguity and redundancy, and enhances the model's ability to distinguish and aggregate field semantics. Furthermore, this structured preprocessing framework is scalable, easily adaptable to semantic standard rules of different enterprises or fields, and helps build a reusable, highly accurate standard field identification and management system, providing underlying support for heterogeneous field alignment, standardized indexing, and cross-system integration in enterprise digital platforms.

[0031] Step S20: Construct the processed structured field information into an embedded input sequence, concatenate the embedded input sequence into a natural language fragment according to a preset template, and input it into the fine-tuned semantic coding model to obtain the corresponding semantic embedding vector.

[0032] Specifically, the processed structured field information is constructed into an embedded input sequence, including: constructing a natural language template for each structured field information containing a field name, field description, and field example value, and using a predefined format and the constructed template to construct a unified text fragment for the corresponding structured field information; encoding the obtained text fragment into a standard token sequence, and processing low-frequency characters or unrecognized words appearing in the token sequence using a sub-word segmentation mechanism to obtain the embedded input sequence for each structured field information; after obtaining the embedded input sequence for each structured field information, the method further includes: performing batch normalization processing on all embedded input sequences, the normalization processing including length padding, mask processing, and field boundary annotation.

[0033] Furthermore, the embedded input sequence is concatenated into natural language fragments according to a preset template and input into a fine-tuned semantic coding model to obtain corresponding semantic embedding vectors. This includes: inputting the natural language fragments in the embedded input sequence into a language model based on a Transformer structure; introducing three types of labeled vectors at the model input end, which are used to label field name segments, field description segments, and example value segments, respectively; extracting sequence-level semantic features through the hidden layer of the semantic coding model, and performing average pooling on the sequence-level semantic features to generate fixed-length semantic embedding vectors; performing layer normalization and Dropout operations on the fixed-length semantic embedding vectors, and using the fixed-length semantic embedding vectors as the unique semantic embedding vectors representing structured field information.

[0034] In this embodiment of the invention, after completing the preprocessing of structured field information from various heterogeneous data sources, in order to further achieve semantic-level field aggregation and standard field generation, it is necessary to construct an embedded representation and semantically encode the preprocessed structured field information. The goal of this step is to transform traditional structured field information into an input form that can be processed by a language model, and then obtain a vector representation with unified semantic expression capabilities through deep semantic modeling, serving as the basis for subsequent semantic clustering and standard field generation.

[0035] Specifically, each processed structured field information is first constructed as an embedded input sequence. Structured field information typically contains three types of content: field name, field description, and field example value. To ensure the consistency and completeness of the input information, a unified natural language template needs to be constructed to organize these three types of content into text segments that can be understood by the language model. For example, the following natural language template structure can be defined: "Field name is [X], its description is [Y], and its value is [Z]." Here, [X], [Y], and [Z] correspond to the field name, field description, and field example value, respectively. This template not only preserves the structural information of the fields but also strengthens the semantic relationships between different components within the fields through language connections, enabling the language model to better capture contextual information during processing.

[0036] Next, the aforementioned natural language template text fragments are encoded into standard token sequences. Tokenization is a prerequisite for language models to understand text. It typically uses the model's built-in vocabulary for encoding, breaking down the string into recognizable word units. During this process, if low-frequency words or domain-specific terms not present in the model's vocabulary appear, to avoid information loss or encoding errors, a sub-word segmentation mechanism (such as Byte Pair Encoding, SentencePiece, etc.) can be introduced to further decompose unknown words into several common sub-word units. This allows the model to extract semantic features from its constituent parts even if it is unfamiliar with the overall vocabulary.

[0037] To improve the efficiency and alignment consistency of subsequent model inference, a unified batch normalization process is required for the token sequences. Normalization includes three aspects: First, length padding: to ensure each input sequence has a consistent length when entering the language model, sequences with insufficient length are padded with special markers (such as [PAD]) to align to the maximum length; second, masking: by generating mask vectors to mask out the padded parts, preventing them from participating in semantic computation; and finally, field boundary labeling: by inserting or appending position markers (such as segment ids) to the token sequences, the start and end positions of the field name segment, description segment, and example value segment are clearly defined, providing guidance for the model to subsequently extract semantics within the structure.

[0038] After constructing the embedded input sequence, it is input into the fine-tuned semantic encoding model according to a preset format. This semantic encoding model adopts a language model based on the Transformer architecture, which has the ability to model contextual semantics and effectively capture deep semantic relationships in the input text. During the model input stage, to further enhance the model's ability to perceive field structure, three types of auxiliary label vectors are introduced, corresponding to field name segments, field description segments, and field example value segments, respectively. Each type of label vector serves as a "structural guidance signal" and is concatenated in the embedding layer, enabling the model to distinguish the differences in semantic roles of different paragraphs when executing the multi-head attention mechanism, thereby learning more discriminative field structure semantic features in the hidden layers.

[0039] During the forward propagation of the model, semantic representations are extracted from the input sequence layer by layer through multiple encoding layers of the Transformer. In the last hidden layer, the contextual representation of each token position is extracted, and the sequence-level features are compressed into a fixed-length vector using a mean pooling strategy, serving as the overall semantic embedding vector for that structured field. Compared to directly using vectors at specific positions (such as the [CLS] position), mean pooling can more fully integrate the information from various parts of the entire field text, improving the stability and generality of the semantic representation.

[0040] To further improve the usability of the representation and the robustness of the training process, layer normalization and dropout operations are performed on the generated fixed-length embedding vectors. On the one hand, layer normalization helps stabilize the vector distribution between different input batches, avoiding the impact of differences in input data distribution on subsequent model performance; on the other hand, the dropout operation effectively prevents overfitting during fine-tuning by randomly masking some neuron outputs, thus enhancing the generalization ability of the embedding representation. Finally, the obtained fixed-length semantic embedding vector serves as a unique semantic representation of each structured field, and can be directly used in the semantic space for field similarity measurement, cluster analysis, and semantic alignment operations. This vector not only integrates field naming information and semantic descriptions but also provides supplementary contextual scenarios through example values, thus possessing stronger semantic expressive power than traditional field name-based matching methods.

[0041] Based on this invention, by using natural language templates and standardized token processing, fields from different sources achieve structural consistency at the model input level, reducing the model's sensitivity to input differences. Field names, descriptions, and example values ​​are uniformly integrated into the embedding sequence to achieve semantic completion and enhance the accuracy of semantic modeling. The introduction of paragraph markers and boundary annotations enables the model to recognize the internal structure of fields, improving its ability to distinguish the semantic roles of fields. Through word segmentation, normalization, and Dropout mechanisms, the model's adaptability to new domain terms and marginal cases is improved. The generated semantic embedding vectors have good context awareness and complete field expression, making them suitable as the input basis for field clustering and standard field generation.

[0042] Step S30: Use a density-based or hierarchical clustering algorithm to identify similar field groups and group each group of structured field information into a semantic cluster.

[0043] Specifically, a vector space index structure is constructed based on all semantic embedding vectors. The index structure uses an inverted index and incorporates locality-sensitive hashing. Initial clustering is performed based on density clustering algorithm and the spatial density of semantic embedding vectors to obtain initial clusters. Principal component analysis is performed on each initial cluster to identify the corresponding principal axis of distribution and determine whether there are ambiguous fields within the corresponding initial cluster. Initial clusters with ambiguous fields are designated as ambiguous clusters. The ambiguous clusters are then further divided using a sub-clustering algorithm to obtain multiple sub-clusters for each ambiguous cluster, with the central vector within each cluster serving as the representative vector for each sub-cluster. Structured field information that fails to be assigned to any cluster is marked as semantically isolated fields and recorded in a candidate field table.

[0044] In this embodiment of the invention, in order to achieve semantic classification and standardization of structured field information from various heterogeneous data sources, after obtaining the semantic embedding vector of each structured field information, the invention uses a density-based or hierarchical clustering algorithm to perform semantic clustering analysis on all semantic embedding vectors to identify field groups with similar meanings or consistent functions, and classifies them into a unified semantic cluster, providing a basis for the subsequent generation of standard field identifiers.

[0045] Specifically, before the clustering operation, a vector space index structure is constructed based on all semantic embedding vectors. Considering that the number of fields may reach tens of thousands or even higher, traditional linear similarity search is inefficient. Therefore, this implementation uses an inverted index structure combined with a Locality-Sensitive Hashing (LSH) mechanism to construct the semantic index. The inverted index buckets the field vectors according to the hash signature, so that vector lookup no longer traverses all fields, but only performs similarity retrieval in the candidate buckets, which greatly improves the efficiency of similarity query before clustering.

[0046] Subsequently, based on the constructed index structure, density-based clustering algorithms (such as DBSCAN, HDBSCAN, etc.) are performed on all semantic embedding vectors, using the spatial distribution density of semantic vectors as the basis for clustering. In this process, for each field vector, its nearest neighbor distribution density in the semantic space is calculated, and this density is used to determine whether it belongs to a high-density core point, boundary point, or discrete point. Density clustering does not require a pre-defined number of clusters, is suitable for field distributions with an unknown number of categories, and is beneficial for dynamically identifying field clusters at different semantic granularities. After clustering is completed, a set of initial clusters is obtained, each cluster representing a group of semantically similar structured field information.

[0047] To further improve the semantic consistency of the clustering results, this implementation performs Principal Component Analysis (PCA) on each initial cluster to identify the principal distribution direction of the field embedding vectors in that cluster. By analyzing the variance distribution along the principal axis, it can be determined whether there is semantic ambiguity in the cluster. If an initial cluster has multiple obvious distribution peaks along the principal axis, it indicates that the cluster may contain multiple semantically similar but different fields, and is therefore classified as an ambiguous cluster.

[0048] For cases identified as ambiguous clusters, to improve clustering granularity and semantic purity, a sub-clustering algorithm (such as hierarchical clustering or K-Means) is further employed to perform secondary partitioning within them. Specifically, using the principal component axis as a distribution reference, sub-clustering operations such as K-Means or Agglomerative Clustering are performed to further subdivide the field vectors in the ambiguous clusters into multiple sub-clusters. Each sub-cluster represents a smaller, semantically purer set of fields. After sub-clustering, the central vector of each sub-cluster (i.e., the field vector in the embedding space that is closest to the average of all field vectors in that sub-cluster) is extracted as the representative vector of that sub-cluster, providing an aggregation basis for subsequent standard field identification.

[0049] Furthermore, fields that were not assigned to any cluster during the initial density clustering process (typically outliers or low-density regions) are marked as semantically isolated fields. Semantically isolated fields represent fields that cannot form clusters with any other fields under the current clustering conditions; these may be newly appearing fields, specially named fields, or incorrectly named fields. The system records these fields in a candidate field table for subsequent manual review, manual clustering, or re-clustering strategies.

[0050] This implementation achieves high-precision semantic clustering of structured field information by introducing an index structure to optimize vector access efficiency, employing a density- and hierarchical clustering approach, performing principal component semantic deconstruction and polysemous identification, and handling isolated fields outside the clusters. Compared to traditional methods that classify fields solely based on field names or literal similarity, this invention can more comprehensively and meticulously handle semantic similarities and subtle differences between fields. It demonstrates excellent aggregation effects and intelligent feature normalization capabilities, especially in enterprise digital transformation platforms facing a large number of heterogeneous, redundant, and fuzzy-named fields.

[0051] Step S40: Generate a corresponding standard field identifier for each semantic cluster and store the generated standard field identifier in the standard field index library of the platform after digital transformation.

[0052] Specifically, for each semantic cluster, a corresponding standard field identifier is generated, including: in each semantic cluster, selecting the structured field information whose semantic embedding vector is closest to the cluster center as the representative field; extracting the field name from the representative field and performing character regular expression cleaning to remove redundant naming structures to obtain the input field name; using a fine-tuned language model to regenerate the standard field name from the input field name, and comparing it with the original input field name for semantic differences, retaining the input field names with similarity higher than the similarity threshold; summarizing the field type information of all structured field information in the corresponding semantic cluster, and determining a unified field type based on majority rules; writing the generated input field name, field type, semantic embedding vector of the representative field, and its source system into the standard field identifier document, and generating a unique identifier code as the standard field identifier for the corresponding semantic cluster.

[0053] In this embodiment of the invention, for each identified semantic cluster, in order to achieve unified field semantics and platform-level standardized management, the invention provides a method for generating standard field identifiers and stores the generated results in the standard field index library of the digital transformation platform to support subsequent data integration, field mapping and query operations.

[0054] Specifically, in the standard field identifier generation process, representative fields are first identified from each semantic cluster. The selection criteria for these representative fields are: calculating the Euclidean distance or cosine similarity between the semantic embedding vectors of all structured field information within the cluster and the cluster center vector, and selecting the structured field information with the smallest distance to the cluster center vector as the representative field. This representative field is considered the most typical field in the cluster that best expresses the common semantic meaning. Subsequently, field names are extracted from the representative fields as candidate sources for standard field names. Character-level regular expression cleaning is then performed on these field names to remove non-semantic naming redundancy structures, such as automatically generated prefix numbers, system identifiers, underscore separators, version numbers, and other irrelevant components, to obtain standardized input field names.

[0055] Furthermore, the input field name is fed into a language generation model that has been fine-tuned for the target business domain, such as a pre-trained language model based on the Transformer architecture (e.g., T5, BART), to generate more semantically accurate and stylistically consistent standard field names. This generation process not only considers the meaning of the field words but also implicitly incorporates the language style and naming conventions within the target business domain, contributing to the standardization of output accuracy and consistency.

[0056] After generating standard field names, they are semantically compared with the original input field names. The comparison method can employ cosine similarity in the pre-trained embedding space to assess semantic closeness, or use an aggregated similarity calculation method based on sentence vectors. If the similarity exceeds a set similarity threshold (e.g., 0.8), the input field name is retained and the generated result is accepted; otherwise, the generated result is marked as suspicious and submitted to a manual proofreading queue to ensure the accuracy and controllability of the standard field identification.

[0057] Furthermore, to enhance the information integrity of standard field identifiers, it is necessary to statistically analyze and integrate the field type information of all structured fields in the current semantic cluster. Field types can include basic data types such as text, integer, floating-point, boolean, date, and currency, and can also be extended to composite types such as enumeration and JSON structures. A majority rule principle is adopted for field type unification, meaning the type with the highest frequency among all field types is used as the final field type for the standard field, ensuring maximum compatibility of the standard field in practical data applications. Finally, the generated standard field names, field types, semantic embedding vectors representing the fields, and system identifier information of the referenced source fields are integrated and encapsulated into a standard field identifier document. Simultaneously, a globally unique identifier (UUID or hash fingerprint) is generated for each standard field identifier to ensure consistent referencing and efficient retrieval across platform components. This standard field identifier document will be written into the standard field index library of the enterprise digital transformation platform.

[0058] Preferably, after obtaining the standard field identifiers of each semantic cluster, the method further includes establishing mapping records for the standard field identifiers; establishing mapping records for the standard field identifiers includes: generating mapping entries for the structured field information associated with each standard field identifier, wherein the mapping entries include field name, field description, field example value, source system identifier, and cluster ID; organizing the mapping entries according to the standard field identifiers to form a mapping relationship table; assigning version numbers to the mapping relationship table, marking the generation time, the version of the clustering algorithm used, and the semantic model identifier; establishing a one-to-one binding relationship between the mapping relationship table and the standard field identifiers to obtain the corresponding mapping records; and writing the mapping records into the standard field index library.

[0059] In this invention, for each standard field identifier, all structured field information within its semantic cluster is traversed. For each structured field information, a corresponding mapping entry is generated. This mapping entry includes the following fields: original field name, field description information, field example value, source system identifier (e.g., system code, system name), and cluster ID, used to locate the field's affiliation within the cluster structure. The field description information and field example value are preprocessed versions to ensure format consistency.

[0060] Furthermore, based on the standard field identifier as the primary key, the above mapping entries are aggregated and organized to construct a structured mapping relationship table. Each standard field identifier corresponds to a unique mapping relationship table, which records all the original field sets represented by that standard field and their key information. To ensure the data change management capability and auditability of this table, each mapping relationship table is assigned a unique version number, and its generation timestamp, the currently used semantic encoding model identifier (such as model training batch, parameter configuration), and the name and version number of the clustering algorithm used are recorded in the table, ensuring the traceability and comparability of each field clustering and standardization operation.

[0061] Furthermore, a one-to-one binding relationship is established between this mapping table and the corresponding standard field identifier. This binding relationship is implemented in the platform through references, ensuring that each standard field identifier has a queryable and updatable linked list of field sources, providing good interface availability. Finally, the generated standard field mapping records are written as independent records to the platform's standard field index library, stored and maintained synchronously with the standard field identifiers, supporting functions such as on-demand querying, batch exporting, and API retrieval.

[0062] Preferably, before storing the generated standard field identifiers in the standard field index library of the platform after digital transformation, the method further includes performing a uniqueness check on the standard field identifiers; the uniqueness check on the standard field identifiers includes: calculating the pairwise cosine similarity between the semantic embedding vector of the standard field identifier to be stored and the embedding vector of all existing standard field identifiers in the index library; performing aggregation analysis on standard field identifiers with similarity exceeding a set threshold to determine whether they are duplicate definitions; if they are duplicate definitions, calling the manual review interface and providing an embedding vector distribution map and original field example content as auxiliary information, responding to the review instruction, performing retention, merging or replacement of existing standard field identifiers, and recording the version number and operator ID for the operation behavior; if there are no duplicates, directly storing them in the standard field index library and recording them as new fields.

[0063] In this embodiment of the invention, the semantic embedding vector corresponding to the standard field identifier to be stored is obtained as the basis for the semantic representation of the current field. Simultaneously, the embedding vectors of all existing standard field identifiers are retrieved from the standard field index library to construct a comparison vector set. All semantic embedding vectors are fixed-length vectors with uniform dimensions and vector space normalization attributes to support efficient and comparable vector similarity calculation. A vector cosine similarity algorithm is used to perform pairwise similarity calculations between the semantic embedding vector of the current standard field identifier and all existing embedding vectors in the index library, resulting in a similarity score list. To optimize efficiency, this process can be based on a pre-built vector index structure (such as FAISS or HNSW) to achieve fast nearest neighbor retrieval.

[0064] After obtaining all similarity results, standard field identifiers with similarity scores higher than a set threshold (e.g., 0.92) are selected as potential duplicate candidates and proceed to the next step of aggregation analysis. Aggregation analysis includes semantic distribution visualization of these high-similarity field identifiers, generating a two-dimensional distribution map of embedding vectors based on principal component analysis (PCA) or t-SNE algorithms to help users identify semantically overlapping regions between fields.

[0065] If multiple field identifiers with abnormally close similarity exist, indicating a potential duplicate field definition, the system will invoke the manual review interface. This interface will visually display: 1) the name, description, and example values ​​of the field to be saved and candidate duplicate fields; 2) a two-dimensional distribution diagram of the semantic embedding vectors of each field; and 3) its source system and its corresponding cluster information. Based on the visual information and experience, reviewers can execute the following instructions: retain the current standard field identifier, merge the current field into an existing field identifier, or replace an existing field identifier with the current field.

[0066] In response to audit instructions, update the standard field index library and record the operation log. This log includes metadata such as: change type (retain / merge / replace), target field ID, operated field ID, auditer ID, operation timestamp, and field version number, for subsequent auditing and version tracking.

[0067] If no existing field identifier exceeding the similarity threshold is found during the above comparison process, the current standard field identifier will be directly stored in the index and registered as a "new field". The system will also generate a unique identifier and version record for this field, marking its creation time, generation process, semantic coding model version, and other metadata.

[0068] The aforementioned standard field identifier uniqueness verification process significantly improves the global consistency and naming standardization of standard fields, effectively avoiding redundant field semantic definitions and the ineffective expansion of platform field resources. Simultaneously, the introduction of a manual review mechanism and visual semantic comparison technology enhances the system's controllability and interpretability, adapting to the real-world challenges of ambiguous field semantic boundaries, multiple meanings, and duplicate names in complex enterprise scenarios, thereby improving the standardization capabilities of enterprise digital transformation platforms at the data governance level.

[0069] Figure 2 This is a system architecture diagram of a data processing system for an enterprise digital transformation platform provided by one embodiment of the present invention. (See diagram below.) Figure 2As shown, this invention provides a data processing system for an enterprise digital transformation platform. The system includes: a data acquisition unit, used to acquire structured field information from various heterogeneous data sources when performing data aggregation from these sources, and to perform preprocessing operations on the structured field information; a vector generation unit, used to construct an embedded input sequence from the processed structured field information, concatenate the embedded input sequence into a natural language fragment according to a preset template, and input it into a fine-tuned semantic coding model to obtain a corresponding semantic embedding vector; a clustering unit, used to identify similar field groups using a density-based or hierarchical clustering algorithm, and to group each group of structured field information into a semantic cluster; and a database storage unit, used to generate a corresponding standard field identifier for each semantic cluster and store the generated standard field identifier in the standard field index database of the platform after digital transformation.

[0070] The present invention also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned data processing method for an enterprise digital transformation platform.

[0071] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0072] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details described above. Within the scope of the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention. It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not further describe the various possible combinations.

[0073] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the embodiments of the present invention, they should also be regarded as the content disclosed by the embodiments of the present invention.

Claims

1. A data processing method for an enterprise digital transformation platform, characterized in that, The method includes: When aggregating data from various heterogeneous data sources, structured field information from each heterogeneous data source is obtained, and preprocessing operations are performed on the structured field information; wherein, Perform preprocessing operations on the structured field information, including: Perform unified character encoding conversion, all-lowercase conversion, and punctuation cleanup on the field names, field descriptions, and field example values ​​in the structured field information, respectively; The built-in tokenizer performs language segmentation on the field description and the field example value, and removes stop words; Based on a pre-defined domain terminology library, the terms in the field description are mapped and replaced. The pre-defined domain terminology library contains synonyms and their standard term correspondences existing in multiple heterogeneous systems. Perform format recognition operations on the field example values ​​to identify date formats, numerical units and / or currency symbols, and generate structured type tags based on the recognition results to obtain preprocessed structured field information; The processed structured field information is constructed into an embedded input sequence. The embedded input sequence is then concatenated into a natural language fragment according to a preset template and input into a fine-tuned semantic coding model to obtain the corresponding semantic embedding vector. Clustering algorithms based on density or hierarchy are used to identify similar field groups, and each group of structured field information is grouped into a semantic cluster. For each semantic cluster, a corresponding standard field identifier is generated, and the generated standard field identifier is stored in the standard field index library of the platform after digital transformation.

2. The method according to claim 1, characterized in that, The processed structured field information is constructed into an embedded input sequence, including: For each structured field information, a natural language template containing the field name, field description, and field example value is constructed, and the corresponding structured field information is constructed into a unified text fragment using a predefined format and the constructed template; The obtained text fragments are encoded into standard token sequences, and low-frequency characters or unrecognized words appearing in the token sequences are processed using a sub-word segmentation mechanism to obtain the embedded input sequence of each structured field information. After obtaining the embedded input sequence of each structured field information, the method further includes: All embedded input sequences are subjected to batch normalization processing, which includes length padding, mask processing, and field boundary annotation.

3. The method according to claim 1, characterized in that, The embedded input sequence is concatenated into a natural language fragment according to a preset template and input into a fine-tuned semantic coding model to obtain the corresponding semantic embedding vector, including: Input natural language segments embedded in the input sequence into a language model based on the Transformer structure; Three types of label vectors are introduced at the model input, which are used to label the field name segment, the field description segment, and the example value segment, respectively. Sequence-level semantic features are extracted through the hidden layer of the semantic coding model, and average pooling is performed on the sequence-level semantic features to generate a fixed-length semantic embedding vector. Perform layer normalization and Dropout operations on the fixed-length semantic embedding vector, and use the fixed-length semantic embedding vector as the semantic embedding vector that uniquely represents the structured field information.

4. The method according to claim 1, characterized in that, Clustering algorithms based on density or hierarchy are used to identify similar field groups, and each group of structured field information is grouped into a semantic cluster, including: A vector space index structure is constructed based on all semantic embedding vectors. The index structure adopts an inverted index and is combined with locality-sensitive hashing. Preliminary clustering is performed based on density clustering algorithm and spatial density of semantic embedding vectors to obtain initial clusters; Principal component analysis is performed on each initial cluster to identify the corresponding principal axis of distribution and to determine whether there are ambiguous fields within the corresponding initial cluster. Initial clusters with ambiguous fields are designated as ambiguous clusters. The polysemous clusters are divided into multiple subclusters using a sub-clustering algorithm to obtain multiple subclusters for each polysemous cluster, and the center vector within the cluster is used as the representative vector of each subcluster. Structured field information that fails to be assigned to any cluster is marked as semantically isolated and recorded in the candidate field table.

5. The method according to claim 1, characterized in that, Generate a corresponding standard field identifier for each semantic cluster, including: In each semantic cluster, the structured field information whose semantic embedding vector is closest to the cluster center is selected as the representative field; Extract the field names from the representative fields, perform character regular expression cleaning to remove redundant naming structures, and obtain the input field names; The standard field names are regenerated using the fine-tuned language model and compared with the original input field names semantically. Input field names with similarity higher than the similarity threshold are retained. Summarize the field type information of all structured fields in the corresponding semantic clusters, and determine a unified field type based on majority rules; The generated input field names, field types, semantic embedding vectors representing the fields, and their source systems are written into a standard field identifier document, and a unique identifier is generated as the standard field identifier for the corresponding semantic cluster.

6. The method according to claim 1, characterized in that, After obtaining the standard field identifiers of each semantic cluster, the method further includes establishing mapping records for the standard field identifiers; Establishing mapping records for standard field identifiers includes: For each standard field identifier associated with the structured field information, a mapping entry is generated. The mapping entry includes the field name, field description, field example value, source system identifier, and cluster ID. The mapping entries are organized according to standard field identifiers to form a mapping relationship table; The mapping table is version-numbered and marked with the generation time, the version of the clustering algorithm used, and the semantic model identifier. Establish a one-to-one binding relationship between the mapping table and the standard field identifier to obtain the corresponding mapping record; Write the mapping records into the standard field index library.

7. The method according to claim 1, characterized in that, Before storing the generated standard field identifier in the standard field index library of the platform after digital transformation, the method further includes performing a uniqueness check on the standard field identifier; Standard field identifier uniqueness verification includes: Perform pairwise cosine similarity calculations between the semantic embedding vector of the standard field identifier to be stored and the embedding vectors of all existing standard field identifiers in the index. Perform aggregation analysis on standard field identifiers with similarity exceeding a set threshold to determine whether they are duplicate definitions; If it is a duplicate definition, the manual review interface will be called, and the embedded vector distribution diagram and the original field example content will be provided as auxiliary information. In response to the review instructions, the existing standard field identifier will be retained, merged or replaced, and the version number and operator ID will be recorded for the operation. If there are no duplicates, the data will be directly stored in the standard field index and recorded as a new field.

8. A data processing system for an enterprise digital transformation platform, characterized in that, The system is used to execute the data processing method for an enterprise digital transformation platform as described in any one of claims 1-7, the system comprising: The acquisition unit is used to acquire structured field information from each heterogeneous data source when performing data aggregation from each heterogeneous data source, and to perform preprocessing operations on the structured field information. The vector generation unit is used to construct the processed structured field information into an embedded input sequence, concatenate the embedded input sequence into a natural language fragment according to a preset template, and input it into the fine-tuned semantic coding model to obtain the corresponding semantic embedding vector. Clustering units are used to identify similar field groups using density-based or hierarchical clustering algorithms, and to group each group of structured field information into a semantic cluster. The data entry unit is used to generate corresponding standard field identifiers for each semantic cluster and store the generated standard field identifiers in the standard field index library of the platform after digital transformation.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the data processing method for an enterprise digital transformation platform as described in any one of claims 1-7.

Images