A large model driven catalog data cleaning and reorganization method and system
By using a large model-driven method for cleaning and organizing cataloging data, the problem of automatically identifying and aggregating synonymous and heterogeneous cataloging data was solved, generating a standardized cataloging dataset and improving the usability and consistency of cataloging data.
Patent Information
- Application Number
- CN202611496126.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-24
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies cannot identify and aggregate cataloging data with different meanings, resulting in poor availability and consistency of cataloging data in business systems.
A large-model-driven cataloging data cleaning and reorganization method is adopted to generate a standardized cataloging dataset through multi-source heterogeneous data verification, basic cleaning, semantic encoding and clustering.
It enables automatic identification and aggregation of synonymous and heterogeneous cataloging data, improving the usability and consistency of cataloging data and reducing subsequent processing deviations caused by data quality issues.
Smart Images

Figure CN123068801A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data processing technology, and in particular relates to a method and system for cleaning and organizing cataloging data driven by a large model. Background Technology
[0002] Industrial material cataloging data is a core foundation for supply chain management and warehousing control, and its quality directly impacts the efficiency of operations such as procurement sourcing, inventory counting, and material matching. However, in the long term, due to factors such as dispersed data sources, inconsistent data entry standards, and barriers to cross-system data synchronization, cataloging data commonly suffers from quality issues such as multiple names for the same item, non-standard naming formats, and information redundancy. This results in the same material being given different names in different systems or batches, severely restricting the availability and consistency of cataloging data in business systems.
[0003] To address the quality issues of cataloging data, existing technologies primarily employ rule-based cleaning schemes. These schemes utilize predefined rule bases and industry dictionaries combined with regular expressions and string matching techniques to detect and process format errors, null values, and duplicate data in the cataloging data. Such schemes can eliminate character-level formatting problems to a certain extent and are suitable for small-scale datasets with a high degree of standardization.
[0004] However, existing rule-based cleaning schemes can only handle character-level format differences and cannot identify semantic-level equivalence relationships, resulting in the inability to automatically identify and aggregate synonymous heterogeneous catalog data. Summary of the Invention
[0005] This invention provides a large model-driven method and system for cleaning and organizing cataloging data, which at least solves the problem in the prior art that the semantic equivalence relationship of cataloging data cannot be identified, resulting in the inability to automatically identify and aggregate synonymous and heterogeneous cataloging data.
[0006] In a first aspect, embodiments of this application provide a large-model-driven method for cleaning and organizing cataloging data, the method comprising: Step S1: Access multi-source heterogeneous catalog data, perform basic verification on the multi-source heterogeneous encoded data, and generate a structured dataset to be processed; Step S2: Perform basic cleaning on the dataset to be processed to obtain a text dataset with a uniform format; Step S3: Calculate the semantic complexity score of each cataloged text in the text dataset, match the embedding model based on the semantic complexity score distribution of all cataloged texts in the text dataset, perform semantic encoding on all text data in the text dataset based on the matched embedding model, and output a high-dimensional feature vector of the same dimension. Step S4: Based on the set of high-dimensional feature vectors of the same dimension, the cataloging data is clustered and grouped according to the number of vectors in the set of high-dimensional feature vectors of the same dimension and the distribution pattern of the vectors in space, and the synonymous heterogeneous cataloging data referring to the same entity are aggregated into the same data cluster. Step S5: For each data cluster, use a large language model to generate a standardized name for the corresponding data cluster; combine each data cluster and its corresponding standardized name into a standardized catalog data entry, and summarize to generate a standardized catalog dataset.
[0007] Further, in step S2, the dataset to be processed undergoes basic cleaning to obtain a text dataset with a uniform format, specifically including: Basic cleaning is performed on each cataloged data in the dataset to be processed. The basic cleaning includes removing redundant spaces and special symbols, unifying capitalization, removing null values and completely duplicate data, filtering stop words and redundant information, and performing semantic correction and restoration on abbreviations, acronyms and typos. The cleaned data is output as a text dataset with a uniform format.
[0008] Further, in step S3, the semantic complexity score of each cataloged text in the text dataset is calculated, and its expression is:
[0009] in, The score is normalized based on the length of the cataloged text; Density of technical terms; The score is the character information entropy normalization score; The proportion of non-standard characters; , , , These are the weighting coefficients.
[0010] Further, in step S3, the embedding model is matched based on the semantic complexity score distribution of all cataloged texts in the text dataset, specifically including: The semantic complexity score distribution of all cataloged texts in the text dataset is statistically analyzed, and combined with the P80 quantile value, an embedding model is matched for the text dataset: When the P80 quantile is greater than or equal to 0.7 or the proportion of highly complex data is greater than or equal to 30%, the BGE-M3 model is matched as the embedding model. When P80 is greater than or equal to 0.4 and less than 0.7, and the proportion of highly complex data is less than 30%, the BGE-large model is matched as the embedding model. When P80 is less than 0.7 and the proportion of low-complexity data is greater than or equal to 70%, the Snowflake / Nomic lightweight model is matched as the embedding model. High-complexity data is cataloged text with a semantic complexity score greater than or equal to 0.7; The proportion of highly complex data is the ratio of the number of cataloged texts with a semantic complexity score greater than or equal to 0.7 to the total number of cataloged texts in the text dataset; Low-complexity data refers to cataloged text with a semantic complexity score of less than 0.7; The proportion of low-complexity data is the ratio of the number of cataloged texts with a semantic complexity score of less than 0.7 to the total number of cataloged texts in the text dataset.
[0011] Furthermore, based on the matched embedding model, semantic encoding is performed on all text data in the text dataset to output a high-dimensional feature vector of the same dimension, specifically including: When the matched embedding model is the BGE-M3 model, dense encoding and sparse encoding are performed on each cataloged text in the text dataset to obtain dense vectors and sparse vectors. The dense vectors and sparse vectors are concatenated and fused to output a high-dimensional feature vector. When the matched embedding model is the BGE-large model, each cataloged text in the text dataset is densely encoded using a single vector, and a high-dimensional feature vector is output after pooling. When the matched embedding model is the Snowflake / Nomic lightweight model, each cataloged text in the text dataset is distilled and then densely encoded, and a high-dimensional feature vector is output after pooling. Project all high-dimensional feature vectors uniformly into a space of the same dimension, and output a set of high-dimensional feature vectors of the same dimension.
[0012] Further, in step S4, based on the set of high-dimensional feature vectors of the same dimension, the cataloging data is clustered and grouped according to the number of vectors in the set of high-dimensional feature vectors of the same dimension and the distribution pattern of the vectors in space, and synonymous heterogeneous cataloging data referring to the same entity are aggregated into the same data cluster, specifically including: Based on the number of vectors and their spatial distribution in the set of high-dimensional feature vectors of the same dimension, a clustering algorithm is matched from a set of preset clustering algorithms, including hierarchical clustering, K-Means algorithm, DBSCAN algorithm and GMM Gaussian mixture clustering algorithm. The matched clustering algorithm is used to cluster the vectors in the high-dimensional feature vector set of the same dimension, and the cataloging data with similar semantics are aggregated into the same data cluster.
[0013] Furthermore, a hierarchical clustering algorithm is used to cluster and group the vectors in the high-dimensional feature vector set of the same dimension, aggregating semantically similar cataloging data into the same data cluster, specifically including: Each vector in the set of high-dimensional feature vectors of the same dimension is used as an initial cluster; Calculate the similarity between any two clusters, iteratively merge the two clusters with the highest similarity, until the number of clusters reaches the preset target number of clusters or the minimum similarity between clusters is lower than the set threshold; Output the final cluster partitioning results, with each cluster containing semantically similar cataloging data; The similarity between any two clusters is calculated using the following expression:
[0014] in, and Clusters and cluster The central vector (i.e., the mean of all vectors within the cluster).
[0015] Furthermore, the K-Means algorithm is used to cluster the vectors in the high-dimensional feature vector set of the same dimension, aggregating semantically similar cataloging data into the same data cluster, specifically including: A predetermined number of vectors are randomly selected from the set of high-dimensional feature vectors of the same dimension as the initial cluster centers; Calculate the distance between each vector and each cluster center, and assign each vector to the cluster containing the nearest cluster center; Calculate the mean of all vectors in each cluster and update the cluster center of that cluster; Repeat the vector assignment and cluster center update steps until the change in cluster centers is less than the convergence threshold or the maximum number of iterations is reached. Output the final cluster partitioning results, with each cluster containing semantically similar catalog data.
[0016] Furthermore, the DBSCAN algorithm is used to cluster the vectors in the high-dimensional feature vector set of the same dimension, aggregating semantically similar cataloging data into the same data cluster, specifically including: Traverse each vector from the set of high-dimensional feature vectors of the same dimension and calculate the number of neighbor vectors of that vector within the neighborhood radius. When the number of neighbor vectors is greater than or equal to the minimum sample number threshold, the vector is marked as the core point, and a new cluster is created with the vector as the center. Add all vectors within the radius of the core point's neighborhood to the cluster, and recursively perform neighborhood search on the newly added vectors to expand the cluster's boundary; Vectors whose number of neighbor vectors is less than the minimum sample number threshold and are not in the neighborhood of any core point are marked as noise points; Output the final cluster partitioning results. Each cluster contains semantically similar cataloged data. Noise points are not assigned to any cluster. The Gaussian mixture model (GMM) clustering algorithm is used to cluster the vectors in the high-dimensional feature vector set of the same dimension, aggregating semantically similar cataloging data into the same data cluster. Specifically, this includes: Initialize a preset number of Gaussian distribution components, each component containing a mean vector, a covariance matrix, and mixing coefficients; Calculate the posterior probability of each vector under each Gaussian distribution component, and assign the vector to the cluster to which the component with the highest posterior probability belongs; Based on the current allocation results, re-estimate the mean vector, covariance matrix, and mixing coefficients for each component; Repeat the steps of posterior probability calculation and parameter estimation until the change in the likelihood function is less than the convergence threshold or the maximum number of iterations is reached. Output the final cluster partitioning results, with each cluster containing semantically similar catalog data.
[0017] Secondly, embodiments of this application also provide a system for the large model-driven cataloging data cleaning and organization method described in the above aspects, the system comprising: The data access verification module is used to access multi-source heterogeneous cataloged data, perform basic verification on multi-source heterogeneous encoded data, and generate a structured dataset to be processed. The basic data cleaning module is used to perform basic cleaning on the dataset to be processed, so as to obtain a text dataset with a uniform format. The semantic encoding module is used to calculate the semantic complexity score of each cataloged text in the text dataset, match the embedding model based on the semantic complexity score distribution of all cataloged texts in the text dataset, perform semantic encoding on all text data in the text dataset based on the matched embedding model, and output a high-dimensional feature vector of the same dimension. The clustering and grouping module is used to cluster and group cataloging data based on the set of high-dimensional feature vectors of the same dimension, according to the number of vectors in the set of high-dimensional feature vectors of the same dimension and the distribution pattern of the vectors in space, and to aggregate synonymous heterogeneous cataloging data that refer to the same entity into the same data cluster. The standardization generation module is used to generate standardized names for each data cluster using a large language model; and to combine each data cluster and its corresponding standardized name into standardized catalog data entries, and then aggregate them to generate a standardized catalog dataset.
[0018] As can be seen from the above technical solutions, the present invention has the following advantages: The large model-driven cataloging data cleaning and organization method provided in this application performs basic verification on the accessed multi-source heterogeneous cataloging data, removes invalid dirty data, and generates a structured dataset to be processed. This provides a reliable data entry foundation for subsequent semantic processing and standardization governance, and reduces the risk of subsequent processing deviations caused by source data quality issues.
[0019] This invention performs basic cleaning on the dataset to be processed, resulting in a text dataset with a uniform format. This eliminates the format inconsistencies caused by differences in input habits and system encoding in cataloging data, providing uniform and clean input data for subsequent semantic encoding, thereby improving the stability and accuracy of semantic encoding.
[0020] This invention calculates the semantic complexity score of each cataloging text, matches the embedding model based on the semantic complexity score distribution of the full text data, performs semantic encoding on the full text data, and outputs a high-dimensional feature vector of the same dimension. This realizes a differentiated encoding strategy for cataloging data with different complexities, enabling complex cataloging texts to obtain high-precision encoding while simple cataloging texts can also be encoded with low computational cost, thus balancing encoding accuracy and processing efficiency.
[0021] This invention is based on a set of high-dimensional feature vectors of the same dimension. It uses a clustering algorithm to cluster cataloging data according to the size and spatial distribution of the vectors. It aggregates synonymous and heterogeneous cataloging data that refer to the same entity into the same data cluster. This solves the fundamental limitation of existing rule matching schemes that cannot identify semantic equivalence relations. It achieves a technical leap from character-level matching to semantic-level aggregation and effectively improves the governance of the problem of one item having multiple names in cataloging data.
[0022] This invention generates standardized names for each data cluster using a large language model, and combines each data cluster and its standardized names into standardized catalog data entries, which are then aggregated to generate a standardized catalog dataset. This achieves end-to-end automated processing from raw dirty data to standardized catalog data. The output standardized catalog dataset can be directly applied to business systems, improving the usability and consistency of catalog data. Attached Figure Description
[0023] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1This is a flowchart of the large model-driven cataloging data cleaning and organization method described in this invention. Detailed Implementation
[0025] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0026] This application provides a large model-driven method and system for cleaning and organizing cataloging data, which solves the current urgent technical problem of automatically identifying and aggregating semantically equivalent but character-formally different synonymous entries in industrial material cataloging data.
[0027] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0028] Figure 1 A flowchart illustrating a large-model-driven cataloging data cleaning and organization method and system provided in this application embodiment. Figure 1 As shown in the embodiments of this application, a large model-driven cataloging data cleaning and organization method and system are provided, which specifically includes the following steps: Step S1: Access multi-source heterogeneous catalog data, perform basic verification on the multi-source heterogeneous coded data, remove invalid dirty data, and generate a structured dataset to be processed; the multi-source heterogeneous catalog data includes Excel and CSV; the basic verification includes: data format unification, null value screening, and initial screening of duplicate data.
[0029] In this embodiment, the data format unification specifically includes: converting date, numerical, and text fields in multi-source heterogeneous cataloging data into a unified standard format; The null value screening specifically includes: detecting null and missing values in catalog data, and marking or removing data records with empty key fields; The initial screening of duplicate data specifically includes: identifying and removing completely duplicate data records based on exact matching of key fields.
[0030] According to another embodiment of the present invention, for unstructured tables and heterogeneous files with misaligned fields, the catalog name field, specification field, and category field of the heterogeneous files are identified, field mapping and alignment are completed, and basic data verification is performed to generate a structured dataset to be processed, thereby reducing the manual cost of accessing heterogeneous files.
[0031] Step S2: Perform basic cleaning on the dataset to be processed to obtain a text dataset with a uniform format; Step S3: Calculate the semantic complexity score of each cataloged text in the text dataset, match the embedding model based on the semantic complexity score distribution of all cataloged texts in the text dataset, perform semantic encoding on all text data in the text dataset based on the matched embedding model, and output a high-dimensional feature vector of the same dimension. Step S4: Based on the set of high-dimensional feature vectors of the same dimension, the cataloging data is clustered and grouped according to the number of vectors in the set of high-dimensional feature vectors of the same dimension and the distribution pattern of the vectors in space, and the synonymous heterogeneous cataloging data referring to the same entity are aggregated into the same data cluster. Step S5: For each data cluster, use a large language model to generate a standardized name for the corresponding data cluster; combine each data cluster and its corresponding standardized name into a standardized catalog data entry, and summarize to generate a standardized catalog dataset.
[0032] It should be noted that in step S2, the dataset to be processed undergoes basic cleaning to obtain a text dataset with a uniform format, specifically including: Basic cleaning is performed on each cataloged data in the dataset to be processed. The basic cleaning includes removing redundant spaces and special symbols, unifying capitalization, removing null values and completely duplicate data, filtering stop words and redundant information, and performing semantic correction and restoration on abbreviations, acronyms and typos. Record a full cleaning operation log and generate a data comparison report before and after cleaning; The cleaned data is output as a text dataset with a uniform format.
[0033] In an exemplary embodiment, in step S3, the semantic complexity score of each cataloged text in the text dataset is calculated, and its expression is:
[0034] in, The normalized score for catalog text length is calculated by dividing the number of characters by 30, with an upper limit of 1.0. The terminology density is calculated by dividing the number of terms hits in the industry knowledge base by the total number of words. The character information entropy normalization score is calculated by dividing the information entropy by 4.0, with an upper limit of 1.0. The percentage of non-standard characters is calculated by dividing the number of characters not in the standard character set by the total number of characters. , , , These are the weighting coefficients, with default values of 0.2, 0.4, 0.2, and 0.2.
[0035] Cataloging text length normalized score The expression is:
[0036] in, The number of characters in the catalog text; Terminology density The expression is:
[0037] in, This represents the number of times industry knowledge base terms are encountered in the current cataloging text. This represents the total number of words in the currently cataloged text. The expression for the character information entropy normalization score is:
[0038] in, The character information entropy is expressed as:
[0039] In the formula, The number of different character types in the cataloged text. For the first Character The probability of a character appearing in the catalog text is the number of times the character appears divided by the total number of characters in the text. The expression for the proportion of non-standard characters is:
[0040] in, This represents the number of characters in the current catalog text that are not within the standard character set.
[0041] According to another embodiment of the present invention, in step S3, matching the embedding model based on the semantic complexity score distribution of all cataloged texts in the text dataset specifically includes: The semantic complexity score distribution of all cataloged texts in the text dataset is statistically analyzed, and combined with the P80 quantile value, an embedding model is matched for the text dataset: When the P80 quantile is greater than or equal to 0.7 or the proportion of highly complex data is greater than or equal to 30%, the BGE-M3 model is matched as the embedding model. When P80 is greater than or equal to 0.4 and less than 0.7, and the proportion of highly complex data is less than 30%, the BGE-large model is matched as the embedding model. When P80 is less than 0.7 and the proportion of low-complexity data is greater than or equal to 70%, the Snowflake / Nomic lightweight model is matched as the embedding model. High-complexity data is cataloged text with a semantic complexity score greater than or equal to 0.7; The proportion of highly complex data is the ratio of the number of cataloged texts with a semantic complexity score greater than or equal to 0.7 to the total number of cataloged texts in the text dataset; Low-complexity data refers to cataloged text with a semantic complexity score of less than 0.7; The proportion of low-complexity data is the ratio of the number of cataloged texts with a semantic complexity score of less than 0.7 to the total number of cataloged texts in the text dataset.
[0042] It's important to note that the P80 quantile refers to the value at the 80th percentile when a set of data is sorted from smallest to largest. Specifically, if the semantic complexity scores of all cataloged texts in the dataset are sorted from smallest to largest, the score at the 80th percentile is the P80 quantile. This means that 80% of the cataloged texts have a complexity score less than or equal to this value, and 20% have a complexity score greater than this value. For example, if there are 100 cataloged texts, the 80th score, when sorted from smallest to largest, is the P80 quantile.
[0043] According to embodiments of this application, semantic encoding is performed on all text data in the text dataset based on the matched embedding model, outputting a high-dimensional feature vector of the same dimension, specifically including: When the matched embedding model is the BGE-M3 model, dense encoding and sparse encoding are performed on each cataloged text in the text dataset to obtain dense vectors and sparse vectors. The dense vectors and sparse vectors are concatenated and fused to output a high-dimensional feature vector. The cataloged text is input into the Tokenizer layer of the BGE-M3 model, where it is segmented and tokenized to generate a token index sequence. The token index sequence is input into the dense coding branch of the BGE-M3 model. Contextual semantic encoding is performed through a multi-layer Transformer encoder to extract the deep semantic representation of each token. Mean pooling is then performed on the hidden representations of all tokens to generate a dense vector of fixed length with a dimension of 1024. The token index sequence is input into the sparse coding branch of the BGE-M3 model. Based on the pre-trained word frequency statistics and importance weight mapping, the sparse weight distribution of each token in the vocabulary is calculated. Only non-zero dimensions with weights higher than a set threshold are retained to generate a sparse vector. The sparse vector has the same dimension as the vocabulary size and most of the dimensions are zero. The dense vector and the sparse vector are concatenated and fused along the dimensional direction to output a high-dimensional feature vector with a dimension of 1024.
[0044] When the matched embedding model is the BGE-large model, each cataloged text in the text dataset is densely encoded using a single vector, and a high-dimensional feature vector is output after pooling. The cataloged text is input into the Tokenizer layer of the BGE-large model, where it is segmented and tokenized, and [CLS] and [SEP] tags are added to generate a token index sequence. The token index sequence is input into the Transformer encoder of the BGE-large model, and is encoded layer by layer through a 24-layer multi-head attention mechanism and a feedforward neural network to extract the contextual semantic representation of each token and generate a token-level hidden layer vector sequence. Perform a pooling operation on the hidden vector sequence, extract the hidden vector corresponding to the [CLS] tag as the semantic representation of the entire catalog text, or perform mean pooling on the hidden vectors of all tokens to generate a global semantic representation; The pooled semantic vector is subjected to L2 normalization to output a high-dimensional feature vector with a dimension of 1024.
[0045] When the matched embedding model is the Snowflake / Nomic lightweight model, each cataloged text in the text dataset is distilled and then densely encoded, and a high-dimensional feature vector is output after pooling. The cataloged text is input into the Tokenizer layer of the lightweight model, where it is segmented and tokenized to generate a token index sequence. The token index sequence is input into the distilled Transformer encoder of the lightweight model. The distilled Transformer encoder is compressed from the large-scale teacher model through knowledge distillation technology. It has fewer layers than the teacher model and the hidden layer dimension is reduced. The token index sequence is forward encoded to extract the lightweight semantic representation of each token and generate a token-level hidden layer vector sequence. A pooling operation is performed on the hidden vector sequence to generate a global semantic representation by mean pooling of the hidden vectors of all tokens, or the hidden vector corresponding to the first token is extracted as the semantic representation of the entire catalog text. The pooled semantic vector is subjected to L2 normalization to output a high-dimensional feature vector with a dimension of 768.
[0046] Project all high-dimensional feature vectors uniformly into a space of the same dimension, and output a set of high-dimensional feature vectors of the same dimension.
[0047] In one embodiment, in step S4, based on the set of high-dimensional feature vectors of the same dimension, the cataloging data is clustered and grouped according to the number of vectors in the set of high-dimensional feature vectors of the same dimension and the distribution pattern of the vectors in space using a clustering algorithm. Synonymous and heterogeneous cataloging data referring to the same entity are aggregated into the same data cluster. Specifically, this includes: Based on the number of vectors and their spatial distribution in the set of high-dimensional feature vectors of the same dimension, a clustering algorithm is matched from a set of preset clustering algorithms, including hierarchical clustering, K-Means algorithm, DBSCAN algorithm and GMM Gaussian mixture clustering algorithm. The matched clustering algorithm is used to cluster the vectors in the high-dimensional feature vector set of the same dimension, and the cataloging data with similar semantics are aggregated into the same data cluster.
[0048] It should be noted that when the amount of data is less than or equal to 10,000 records and the number of clusters is predictable, a hierarchical clustering algorithm is used. When the amount of data is between 10,000 and 100,000 records and the cluster shape is approximately spherical, the K-Means algorithm is used. When the data distribution is irregular and there are noisy points, the DBSCAN algorithm is used; When the data has overlapping distributions, the Gaussian Mixture Clustering (GMM) algorithm is used.
[0049] As an example, a hierarchical clustering algorithm is used to cluster and group vectors in the high-dimensional feature vector set of the same dimension, aggregating semantically similar cataloging data into the same data cluster, specifically including: Each vector in the set of high-dimensional feature vectors of the same dimension is used as an initial cluster; Calculate the similarity between any two clusters, iteratively merge the two clusters with the highest similarity, until the number of clusters reaches the preset target number of clusters or the minimum similarity between clusters is lower than the set threshold; Output the final cluster partitioning results, with each cluster containing semantically similar cataloging data; The similarity between any two clusters is calculated using the following expression:
[0050] in, and Clusters and cluster The central vector (i.e., the mean of all vectors within the cluster).
[0051] The K-Means algorithm is used to cluster the vectors in the high-dimensional feature vector set of the same dimension, aggregating semantically similar cataloging data into the same data cluster, specifically including: A predetermined number of vectors are randomly selected from the set of high-dimensional feature vectors of the same dimension as the initial cluster centers; Calculate the distance between each vector and each cluster center, and assign each vector to the cluster containing the nearest cluster center; Calculate the mean of all vectors in each cluster and update the cluster center of that cluster; Repeat the vector assignment and cluster center update steps until the change in cluster centers is less than the convergence threshold or the maximum number of iterations is reached. Output the final cluster partitioning results, with each cluster containing semantically similar catalog data.
[0052] The DBSCAN algorithm is used to cluster and group the vectors in the high-dimensional feature vector set of the same dimension, aggregating semantically similar cataloging data into the same data cluster, specifically including: Traverse each vector from the set of high-dimensional feature vectors of the same dimension and calculate the number of neighbor vectors of that vector within the neighborhood radius. When the number of neighbor vectors is greater than or equal to the minimum sample number threshold, the vector is marked as the core point, and a new cluster is created with the vector as the center. Add all vectors within the radius of the core point's neighborhood to the cluster, and recursively perform neighborhood search on the newly added vectors to expand the cluster's boundary; Vectors whose number of neighbor vectors is less than the minimum sample number threshold and are not in the neighborhood of any core point are marked as noise points; The final clustering results are output, with each cluster containing semantically similar cataloged data. Noise points are not assigned to any cluster.
[0053] The Gaussian mixture model (GMM) clustering algorithm is used to cluster the vectors in the high-dimensional feature vector set of the same dimension, aggregating semantically similar cataloging data into the same data cluster. Specifically, this includes: Initialize a preset number of Gaussian distribution components, each component containing a mean vector, a covariance matrix, and mixing coefficients; Calculate the posterior probability of each vector under each Gaussian distribution component, and assign the vector to the cluster to which the component with the highest posterior probability belongs; Based on the current allocation results, re-estimate the mean vector, covariance matrix, and mixing coefficients for each component; Repeat the steps of posterior probability calculation and parameter estimation until the change in the likelihood function is less than the convergence threshold or the maximum number of iterations is reached. Output the final cluster partitioning results, with each cluster containing semantically similar catalog data.
[0054] According to another embodiment of the present invention, after the clustering grouping is completed, the method further includes: using the manually fine-tuned clusters as constraint anchors, fixing the samples and cluster center vectors within the anchor clusters, and no longer participating in subsequent clustering iteration calculations; for the remaining unassigned samples, calculating the cosine similarity between the sample vector and the center of each anchor cluster, and assigning the sample to the corresponding anchor cluster when the sample similarity is higher than a set threshold, and generating a new cluster for samples that do not meet the threshold through a semi-supervised clustering algorithm.
[0055] In step S5, a standardized prompt word template is constructed using four modules: role positioning, rule constraints, case reference, and knowledge anchors. Matching cataloging specifications and standard terms are retrieved from the industry knowledge base and injected into the prompt word template. A standardized name, multiple candidate names, and naming criteria are generated through a large language model. The retrieval adopts a hybrid recall strategy of keyword retrieval and vector semantic retrieval, and the recall result is Top 3 recall with a similarity threshold greater than or equal to 0.7.
[0056] (1) Standardized Prompt Template for Four Modules: The Prompt template is shown below: Role positioning: You are a standardization expert with over 10 years of experience in industrial materials cataloging, proficient in industry cataloging specifications and naming standards, and skilled at extracting accurate and standardized common names from multiple versions of synonym data.
[0057] Rule constraints: 1. Naming must follow a hierarchical structure of "modifier + base name" and conform to the corresponding industry cataloging standards; 2. The name length should be controlled between 8 and 30 characters, concise and without redundancy, and non-standard information such as brand, manufacturer, and non-standard codes should be removed; 3. Prioritize the use of national standard terminology, avoid colloquial or localized expressions, and standardize units of measurement and symbols; 4. When there are differences in specifications among data in the same cluster, take the common feature with the highest coverage and put the differences into the remarks field.
[0058] Case Study: Original cluster data: ["Deep groove ball bearing 6205 / Z3", "Deep groove ball bearing", "Ball bearing"]; Standard name: Deep groove ball bearing; Naming criteria: Extract the core product category "deep groove ball bearings" and remove colloquial expressions, model numbers, and other non-generic information; Knowledge Anchor Points: {{RAG recalls industry standard clauses, standard terms, and historical high-quality samples, with standard numbers and sources noted}}; Current task: Please process the following data clusters and output their normalized names: Original data cluster: {{all catalog entries within the cluster}}; Please strictly adhere to the JSON format when outputting: { "standard_name": "Standardization benchmark name", "candidates": ["candidate name1", "candidate name2"], "reason": Explanation of the naming criteria } (2) RAG search enhancement mechanism: Search method: A hybrid recall strategy of "keyword search + vector semantic search" is adopted to improve the recall accuracy through bidirectional complementarity; Search fields: standard terminology, cataloging specifications, and historical high-quality naming samples from the industry knowledge base; Recall rules: Top 3 are recalled, with a similarity threshold of ≥0.7; otherwise, no injection is performed. Priority rule: National mandatory standards > Industry specifications > Historical high-quality samples; In case of conflict between search results and rule constraints, the standards and specifications shall prevail. Injection method: The knowledge anchor module of Prompt is embedded and presented in the form of referenced articles, with the standard number clearly marked.
[0059] (3) Large model call and output constraints: Model selection: DeepSeek-V2 / Qwen2-72B is selected for high-precision scenarios, and Qwen2-7B / 14B is selected for fast processing scenarios; Hyperparameter configuration: temperature=0.1, top_p=0.3, max_tokens=512, to ensure stable output with low divergence; Output parsing: Forces JSON Schema validation. Output fields include standard_name (string), candidates (array, length 2-3), and reason (string). If parsing fails, it will automatically retry twice. If it still fails, it will be marked as pending manual processing.
[0060] The method also includes: reviewing the standardized names generated by the large model, and performing adoption, modification, rewriting, and rejection operations; the system automatically records all review results and modification content in a structured manner, constructs a three-level closed-loop self-iterative mechanism of "knowledge base layer - strategy layer - model layer", continuously optimizes vector embedding model selection, clustering algorithm parameters, and large model prompt word templates, forming a self-iterative closed loop of system capabilities, and gradually improving processing accuracy.
[0061] The approved standardized names and corresponding original data clusters are stored in the high-quality sample library of the industry knowledge base, and manually corrected terms are updated to the standard terminology library. The strategy layer calculates the approval rate by data category. When the approval rate of a certain type of data is lower than 80% for consecutive batches, the embedding model is automatically upgraded. The initial parameter recommendation rules of the clustering algorithm are iteratively updated based on the results of manual clustering fine-tuning. The prompt word template and constraint rules are optimized based on manual modification records. After accumulating tens of thousands of labeled samples, the model layer performs lightweight fine-tuning on the embedding model and the large language model based on domain data.
[0062] It also includes: generating standardized datasets from the approved full-volume standardized catalog data, supporting export in multiple formats, and providing API interfaces to synchronize with business systems.
[0063] Taking industrial sealing materials as an example, the complete processing chain is as follows: Raw dirty data cluster: ["O-ring", "O-ring", "O-ring", "O-ring"]; Text after intelligent cleaning: ["O-ring", "O-ring", "O-ring"]; Cleaning process: Remove extra spaces, standardize character format, and remove completely duplicate entries; Model matching: The semantic complexity of calculating the full dataset is medium, so the BGE-large embedding model is selected uniformly. Vector generation: Output four 1024-dimensional semantic vectors, with cosine similarity between the vectors ≥ 0.85; Clustering: The DBSCAN algorithm was used to aggregate 4 data points into the same data cluster; RAG Recall: The standard clause of GB / T 5719-2006 "Glossary of Rubber Sealing Products" was retrieved, and the standard term "O-ring" was extracted; Large model output: json { "standard_name": "O-ring seal", "candidates": ["O-rings", "O-rings"], "reason": "According to the GB / T 5719-2006 standard 'Glossary of Rubber Sealing Products,' the unified category name is O-ring seal, eliminating redundant information." } Expert review: The name "O-ring seal" has been confirmed and adopted. Closed-loop feedback: Confirmed samples are stored in a high-quality sample library, and the RAG search index is updated synchronously.
[0064] Semantic complexity scoring core pseudocode: Python run def calc_semantic_complexity(text, term_lib): # Text length normalization L_norm = min(len(text) / 30, 1.0) # Terminology Density word_list = word_segment(text) term_hit = sum(1 for w in word_list if w in term_lib) D = term_hit / len(word_list) if word_list else 0 # Information Entropy Normalization entropy = calc_char_entropy(text) E_norm = min(entropy / 4.0, 1.0) # Percentage of unconventional characters common_chars = set("abcdefghijklmnopqrstuvwxyz0123456789-() / ") rare_char = sum(1 for c in text.lower() if c not in common_chars) R = rare_char / len(text) if text else 0 # Weighted calculation of total score S = 0.2 L_norm + 0.4 D + 0.2 E_norm + 0.2 R return S def select_embedding_model(S): if S>= 0.7: return "bge-m3" else: return "snowflake-base" The above function is used to calculate the complexity of a single text. After the full calculation is completed, the selection of a unified embedding model at the dataset level is finally determined by using the statistical data set P80 quantile and percentage indicators.
[0065] This invention also provides a large-model-driven cataloging data cleaning and organization system, the system comprising: The data access verification module is used to access multi-source heterogeneous cataloged data, perform basic verification on multi-source heterogeneous encoded data, and generate a structured dataset to be processed. The basic data cleaning module is used to perform basic cleaning on the dataset to be processed, so as to obtain a text dataset with a uniform format. The semantic encoding module is used to calculate the semantic complexity score of each cataloged text in the text dataset, match the embedding model based on the semantic complexity score distribution of all cataloged texts in the text dataset, perform semantic encoding on all text data in the text dataset based on the matched embedding model, and output a high-dimensional feature vector of the same dimension. The clustering and grouping module is used to cluster and group cataloging data based on the set of high-dimensional feature vectors of the same dimension, according to the number of vectors in the set of high-dimensional feature vectors of the same dimension and the distribution pattern of the vectors in space, and to aggregate synonymous heterogeneous cataloging data that refer to the same entity into the same data cluster. The standardization generation module is used to generate standardized names for each data cluster using a large language model; and to combine each data cluster and its corresponding standardized name into standardized catalog data entries, and then aggregate them to generate a standardized catalog dataset.
[0066] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0067] Any changes, modifications, substitutions, and variations made to the embodiments without departing from the principles and spirit of the present invention still fall within the protection scope of the present invention.
Claims
1. A method for cleaning and organizing cataloging data driven by a large model, characterized in that, The method includes: Step S1: Access multi-source heterogeneous catalog data, perform basic verification on the multi-source heterogeneous encoded data, and generate a structured dataset to be processed; Step S2: Perform basic cleaning on the dataset to be processed to obtain a text dataset with a uniform format; Step S3: Calculate the semantic complexity score of each cataloged text in the text dataset, match the embedding model based on the semantic complexity score distribution of all cataloged texts in the text dataset, perform semantic encoding on all text data in the text dataset based on the matched embedding model, and output a high-dimensional feature vector of the same dimension. Step S4: Based on the set of high-dimensional feature vectors of the same dimension, the cataloging data is clustered and grouped according to the number of vectors in the set of high-dimensional feature vectors of the same dimension and the distribution pattern of the vectors in space, and the synonymous heterogeneous cataloging data referring to the same entity are aggregated into the same data cluster. Step S5: For each data cluster, use a large language model to generate a standardized name for the corresponding data cluster; combine each data cluster and its corresponding standardized name into a standardized catalog data entry, and summarize to generate a standardized catalog dataset.
2. The method according to claim 1, characterized in that, In step S2, the dataset to be processed undergoes basic cleaning to obtain a text dataset with a uniform format, specifically including: Basic cleaning is performed on each cataloged data in the dataset to be processed. The basic cleaning includes removing redundant spaces and special symbols, unifying capitalization, removing null values and completely duplicate data, filtering stop words and redundant information, and performing semantic correction and restoration on abbreviations, acronyms and typos. The cleaned data is output as a text dataset with a uniform format.
3. The method according to claim 1, characterized in that, In step S3, the semantic complexity score of each cataloged text in the text dataset is calculated, and its expression is: in, The score is normalized based on the length of the cataloged text; Density of technical terms; The score is the character information entropy normalization score; The proportion of non-standard characters; , , , These are the weighting coefficients.
4. The method according to claim 3, characterized in that, In step S3, the embedding model is matched based on the semantic complexity score distribution of all cataloged texts in the text dataset, specifically including: The semantic complexity score distribution of all cataloged texts in the text dataset is statistically analyzed, and combined with the P80 quantile value, an embedding model is matched for the text dataset: When the P80 quantile is greater than or equal to 0.7 or the proportion of highly complex data is greater than or equal to 30%, the BGE-M3 model is matched as the embedding model. When P80 is greater than or equal to 0.4 and less than 0.7, and the proportion of highly complex data is less than 30%, the BGE-large model is matched as the embedding model. When P80 is less than 0.7 and the proportion of low-complexity data is greater than or equal to 70%, the Snowflake / Nomic lightweight model is matched as the embedding model. High-complexity data is cataloged text with a semantic complexity score greater than or equal to 0.7; The proportion of highly complex data is the ratio of the number of cataloged texts with a semantic complexity score greater than or equal to 0.7 to the total number of cataloged texts in the text dataset; Low-complexity data refers to cataloged text with a semantic complexity score of less than 0.7; The proportion of low-complexity data is the ratio of the number of cataloged texts with a semantic complexity score of less than 0.7 to the total number of cataloged texts in the text dataset.
5. The method according to claim 4, characterized in that, Based on the matched embedding model, semantic encoding is performed on all text data in the text dataset to output a high-dimensional feature vector of the same dimension, specifically including: When the matched embedding model is the BGE-M3 model, dense encoding and sparse encoding are performed on each cataloged text in the text dataset to obtain dense vectors and sparse vectors. The dense vectors and sparse vectors are concatenated and fused to output a high-dimensional feature vector. When the matched embedding model is the BGE-large model, each cataloged text in the text dataset is densely encoded using a single vector, and a high-dimensional feature vector is output after pooling. When the matched embedding model is the Snowflake / Nomic lightweight model, each cataloged text in the text dataset is distilled and then densely encoded, and a high-dimensional feature vector is output after pooling. Project all high-dimensional feature vectors uniformly into a space of the same dimension, and output a set of high-dimensional feature vectors of the same dimension.
6. The method according to claim 1, characterized in that, In step S4, based on the set of high-dimensional feature vectors of the same dimension, the cataloging data is clustered and grouped according to the number of vectors in the set and their spatial distribution using a clustering algorithm. Synonymous and heterogeneous cataloging data referring to the same entity are aggregated into the same data cluster. Specifically, this includes: Based on the number of vectors and their spatial distribution in the set of high-dimensional feature vectors of the same dimension, a clustering algorithm is matched from a set of preset clustering algorithms, including hierarchical clustering, K-Means algorithm, DBSCAN algorithm and GMM Gaussian mixture clustering algorithm. The matched clustering algorithm is used to cluster the vectors in the high-dimensional feature vector set of the same dimension, and the cataloging data with similar semantics are aggregated into the same data cluster.
7. The method according to claim 6, characterized in that, Hierarchical clustering algorithm is used to cluster and group vectors in the high-dimensional feature vector set of the same dimension, aggregating semantically similar cataloging data into the same data cluster, specifically including: Each vector in the set of high-dimensional feature vectors of the same dimension is used as an initial cluster; Calculate the similarity between any two clusters, iteratively merge the two clusters with the highest similarity, until the number of clusters reaches the preset target number of clusters or the minimum similarity between clusters is lower than the set threshold; Output the final cluster partitioning results, with each cluster containing semantically similar cataloging data; The similarity between any two clusters is calculated using the following expression: in, and Clusters and cluster The central vector (i.e., the mean of all vectors within the cluster).
8. The method according to claim 6, characterized in that, The K-Means algorithm is used to cluster the vectors in the high-dimensional feature vector set of the same dimension, aggregating semantically similar cataloging data into the same data cluster, specifically including: A predetermined number of vectors are randomly selected from the set of high-dimensional feature vectors of the same dimension as the initial cluster centers; Calculate the distance between each vector and each cluster center, and assign each vector to the cluster containing the nearest cluster center; Calculate the mean of all vectors in each cluster and update the cluster center of that cluster; Repeat the vector assignment and cluster center update steps until the change in cluster centers is less than the convergence threshold or the maximum number of iterations is reached. Output the final cluster partitioning results, with each cluster containing semantically similar catalog data.
9. The method according to claim 6, characterized in that, The DBSCAN algorithm is used to cluster and group the vectors in the high-dimensional feature vector set of the same dimension, aggregating semantically similar cataloging data into the same data cluster, specifically including: Traverse each vector from the set of high-dimensional feature vectors of the same dimension and calculate the number of neighbor vectors of that vector within the neighborhood radius. When the number of neighbor vectors is greater than or equal to the minimum sample number threshold, the vector is marked as the core point, and a new cluster is created with the vector as the center. Add all vectors within the radius of the core point's neighborhood to the cluster, and recursively perform neighborhood search on the newly added vectors to expand the cluster's boundary; Vectors whose number of neighbor vectors is less than the minimum sample number threshold and are not in the neighborhood of any core point are marked as noise points; Output the final cluster partitioning results. Each cluster contains semantically similar cataloged data. Noise points do not belong to any cluster. The Gaussian mixture model (GMM) clustering algorithm is used to cluster the vectors in the high-dimensional feature vector set of the same dimension, aggregating semantically similar cataloging data into the same data cluster. Specifically, this includes: Initialize a preset number of Gaussian distribution components, each component containing a mean vector, a covariance matrix, and mixing coefficients; Calculate the posterior probability of each vector under each Gaussian distribution component, and assign the vector to the cluster to which the component with the highest posterior probability belongs; Based on the current allocation results, re-estimate the mean vector, covariance matrix, and mixing coefficients for each component; Repeat the steps of posterior probability calculation and parameter estimation until the change in the likelihood function is less than the convergence threshold or the maximum number of iterations is reached. Output the final cluster partitioning results, with each cluster containing semantically similar catalog data.
10. A system applied to the large model-driven cataloging data cleaning and organization method as described in any one of claims 1-9, characterized in that, The system includes: The data access verification module is used to access multi-source heterogeneous cataloged data, perform basic verification on multi-source heterogeneous encoded data, and generate a structured dataset to be processed. The basic data cleaning module is used to perform basic cleaning on the dataset to be processed, so as to obtain a text dataset with a uniform format. The semantic encoding module is used to calculate the semantic complexity score of each cataloged text in the text dataset, match the embedding model based on the semantic complexity score distribution of all cataloged texts in the text dataset, perform semantic encoding on all text data in the text dataset based on the matched embedding model, and output a high-dimensional feature vector of the same dimension. The clustering and grouping module is used to cluster and group cataloging data based on the set of high-dimensional feature vectors of the same dimension, according to the number of vectors in the set of high-dimensional feature vectors of the same dimension and the distribution pattern of the vectors in space, and to aggregate synonymous heterogeneous cataloging data that refer to the same entity into the same data cluster. The standardization generation module is used to generate standardized names for each data cluster using a large language model; and to combine each data cluster and its corresponding standardized name into standardized catalog data entries, and then aggregate them to generate a standardized catalog dataset.
