A method and system for large-scale item classification

Abstract

This application discloses a method and system for large-scale item classification, relating to the field of intelligent supervision. The method includes: extracting visual feature vectors, semantic embedding vectors, and attribute field vectors from an original item set and concatenating and fusing them to obtain a multimodal vector, then reducing its dimensionality to obtain a low-dimensional latent vector; clustering the low-dimensional latent vector to obtain hierarchical labels for each item; constructing a baseline name library; and calculating the structured attribute feature set corresponding to the baseline model. F p and the structured attribute feature set of clusters F c The method involves determining the overlap rate to identify the category model that matches the cluster; generating candidate names; checking if the candidate names already exist in the baseline name library; if they do, calculating a list of candidate names that are closest to the candidate name for selection; if they do not exist, directly using this candidate name to obtain the classification result for each item. This application enables structured classification and standardized naming of massive amounts of item data with low reliance on manual intervention.

Description

Technical Field

[0001] This application relates to the field of intelligent supervision, and in particular to a method and system for large-scale item classification. Background Technology

[0002] Equipment cataloging is a fundamental and prerequisite step in the construction and management of equipment data resources. Due to the low efficiency and limited scalability of traditional cataloging methods, as well as the inherent bottlenecks of existing methods such as manual processing, threshold matching, and supervised learning, the cataloging process suffers from problems such as efficiency bottlenecks, high misclassification rates, lagging updates to classification standards, inconsistent coding standards across systems, and poor generalization of automated methods. This leads to consequences such as delayed resource allocation, chaotic inventory information, soaring manual maintenance costs, and inability to trace audit responsibilities.

[0003] With the emergence of massive and heterogeneous equipment data in fields such as the Internet of Things, intelligent manufacturing, and defense logistics, traditional equipment cataloging methods, which mainly rely on manual operation, fixed rule matching, or supervised models, are increasingly facing efficiency bottlenecks and limited scalability. These methods are ill-suited to today's large-scale, heterogeneous, and multi-source equipment data environment. Manual classification requires professionals to compare item names and attributes one by one. Faced with massive amounts of diverse data from various sources and sudden emergency tasks, this is not only inefficient but also prone to errors and omissions. Related research indicates that in the medical device industry, manual cataloging methods involve more than 80% of the time spent searching for similar records and verifying standards, rather than actual data entry. Meanwhile, equipment cataloging work must strictly adhere to GBT 37936-2019 Information Classification and Coding Requirements, GBT37944-2019 Data Model Compilation Requirements, and GBT 37948-2019 Data Element Compilation Requirements. These standards set forth clear requirements for material classification coding, baseline name construction, and the compilation of data models and data elements. Strictly implementing multiple standards under manual rule-based methods further increases the complexity and difficulty of the work.

[0004] Matching methods based on preset rules also have obvious limitations—the rule set is difficult to cover all complex and ever-changing descriptions. Once a new, undefined expression or data type appears, rules must be added or adjusted, causing cataloging to lag behind data updates. In the classification of manufacturing parts, TF-IDF + cosine similarity can achieve an automatic classification accuracy of 85%, but the authors point out that it lacks support for new models and low-frequency terms, and the recall rate drops by more than 20% in long-tail categories. In the field of bioinformatics, the method proposed by PLOS ONE to minimize the false positive / false negative ratio threshold can provide an "optimal" threshold selection reference for any semantic similarity task. For data directory structure matching, Sheen's similarity flooding iterative algorithm can perform global alignment of pattern elements at the graph structure level, but it relies on manual threshold setting for text attribute similarity and is difficult to autonomously extend to new nodes.

[0005] Existing automated cataloging methods also have shortcomings: simple threshold matching algorithms lack dynamic adaptability and are difficult to adjust in real time as data changes; Joachims (1998) first proved that SVM can significantly outperform traditional KNN and Naive Bayes in text classification tasks, and is robust to high-dimensional sparse features, and is widely used in product title classification and document cataloging scenarios; Dalal and Zaveri (2011) reviewed automatic text classification technology, pointing out that supervised models have insufficient generalization ability in cold-start and long-tail categories, and need to be supplemented with incremental annotation and transfer learning strategies to cope with dynamic category expansion; recently, the hybrid neural network architecture of GCNN+BiLSTM+attention has surpassed the BERT baseline in short text classification, but still relies on large-scale high-quality labeled corpora, and lacks generalization efforts for unseen categories. At the same time, its "black box" decision-making process is difficult to explain and lacks transparent interpretability, which affects the credibility of the results.

[0006] While existing methods each have their advantages, the lack of a mature, systematic solution to support the classification and cataloging needs of "large-scale, dynamic item classification" necessitates a new, more scalable, intelligent, and interpretable method and system to meet the growing challenges of equipment data management. These issues collectively lead to a series of negative consequences during the cataloging process, including resource allocation delays, information redundancy and conflicts, frequent coding conflicts, soaring manual maintenance costs, and difficulty in assigning blame for errors, severely hindering the standardized management and efficient utilization of equipment data. To address this, this application proposes an automatic classification method system centered on "cluster analysis + multimodal semantic matching + attribute recommendation." This system can achieve structured classification and standardized naming of massive amounts of item data with low reliance on manual intervention, exhibiting strong generalization capabilities and good structural interpretability. It aims to build a highly adaptable intelligent system for future complex cataloging tasks. Summary of the Invention

[0007] The purpose of this application is to provide a method and system for large-scale item classification, which can achieve structured classification and standardized naming of massive item data with low reliance on manual labor. It has strong generalization ability and good structural interpretability, and aims to build a highly adaptable intelligent system for future complex cataloging tasks.

[0008] To achieve the above objectives, this application provides the following solution:

[0009] Firstly, this application provides a method for large-scale item classification, including:

[0010] Obtain the original item set; the original item set includes: images, text descriptions, and structured attribute fields;

[0011] Extract visual feature vectors, semantic embedding vectors, and attribute field vectors from the original set of items;

[0012] The visual feature vector, semantic embedding vector, and attribute field vector are concatenated and fused to obtain the multimodal vector of the item;

[0013] The multimodal vectors are dimensionality reduced to obtain low-dimensional latent vectors;

[0014] The low-dimensional latent vectors are subjected to first-level clustering, coarsely divided into... K (1) Several major clusters;

[0015] For each major cluster, perform secondary clustering again, further subdividing it into... K (2) Each item is assigned a hierarchical label by creating several smaller clusters; these larger and smaller clusters together form a cluster.

[0016] Construct a benchmark name library; the benchmark name library includes benchmark models and the structured attribute feature sets corresponding to the benchmark models. F p ;

[0017] Obtain the structured attribute feature set of the cluster. F c ;

[0018] Calculate the structured attribute feature set corresponding to the benchmark model. F p and the structured attribute feature set of clusters F c The overlap rate is used to determine the category model that matches the cluster; the category model includes: a reused model, an extended model, and a newly created model;

[0019] For clusters that require the creation of new models, candidate names are generated based on the cluster's characteristic content or domain terminology.

[0020] Check if the candidate name already exists in the baseline name library. If the candidate name already exists, it indicates a naming conflict. The system will prompt the name conflict and use edit distance or a thesaurus to calculate a list of candidate names that are closest to the candidate name for selection. If the candidate name does not exist, use this candidate name directly, assign it a new unique ID and register it in the library to obtain the classification result for each item.

[0021] Optionally, the method for classifying large-scale items further includes, after the last step:

[0022] The classification result of each item is represented as a tuple.

[0023] Optionally, the visual feature vector, semantic embedding vector, and attribute field vector are extracted from the original set of items using the following formula:

[0024] ;

[0025] in, For visual feature vectors, For semantic embedding vectors, For attribute field vectors, For image, For text description, For structured attribute fields, For visual feature space, For text semantic embedding space, Encode the structured attributes space.

[0026] Optionally, the visual feature vector, semantic embedding vector, and attribute field vector are concatenated and fused to obtain the multimodal vector of the item using the following formula:

[0027] ;

[0028] in, This is a vector concatenation symbol. For visual feature vectors, For semantic embedding vectors, For attribute field vectors, Let be the Euclidean space containing the three types of vectors. For the first i A high-dimensional multimodal representation vector of an item. The dimension of the visual subvector; The semantic sub-vector dimension; The dimension of the attribute sub-vector; It is the sum of the dimensions of the three types of sub-vectors.

[0029] Optionally, dimensionality reduction of the multimodal vector to obtain a low-dimensional latent vector is achieved by introducing an autoencoder to reduce the dimensionality of the multimodal vector, as shown in the following formula:

[0030] coding: ;

[0031] decoding: ;

[0032] loss: ;

[0033] in, It is a low-dimensional latent vector; The encoder network Enc (Encoder) is represented by parameters. For control, input samples Output a latent space vector , indicating a compressed representation of the sample; The latent mean vector output by the encoder; For the first i A high-dimensional multimodal representation vector of an item; The standard deviation vector representing the latent distribution of the encoder output is The scale of uncertainty mapped onto the latent space; The noise vector follows a standard normal distribution; This represents a random vector sampled from a standard normal distribution; Let be the representation vector reconstructed by the decoder. If the result is the same as... The closer they are, the more it proves... The more complete the information; The decoder network is represented by parameters. control, This is an abbreviation for decoder function; and These represent the learnable parameters of the encoder and decoder, respectively. The total loss function of the variational autoencoder; N This represents the total number of item samples in the batch processing. i For sample index number; The square of the Euclidean distance; This is the balance coefficient; The Kullback–Leibler divergence; It is an approximate posterior distribution; This is the prior distribution.

[0034] Optionally, the low-dimensional latent vectors are subjected to first-level clustering, coarsely divided into... K(1) The specific formulas for each major cluster are as follows:

[0035] ;

[0036] in, The number of major categories is preset; For the first j A set of sample indexes for each major category; For low-dimensional latent vectors, As a major category The center of mass; The square of the Euclidean distance;

[0037] For each major cluster, perform secondary clustering again, further subdividing it into... K (2) The hierarchical labels for each item are obtained from the sub-clusters using the following formula:

[0038] ;

[0039] in, Pre-determine the number of subcategories to be subdivided within each major category; For small cluster sample sets, k For the sub-category index number, For the first Among the major categories, the first The centroid of a small cluster;

[0040] The hierarchical label for each item is: ,in, For the first i The category label assigned to each item in the first-level cluster; These are the subdivision labels for the same item within a secondary cluster.

[0041] Optionally, for clusters that require a new model, the following formula is used to generate candidate names based on the cluster's features or domain terminology:

[0042] ;

[0043] in, Cosine similarity; It serves as a secondary cluster center; Name vector; Candidate names; To select the item with the highest similarity among all candidate models, arg This indicates the value of the variable that makes a certain expression take its maximum (or minimum) value.

[0044] Optionally, the structured attribute feature set corresponding to the benchmark model is calculated.F p and the structured attribute feature set of clusters F c The overlap rate is used to determine the category model that matches the cluster, specifically using the following formula:

[0045] ;

[0046] in, for F p and F c The overlap rate; when When, a reusable model is recommended; when When, it is recommended to extend the model; when When this is the case, it is recommended to create a new model.

[0047] Optionally, the classification result of each item can be represented as a tuple as follows:

[0048] ;

[0049] in, It is an item identifier. For the first i The category label assigned to each item in the first-level cluster; These are the detailed labels for the same item within a secondary cluster. The final determined reference name, This indicates the matching status of the category with the baseline model, where the matching status includes reuse, extension, or creation. This refers to a referenced or newly created attribute model; if newly created, it is the new model ID. This is the overlap rate calculated during matching.

[0050] Secondly, this application provides a large-scale item classification system, which includes:

[0051] The original item set acquisition module is used to acquire the original item set; the original item set includes: images, text descriptions, and structured attribute fields;

[0052] The vector extraction module is used to extract visual feature vectors, semantic embedding vectors, and attribute field vectors from the original set of items.

[0053] The splicing and fusion module is used to splice and fuse the visual feature vector, semantic embedding vector, and attribute field vector to obtain the multimodal vector of the item;

[0054] The dimensionality reduction module is used to reduce the dimensionality of the multimodal vector to obtain a low-dimensional latent vector;

[0055] The first-level clustering module is used to perform first-level clustering on the low-dimensional latent vectors, roughly dividing them into... K (1) Several major clusters;

[0056] The secondary clustering module is used to perform secondary clustering on each major cluster, further subdividing it into... K (2) Each item is assigned a hierarchical label by creating several smaller clusters; these larger and smaller clusters together form a cluster.

[0057] The benchmark name library construction module is used to construct a benchmark name library; the benchmark name library includes benchmark models and structured attribute feature sets corresponding to the benchmark models. F p ;

[0058] The cluster structured attribute feature set acquisition module is used to acquire the structured attribute feature set of the cluster. F c ;

[0059] The overlap rate calculation module is used to calculate the structured attribute feature set corresponding to the benchmark model. F p and the structured attribute feature set of clusters F c The overlap rate is used to determine the category model that matches the cluster; the category model includes: a reused model, an extended model, and a newly created model;

[0060] The candidate name determination module is used to generate candidate names for clusters that need to be newly built, based on the characteristics or domain terms of the clusters.

[0061] The retrieval module is used to check whether the candidate name already exists in the baseline name database. If the candidate name already exists, it indicates a naming conflict. The system will prompt the name conflict and use the edit distance or the thesaurus to calculate a list of candidate names that are closest to the candidate name for selection. If the candidate name does not exist, the candidate name is used directly, a new unique ID is assigned to it and it is registered in the database to obtain the classification result for each item.

[0062] According to the specific embodiments provided in this application, this application has the following technical effects:

[0063] This application provides a method and system for large-scale item classification. First, it addresses the problems of high subjectivity and misclassification rate in manual classification by employing two-level clustering combined with cosine semantic matching and field overlap rate judgment to provide quantification for each category. and First, this application addresses several key issues: 1) Improving overall classification accuracy; 2) Solving the problem of low efficiency in manual processing, enabling end-to-end batch processing of ≥10,000 items with only one offline pipeline run, increasing classification speed by tens of times compared to traditional manual entry; 3) Addressing the problem of lagging classification and coding standard updates by automatically connecting old and new standards through a three-state model of "reuse / expansion / new creation," while introducing name uniqueness verification to ensure cross-system coding consistency; 4) Solving the problem of poor generalization ability and difficulty in identifying new categories in existing methods by using unsupervised clustering to discover potential new clusters first, supplemented by automatic generation of attribute drafts, achieving dynamic expansion capabilities for unknown items; 5) Solving the problem of lack of interpretability in black-box models by outputting cluster centers, similarity heatmaps, and field difference lists, forming an auditable and traceable explanation chain, enhancing the credibility of the results. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A flowchart illustrating a method for large-scale item classification provided in an embodiment of this application;

[0066] Figure 2 A flowchart of a method for large-scale item classification provided in one embodiment of this application;

[0067] Figure 3 This is a structural diagram of a large-scale item classification system provided in one embodiment of this application. Detailed Implementation

[0068] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] In one exemplary embodiment, such as Figure 1 and Figure 2 As shown, a classification method for large-scale items is provided. See [link / reference] Figure 2 The system starts with input multimodal item data, and through feature processing, dimensionality reduction, and cluster analysis, forms preliminary categories. Based on this, it compares each cluster with a baseline category model library to determine if it corresponds to an existing standard category model. For clusters that match existing categories with high similarity, the system directly adopts the corresponding baseline name and reuses its attribute model; if only partially matched, it triggers a "suggested expansion" process: the system recommends adjusting the existing model or adding new attribute fields to improve the category definition; for clusters that do not match any existing categories, the system determines that a new category is needed and enters the "new model generation" process, including automatically extracting common attributes of clusters to construct candidate attribute models and recommending a name for the new category. Next, the system performs uniqueness verification on the new name and handles naming conflicts (if duplicates exist, alternative names are suggested). Finally, the system outputs a result file or data table containing information about the category to which each item belongs and its baseline name / model, and updates the internal knowledge base to support subsequent classification tasks. The entire process achieves automatic classification conversion from data to information, ensuring the scalability and standardization of the classification system while reducing manual intervention.

[0071] See Figure 1 The method for classifying large-scale items in this application includes steps 201 to 208. Wherein:

[0072] Step 101: Obtain the original item set; the original item set includes: images, text descriptions, and structured attribute fields.

[0073] Let the original set of items be:

[0074] ;

[0075] in, For item serial number, the range of values ​​is: each Corresponding to one original item record, For image, For text description, This is a structured attribute field.

[0076] Then, the raw item data is collected and cleaned. Specifically, the input consists of raw item records containing images, text descriptions, and structured attributes, and the output is a standardized data format. For example, images are normalized in size and noise is removed, text is segmented, stop words are removed, and encoding is performed, and missing structured fields are filled in or marked, etc., to ensure the quality of input data for subsequent processing.

[0077] Step 102: Extract visual feature vectors, semantic embedding vectors, and attribute field vectors from the original set of items.

[0078] Three types of deep network models are used to extract image, text, and attribute features, respectively. The feature vectors are represented as follows:

[0079] ;

[0080] in, Visual feature vectors: generated by a visual Transformer or CNN on an object image. Extracting local texture, edge orientation, or higher-order appearance semantics from a single-dimensional response image; Semantic embedding vectors: Text descriptions derived from contrastive distillation-BERT Encoding is used to obtain geometric distance, which characterizes the similarity of texts in the semantic space; For attribute field vectors: structured attributes The numerical fields are normalized, and the categorical fields are one-hot / embedded and then concatenated. The dimensions correspond to the scales of real physical or engineering properties. Although the Euclidean space in which the three types of vectors reside is denoted as... However, they are completely different in terms of feature source, physical interpretation, and scale distribution. Subsequently, through splicing and VAE compression, these three signals are fused into a unified "multimodal representation".

[0081] Step 103: Concatenate and fuse the visual feature vector, semantic embedding vector, and attribute field vector to obtain the multimodal vector of the item.

[0082] After obtaining three complementary and overlapping vectors—image, text, and attributes—they are directly concatenated along the column direction to generate a multimodal representation of the item. This refers to an "integrated vector tag" that can simultaneously depict the appearance, textual semantics, and engineering attributes of an object, expressed as:

[0083] ;

[0084] in, This is a vector concatenation symbol that connects three sub-vectors sequentially in the column direction. It is a multimodal representation vector that carries an integrated label containing visual, semantic, and attribute information, which is used for subsequent dimensionality reduction, clustering, and semantic retrieval. The visual sub-vector dimension is determined by the visual network architecture and feature extraction layer. The semantic subvector dimension is determined by the size of the hidden layer. The dimension of the attribute sub-vector is determined by the number of attribute fields and their encoding method. The sum of the dimensions of the three types of sub-vectors is the sum of the dimensions, which are compressed to the latent dimension during dimensionality reduction encoding. .

[0085] Step 104: Reduce the dimension of the multi-modal vector to obtain a low-dimensional latent vector.

[0086] Multi-modal representation The dimension can reach up to hundreds, which is not conducive to calculation and storage. Therefore, a variational autoencoder is introduced , whose purpose is to compress the high-dimensional multi-modal representation into the latent space (k << d), so that has good properties of "continuous, interpolable, and probabilistically measurable". While ensuring information recoverability, learn the non-linear mapping from high dimension to low dimension, and its training objective is:

[0087] Encoding: ;

[0088] Decoding: ;

[0089] Loss: ;

[0090] where is the low-dimensional latent representation, which is used as the clustering basis in the follow-up; represents that the encoder network Enc (Encoder) takes the parameter as the control, inputs the sample , and outputs a latent space vector , representing the compressed representation of the sample; is the latent mean vector output by the encoder; is the i high-dimensional multi-modal representation vector of the item; represents the standard deviation vector of the latent distribution output by the encoder, which is [[ID=5​​​​​​​​​​​​​​​​​​​​​​Uncertainty; The prior distribution is used to ensure consistent sampling of the latent space; Kullback–Leibler divergence is used to measure the difference between two distributions; This is the balance coefficient.

[0091] Step 105: Perform first-level clustering on the low-dimensional latent vectors, coarsely dividing them into... K (1) Large clusters.

[0092] First-level clustering (major clusters): Using an improved K-means clustering method to group all potential vectors Roughly divided into Major categories Examples include: electrical appliances, machinery, and textiles. The optimization objective is:

[0093] ;

[0094] in, The number of major categories is preset; Let be the set of sample indices for the j-th major class, where all major classes are pairwise disjoint and their union is the entire set; As a major category The center of mass; It is the squared Euclidean distance, used to measure the density of a sample to the centroid.

[0095] The explanation for the condition that all major categories intersect each other and their union forms the universal set is as follows:

[0096] In the latent space, clustering multimodal vectors can form several large classes (clusters). Let... To include all The sample space of each sample. The clustering results will Divided into Each of the following major categories corresponds to a sample subset (Right now ), For the first j The sample index set of each major category, formally, the above subset satisfies the following two conditions:

[0097] Pairwise disjoint: for any They all In other words, no single sample belongs to two different subsets simultaneously.

[0098] Union of sets forms the complete set: In other words, the elements of these subsets, when added together, exactly cover the entire set. There are no missing samples.

[0099] Satisfying the above properties means that each sample falls exactly into a subset. In this context, the subsets are neither repeated in multiple subsets nor omitted from all subsets. Therefore, the subset set... This constitutes the complete sample set. A division.

[0100] For example, suppose the total sample set Divide it into Class, can be taken , , Obviously They do not interact with each other, and Therefore, each sample belongs to only one of the three subsets above, and all five samples are included in the partitioned subsets.

[0101] Step 106: Perform secondary clustering again for each major cluster, further subdividing it into... K (2) Each item is assigned a hierarchical label by creating a small cluster; the large and small clusters together form a cluster.

[0102] Secondary clustering (minor clusters): K-means is performed again within each major cluster, and... Subdivided into Subcategories The objective function is the same as above, but constrained to local fine-grained clusters. The optimization objective is:

[0103] ;

[0104] in, The number of subcategories to be subdivided within each major category is preset. For example, if "Major Category = Charger" is further divided into 5 subcategories, then... =5; For small cluster sample sets, k For the sub-category index number, For the first Among the major categories, the first Centroid (mean vector) of each small cluster.

[0105] The hierarchical label for each item is: ,in, For the first i The category label assigned to each item in the first-level cluster; Subdivision labels for the same item in secondary clusters

[0106] Therefore, each item is given a pair of hierarchical tags. This lays the structural foundation for subsequent name and attribute matching, and can effectively reduce the problem of long-tail categories being ignored.

[0107] Step 107: Construct a benchmark name library; the benchmark name library includes benchmark models and the structured attribute feature sets corresponding to the benchmark models. F p .

[0108] The system pre-configures a set of baseline category names that have been manually approved as a standard name library. Each It uniquely corresponds to one industry or national standard category. This indicates the capacity of the name database, which is the total number of registered base names.

[0109] Step 108: Obtain the structured attribute feature set of the cluster. F c .

[0110] Step 109: Calculate the structured attribute feature set corresponding to the baseline model. F p and the structured attribute feature set of clusters F c The overlap rate is used to determine the category model that matches the cluster; the category model includes: a reused model, an extended model, and a newly created model.

[0111] Step 110: For clusters that require a new model, generate candidate names based on the cluster's features or domain terminology.

[0112] Indicates name The semantic embedding vector aims to embed the base name (A short text / term) is mapped to a k-dimensional real vector so that it can represent the semantics of the name in the vector space and can be associated with cluster centers. Perform a similarity comparison. Second-order clustering yields several subclasses, each denoted as . For secondary cluster centers With name vector affected cosine similarity Find the largest one As a candidate benchmark name, the calculation process is represented as follows:

[0113] ;

[0114] like Then we directly use cosine similarity. The largest name As a subcategory The candidate baseline name is selected; otherwise, it is marked as "to be renamed". This step quickly aligns the unsupervised clustering results to the existing semantic system and promptly discovers potential new categories.

[0115] Step 111: Check whether the candidate name already exists in the baseline name library. If the candidate name already exists, it indicates a naming conflict. The system will prompt the name conflict and use the edit distance or the thesaurus to calculate a list of candidate names that are closest to the candidate name for selection. If the candidate name does not exist, this candidate name can be used directly, a new unique ID can be assigned to it and it can be registered in the library to obtain the classification result of each item.

[0116] Candidate benchmark name Calling the hash index: If a name is already occupied by another category in the standard name library, conflict resolution is triggered, and a list of alternative names is generated sequentially using edit distance and semantic distance. If no category occupies the name, a globally unique ID is immediately assigned, along with a batch of clustering labels and attribute model association records generated by the new name. Based on threshold rules, a decision is made to reuse, expand, or create a new model, ensuring that the attribute structure aligns with the standard system. For example, the process of finding the most similar old template for a new item: if it's more than 80% similar, the old template is used directly; if it's 60-80% similar, adjustments are made based on the old template; if it's less than 60% similar, new content must be revised and added to the template. (And the model library) Chinese model field set Calculate the overlap rate Represented as:

[0117] ;

[0118] when Reusable model ; Considered as an extended scenario, in Add fields to the existing ones; if It was determined that a new model needed to be created. This mechanism ensures that the attribute structure both inherits from history and covers new item characteristics.

[0119] Finally, the classification result for each item is represented as a tuple, specifically as follows:

[0120] ;

[0121] in, It serves as an item identifier, used to uniquely locate the item in the database; For the first i The category label assigned to each item in the first-level cluster; For the same item, further subdivision labels in the secondary cluster; The final benchmark name will be aligned with the industry standard database. The name-model matching result status indicates how the item (or its subcategory) handles the attribute model. Possible values ​​are "Reuse", "Extend", and "Create". The referenced attribute model ID: if match_status=reuse, it points to the existing model; if it is "extended", it points to the derived model; if it is "new", it is the new model number generated by the system for this cluster. The overlap rate, with values ​​[0,1], quantifies the degree of fit between the model and the actual data. All By using Spark to batch write data to the Parquet dataset and push it to the review end, a rapid closed loop of "algorithm clustering first, expert confirmation later" is achieved.

[0122] Based on the same inventive concept, this application also provides a large-scale item classification system for implementing the aforementioned method for large-scale item classification. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the large-scale item classification system provided below can be found in the limitations of the large-scale item classification method described above, and will not be repeated here.

[0123] In one exemplary embodiment, such as Figure 3 As shown, a large-scale item classification system is provided, including: an original item set acquisition module, a vector extraction module, a splicing and fusion module, a dimensionality reduction module, a first-level clustering module, a second-level clustering module, a benchmark name database construction module, a structured attribute feature set acquisition module for clusters, an overlap rate calculation module, a candidate name determination module, and a retrieval module;

[0124] The original item set acquisition module is used to acquire the original item set, which includes images, text descriptions, and structured attribute fields. It is also used for collecting and cleaning the original item data. The input is the original item record containing images, text descriptions, and structured attributes, and the output is a standardized data format. For example, it performs size normalization and noise removal on images, word segmentation, stop word removal, and encoding on text, and imputation or marking of missing structured fields to ensure the quality of input data for subsequent processing.

[0125] The vector extraction module extracts visual feature vectors, semantic embedding vectors, and attribute field vectors from the original item set. This module includes image feature extraction submodules, text feature extraction submodules, and structured attribute encoding submodules. The input is a preprocessed image. Text description and attribute fields The output is the corresponding image feature vector. Text feature vectors and attribute feature vector The output of this module can be used as the result of a pre-trained model.

[0126] The splicing and fusion module is used to splice and fuse the visual feature vector, semantic embedding vector, and attribute field vector to obtain the multimodal vector of the item. The input to this module is the image, text, and attribute feature vector of the same item. These vectors are spliced ​​and fused according to a specified dimension, and the output is the comprehensive feature vector of the item. ,in For example, let the total number of items be... , No. The image features of the item are Text features are Attribute characteristics are Then we have:

[0127] ;

[0128] In the above formula, " "" indicates a vector concatenation operation. After fusion, the output vector of this module contains visual, semantic, and attribute information about the items, serving as a representation space for dimensionality reduction and clustering.

[0129] The dimensionality reduction module is used to reduce the dimensionality of the multimodal vectors to obtain low-dimensional latent vectors. The input to this module is the fused feature vector. The output is a low-dimensional, compact representation. A self-encoder contains an encoder. and decoder Two parts, through unsupervised learning Mapped to The encoder aims to reconstruct the original input in a low-dimensional latent space as much as possible. Let the encoder parameters be... The encoding process is as follows: Decoding and reconstructing into The training objective of an autoencoder is to minimize the reconstruction error:

[0130] ;

[0131] By minimizing the loss function described above, the model can learn a low-dimensional representation of multimodal data. This effectively removes redundant information. In practical applications, an appropriate dimensionality reduction dimension can be selected based on experience. (For example, the original) Dimensional compression to The dimensionality reduction encoding module outputs a low-dimensional representation of each item to balance information fidelity and computational cost. These representations will be used for cluster analysis.

[0132] The first-level clustering module is used to perform first-level clustering on the low-dimensional latent vectors, roughly dividing them into... K (1) Large clusters.

[0133] The secondary clustering module is used to perform secondary clustering on each major cluster, further subdividing it into... K (2) Each item is assigned a hierarchical label by creating a small cluster; the large and small clusters together form a cluster.

[0134] The aforementioned first-level and second-level clustering modules are used for unsupervised clustering analysis of the dimensionality-reduced item representations, enabling automatic classification of similar items. Specifically, a two-level clustering architecture is adopted: first, "major category" clustering is performed, and then the subcategories are further subdivided. The input is the set of representation vectors for all items. The output is the major category and minor category to which each item belongs. First, the first-level clustering (major category) stage uses... -means algorithm will all The vector partitioning of the items is as follows Initial clusters; in the subsequent secondary clustering (minor cluster) stage, samples within each major cluster are further processed using... -means is further subdivided into several sub-clusters. The output of the clustering module includes the clustering results for each item: category label. and subcategories The output of this module It can be further refined into a number under its major category.

[0135] The benchmark name library construction module is used to construct a benchmark name library; the benchmark name library includes benchmark models and the structured attribute feature sets corresponding to the benchmark models. F p .

[0136] The cluster structured attribute feature set acquisition module is used to acquire the structured attribute feature set of the cluster. F c .

[0137] The overlap rate calculation module is used to calculate the structured attribute feature set corresponding to the baseline model. F p and the structured attribute feature set of clusters F c The overlap rate is used to determine the category model that matches the cluster; the category model includes: a reused model, an extended model, and a newly created model.

[0138] Specifically, the first step is to obtain the structured attribute feature set of the clusters. and compared with the pre-existing baseline attribute model library of the system. Comparisons were performed on each baseline model. A set of structured fields is predefined. This corresponds to the attribute template of a certain standard item category. The overlap rate between the cluster attribute set and the model attribute set is calculated. This measures the similarity between the cluster and each known model. Then, based on... The value is determined using a set threshold rule: when there is... When designing a model, if the current cluster is considered to closely match an existing category model, the matching model is output. This means that existing models are recommended for reuse; when the highest overlap rate falls within... When dealing with intervals, it is assumed that the cluster features partially match the existing model, and the output is the closest model. And the difference fields; it is recommended to modify and extend the model based on these. If a cluster fails to match a suitable existing model, a new model needs to be created to define a new category for the cluster. In the scenario of creating a new model, the system will automatically count the most common important attribute fields in the items of the cluster to form a preliminary attribute structure draft for subsequent manual confirmation and improvement.

[0139] The candidate name determination module generates candidate names for clusters requiring a new model, based on the cluster's features or domain terminology. The module takes the clusters to be named and their matching status as input, and outputs a recommended name and a unique identifier for that cluster. For clusters that directly match an existing model, the module inherits that model's standard name. As a category name; for a category cluster for which a new model needs to be created, the system will generate a candidate name based on the cluster's features or domain terminology. For example, based on the most frequent keywords in the cluster, or temporarily named "New Benchmark," the system then searches the benchmark name database for this candidate name: if the name... If the name already exists, it indicates a naming conflict. The system will prompt a name conflict and calculate the result using edit distance or a thesaurus. A list of similar candidate names is available for selection; if the name If the name does not exist, it can be used directly, assigned a new unique ID, and registered in the database. This mechanism ensures the standardization and uniqueness of newly added category names, avoiding confusion with existing categories.

[0140] The retrieval module is used to check whether the candidate name already exists in the baseline name database. If the candidate name already exists, it indicates a naming conflict. The system will prompt the name conflict and use the edit distance or the thesaurus to calculate a list of candidate names that are closest to the candidate name for selection. If the candidate name does not exist, the candidate name can be used directly, a new unique ID can be assigned to it and it can be registered in the database to obtain the classification result for each item.

[0141] The following is a clustering case study based on the medical supplies field, clearly demonstrating how the system can complete a one-time large-scale item classification task through clustering:

[0142] This case study focuses on the medical supplies industry and addresses the standardization needs of medical devices and their components by constructing a multimodal clustering classification system capable of processing over 10,000 records per batch. The system's core objective is to resolve issues such as the large number of medical item categories, inconsistent naming conventions, and high structural similarity with ambiguous definitions, thereby achieving standardized classification, baseline name archiving, and attribute model generation.

[0143] The data source comprises 12,000 samples of medical components, covering several typical products such as syringes, breathing masks, infusion sets, and surgical forceps. Each sample contains three dimensions: image (actual product photo), text (product name and description), and structured parameter fields (such as material, model, caliber, connector standard, etc.). The system first uses ResNet to extract 256-dimensional visual embeddings from the images, extracts 768-dimensional semantic vectors from the text using the BERT model, and normalizes the attribute fields to form 64-dimensional structured field vectors. These three are concatenated into a unified 1088-dimensional multimodal feature vector for subsequent clustering and classification tasks.

[0144] The system employs an autoencoder to compress and reduce the dimensionality of multimodal features to a 64-dimensional latent space, and then uses a two-level KMeans clustering algorithm for grouping. The first-level clustering divides items into 15 major categories, such as "respiratory equipment," "injection components," and "surgical aids." The second-level clustering further subdivides each major category into smaller categories, resulting in a total of 80 sub-clusters. The center vector of each sub-category represents the semantic features of typical items in that category, serving as a key reference for subsequent name and model matching.

[0145] In the baseline name recommendation phase, the system introduces a standard name knowledge base containing existing names, aliases, and their embedded representations. By calculating the semantic cosine similarity between the center vector of each sub-cluster and the baseline name vector, the system can determine whether there are highly similar matches. For matches with a similarity higher than 0.85, the system suggests using the original name; for matches below the threshold, the system automatically recommends three highly similar reference names for manual selection and generates a suggested new baseline name and a unique code. This mechanism effectively solves the problem of inconsistent and non-standardized names and expressions for medical products across multiple scenarios.

[0146] Simultaneously, the system analyzes the common structured fields within each cluster (such as ventilation resistance, interface diameter, and airtightness level) and compares them with existing attribute models. If the overlap rate of structural fields exceeds 85%, the system suggests reusing the existing model; if it falls between 65% and 85%, the system suggests adding or modifying fields based on a template; if the overlap rate is low, the system automatically summarizes the fields and outputs a draft structure for "suggesting a new model." This ensures both structural inheritance and supports the scalability of new models.

[0147] Finally, the system integrates the classification suggestions, baseline name recommendations, and attribute model comparison results for each cluster into a "review package," which is then submitted to the human review platform in a structured format. Reviewers can confirm or correct naming suggestions, structural fields, and coding conflicts. The corrections are written to the official database and used for subsequent model optimization. By periodically feeding human feedback back into the training data, the system achieves adaptive optimization capabilities, ensuring that the model continuously evolves as the data scale grows, adapting to the ever-expanding medical category system.

[0148] Database table structure design (MySQL)

[0149] Table 1 Category Table – Classification Hierarchy (Major Category / Subcategory)

[0150]

[0151] Table 2 BenchmarkName Table – Benchmark Name and Unique Code

[0152]

[0153] Table 3 Alias ​​Table – Name Alias ​​Management

[0154]

[0155] Table 4 AttributeModel Table – Attribute Model Structure

[0156]

[0157] Table 5 ClusterSuggestion Table – Clustering Results and Suggestion Records

[0158]

[0159] Recommendation system evaluation metrics:

[0160] In addition to commonly used metrics such as precision, recall, F1 score, and human review acceptance rate, a comprehensive evaluation should be conducted from multiple dimensions:

[0161] 1. Multi-objective ranking quality: NDCG (Normalized Discount Cumulative Gain) and MAP (Mean Precision) are used to measure the reasonableness and accuracy of the ranking of the recommended results in the list.

[0162] 2. Coverage: Recommendation coverage is defined as the proportion of recommended items to the total number of items. High coverage means that the recommendation system can serve a wider range of users, reducing the number of users who feel that the system is not responding to their needs, thereby improving user satisfaction and the overall utility of the system. This metric reflects the system's ability to expose long-tail products.

[0163] 3. Diversity: This measures the degree to which the recommended list involves different categories or topics. In recommendation systems, similar content clustering, overly boring content, and a narrow range of information for users can easily lead to "information cocoons," poor user experience due to broad or ambiguous interests, and the inability to generate recommendations quickly. High diversity helps uncover users' potential interests and avoids monotonous recommendation results.

[0164] 4. Novelty: The recommendation system should focus on recommending item types that users have not yet encountered, stimulating their interest in exploration by expanding their cognitive boundaries. The system needs to dynamically optimize its recommendation strategy while maintaining accuracy, achieving a balance between novelty and precision. Simultaneously, novelty also requires the item cataloging system to have dynamic update capabilities, continuously optimizing the item cataloging within the system: regularly integrating and classifying low-frequency matching items; for newly updated items, generating cataloging baseline names or attribute models that conform to data standards and classification patterns; and continuously adjusting and optimizing its re-clustering algorithm.

[0165] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for classifying large-scale items, characterized in that, The methods for large-scale item classification include: Obtain the original item set; the original item set includes: images, text descriptions, and structured attribute fields; Extract visual feature vectors, semantic embedding vectors, and attribute field vectors from the original set of items; The visual feature vector, semantic embedding vector, and attribute field vector are concatenated and fused to obtain the multimodal vector of the item; The multimodal vectors are dimensionality reduced to obtain low-dimensional latent vectors; The low-dimensional latent vectors are subjected to first-level clustering, coarsely divided into... K (1) Several major clusters; For each major cluster, perform secondary clustering again, further subdividing it into... K (2) Each item is assigned a hierarchical label by creating several smaller clusters; these larger and smaller clusters together form a cluster. Construct a benchmark name library; the benchmark name library includes benchmark models and the structured attribute feature sets corresponding to the benchmark models. F p ; Obtain the structured attribute feature set of the cluster. F c ; Calculate the structured attribute feature set corresponding to the benchmark model. F p and the structured attribute feature set of clusters F c The overlap rate is used to determine the category model that matches the cluster; the category model includes: a reused model, an extended model, and a newly created model; For clusters that require the creation of new models, candidate names are generated based on the cluster's characteristic content or domain terminology. Check if the candidate name already exists in the baseline name library. If the candidate name already exists, it indicates a naming conflict. The system will prompt the name conflict and use edit distance or a thesaurus to calculate a list of candidate names that are closest to the candidate name for selection. If the candidate name does not exist, use this candidate name directly, assign it a new unique ID and register it in the library to obtain the classification result for each item.

2. The method for large-scale item classification according to claim 1, characterized in that, The method for classifying large-scale items also includes the following after the last step: The classification result of each item is represented as a tuple.

3. The method for large-scale item classification according to claim 1, characterized in that, The following formula is used to extract visual feature vectors, semantic embedding vectors, and attribute field vectors from the original set of items: ； in, For visual feature vectors, For semantic embedding vectors, For attribute field vectors, For image, For text description, For structured attribute fields, For visual feature space, For text semantic embedding space, Encode the structured attributes space.

4. The method for large-scale item classification according to claim 1, characterized in that, The visual feature vector, semantic embedding vector, and attribute field vector are concatenated and fused to obtain the multimodal vector of the item using the following formula: ； in, This is a vector concatenation symbol. For visual feature vectors, For semantic embedding vectors, For attribute field vectors, Let be the Euclidean space containing the three types of vectors. For the first i A high-dimensional multimodal representation vector of an item. For visual subvector dimensions; The semantic sub-vector dimension; The dimension of the attribute sub-vector; It is the sum of the dimensions of the three types of sub-vectors.

5. The method for large-scale item classification according to claim 1, characterized in that, To reduce the dimensionality of the multimodal vectors to obtain low-dimensional latent vectors, an autoencoder is introduced to perform dimensionality reduction on the multimodal vectors. The specific formula is as follows: coding: ; decoding: ; loss: ; in, It is a low-dimensional latent vector; The encoder network Enc is represented by parameters. For control, input samples Output a latent space vector , indicating a compressed representation of the sample; The latent mean vector output by the encoder; For the first i A high-dimensional multimodal representation vector of an item; The standard deviation vector representing the latent distribution of the encoder output is The scale of uncertainty mapped onto the latent space; The noise vector follows a standard normal distribution; This represents a random vector sampled from a standard normal distribution; Let be the representation vector reconstructed by the decoder. If the result is the same as... The closer they are, the more it proves... The more complete the information; The decoder network is represented by parameters. control; This is an abbreviation for decoder function; and These represent the learnable parameters of the encoder and decoder, respectively. The total loss function of the variational autoencoder; N This represents the total number of item samples in the batch processing. i For sample index number; The square of the Euclidean distance; This is the balance coefficient; The Kullback–Leibler divergence; It is an approximate posterior distribution; This is the prior distribution.

6. The method for large-scale item classification according to claim 1, characterized in that, The low-dimensional latent vectors are subjected to first-level clustering, coarsely divided into... K (1) The specific formulas for each major cluster are as follows: ； in, The number of major categories is preset; For the first j A set of sample indexes for each major category; For low-dimensional latent vectors, As a major category The center of mass; The square of the Euclidean distance; For each major cluster, perform secondary clustering again, further subdividing it into... K (2) The hierarchical labels for each item are obtained from the sub-clusters using the following formula: ； in, Pre-determine the number of subcategories to be subdivided within each major category; For small cluster sample sets, k For the sub-category index number, For the first Among the major categories, the first The centroid of a small cluster; The hierarchical label for each item is: ,in, For the first i The category label assigned to each item in the first-level cluster; These are the subdivision labels for the same item within a secondary cluster.

7. The method for large-scale item classification according to claim 1, characterized in that, For clusters that require a new model, the following formula is used to generate candidate names based on the cluster's features or domain terminology: ； in, Cosine similarity; It serves as a secondary cluster center; Name vector; Candidate names; To select the item with the highest similarity among all candidate models, arg This indicates the value of the variable that makes a certain expression take its maximum or minimum value.

8. The method for large-scale item classification according to claim 1, characterized in that, Calculate the structured attribute feature set corresponding to the benchmark model. F p and the structured attribute feature set of clusters F c The overlap rate is used to determine the category model that matches the cluster, specifically using the following formula: ； in, for F p and F c The overlap rate; when When, a reusable model is recommended; when When, it is recommended to extend the model; when When this is the case, it is recommended to create a new model.

9. The method for large-scale item classification according to claim 1, characterized in that, The classification result of each item is represented as a tuple as follows: ； in, It is an item identifier. For the first i The category label assigned to each item in the first-level cluster; These are the detailed labels for the same item within a secondary cluster. The final determined reference name, This indicates the matching status of the category with the baseline model, where the matching status includes reuse, extension, or creation. This refers to a referenced or newly created attribute model; if newly created, it is the new model ID. This is the overlap rate calculated during matching.

10. A large-scale item classification system, characterized in that, The large-scale item classification system includes: The original item set acquisition module is used to acquire the original item set; the original item set includes: images, text descriptions, and structured attribute fields; The vector extraction module is used to extract visual feature vectors, semantic embedding vectors, and attribute field vectors from the original set of items. The splicing and fusion module is used to splice and fuse the visual feature vector, semantic embedding vector, and attribute field vector to obtain the multimodal vector of the item; The dimensionality reduction module is used to reduce the dimensionality of the multimodal vector to obtain a low-dimensional latent vector; The first-level clustering module is used to perform first-level clustering on the low-dimensional latent vectors, roughly dividing them into... K (1) Several major clusters; The secondary clustering module is used to perform secondary clustering on each major cluster, further subdividing it into... K (2) Each item is assigned a hierarchical label by creating several smaller clusters; these larger and smaller clusters together form a cluster. The benchmark name library construction module is used to construct a benchmark name library; the benchmark name library includes benchmark models and structured attribute feature sets corresponding to the benchmark models. F p ; The cluster structured attribute feature set acquisition module is used to acquire the structured attribute feature set of the cluster. F c ; The overlap rate calculation module is used to calculate the structured attribute feature set corresponding to the benchmark model. F p and the structured attribute feature set of clusters F c The overlap rate is used to determine the category model that matches the cluster; the category model includes: a reused model, an extended model, and a newly created model; The candidate name determination module is used to generate candidate names for clusters that need to be newly built, based on the characteristics or domain terms of the clusters. The retrieval module is used to check whether the candidate name already exists in the baseline name database. If the candidate name already exists, it indicates a naming conflict. The system will prompt the name conflict and use the edit distance or the thesaurus to calculate a list of candidate names that are closest to the candidate name for selection. If the candidate name does not exist, the candidate name is used directly, a new unique ID is assigned to it and it is registered in the database to obtain the classification result for each item.

Images