Industrial material hierarchical classification method fusing retrieval enhancement and explicit hierarchical constraint
By constructing a standard material vector knowledge base, data balancing, and enterprise standard classification prefix trees, combined with large language model fine-tuning and Logits masking, the problems of large label space and long-tail distribution in industrial material classification are solved, achieving accurate and reliable enterprise standard classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUZHOU TONGDA HECHUANG INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2026-04-22
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies are unable to effectively solve the problems of large label space, significant long-tail distribution and lack of explicit hierarchical constraints in industrial material classification, resulting in inaccurate classification results and failure to meet enterprise needs.
The method integrates retrieval enhancement and explicit hierarchical constraints. By constructing a standard material vector knowledge base, a data balancing strategy, and a prefix tree of the enterprise standard classification system, combined with fine-tuning of a large language model and Logits probability masking, it ensures that the classification results conform to enterprise standards.
It improves the accuracy and reliability of industrial material classification, solves the problems of long-tail distribution and hierarchical constraints, and ensures that the classification results meet enterprise standards.
Smart Images
Figure CN122432767A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of industrial material classification, and particularly relates to a hierarchical classification method for industrial materials that integrates enhanced retrieval and explicit hierarchical constraints. Background Technology
[0002] As global manufacturing accelerates its digitalization and intelligentization, industrial material data has become a core asset for enterprise production scheduling, supply chain collaboration, and cost control. Industrial material classification is a core link in manufacturing enterprises' data governance and production scheduling, with the goal of mapping non-standardized, colloquial material descriptions to the enterprise's material coding system. For a long time, enterprises have mainly relied on manual experience or rule-based systems for classification, which is not only inefficient and costly to maintain, but also highly susceptible to errors in material data classification due to differences in human perception, thus hindering the level of refined management.
[0003] Traditional machine learning methods, such as Support Vector Machines (SVMs) and TextCNN, have been applied to material classification tasks. However, industrial material descriptions are generally characterized by short texts, sparse semantic features, and specialized, low-frequency naming conventions. Furthermore, industrial classification systems have strict hierarchical constraints, making it difficult for traditional methods to capture fine-grained semantic relationships and deep dependencies. Additionally, hierarchical classification methods based on graph neural networks, such as HiAGM and LCGCN, improve classification performance by explicitly modeling parent-child dependencies between labels. However, these methods heavily rely on the statistical features of training samples. When faced with the significant long-tail distribution phenomenon in industrial materials, the sparse sample size of tail categories leads to severely insufficient generalization ability and poor interpretability. Currently, large language models are emerging, providing a new path for industrial material classification. However, general-purpose large language models lack domain-specific knowledge, making them prone to cognitive blind spots. Moreover, their inherently probabilistic prediction mechanisms are susceptible to generating semantically reasonable but non-standard coding schemes, resulting in misclassification of industrial materials. Summary of the Invention
[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a hierarchical classification method for industrial materials that integrates enhanced retrieval with explicit hierarchical constraints. This method solves the problems that existing hierarchical text classification methods struggle to address the issues of long-tail semantic understanding, inaccurate classification results, and inability to meet the industrial material classification needs of enterprises due to the large label space, significant long-tail distribution, and lack of explicit hierarchical constraints in industrial material classification.
[0005] To achieve the above objectives, the technical solution adopted by this invention is: a hierarchical classification method for industrial materials that integrates enhanced retrieval and explicit hierarchical constraints, comprising the following steps: Historical industrial material data is acquired, vectorized by embedding an encoder to obtain dense vectors, and a standard material vector knowledge base is constructed. Based on the industrial material to be classified, the embedded encoder is used to calculate the query vector, and a standard material sample is retrieved from the standard material vector knowledge base based on similarity. The standard material sample is then concatenated with the industrial material to be classified to obtain the composite input text. Historical industrial material classification samples were obtained. Based on the data balancing strategy of random undersampling of head categories and generative augmentation of tail categories, a fine-tuned dataset was constructed, and the large language model was fine-tuned and trained to obtain an industrial material classification model. Based on the enterprise standard classification system, a legal word prefix tree is obtained through serialization and tree mapping. Then, a legal next-hop word set is constructed based on each node of the legal word prefix tree. The composite input text is fed into the industrial material classification model. By using the Logits probability mask based on the set of legal next-hop words, the industrial material classification model is constrained to sample only from legal words, thus obtaining industrial material classification results that conform to the enterprise's standard classification system.
[0006] In real-world industrial scenarios, industrial material classification primarily relies on human experience or rule-based systems. However, industrial materials are characterized by a long-tail distribution and semantic ambiguity, leading to low efficiency and potential classification errors due to differences in human cognition. This invention addresses this by establishing a standard material vector knowledge base, transforming unstructured historical industrial material data into a real-time searchable structured knowledge base. This not only improves retrieval speed but also preserves the hierarchical tags and attributes of industrial materials, laying a solid data foundation for solving the problem of knowledge gaps caused by long-tail distribution.
[0007] While traditional machine learning methods have been applied to text classification tasks, including industrial material classification, industrial materials are characterized by short texts with sparse features and deep classification systems, making it difficult to capture complex semantic relationships and fine-grained features. This results in poor generalization ability and easy classification errors. To address this, this invention uses a standard material vector knowledge base as external prior knowledge and enhances retrieval through an embedded encoder. This external prior knowledge is introduced to alleviate classification errors caused by sparse samples. Subsequently, a data balancing strategy of random undersampling of head categories and generative enhancement of tail categories is used to alleviate the problem of low classification accuracy caused by long-tail distribution. Currently, large language models are used for industrial material classification. However, the industrial field has a large number of low-frequency, non-standard specialized terms and strict hierarchical constraints. Furthermore, general-purpose large language models lack specific enterprise-specific classification knowledge, leading to illusionary errors in the output results. This invention constructs a legal word prefix tree for classification based on the enterprise's standard classification system through serialization and tree mapping, and maintains a legal next-hop word set for each node. During the decoding stage, the Logits probability distribution output by the large language model is masked based on the legal next-hop word set, forcing sampling only from legal words, thereby eliminating possible illusions and outputting hierarchical classification results that conform to the enterprise's standard classification system.
[0008] Furthermore: the construction of the standard material vector knowledge base specifically includes: By acquiring historical industrial material data and filtering, eliminating, and standardizing material naming, several standard industrial material text descriptions and corresponding classification tags are obtained. Based on several standard industrial material text descriptions and corresponding classification tags, vectorization encoding is performed through an embedded encoder to map several dense vectors. The dense vector corresponding to each standard industrial material is stored as a data point in the vector index field, and the material information corresponding to the standard industrial material is encapsulated as metadata and stored in association with the vector index field to construct a standard material vector knowledge base.
[0009] The further beneficial effects mentioned above are as follows: Existing industrial material data is often stored in a scattered manner in the form of unstructured raw text, lacking a unified semantic representation, which makes it difficult for classification tasks to make full use of historical industrial material data; This invention maps historical industrial material data into dense vectors and uses a vector-associated metadata storage strategy to build a knowledge base, transforming unstructured historical materials into structured knowledge assets that can be retrieved in real time, improving semantic retrieval efficiency, and completely preserving the hierarchical tags and attribute information of industrial materials.
[0010] Further: obtaining the composite input text specifically includes: The system obtains the user's query input text based on the industrial materials to be classified, processes it using the same embedding encoder as that used to build the standard material vector knowledge base, and obtains the query vector. Calculate the similarity between the query vector and each dense vector in the standard material vector knowledge base, and extract several standard material samples with the highest similarity based on the similarity to obtain a set of similar industrial material samples; each standard material sample includes the standard material name and its corresponding complete hierarchical classification path; The query input text, system instructions, standard material names and their corresponding complete hierarchical classification paths are concatenated according to a preset template to obtain composite input text.
[0011] Furthermore, the expression for the composite input text is as follows:
[0012]
[0013]
[0014]
[0015] in, For compound input text, For system commands, For a set of similar industrial material samples, Enter the text for the query. This is the first standard material sample with the highest similarity. The first standard material name that has the highest similarity to the standard material sample. This is the complete hierarchical classification path for the first standard material sample with the highest similarity. The second sample is the standard material with the highest similarity. The second standard material name that has the highest similarity to the standard material sample. This is the complete hierarchical classification path for the second standard material sample with the highest similarity. For the first The most similar standard material sample For the first The name of the standard material that is most similar to the standard material sample. For the first A complete hierarchical classification path for the standard material sample with the highest similarity. To preset the maximum number of similarity searches, To match the query input text The most similar A standard material sample, For similarity calculation, For query vector, For the density vector of standard industrial materials, For the first Metadata corresponding to a standard material It is the Euclidean norm.
[0016] The further beneficial effects mentioned above are as follows: When directly inputting query text into a large language model for material classification, the accuracy is low due to the long-tail distribution characteristics of industrial physical data. This invention uses retrieval enhancement technology to dynamically recall semantically similar standardized material samples from a vector knowledge base as context examples, and concatenates them to obtain composite input text with enhanced context. This can make up for the semantic loss of long-tail samples, reduce the prediction entropy value on low resource categories, and alleviate the classification error problem caused by sample sparsity.
[0017] Furthermore: the data equalization strategy based on random undersampling of head categories and generative augmentation of tail categories specifically includes: Obtain historical industrial material classification samples and count the number of samples in each classification category within the historical industrial material classification samples; For classification categories where the number of samples exceeds a preset head count threshold, truncation is performed, and the same number of samples as the head count threshold are randomly retained to obtain the first sample. For classification categories where the number of samples is less than the preset head number threshold but greater than the preset tail number threshold, no processing is performed, and a second sample is obtained. For a classification category with a sample size less than a preset tail number threshold, a real sample of historical industrial material in that classification category is used as a seed example. Diverse synthetic data is obtained through a generative model until the sample size of that classification category is the same as the preset tail number threshold, at which point a third sample is obtained. The first, second, and third samples are converted into an instruction format suitable for supervised fine-tuning of large models to obtain a fine-tuning dataset, which includes task instructions, user input, and target output.
[0018] The further beneficial effects mentioned above are as follows: Industrial material data exhibits a significant long-tail distribution characteristic, with the head category occupying most of the samples and dominating the model training direction, while the tail category suffers from insufficient classification accuracy due to the sparse sample size, making it difficult for the model to learn effective classification boundaries. This invention employs random undersampling for truncation of the head category and generative augmentation for the tail category to expand the sample size. Through a data balancing strategy, the industrial material data is smoothed and the classification accuracy decline caused by oversampling is avoided, thereby improving the model's classification accuracy and generalization performance for industrial materials.
[0019] Furthermore: the fine-tuning and training of the large language model to obtain the industrial material classification model specifically includes: Based on the large language model, the parameters of all linear layers are fine-tuned using low-rank adaptation techniques, the original weights are frozen, and two trainable low-rank matrices are introduced to simulate incremental weight updates, thus obtaining a pre-trained model. Based on the pre-trained model, the Cosine learning rate decay strategy is adopted and repeated training is carried out in conjunction with a pre-set proportion of Warmup stage until the loss function based on causal language modeling is less than the set loss threshold, thus obtaining the industrial material classification model. The expression for the pre-trained model is as follows:
[0020] in, The output of the pre-trained model, For the original weights, Given the input sequence, To update the weights incrementally, This is the scaling factor. Let be the rank of the low-rank decomposition. and All are trainable low-rank matrices; The expression for the loss function based on causal language modeling is as follows:
[0021] in, For loss functions based on causal language modeling, For a moment, The length of the target sequence. To maximize a given input sequence Below, standard classification path The conditional generation probability, For the first in the standard classification path A true label for a moment For the first Context information from a specific point in time.
[0022] The further beneficial effects mentioned above are as follows: Existing technologies involving full-scale fine-tuning of large language models are not only computationally expensive but also easily damage the model's original general semantic capabilities. This invention employs LoRA technology to adapt the fully linear layers, efficiently injecting industrial domain classification logic while maintaining the model's general knowledge; by setting a higher rank (r=32) to enhance the domain knowledge memory capacity, and combined with cosine annealing and gradient accumulation strategies, it significantly improves the model's professional performance and training stability in vertical domains. Furthermore: the obtained set of legal next-hop lexical units specifically includes: Obtain the enterprise standard classification system, which includes a set of several legal hierarchical paths, and each legal hierarchical path includes several classification nodes; The word segmenter is used to segment several category nodes of each legal hierarchical path, and the resulting word sequence is obtained. Insert all word sequences into a prefix tree to obtain a legal word classification prefix tree. The nodes of the legal word classification prefix tree represent word states, the edges represent the adjacency relationships between words, the root node is the starting state, and the leaf nodes contain the ending symbol words representing the end of the path. For each node in the legal word prefix tree, maintain the corresponding set of legal next-hop words; The set of valid next-hop lexical elements includes lexical elements corresponding to all child nodes starting from the current node; for a leaf node representing the end of a complete classification path, its valid next-hop set only includes terminator lexical elements.
[0023] The further beneficial effects mentioned above are as follows: Given the inconsistent naming and different classification structures of industrial material data from various enterprises, a general large language model, lacking constraints, may output illusory content not present in the enterprise's standard classification system, leading to erroneous and non-compliant classification results. This invention transforms the enterprise's standard classification system into a prefix tree and constructs a set of legal next-hop lexical units for each node, converting the static encoding standard into a dynamic finite state automaton. This provides an executable structural foundation for explicit structural constraints during the decoding process, eliminates the generation of illegal lexical units, and ensures the legality of the hierarchical structure of industrial material classification.
[0024] Furthermore: the output of industrial material classification results conforming to the enterprise standard classification system specifically includes: Input the composite input text into the industrial material classification model, and initialize the state pointer to the root node of the legal word prefix tree at the initial decoding moment; The industrial material classification model outputs the original Logits vector at the current decoding time, and reads the set of legal next-hop tokens of the current node from the legal token classification prefix tree according to the state pointer at the current decoding time. Based on the original Logits vector and the set of legal next-hop words, a hard constraint mask vector with the same dimension as the original Logits vector is constructed. The hard constraint mask vector is superimposed element by element with the original Logits vector, and the constrained conditional probability distribution is calculated by the Softmax function. This makes the probability of illegal words 0, and only legal words participate in normalization. The word with the highest probability is selected as the word output at the current decoding time. Based on the word elements output at the current decoding moment, update the state pointer to point to the next node in the legal word element classification prefix tree; Repeat the decoding process until the state pointer reaches a leaf node, output the end symbol word, and obtain the industrial material classification result that conforms to the enterprise standard classification system.
[0025] The further beneficial effects mentioned above are as follows: The lack of structural constraints in the autoregressive generation of general large language models leads to the illusion of semantically reasonable but non-compliant classification systems; this invention tracks the current decoding position through a state pointer, dynamically reads the set of legal next-hop words and constructs a hard constraint mask vector, forcing the probability of illegal words to be 0, ensuring that the large language model only samples from legal words, eliminating illegal paths and illusion outputs from the root, and improving the structural consistency and reliability of classification results.
[0026] The beneficial effects of this invention are: This invention constructs a standardized material vector knowledge base and retrieves similar samples as contextual prior knowledge, which makes up for the semantic loss of long-tail samples and solves the problem of low classification accuracy caused by the sparse tail category samples in the existing technology. Compared with the existing methods, it enhances the ability of large language models to recognize tail categories. This invention truncates the head categories of industrial material data and enhances the generative representation of the tail categories, which not only smooths the peaks in the distribution of data but also expands the capacity of sparse samples, providing high-quality supervision signals for model fine-tuning and significantly improving the overall classification performance. Existing large-scale models are prone to generating illegal paths during free decoding. This invention transforms the classification system into a prefix tree, dynamically maintains the set of legal lexical units, and constructs a hard constraint mask, forcing each step to sample only from the legal space. This improves the accuracy of hierarchical classification while ensuring that the output results conform to the enterprise standard classification system. Attached Figure Description
[0027] Figure 1 A flowchart illustrating the hierarchical classification method for industrial materials that integrates enhanced retrieval with explicit hierarchical constraints. Detailed Implementation
[0028] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0029] Example 1 like Figure 1 The diagram shown illustrates the process of a hierarchical classification method for industrial materials that integrates enhanced retrieval with explicit hierarchical constraints. The method includes the following steps: Historical industrial material data is acquired, vectorized by embedding an encoder to obtain dense vectors, and a standard material vector knowledge base is constructed. Based on the industrial material to be classified, the embedded encoder is used to calculate the query vector, and a standard material sample is retrieved from the standard material vector knowledge base based on similarity. The standard material sample is then concatenated with the industrial material to be classified to obtain the composite input text. Historical industrial material classification samples were obtained. Based on the data balancing strategy of random undersampling of head categories and generative augmentation of tail categories, a fine-tuned dataset was constructed, and the large language model was fine-tuned and trained to obtain an industrial material classification model. Based on the enterprise standard classification system, a legal word prefix tree is obtained through serialization and tree mapping. Then, a legal next-hop word set is constructed based on each node of the legal word prefix tree. The composite input text is fed into the industrial material classification model. By using the Logits probability mask based on the set of legal next-hop words, the industrial material classification model is constrained to sample only from legal words, thus obtaining industrial material classification results that conform to the enterprise's standard classification system.
[0030] In one embodiment of the present invention, industrial material description texts typically exhibit characteristics such as sparse semantic information, short text length, and diverse naming formats. Especially in long-tail categories, material names often contain only a few keywords and lack clear category indication information. If a single material text is directly input into a material classification model, the model struggles to fully understand its true semantic meaning, thereby reducing classification performance. Therefore, the present invention first constructs a standardized material vector knowledge base. By uniformly vectorizing and modeling historical material semantic information, a semantic foundation is provided for subsequent information enhancement and retrieval. The construction of the standard material vector knowledge base specifically includes: By acquiring historical industrial material data and filtering, eliminating, and standardizing material naming, several standard industrial material text descriptions and corresponding classification tags are obtained. Based on several standard industrial material text descriptions and corresponding classification labels, vectorization encoding is performed using an embedding encoder to obtain several dense vectors. The embedding encoder can be a pre-trained BGE-M3 model, supporting multilingual and multi-granular feature extraction, and effectively aligning the mixed semantics of Chinese descriptions and English model parameters. The expression for the dense vectors is as follows:
[0031] in, For the density vector of standard industrial materials, For embedding encoders, for, The vector dimension can be set to 1024; by mapping to dense vectors, it is possible to accurately represent the subtle semantic differences of industrial terms in vector space. Each standard industrial material's corresponding dense vector is stored as a data point in a vector index field. The corresponding material information is encapsulated as metadata and stored in association with the vector index field, thus constructing a standard material vector knowledge base. By constructing the vector index field, this paper transforms unstructured historical materials into a real-time searchable structured knowledge base. This not only significantly improves retrieval speed but also preserves the material's hierarchical tags and attribute details, laying a solid data foundation for solving the knowledge gap problem of long-tail samples.
[0032] In one embodiment of the present invention, prior knowledge of industrial materials has been structurally stored based on the constructed standard material vector knowledge base. However, to truly apply this static data to solve the problem of extremely sparse training samples for long-tail categories, it is necessary to establish a process from offline storage to online inference. To this end, the present invention calculates a query vector based on the industrial material to be classified using the embedded encoder, and retrieves standard material samples from the standard material vector knowledge base based on similarity. The standard material samples are then concatenated with the industrial material to be classified to obtain composite input text, specifically including: The system obtains the user's query input text based on the industrial materials to be classified, processes it using the same embedding encoder as that used to build the standard material vector knowledge base, and obtains the query vector. Calculate the similarity between the query vector and each dense vector in the standard material vector knowledge base, and extract several standard material samples with the highest similarity based on the similarity to obtain a set of similar industrial material samples; each standard material sample includes the standard material name and its corresponding complete hierarchical classification path; the expression for similarity is as follows:
[0033] in, For similarity calculation, For query vector, For the density vector of standard industrial materials, It is the Euclidean norm; The expressions for the extracted standard material samples with the highest similarity are as follows:
[0034] in, For the first The most similar standard material sample To match the query input text The most similar A standard material sample, For the first Metadata corresponding to a standard material To preset the maximum number of similarity searches, you can Setting it to 5 indicates retrieving the top 5 densest vectors with the highest similarity from the standard material vector knowledge base, extracting the corresponding standard material samples, and obtaining a set of similar industrial material samples. The expression is as follows:
[0035] in, For a set of similar industrial material samples, This is the first standard material sample with the highest similarity. The first standard material name that has the highest similarity to the standard material sample. This is the complete hierarchical classification path for the first standard material sample with the highest similarity. The second sample is the standard material with the highest similarity. The second standard material name that has the highest similarity to the standard material sample. This is the complete hierarchical classification path for the second standard material sample with the highest similarity. For the first The most similar standard material sample For the first The name of the standard material that is most similar to the standard material sample. For the first The complete hierarchical classification path of the standard material sample with the highest similarity; The query input text, system command, standard material name, and its corresponding complete hierarchical classification path are concatenated according to a preset template to obtain a composite input text that combines long-tail semantic details with standard classification logic. The expression of the composite input text is as follows:
[0036] in, For compound input text, For system commands, For a set of similar industrial material samples, Enter text for the query.
[0037] This invention constructs a standard material vector knowledge base and designs a vector retrieval context recall. By injecting the standard material vector knowledge base, which serves as external prior knowledge, into the inference window of a large language model, it achieves analogical reasoning for long-tail materials. This dynamic recall strategy can effectively reduce the prediction entropy value of the model on low resource categories and alleviate the classification error problem caused by sample sparsity.
[0038] In one embodiment of the present invention, constructing a high-quality, evenly distributed training dataset is a prerequisite for the model to effectively internalize domain knowledge. Industrial material data exhibits a significant long-tail distribution characteristic, meaning that a few head categories occupy the majority of samples, while a massive number of tail category samples are extremely sparse. This leads to head category samples dominating the training data and driving the model's training direction, while tail category samples suffer from insufficient training data due to sparseness, hindering the model's accurate judgment of industrial material classification boundaries. To address the long-tail distribution problem of industrial material data, the present invention designs a data balancing strategy based on random undersampling of head categories and generative augmentation of tail categories, based on historical industrial material classification samples, to construct a fine-tuned dataset, specifically including: Obtain historical industrial material classification samples and count the number of samples in each classification category within the historical industrial material classification samples; For categories with a sample size greater than the preset head number threshold, truncation is performed, and the same number of samples as the head number threshold are randomly retained to obtain the first sample; for example, if the preset head number threshold is 600, 600 samples are randomly selected from categories with a sample size greater than the preset head number threshold and retained. This operation ensures that the proportion of samples in any single category does not exceed the preset upper limit, effectively smooths the distribution peaks of the head data, and prevents the large language model from overfitting high-frequency patterns and dominating the gradient descent direction. For classification categories where the number of samples is less than the preset head number threshold but greater than the preset tail number threshold, no processing is performed, and a second sample is obtained. For categories with fewer than a preset tail number threshold, real samples of historical industrial materials in that category are used as seed examples. Diverse synthetic data is generated through a generative model until the number of samples in that category is the same as the preset tail number threshold, at which point a third sample is obtained. If the preset tail number threshold is 50, for categories with fewer than 50 samples, real samples of historical industrial materials in that category are used as seed examples. A large language model is invoked to generate diverse synthetic data by mimicking semantic styles and specification description patterns until the total number of samples in that category is increased to 50. Through tail category generative enhancement, the large language model can acquire the form of the same long-tail material under more diverse descriptions during the training phase, thereby improving the recognition and generalization ability of tail categories.
[0039] The first, second, and third samples were transformed into an instruction format suitable for supervised fine-tuning of large models, resulting in a fine-tuning dataset. The fine-tuning dataset includes task instructions, user input, and target output. User input includes industrial material names, core words, specifications, and units, while the target output is structured text containing root class, major class, and leaf class names. This ensures the completeness and consistency of the classification paths generated by the large language model, facilitating subsequent parsing and evaluation.
[0040] After constructing and obtaining the fine-tuned dataset, it is used as input to fine-tune and train the large language model. This addresses the comprehension bias issues that arise when the general-purpose large language model encounters highly specialized professionalism and enterprise-specific classification tree logic in industrial scenarios. The final result is an industrial material classification model, which includes: Based on the large language model, this invention fine-tunes the parameters of all linear layers using low-rank adaptation techniques, freezes the original weights, and introduces two trainable low-rank matrices to simulate incremental weight updates, resulting in a pre-trained model. The large language model can be Qwen-2.5-7B-Instruct. Compared to typical low-rank adaptation LoRA techniques that only fine-tune the query Q and key V of the attention layer, this invention fine-tunes the parameters of all linear layers to maximize the efficiency of domain knowledge injection. The expression of the pre-trained model is as follows:
[0041] in, The output of the pre-trained model, For the original weights, Given the input sequence, To update the weights incrementally, This is a scaling factor used to balance the relative contributions of pre-trained parameters and fine-tuned parameters to the model output. Let be the rank of the low-rank decomposition. and All are trainable low-rank matrices; considering the complex logic and high accuracy requirements of industrial material classification tasks, the rank of the low-rank decomposition can be... The scaling factor is set to 32 instead of the usual 8 or 16 to enhance the model's memory capacity for domain-specific knowledge. Set to 64, keep The ratio is adjusted to balance the contributions of pre-trained weights and fine-tuned weights; Based on the pre-trained model, the Cosine learning rate decay strategy is adopted and repeated training is carried out in conjunction with a pre-set proportion of Warmup stage until the loss function based on causal language modeling is less than the set loss threshold, thus obtaining the industrial material classification model. The expression for the loss function based on causal language modeling is as follows:
[0042] in, For loss functions based on causal language modeling, For a moment, The length of the target sequence. To maximize a given input sequence Below, standard classification path The conditional generation probability, For the first in the standard classification path A true label for a moment For the first The system provides contextual information up to the last minute; at the same time, by using the Cosine learning rate decay strategy and a pre-set proportion of warmup, it is possible to ensure the training stability and convergence effect of the large language model on specific domain data.
[0043] This invention enables large language models to effectively encode the naming rules, specification differences, and hierarchical path information contained in the classification of industrial materials into the parameter space through domain instruction fine-tuning based on low-rank adaptive LoRA technology. It can also accurately grasp the complex naming logic and hierarchical classification rules of industrial materials, thereby improving the model's professional performance in vertical domains.
[0044] In one embodiment of the present invention, the large language model is a generative model, and is essentially an autoregressive prediction model based on probability sampling. In the free generation mode, the model is easily affected by long-tail noise, generating semantically reasonable but non-existent illusory paths in the enterprise standard classification system, i.e., industrial material classification errors occur. To solve this problem, the present invention obtains a legal word prefix tree based on the enterprise standard classification system through serialization and tree mapping, and constructs a legal next-hop word set based on each node of the legal word prefix tree, specifically including: Obtain the enterprise standard classification system, which includes a set of several legal hierarchical paths, and each legal hierarchical path includes several classification nodes; The word segmenter is used to segment several category nodes of each legal hierarchical path, and the resulting word sequence is obtained. Insert all word sequences into a prefix tree to obtain a valid word classification prefix tree. The nodes of the valid word classification prefix tree represent word states, and the edges represent the adjacency relationships between words. The word state includes the root node and leaf nodes. The root node is the starting state, and the leaf nodes contain the end-of-path word. For each node in the legal word prefix tree, maintain the corresponding set of legal next-hop words; The set of legal next-hop lexical elements includes lexical elements corresponding to all child nodes starting from the current node; for a leaf node representing the end of a complete classification path, its legal next-hop set only includes end-signature lexical elements; the legal lexical element classification prefix tree allows the large language model to select all legal lexical elements. For all leaf nodes representing the end of a complete classification path, their legal next-hop set only includes lexical elements of the end-signature node. This means that once the large language model predicts the end of the classification path, the legal lexical element classification prefix tree will force the large language model to stop generating immediately, thereby avoiding the generation of invalid suffix characters.
[0045] By constructing a prefix tree for legal word classification, this invention transforms the static enterprise coding standard into a dynamic finite state automaton; it provides an executable structural foundation for the subsequent autoregressive decoding process based on Logits constraints, ensuring that the model can only make decisions within the path space allowed by the enterprise standard at any generation step.
[0046] In one embodiment of this invention, based on a pre-constructed legal word prefix tree, a Logits probability mask mechanism is designed during the model inference stage. This mechanism progressively constrains the Logits vector output by the model during autoregressive decoding, forcing the model to generate only within the legal search space defined by the enterprise standard by dynamically correcting the probability distribution of candidate words. This invention inputs composite input text into an industrial material classification model, and through a Logits probability mask based on the legal next-hop word set, constrains the industrial material classification model to sample only from legal words, obtaining industrial material classification results that conform to the enterprise standard classification system. Specifically, this includes: The composite input text is input into the industrial material classification model, and at the initial decoding time ( ) Set the state pointer Initialize as the root node of the legal lexical classification prefix tree ; At the current decoding moment The original Logits vector is output through the industrial material classification model. And based on the current state pointer being decoded , pointing to For the corresponding tree node, read the set of legal next-hop words for the current node from the legal word classification prefix tree. :
[0047] in, The full vocabulary for pre-training large models. The current decoding moment status pointer, The set of all edges of the prefix tree for classifying legal lexical terms; It is usually the root node, but in special cases, if If it is a leaf node, then The set This will be directly used as the criterion for constructing the mask vector in the next step. Based on the original Logits vector and the set of valid next-hop words, a vector is constructed that is equivalent to the original Logits vector. Hard constraint mask vectors of the same dimension :
[0048] in, The first word in the full vocabulary for pre-training of a large model For each word, the hard constraint mask vector will completely mask words that do not belong to the set of legal next-hop words in the probability space, and only retain the relative probability relationship of words within the set of legal next-hop words; The hard-constraint mask vector is element-wise superimposed with the original Logits vector, and the constrained conditional probability distribution is calculated using the Softmax function. This ensures that the probability of illegal words is 0, only legal words participate in normalization, and the word with the highest probability is selected as the word output at the current decoding time. The expression for the conditional probability distribution is as follows:
[0049] in, Given a valid prefix tree for word classification, the current decoding time... The probability distribution of generated lexical units, The current decoding moment The output token, The current decoding moment Previously generated words; through hard-constrained mask vectors, the generation probability of all illegal words is strictly suppressed to 0, only legal candidate words participate in normalization and competition, and the word with the highest probability is selected as the word output at the current decoding moment. After selection, the state transition mechanism of the prefix tree is triggered, and the state pointer is pointed to the next level node, thus providing... The construction of time-matrix masks provides new topological basis:
[0050] in, For the state transition function in the prefix tree, for The updated state pointer; this process is executed cyclically until the end symbol word is generated or the leaf node of the classification tree is reached, and finally a classification path that is strictly continuous in the tree structure is output; that is, according to the word output at the current decoding time, the state pointer is updated to point to the next node in the legal word classification prefix tree; Repeat the decoding process until the state pointer reaches a leaf node, output the end symbol word, and obtain the industrial material classification result that conforms to the enterprise standard classification system.
[0051] This invention generates a complete, tree-structured, continuous classification path as the final output through word-by-word guidance using the Logits masking mechanism. This effectively suppresses the illusion of large models and ensures the legitimacy of the classification path, significantly improving the overall quality and usability of industrial material classification.
[0052] The beneficial effects of this invention are as follows: This invention constructs a standardized material vector knowledge base and introduces a retrieval enhancement mechanism to recall similar industrial material samples from an external knowledge base as prior knowledge, compensating for the semantic deficiencies of long-tail samples and enhancing the model's ability to identify tail categories. It employs a data balancing strategy of head-side random undersampling and tail-side generative enhancement to mitigate the negative impact of long-tail differential distribution features on model training, providing high-quality supervision signals for low-rank adaptive fine-tuning. Based on low-rank adaptive LoRA technology, it fine-tunes the parameters of all linear layers, efficiently injecting industry-specific private knowledge while maintaining the general semantic capabilities of the large model, thus improving the model's industrial material classification accuracy. By constructing a prefix tree and introducing a Logits probability mask mechanism, the model is forced to sample only from the legal path space during decoding, eliminating model generation illusions at the source, ensuring the structural compliance of the classification results, and improving the reliability and interpretability of the industrial material classification results.
Claims
1. A hierarchical classification method for industrial materials that integrates enhanced retrieval and explicit hierarchical constraints, characterized in that: Includes the following steps: Historical industrial material data is acquired, vectorized by embedding an encoder to obtain dense vectors, and a standard material vector knowledge base is constructed. Based on the industrial material to be classified, the embedded encoder is used to calculate the query vector, and the standard material sample is retrieved from the standard material vector knowledge base based on similarity. The standard material sample is then concatenated with the industrial material to be classified to obtain the composite input text. Historical industrial material classification samples were obtained. Based on the data balancing strategy of random undersampling of head categories and generative augmentation of tail categories, a fine-tuned dataset was constructed, and the large language model was fine-tuned and trained to obtain an industrial material classification model. Based on the enterprise standard classification system, a legal word prefix tree is obtained through serialization and tree mapping. Then, based on each node of the legal word prefix tree, a legal next-hop word set is constructed. By inputting the composite input text into the industrial material classification model, and using the Logits probability mask based on the set of legal next-hop words, the industrial material classification model is constrained to sample only from legal words, thus obtaining industrial material classification results that conform to the enterprise's standard classification system.
2. The hierarchical classification method for industrial materials based on fusion retrieval enhancement and explicit hierarchical constraints as described in claim 1, characterized in that, The construction of the standard material vector knowledge base specifically includes: By acquiring historical industrial material data and filtering, eliminating, and standardizing material naming, several standard industrial material text descriptions and corresponding classification tags are obtained. Based on several standard industrial material text descriptions and corresponding classification tags, vectorization encoding is performed through an embedded encoder to map several dense vectors. Each standard industrial material's corresponding dense vector is stored as a data point in the vector index field, and the material information corresponding to that standard industrial material is encapsulated as metadata and stored in association with the vector index field, thus constructing a standard material vector knowledge base.
3. The hierarchical classification method for industrial materials based on fusion retrieval enhancement and explicit hierarchical constraints as described in claim 1, characterized in that, The process of obtaining the composite input text specifically includes: The system obtains the user's query input text based on the industrial materials to be classified, processes it using the same embedding encoder as that used to build the standard material vector knowledge base, and obtains the query vector. Calculate the similarity between the query vector and each dense vector in the standard material vector knowledge base, and extract several standard material samples with the highest similarity based on the similarity to obtain a set of similar industrial material samples; each standard material sample includes the standard material name and its corresponding complete hierarchical classification path; The query input text, system instructions, standard material names and their corresponding complete hierarchical classification paths are concatenated according to a preset template to obtain composite input text.
4. The hierarchical classification method for industrial materials based on fusion retrieval enhancement and explicit hierarchical constraints as described in claim 3, characterized in that, The expression for the compound input text is as follows: in, For compound input text, For system commands, For a set of similar industrial material samples, Enter text for the query. This is the first standard material sample with the highest similarity. The first standard material name that has the highest similarity to the standard material sample. This is the complete hierarchical classification path for the first standard material sample with the highest similarity. The second sample is the standard material with the highest similarity. The second standard material name that has the highest similarity to the standard material sample. This is the complete hierarchical classification path for the second standard material sample with the highest similarity. For the first The most similar standard material sample For the first The name of the standard material that is most similar to the standard material sample. For the first A complete hierarchical classification path for the standard material sample with the highest similarity. To preset the maximum number of similarity searches, To match the query input text The most similar A standard material sample, For similarity calculation, For query vector, For the density vector of standard industrial materials, For the first Metadata corresponding to a standard material It is the Euclidean norm.
5. The hierarchical classification method for industrial materials based on fusion retrieval enhancement and explicit hierarchical constraints as described in claim 1, characterized in that, The data equalization strategy based on random undersampling of head categories and generative augmentation of tail categories specifically includes: Obtain historical industrial material classification samples and count the number of samples in each classification category within the historical industrial material classification samples; For classification categories where the number of samples exceeds a preset head count threshold, truncation is performed, and the same number of samples as the head count threshold are randomly retained to obtain the first sample. For classification categories where the number of samples is less than the preset head number threshold but greater than the preset tail number threshold, no processing is performed, and a second sample is obtained. For a classification category with a sample size less than a preset tail number threshold, a real sample of historical industrial material in that classification category is used as a seed example. Diverse synthetic data is obtained through a generative model until the sample size of that classification category is the same as the preset tail number threshold, at which point a third sample is obtained. The first, second, and third samples are converted into an instruction format suitable for supervised fine-tuning of large models to obtain a fine-tuning dataset, which includes task instructions, user input, and target output.
6. The hierarchical classification method for industrial materials based on fusion retrieval enhancement and explicit hierarchical constraints as described in claim 1, characterized in that, The fine-tuning and training of the large language model to obtain the industrial material classification model specifically includes: Based on the large language model, the parameters of all linear layers are fine-tuned using low-rank adaptation techniques, the original weights are frozen, and two trainable low-rank matrices are introduced to simulate incremental weight updates, thus obtaining a pre-trained model. Based on the pre-trained model, the Cosine learning rate decay strategy is adopted and repeated training is carried out in conjunction with a pre-set proportion of Warmup stage until the loss function based on causal language modeling is less than the set loss threshold, thus obtaining the industrial material classification model. The expression for the pre-trained model is as follows: in, The output of the pre-trained model, For the original weights, Given the input sequence, To update the weights incrementally, This is the scaling factor. Let be the rank of the low-rank decomposition. and All are trainable low-rank matrices; The expression for the loss function based on causal language modeling is as follows: in, For loss functions based on causal language modeling, For a moment, The length of the target sequence. To maximize a given input sequence Below, standard classification path The conditional generation probability, For the first in the standard classification path A true label for a moment For the first Context information from a specific point in time.
7. The hierarchical classification method for industrial materials based on fusion retrieval enhancement and explicit hierarchical constraints as described in claim 1, characterized in that, The obtained set of legal next-hop lexical units specifically includes: Obtain the enterprise standard classification system, which includes a set of several legal hierarchical paths, and each legal hierarchical path includes several classification nodes; The word segmenter is used to segment several category nodes of each legal hierarchical path, and the resulting word sequence is obtained. Insert all word sequences into a prefix tree to obtain a legal word classification prefix tree. The nodes of the legal word classification prefix tree represent word states, the edges represent the adjacency relationships between words, the root node is the starting state, and the leaf nodes contain the ending symbol words representing the end of the path. For each node in the legal word prefix tree, maintain the corresponding set of legal next-hop words; The set of valid next-hop lexical elements includes lexical elements corresponding to all child nodes starting from the current node; for a leaf node representing the end of a complete classification path, its valid next-hop set only includes terminator lexical elements.
8. The hierarchical classification method for industrial materials based on fusion retrieval enhancement and explicit hierarchical constraints as described in claim 7, characterized in that, The output, which conforms to the enterprise's standard classification system, includes the following industrial material classification results: Input the composite input text into the industrial material classification model, and initialize the state pointer to the root node of the legal word prefix tree at the initial decoding moment; The industrial material classification model outputs the original Logits vector at the current decoding time, and reads the set of legal next-hop tokens of the current node from the legal token classification prefix tree according to the state pointer at the current decoding time. Based on the original Logits vector and the set of legal next-hop words, a hard constraint mask vector with the same dimension as the original Logits vector is constructed. The hard constraint mask vector is superimposed element by element with the original Logits vector, and the constrained conditional probability distribution is calculated by the Softmax function. This makes the probability of illegal words 0, and only legal words participate in normalization. The word with the highest probability is selected as the word output at the current decoding time. Based on the word elements output at the current decoding moment, update the state pointer to point to the next node in the legal word element classification prefix tree; Repeat the decoding process until the state pointer reaches a leaf node, output the end symbol word, and obtain the industrial material classification result that conforms to the enterprise standard classification system.