Knowledge guide retrieval enhancement generation method for data scarce industrial vertical field

Abstract

The invention discloses a data scarcity industrial vertical field-oriented knowledge-guided retrieval enhancement generation method, belongs to the technical field of natural language processing and industrial intelligence crossing, and can improve the retrieval accuracy and generation reliability of a large language model in an industrial scene. According to the method, a'sparse vector + dense vector 'mixed knowledge base is constructed, and general knowledge and industrial field texts are fused; after the model generates an initial text, judging whether external retrieval is needed or not through multi-dimensional evaluation token confidence; extracting attention weights for the low-confidence tokens, and screening key tokens to generate a retrieval query; dynamically adjusting the weight of a retriever based on a BGE-M3 model, and optimizing a retrieval result through reordering; the retrieval knowledge is converted into a context with an index, and a prompt template is constructed to generate a correction value iteration calibration text; and finally, optimizing the text format, and generating a structured response meeting industrial requirements. The method solves the problems of lack of professional knowledge of large language models in the industrial field, shallow retrieval and generation fusion, lack of knowledge calibration mechanisms and the like.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of natural language processing and industrial intelligence. It relates to a knowledge-guided retrieval enhancement generation method for data-scarce industrial vertical fields. It is applicable to semantic interaction tasks in industrial scenarios and can improve the retrieval accuracy and reliability of generated content of large language models in such scenarios. Background Technology

[0002] Currently, global industry is accelerating its transformation towards intelligence and digitalization. my country has also issued policies such as the "Implementation Guidelines for Digital Transformation of Manufacturing Enterprises," which clearly proposes to break through key technologies in industrial intelligence and improve capabilities such as semantic interaction and knowledge application in industrial scenarios to support the high-quality development of the manufacturing industry. Under this trend, large-scale pre-trained language models (LLMs) such as the GPT series and LLaMa series, with their powerful language representation and generation capabilities, have demonstrated excellent performance in general natural language processing (NLP) tasks, but they face significant adaptation challenges in vertical industrial fields.

[0003] The industrial sector contains a large number of proprietary terms and complex technical principles, making it difficult for LLMs to grasp in-depth professional knowledge through general corpora, resulting in insufficient professionalism in their answers. Furthermore, industrial data often involves trade secrets or requires high collection costs, and labeled data within the field is scarce, making LLMs prone to generating "illusory content," such as incorrect descriptions of industrial equipment troubleshooting procedures. Even though Retrieval Augmentation (RAG) technology attempts to improve output accuracy by integrating external knowledge bases, it still has limitations in the industrial vertical: the fusion of retrieval and generation modules is superficial, often directly concatenating search results to the LLM input, lacking deep interaction based on industrial semantics, preventing LLMs from fully utilizing professional knowledge; traditional single sparse retrieval or fixed-weight fusion retrieval is insufficient to cope with the semantic complexity of industrial queries, easily leading to retrieval bias; and the generation process lacks dynamic calibration logic designed for the specificities of industrial knowledge, failing to effectively correct initial generation errors, resulting in insufficient reliability of the final output.

[0004] To address the aforementioned challenges, this invention provides a knowledge-guided retrieval enhancement generation method for data-scarce industrial verticals. This method aims to overcome the technical bottlenecks caused by data scarcity through multi-dimensional optimization, achieving "high accuracy, strong traceability, and excellent domain adaptability" in semantic interaction within industrial scenarios. Summary of the Invention

[0005] This invention provides a knowledge-guided retrieval enhancement generation method for data-scarce industrial verticals, aiming to solve the problems of low retrieval accuracy and inaccurate generated content in data-scarce industrial verticals.

[0006] The method provided by this invention includes six key steps: constructing an external knowledge base, joint evaluation of token confidence, adaptive query generation, integrated retrieval and re-ranking of knowledge collaboration, iterative calibration and dynamic optimization of industrial knowledge, and outputting semantic instruction interaction results for industrial vertical domains.

[0007] Step 1: Building an external knowledge base

[0008] Step 1.1 Selection and Processing of Knowledge Corpus

[0009] The general foundational knowledge corpus is based on 500,000 English Wikipedia text paragraphs; the industrial vertical knowledge corpus comes from scattered text content in the field, including key materials such as technical manuals, operation guides, and failure cases.

[0010] Step 1.2 BGE-M3 Model Encoding

[0011] The BGE-M3 model is used to encode the corpora of the general basic layer and the industrial vertical layer, forming a hybrid knowledge representation system of "dense vectors + sparse vectors":

[0012] Suppose the document set is D = {d1, d2, ..., dn} m}, where m is the total number of documents, and the sparse coding capability of the BGE-M3 model is used to encode each document d i Encoded as a sparse vector v d (i) :

[0013]

[0014] Where T is the global term set, i.e., the unique token (the smallest unit or basic element of text processing) appearing in all documents, w d (i,k) For document d i Chinese term t k The weights, learned by the BGE-M3 model, are used to determine the relationship between the term and document d. Non-zero values ​​indicate the relationship between the term and document d. i It makes a semantic contribution; the sparse vector set V corresponding to the document set D can be generated using Formula 1:

[0015]

[0016] In the intensive encoding stage, a complex nonlinear transformation is used to transform each document d in the document set D using the BGE-M3 model. i Embedded into a 1024-dimensional dense vector space:

[0017]

[0018] Where f γRepresents the nonlinear transformation function of the BGE-M3 model; e di Document d represents document d i via f γ The resulting 1024-dimensional dense vector; the dense vector set E corresponding to the document set D can be generated using Formula 3:

[0019]

[0020] The vector set E is directly imported into the FAISS database to build an efficient index. FAISS achieves fast subsequent retrieval by indexing E.

[0021] The external knowledge base is constructed through the "feature complementarity" of two types of vectors, Equation 2 and Equation 4: the sparse vector represented by Equation 2 achieves key feature localization through non-zero weight terms; the dense vector represented by Equation 4 captures deep associations through 1024-dimensional semantic embedding; the two together constitute the core data of the knowledge base, and knowledge matching is achieved through the collaboration of sparse retrieval and dense retrieval in subsequent retrieval.

[0022] Step 2: Joint Assessment of Token Confidence

[0023] Step 2.1 Initial Generation

[0024] Based on the input demo, question, and currently generated text (initially empty), the LLM is invoked to generate initial text content. Assuming the vocabulary is V, the initial content sequence generated by the LLM is T = {t1, t2, ..., t}. n}, where t j This represents the token at position j in the sequence.

[0025] Step 2.2 Uncertainty Assessment

[0026] For the sequence T = {t1, t2, ..., t n For each token in}, calculate the logarithmic entropy S. i Uncertainty generated by quantification:

[0027]

[0028] Its sss

[0029] Step 2.3 Impact Assessment

[0030] By leveraging the inherent self-attention mechanism of the Transformer architecture, an attention matrix is ​​constructed:

[0031]

[0032] Where Q is the query matrix, K represents the key matrix, and d kd represents the dimension of the key vector, and Mask represents the masking function, which acts on the scaled dot product matrix W = QK T / √d k ; specifically, when the number of rows of matrix W is equal to the number of columns, the masking function processes the elements of the square matrix W according to the following rules:

[0033]

[0034] Traverse the row index (i) and column index (j) of matrix W. When i < j, it means the current position is at a subsequent position, and the matrix element W i,j is assigned -∞, which can mask the upper triangular part of the square matrix; when i ≥ j, corresponding to the historical or current position, the value of the matrix element W i,j remains unchanged. After the masking operation, apply the Softmax function to each row of the matrix to obtain the attention weight matrix A. Accordingly, for token t i , its maximum attention value a max(i) is determined by screening the maximum value of A i,j in all cases where i < j, thereby quantifying the influence of t i on the subsequent context:

[0035] a max ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​i )<θ, indicating that token t i There is a certain level of confidence, so there is no need to perform external knowledge retrieval; the information can be directly concatenated into the current result.

[0042] The threshold θ needs to be determined experimentally based on the actual industrial scenario and model performance. If θ is set too low (e.g., 0.3), a large number of high-confidence tokens will be misjudged as low-confidence tokens, increasing the retrieval burden. If θ is set too high (e.g., 1.2), some low-confidence tokens will be missed, affecting the accuracy of the generated content. Generally, a value of 0.5-1.0 is more appropriate.

[0043] Step 3: Adaptive Query Generation

[0044] Step 3.1 Extract the attention weights of the preceding tokens

[0045] For judging token t with low confidence i Extract the preceding token sequence {t} from the attention matrix A in Formula 2. i-1 ,t i-2 The attention weights of {t1, ...,t2} are used to reflect the role of each preceding token in generating t. i The importance of process:

[0046] A i ={a i,1 ,a i,2 ,...,a i,i-1} (10)

[0047] Among them, a i,j (j = 1, 2, ..., i-1) indicates that during the generation of token t i At that time, token t j For t i Attention weights, which are used to measure t j In t i The importance of the information provided during the generation process;

[0048] Step 3.2 Token Filtering

[0049] Sort the attention weights in descending order and select the n tokens with the highest weights to form a set:

[0050] T n ={t k |rank(a i,k )≤n} (11)

[0051] The value of n needs to be determined based on the specific task requirements. For example, if the task has a high requirement for the concentration of key contextual information, a smaller n can be selected; if more comprehensive contextual support is required, a larger n can be selected, but computational efficiency should also be considered to avoid excessive computational costs in subsequent processing due to an excessively large n. Generally, the value of n is in the range of 20-40.

[0052] Use the indicator function in Formula 5 to filter out the input key token set T. key Token T s :

[0053]

[0054] Step 3.3 Generation of the query term set

[0055] Extracting T from the vocabulary s The words corresponding to the tokens are arranged in the order of their appearance in the original text to construct a query term set W that reflects the actual information needs of LLM. q ;

[0056] Step 4: Integrated Retrieval Reordering Based on Knowledge Collaboration

[0057] Step 4.1 Calculation of keyword density and semantic feature complexity

[0058] Keyword density reflects the specialization and concentration of a query in its semantic information expression; for a given query Q = {q1, q2, ..., q...} n The spaCy model is used for word segmentation and part-of-speech tagging, and a vocabulary weight mapping table W is constructed. P Based on the lexical attributes, each word q i Weights are assigned to reflect the importance of different lexical attributes in semantic contribution; simultaneously, a custom terminology dictionary D is constructed. lex To assign higher weight to specialized terms and adapt to the needs of the field, the keyword density KD(Q) calculation formula is as follows:

[0059]

[0060] Where T(Q) is the set of effective query terms, that is, the set of terms selected from query Q using the SPacy tool that participate in the keyword density calculation. These terms can truly affect the semantic expression, specialization, and concentration of the query; |T(Q)| represents the number of terms contained in T(Q); w c (t) represents the custom term weight, a weight value specifically set for professional terms to reflect their importance in the semantic expression of a particular domain; w p(t) represents the lexical weight, which is assigned to lexical words based on their part of speech, reflecting the importance of lexical words in general semantic contribution. For the term matching indicator function, i.e., when t∈D lex hour =1, at this time w is used c (t) is used as the weight of the term to avoid unreasonable weight allocation of technical terms due to issues such as routine part-of-speech tagging; conversely, At this point, w is used. p (t) represents the weight of the word; ε is a very small positive number used to prevent the denominator of the formula from being 0; the output value is obtained by max(w p Normalization maps to the interval [0,1].

[0061] Semantic feature complexity measures the degree of query expansion in the semantic space and the semantic changes between layers, including differences between two layers and semantic diffusion; given a query Q = {q1,q2,....,q...} n After vectorization and extraction of hierarchical semantic features by the BGE-M3 model, H is defined. (l) Let h be the hidden state matrix of the l-th layer. l =Flatten(H (l) The hidden state vector is vectorized (Flatten represents the dimensionality reduction function). The focus is on three feature layers: the input layer, the intermediate layer, and the output layer. Here, "layer" refers to the network layers of the pre-trained BGE-M3 model. The input layer represents the model's initial hidden layer (layer 0), receiving text vectors processed by previous embedding layers as input and outputting surface features of the text (such as the semantics of the words themselves), containing a significant amount of original information, which can be used to capture surface features and local co-occurrence patterns. The intermediate layer represents the model's intermediate hidden layers (defaulting to layers 2, 4, 8, and 10), responsible for progressively abstracting and processing text features, outputting abstract features that incorporate contextual information, retaining some surface information while containing preliminary semantic associations, used to encode syntactic dependencies and phrase-level semantics. The output layer represents the model's final hidden layer (layer 12), outputting high-level semantic features processed by all layers, used to characterize logical structure and deep semantics. The semantic embedding of the text at each layer progressively abstracts from the surface to the deepest levels.

[0062] Interlayer difference ID(Q) is defined as the squared mean of the JS divergence between adjacent layers, and is used to measure the degree of variation in semantic representations between different layers:

[0063]

[0064] Where L=6 is the total number of layers, and intermediate layers can be added according to the specific scenario; P i h represents the hidden state vector of the i-th layer. iAfter normalization, Q i h represents the hidden state vector of layer i+1. i+1 The values ​​are normalized; semantic diffusion is used to evaluate the breadth of semantic distribution in multidimensional space, and all selected layers are connected as E = [h l0 ,h l1 ,...h lL ], and PCA is applied for dimensionality reduction, where semantic diffusion is defined as the Euclidean distance between the maximum and minimum points in the reduced dimensionality space:

[0065]

[0066] PCA(E) i Let represent the i-th embedding vector after dimensionality reduction; the semantic feature complexity is defined as the weighted sum of inter-layer differences and semantic diffusion:

[0067] SC(Q)=αLD(Q)+βSD(Q) (16)

[0068] α and β are weight parameters, with default values ​​of 0.6 and 0.4, respectively. These values ​​can be flexibly chosen according to the specific task scenario. If the task focuses more on the hierarchical depth of text semantics, the value of α can be increased. If the task focuses more on the richness of text semantics, the value of β can be increased.

[0069] Step 4.2 External Knowledge Retrieval

[0070] The sparse retrieval tool SR is built upon the sparse coding capabilities of the BGE-M3 model to achieve efficient information retrieval; for the generated query term set W... q For each query q in the algorithm, q is first sparsely encoded to obtain a sparse vector v. q :

[0071]

[0072] Its structure and parameters are consistent with the document vector in Formula 1, retaining only terms with non-zero weights; SR achieves retrieval by calculating the similarity score between the document and the query, and its core is the summation of the product of the weights of co-occurring terms:

[0073]

[0074] Where s(q,d) i ) indicates query q and document d i The similarity score between them, T q,i The set of co-occurring terms, i.e., terms that appear in both query q and document d. i The set of terms in w q (k)This represents the weight of term k in query q. A higher weight indicates that the term plays a more crucial role in expressing the core semantics of query q; w d (k) Document d represents document d i The weight of term k in document d; the higher the weight, the more relevant the term is to document d. i The closer the association, the better; the weighted product reflects the synergistic contribution of "query term importance" and "document term relevance", and the summation yields the overall matching degree; the documents are sorted in descending order of similarity score and a certain number of documents (usually 5-20) are selected as the final sparse search results as needed;

[0075] Similar to Equation 2, the dense retrieval tool DR is built upon the dense encoding capability of the BGE-M3 model. First, each query q is encoded and embedded into a 1024-dimensional dense vector space. A similarity score is calculated based on the cosine similarity between the vectors. Assuming the encoded document vector is 'a' and the query vector is 'b', the cosine similarity can be expressed as:

[0076]

[0077] Where a i and b i Let i represent the i-th component of vectors a and b; sort the search results in descending order based on cosine similarity, and select a certain number of documents (usually 5-20) as the final sparse search results as needed;

[0078] Step 4.3 Dynamic Weight Adjustment

[0079] Based on the keyword density KD(Q) and semantic feature complexity SFC(Q) calculated using Formulas 9 and 12, the weights of the sparse search engine SR and the dense search engine DR are dynamically adjusted:

[0080]

[0081] Where w S With w D w represents the weights of the sparse search engine SR and the dense search engine DR, respectively. b,S and w b,D This represents the initial weight, with a value range of [0,1]. The initial setting defaults to 0.5 and 0.5, which can be adjusted according to the actual application scenario. If the business scenario relies more on keyword matching-based retrieval, the initial weight w of the sparse search engine can be increased. b,S Increase the initial weight w of the dense search engine if the task focuses on precise semantic retrieval. b,D γ1 and γ2 represent weight parameters controlling the relative importance of KD and SFC, with a fixed value of 0.2, which is the optimal result obtained from experiments; then normalization is performed:

[0082]

[0083] Among them w' S and w' D These represent the weights of the normalized sparse search engine SR and the dense search engine DR, respectively, ensuring that the sum of the weights is 1;

[0084] Step 4.4 Retrieval Result Fusion and Re-ranking

[0085] Based on the weights of SR and DR obtained from Formula 19, the weighted inverse ranking fusion (WRRF) algorithm is further used to fuse the retrieval results of the retrieval machines SR and DR; assuming that the merged and deduplicated SR and DR retrieval results are a set D = {d1, d2, ..., d...} n}, where n represents the number of documents, for document d i Calculate the fusion score:

[0086]

[0087] Where c=60 is a smoothing constant, which can avoid the influence of abnormal denominators and smooth sorting positions on the score, and can also balance the sorting contributions of different retrieval machines, making the fused score more stable and reasonable; rank S (d i ) indicates document d i Rank is the sorting position in the search results of the SR search engine. D (d i ) indicates document d i The sorting position in the DR search results; and The indicator function is set to 1 when a document belongs to the corresponding retrieval result, ensuring that only documents existing in the corresponding retrieval result have their ranking and other information included in the retrieval's contribution to the fusion score. After fusion, a cross-coding re-ranking model is constructed using a pre-trained ms-marco-MiniLM-L-6-v2 model to achieve deep semantic interaction between queries and documents. For query q and document d, a joint representation is constructed using a pre-trained language model M.

[0088]

[0089] in This indicates a concatenation operation, where [CLS] is the classification symbol and [SEP] is the delimiter. This architecture employs a multi-head cross-attention mechanism to accurately capture fine-grained interaction relationships. The relevance score s(q,d) is generated by the classification head C.

[0090]

[0091] Where dim(C) represents the dimension of the classifier head C; when dim(C) = 1, w and b represent the weight vector and bias scalar, respectively; when dim(C) > 1, W c and b c These correspond to the weight matrix and bias vector in the multidimensional case, respectively; softmax(·)[1] represents the positive class probability value in the softmax output;

[0092] For the candidate document set fused by Formula 15, the relevance score s(q,d) between each document and query q is calculated. i The sorting process is performed to reorder the documents, ultimately returning k (default is 5) documents as the final retrieved knowledge set D. * :

[0093] D * ={d i |rank(d i )≤k} (25)

[0094] Where rank(·) represents the relevance score s(q,d) i Perform sorting operations;

[0095] Step 5: Iterative Calibration and Dynamic Optimization of Industrial Knowledge

[0096] Based on the retrieved knowledge set D * The generated content is iteratively optimized to establish a verifiable knowledge integration loop, specifically including:

[0097] Step 5.1 Construct an industrial-scenario-based calibration mechanism

[0098] Define the calibration function:

[0099] T * =G(P;η) (26)

[0100] Where T * G represents the calibrated text, G is the LLM's generation function, P represents the structured prompt information generated by the prompt template, which includes the question, retrieved knowledge, current text and correction instructions, and is the input carrier that triggers the model to perform calibration inference. η is a set of learnable parameters inside the LLM, which includes the model's weights and biases.

[0101] Step 5.2 Knowledge Integration

[0102] To achieve efficient knowledge fusion, a dual mechanism of contextual enhancement and prompting engineering is introduced; in the contextual semantic information enhancement stage, the retrieved knowledge set D... * It is transformed into linear context information, specifically in the following form:

[0103]

[0104] Where C represents the enhanced context, and [i] represents document d. i index tags, This indicates a concatenation operation. This means adding an index tag before each document, which preserves the original order of the documents and identifies the source of each segment through the index, making it easier for subsequent traceability verification;

[0105] Hint engineering constructs a complete hint message P, which combines the question q, the reinforcement context C, and the current text T. t Integrate into a logical hierarchical structure to form an input format that the model can recognize, ensuring that knowledge is effectively utilized during the generation process;

[0106] Step 5.3 Iterative optimization

[0107] Iterative optimization follows the incremental update principle; let the text at time t be T. t Based on the prompt message P, the LLM is invoked to generate the correction amount Δ. t Update the generated text:

[0108]

[0109] Where the correction amount Δ t Defined as:

[0110] Δ t =arg max δ P(δ∣P t ;η) (29)

[0111] Where η is consistent with Equation 26, representing the parameters of LLM, P(δ|P t η) represents the generation probability of the correction amount; this means that at time point t, the system will select the correction content with the highest selection probability as the update amount based on the prompt information P and the model parameter η.

[0112] This iterative optimization process is implemented through a closed-loop mechanism: In the initial stage, T0 represents the original text generated by the model, and the initial prompt information P0 is constructed based on T0; in each iteration, T0 is first optimized... t Perform joint evaluation of token confidence to determine if there is any content that conflicts with knowledge; if a low-confidence token is detected, perform relevant knowledge retrieval and call the calibration function T. * Update the text to T t+1 Subsequently based on T t+1 Reconstruction prompt message P t+1 Then proceed to the next iteration, and this process continues until the termination condition is met.

[0113] Step 5.4 Iteration Termination Control

[0114] To balance calibration effectiveness and computational efficiency, a multi-condition collaborative termination strategy is adopted, which explicitly includes three triggering scenarios:

[0115] ① Natural termination: When a predefined end symbol ( <eos>) appears in the correction amount Δ t When the time is right, it indicates that the model has determined that the current text does not require further correction;

[0116] ② Length constraint: When the text length |T t |Exceeds the maximum threshold L max (Default is 64, can be adjusted flexibly) When the iteration is terminated, it is to avoid the accumulation of redundant information;

[0117] ③ Stable convergence: When the distance metric between two consecutive corrections is less than ∈, that is:

[0118] d(Δ t ,Δ t-1 )<∈ (30)

[0119] Where d represents the distance metric function of the correction amount (here it is based on the number of tokens, but in practical applications it can be flexibly selected, such as based on edit distance or similarity), ∈ takes a value of 1-2, representing the difference of tokens, and its value depends on the selection of d, indicating that the text has reached a stable state, further iterations are unlikely to significantly improve the quality, and the calibration process tends to converge;

[0120] Step 6: Output the semantic command interaction results for the industrial vertical domain.

[0121] The final calibration text T at the end of the iteration final As the output.

[0122] The inventiveness of this invention is mainly reflected in:

[0123] 1. A three-dimensional joint assessment of token confidence based on "uncertainty, influence, and key token filtering" enables accurate judgment of search needs.

[0124] Traditional search augmented generation (RAG) techniques often rely on fixed user queries or simple keyword matching, which can easily lead to a disconnect between the retrieval and generation processes, especially in industrial verticals where there is a high density of technical terms and complex semantic logic. This invention innovatively constructs a closed loop of "uncertainty-influence-key token filtering-precise query generation," achieving a paradigm shift from "passively relying on external knowledge" to "actively verifying facts."

[0125] 2. Knowledge-based collaborative integrated retrieval and reordering to adapt to the semantic complexity of industrial queries.

[0126] Traditional retrieval methods, whether employing a single sparse retrieval (such as BM25) or a fixed-weight fusion of sparse and dense retrieval, struggle to handle the dynamic semantic features of industrial queries. Some industrial queries are dense with technical terms, requiring strong terminology matching capabilities; others require deep semantic associations, relying on contextual understanding. This invention innovatively designs a dynamic weight adjustment mechanism based on query features: it quantifies the concentration of industrial technical terms in the query by keyword density, calculates semantic feature complexity by combining inter-layer differences and semantic diffusion extracted by the BGE-M3 model, and dynamically allocates weights to the sparse and dense retrieval machines; simultaneously, it introduces weighted reciprocal ranking fusion and cross-coding reordering, first fusing the results of the two retrieval machines, and then optimizing the ranking through deep semantic interaction, thereby improving retrieval accuracy in industrial scenarios.

[0127] 3. Iterative optimization of dynamic and progressive knowledge calibration to solve the problem of knowledge scarcity in the industrial field.

[0128] Data scarcity leads to incomplete coverage of external knowledge bases in industrial verticals. Traditional RAG (Research and Analysis Group) methods rely solely on single retrievals to supplement knowledge, failing to achieve dynamic knowledge reconstruction and accurate calibration. This results in biased generation due to knowledge gaps. This invention innovatively proposes a dynamic, progressive knowledge calibration based on iterative optimization: retrieved knowledge is converted into an indexed linear context, preserving the traceability of knowledge sources and facilitating model association; a structured input of "problem-knowledge-current text" is constructed through prompting engineering, driving a large language model to generate correction values; and three conditions—"natural termination," "length constraint," and "stable convergence"—are used to control the iterative process, gradually calibrating the generated text to conform to industrial realities. This mechanism does not rely on large-scale labeled data, efficiently utilizing limited industrial knowledge through knowledge reconstruction, effectively mitigating the unreliable generation problem caused by data scarcity. Attached Figure Description

[0129] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0130] Figure 1 Overall flowchart

[0131] Figure 2 Radar chart showing the performance of integrated retrieval re-ranking for knowledge collaboration compared to other retrieval methods.

[0132] Specific implementation measures

[0133] The specific implementation steps of this invention mainly include:

[0134] Step 1: Building an external knowledge base

[0135] Step 1.1 Selection and Processing of Knowledge Corpus

[0136] The general foundational knowledge corpus is based on 500,000 English Wikipedia text paragraphs. The "500,000" is a threshold determined based on the industry's need for a broad coverage of general knowledge. This number provides sufficient general semantic support to avoid semantic comprehension biases caused by insufficient general knowledge, while also preventing excessive consumption of subsequent encoding and storage resources due to an excessive amount of corpus. The industry vertical knowledge corpus comes from scattered text content within the field, including 100 key documents such as technical manuals, operation guides, and fault cases. Each document can be divided into 10 document fragments.

[0137] Step 1.2 BGE-M3 Model Encoding

[0138] The BGE-M3 model is used to encode the corpora of the general basic layer and the industrial vertical layer, forming a hybrid knowledge representation system of "dense vectors + sparse vectors":

[0139] Suppose the document set is D = {d1, d2, ..., dn} m }, where m = 501000 is the total number of documents. The value of m depends on the size of the knowledge corpus and can be reduced. Its size directly affects the coverage of the subsequent knowledge base. The sparse coding capability of the BGE-M3 model is used to encode each document d i Encoded as a sparse vector v d (i) :

[0140]

[0141] Where T is the global term set, i.e., the unique token appearing in all documents, and its size varies with the number of documents and the complexity of the content; w d (i,k) For document d i Chinese term t k The weights, learned by the BGE-M3 model, are used to determine the relationship between the term and document d. Non-zero values ​​indicate the relationship between the term and document d. i It makes a semantic contribution; the sparse vector set V corresponding to the document set D can be generated using Formula 1:

[0142]

[0143] In the intensive encoding stage, a complex nonlinear transformation is used to transform each document d in the document set D using the BGE-M3 model. i Embedded into a 1024-dimensional dense vector space:

[0144]

[0145] Where f γ This represents a nonlinear transformation function for an embedded model, containing a multi-layer neural network structure, capable of transforming document d. i The text information is transformed into a high-dimensional vector representation to capture the deep semantic features in the document; di This indicates that document d i via f γ The resulting 1024-dimensional dense vector can be generated using Equation 3:

[0146]

[0147] The vector set E is directly imported into the FAISS database to build an efficient vector index, which is an L2 (Euclidean distance) index. FAISS enables fast subsequent retrieval by indexing E.

[0148] The external knowledge base is constructed through the "feature complementarity" of two types of vectors, Equation 2 and Equation 4: the sparse vector represented by Equation 2 achieves key feature localization through non-zero weight terms; the dense vector represented by Equation 4 captures deep associations through 1024-dimensional semantic embedding; the two together constitute the core data of the knowledge base, and knowledge matching is achieved through the collaboration of sparse retrieval and dense retrieval in subsequent retrieval.

[0149] Step 2: Joint Assessment of Token Confidence

[0150] Step 2.1 Initial Generation

[0151] Based on the input demo, question, and currently generated text (initially empty), LLM is invoked to generate initial text content. For example, in an industrial equipment fault diagnosis task, the demo might contain content such as "Problem: The temperature of the reactor is abnormally high; Output: Possible causes include heating system failure, temperature control sensor malfunction, and it is recommended to first check whether the heating tube is damaged, and then calibrate the sensor." The question is a user-proposed industrial-related requirement, such as "How to solve the problem of decreased machining accuracy of CNC machine tools." The question must be clearly and accurately stated, including key domain terms and requirements, to avoid ambiguity. The currently generated text is initially empty, and in subsequent iterations, it will be the text content generated in the previous round.

[0152] Assume the vocabulary is V, which contains all tokens that the model can recognize and generate, covering general vocabulary and a certain number of industry-specific terms; the initial content sequence generated by LLM is T = {t1, t2, ..., t}. n }, where t i This represents the token at position i in the sequence, and n = 64 is the maximum number of tokens that can be generated.

[0153] Step 2.2 Uncertainty Assessment

[0154] Calculate the logarithmic entropy S for each token in the sequence T = {t1, t2,..., t n} i Quantify the generated uncertainty:

[0155]

[0156] where p i (v) is the probability of generating token v at the i-th position in the sequence. For example, when generating the expression "the spindle speed of a CNC machine tool", the model has a relatively high p i (v) value for the token "spindle speed", while the p i (v) value for an irrelevant token (such as "apple") is close to 0; S i Converts the probability distribution output by the model into a quantifiable risk metric, with a value range of [0, +∞). The higher the S i value, the greater the uncertainty of the model's prediction for that token;

[0157] Step 2.3 Influence Assessment

[0158] Utilize the inherent self-attention mechanism of the Transformer architecture to construct an attention matrix:

[0159]

[0160] where Q is the query matrix, K represents the key-value matrix, and both are generated by the encoder or decoder of the Transformer architecture; d k represents the dimension of the key vector and is related to the hidden layer dimension of the model; Mask represents the masking function, which acts on the scaled dot product matrix W = QK T / √d k ; Specifically, when the number of rows of matrix W is equal to the number of columns, the masking function processes the elements of the square matrix W according to the following rules:

[0161]

[0162] Traverse the row index (i) and column index (j) of matrix W. When i < j, it means the current position is at a subsequent position, and the matrix element W i,j is assigned -∞. This operation can mask the upper triangular part of the square matrix; when i ≥ j, corresponding to the historical or current position, the matrix element W i,j The value remains unchanged. This operation ensures that when the model generates the current token, it only focuses on the tokens at historical and current positions and does not obtain information about future positions in advance, which conforms to the temporal characteristics of natural language generation. After the masking operation, the Softmax function is applied to each row of the matrix to obtain the attention weight matrix A. Accordingly, for token t i , its maximum attention value a max(i) is determined by screening the maximum value of A i,j in all cases where i < j, thereby quantifying the influence of t i on the subsequent context:

[0163] a max (i) = max i＜j A j,i (8)

[0164] a max(i) ranges from [0, 1]. The larger the value, the higher the attention weight of t i to a subsequent token, and the greater the influence on the subsequent context;

[0165] Step 2.3 Token confidence score calculation

[0166] Use the spaCy tool to filter out tokens with little semantic contribution, such as function words like "的" and "进行", to obtain the key token set T key , and combine the uncertainty and influence evaluation of Formula 1 and Formula 4 to define the token confidence score C(t i ):

[0167]

[0168] where[[ID=4l]] is an indicator function used to screen key tokens. It is 1 when t i belongs to T key and 0 otherwise, to exclude the interference of function words on confidence evaluation;

[0169] Step 2.4 External knowledge retrieval judgment

[0170] Set a threshold θ to quantitatively define the critical level of token confidence. If the calculated Token confidence score C(t i ) > θ, it indicates that the confidence of token t i is low. Record the current position i and perform query construction; if C(t i ) < θ, it indicates that token t i has a certain confidence and does not require external knowledge retrieval. It is directly concatenated to the current result;

[0171] The threshold θ needs to be determined experimentally based on the actual industrial scenario and model performance. If θ is set too low (e.g., 0.3), a large number of high-confidence tokens will be misjudged as low-confidence tokens, increasing the retrieval burden. If θ is set too high (e.g., 1.2), some low-confidence tokens will be missed, affecting the accuracy of the generated content. Generally, a value of 0.5-1.0 is more appropriate.

[0172] Step 3: Adaptive Query Generation

[0173] Step 3.1 Extract the attention weights of the preceding tokens

[0174] For judging token t with low confidence i Extract the preceding token sequence {t} from the attention matrix A in Formula 2. i-1 ,t i-2 The attention weights of {t1, ...,t2} are used to reflect the role of each preceding token in generating t. i The importance of process:

[0175] A i ={a i,1 ,a i,2 ,...,a i,i-1 } (10)

[0176] Among them, a i,j (j = 1, 2, ..., i-1) indicates that during the generation of token t i At that time, token t j For t i Attention weights, which are used to measure t j In t i The importance of the information provided during the generation process;

[0177] For example, if t i Given "bearing wear" (low confidence token), and the preceding token sequence "abnormal noise from CNC machine tool spindle, possibly caused by", then the attention weight A corresponding to "abnormal spindle noise" is... i,i-2 It will be significantly higher than the A corresponding to "possible reason". i,i-1 This indicates that "spindle noise" provides stronger semantic support for the conclusion of "bearing wear";

[0178] Step 3.2 Token Filtering

[0179] Sort the attention weights in descending order and select the n tokens with the highest weights to form a set:

[0180] T n ={t k |rank(a i,k )≤n} (11)

[0181] The value of n needs to be determined based on the specific task requirements. For example, if the task has a high requirement for the concentration of key contextual information, a smaller n can be selected; if more comprehensive contextual support is required, a larger n can be selected, but computational efficiency should also be considered to avoid excessive computational costs in subsequent processing due to an excessively large n. Generally, the value of n is in the range of 20-40.

[0182] Use the indicator function in Formula 5 to filter out the input key token set T. key Token T s :

[0183]

[0184] Step 3.3 Generation of the query term set

[0185] Extracting T from the vocabulary s The words corresponding to the tokens are arranged in the order of their appearance in the original text to construct a query term set W that reflects the actual information needs of LLM. q ;

[0186] Step 4: Integrated Retrieval Reordering Based on Knowledge Collaboration

[0187] Step 4.1 Calculation of keyword density and semantic feature complexity

[0188] Keyword density reflects the specialization and concentration of a query in its semantic information expression; for a given query Q = {q1, q2, ..., q...} n The spaCy model is used for word segmentation and part-of-speech tagging, and a vocabulary weight mapping table W is constructed. P Based on the lexical attributes, each word q i Weights are assigned to reflect the importance of different lexical attributes in semantic contribution; simultaneously, a custom terminology dictionary D is constructed. lex To assign higher weight to specialized terms and adapt to the needs of the field; the keyword density calculation formula is as follows:

[0189]

[0190] Where T(Q) is the set of effective query terms, that is, the set of terms selected from query Q using the SPacy tool that participate in the keyword density calculation. These terms can truly affect the semantic expression, specialization, and concentration of the query; |T(Q)| represents the number of terms contained in T(Q); w c (t) represents the custom term weight, a weight value specifically set for professional terms to reflect their importance in the semantic expression of a particular domain; w p (t) represents the lexical weight, which is assigned to lexical words based on their part of speech, reflecting the importance of lexical words in general semantic contribution. For the term matching indicator function, i.e., when t∈D lex hour =1, at this time w is used c (t) is used as the weight of the term to avoid unreasonable weight allocation of technical terms due to issues such as routine part-of-speech tagging; conversely, At this point, w is used. p (t) represents the weight of the word; ε is a very small positive number used to prevent the denominator of the formula from being 0; the output value is obtained by max(w p Normalization is mapped to the interval [0,1]. The closer KD(Q) is to 1, the higher the concentration of technical terms in the query, and the more it needs to rely on the strong term matching ability of sparse retrieval.

[0191] For example, to query Q = {"PLC controller", "fault", "troubleshooting"}, the industrial dictionary D... lex Containing "PLC controller", after spaCy word segmentation, T(Q) = {"PLC controller", "fault", "troubleshooting"}, with no stop words, W P In Chinese, nouns have a weight of 0.7, and verbs have a weight of 0.5; "PLC controller" is a term (lex=1), "fault" (noun, lex=0, w) p =0.7), "investigate" (verb, lex=0, w p =0.5); KD(Q) = 0.77 is calculated according to the formula, which indicates that the terminology concentration of this query is high and the term matching of sparse retrieval needs to be emphasized.

[0192] Semantic feature complexity measures the degree of query expansion in the semantic space and the semantic changes between layers, including differences between two layers and semantic diffusion; given a query Q = {q1,q2,....,q...} n After vectorization and extraction of hierarchical semantic features by the BGE-M3 model, H is defined. (l) Let h be the hidden state matrix of the l-th layer. l =Flatten(H (l) The hidden state vector is vectorized (Flatten represents the dimensionality reduction function). The focus is on three feature layers: the input layer, the intermediate layer, and the output layer. Here, "layer" refers to the network layers of the pre-trained BGE-M3 model. The input layer represents the model's initial hidden layer (layer 0), receiving text vectors processed by previous embedding layers as input and outputting surface features of the text (such as the semantics of the words themselves), containing a significant amount of original information, which can be used to capture surface features and local co-occurrence patterns. The intermediate layer represents the model's intermediate hidden layers (defaulting to layers 2, 4, 8, and 10), responsible for progressively abstracting and processing text features, outputting abstract features that incorporate contextual information, retaining some surface information while containing preliminary semantic associations, used to encode syntactic dependencies and phrase-level semantics. The output layer represents the model's final hidden layer (layer 12), outputting high-level semantic features processed by all layers, used to characterize logical structure and deep semantics. The semantic embedding of the text at each layer progressively abstracts from the surface to the deepest levels.

[0193] Interlayer difference ID(Q) is defined as the squared mean of the JS divergence between adjacent layers, and is used to measure the degree of variation in semantic representations between different layers:

[0194]

[0195] Where L=6 is the total number of layers, and intermediate layers can be added according to the specific scenario; P i h represents the hidden state vector of the i-th layer. i After normalization, Q i h represents the hidden state vector of layer i+1. i+1 The values ​​are normalized; semantic diffusion is used to evaluate the breadth of semantic distribution in multidimensional space, and all selected layers are connected as E = [h l0 ,h l1 ,...h lL ], and PCA is applied for dimensionality reduction, where semantic diffusion is defined as the Euclidean distance between the maximum and minimum points in the reduced dimensionality space:

[0196]

[0197] PCA(E) i Let represent the i-th embedding vector after dimensionality reduction; the semantic feature complexity is defined as the weighted sum of inter-layer differences and semantic diffusion:

[0198] SC(Q)=αLD(Q)+βSD(Q) (16)

[0199] α and β are weight parameters, with default values ​​of 0.6 and 0.4. The core advantage of these values ​​lies in balancing the priority of core semantics with the need for extended associations. They can be dynamically adjusted according to the scenario. If the task focuses more on the hierarchical depth of text semantics, the value of α can be increased; if the task focuses more on the richness of text semantics, the value of β can be increased.

[0200] For example, to query Q = {"motor", "overload", "troubleshooting"}, we take the BGE-M3 model with four layers (L = 4): input layer (l = 0), intermediate layer (l = 4, 8), and output layer (l = 12). After normalization, the JS divergence of adjacent layers is JS(0,4) = 0.15, JS(4,8) = 0.2, and JS(8,12) = 0.18. According to the inter-layer difference formula, ID(Q) = 0.03. After reducing the dimensionality of the four hidden state vectors (k = 2), the coordinates of each layer are (1.2, 2.3), (2.5, 3.1), (0.8, 1.5), and (3.2, 4.0). The maximum Euclidean distance SD(Q) is calculated to be 3.8, so SFC(Q) = 1.53. This indicates that this query requires intensive retrieval to capture deep semantic associations.

[0201] Step 4.2 External Knowledge Retrieval

[0202] The sparse retrieval tool SR is built upon the sparse coding capabilities of the BGE-M3 model to achieve efficient information retrieval; for the generated query term set W... q For each query q in the algorithm, q is first sparsely encoded to obtain a sparse vector v. q :

[0203]

[0204] Its structure and parameters are consistent with the document vector in Formula 1, retaining only terms with non-zero weights; SR achieves retrieval by calculating the similarity score between the document and the query, and its core is the summation of the product of the weights of co-occurring terms:

[0205]

[0206] Where s(q,d) i ) indicates query q and document d i The similarity score between them, T q,i The set of co-occurring terms, i.e., terms that appear in both query q and document d. i The set of terms in w q (k) This represents the weight of term k in query q. A higher weight indicates that the term plays a more crucial role in expressing the core semantics of query q; w d (k) Document d represents document d i The weight of term k in document d; the higher the weight, the more relevant the term is to document d. i The closer the association, the better; the weighted product reflects the synergistic contribution of "query term importance" and "document term relevance", and the summation yields the overall matching degree; the documents are sorted in descending order of similarity score and a certain number of documents (usually 5-20) are selected as the final sparse search results as needed;

[0207] Similar to Equation 2, the dense retrieval tool DR is built upon the dense encoding capability of the BGE-M3 model. First, each query q is encoded and embedded into a 1024-dimensional dense vector space. A similarity score is calculated based on the cosine similarity between the vectors. Assuming the encoded document vector is 'a' and the query vector is 'b', the cosine similarity can be expressed as:

[0208]

[0209] Where a i and b i Let i represent the i-th component of vectors a and b; sort the search results in descending order based on cosine similarity, and select a certain number of documents (usually 5-20) as the final sparse search results as needed;

[0210] The final search results from the two search engines are kept consistent to ensure that the number of search results of the two types are roughly equal during subsequent fusion, thus avoiding the excessive proportion of one type of search result in the fusion process and affecting the fusion effect.

[0211] Step 4.3 Dynamic Weight Adjustment

[0212] Based on the keyword density KD(Q) and semantic feature complexity SFC(Q) calculated using Formulas 9 and 12, the weights of the sparse search engine SR and the dense search engine DR are dynamically adjusted:

[0213]

[0214] Where w S With w D w represents the weights of the sparse search engine SR and the dense search engine DR, respectively. b,S and w b,D This represents the initial weight, with a value range of [0,1]. The initial setting defaults to 0.5 and 0.5, which can be adjusted according to the actual application scenario. If the business scenario relies more on keyword matching-based retrieval, the initial weight w of the sparse search engine can be increased. b,S Increase the initial weight w of the dense search engine if the task focuses on precise semantic retrieval. b,D γ1 and γ2 represent weight parameters controlling the relative importance of KD and SFC, with a fixed value of 0.2, which is the optimal result obtained from experiments; then normalization is performed:

[0215]

[0216] Among them w' S and w' D These represent the weights of the normalized sparse search engine SR and the dense search engine DR, respectively, ensuring that the sum of the weights is 1;

[0217] Step 4.4 Retrieval Result Fusion and Re-ranking

[0218] Based on the weights of SR and DR obtained from Formula 19, the retrieval results of SR and DR are further obtained using a weighted inverse ranking fusion (WRRF) retrieval tool; assuming that the merged and deduplicated SR and DR retrieval results are a set D = {d1, d2, ..., d...} n }, where n represents the number of documents, for document d i Calculate the fusion score:

[0219]

[0220] Where c=60 is a smoothing constant, which can avoid the influence of abnormal denominators and smooth sorting positions on the score, and can also balance the sorting contributions of different retrieval machines, making the fused score more stable and reasonable; rank S (d i ) indicates document d i Rank is the sorting position in the search results of the SR search engine. D (d i ) indicates document d i The sorting position in the DR search results; and The indicator function is set to 1 when a document belongs to the corresponding retrieval result, ensuring that only documents existing in the corresponding retrieval result have their ranking and other information included in the retrieval's contribution to the fusion score. After fusion, a cross-coding re-ranking model is constructed using a pre-trained ms-marco-MiniLM-L-6-v2 model to achieve deep semantic interaction between queries and documents. For query q and document d, a joint representation is constructed using a pre-trained language model M.

[0221]

[0222] in This indicates a concatenation operation, where [CLS] is the classification symbol and [SEP] is the delimiter. This architecture employs a multi-head cross-attention mechanism to accurately capture fine-grained interaction relationships. The relevance score s(q,d) is generated by the classification head C.

[0223]

[0224] Where dim(C) represents the dimension of the classifier head C; when dim(C) = 1, w and b represent the weight vector and bias scalar, respectively; when dim(C) > 1, W c and b c These correspond to the weight matrix and bias vector in the multidimensional case, respectively; softmax(·)[1] represents the positive class probability value in the softmax output;

[0225] For the candidate document set fused by Formula 15, the relevance score s(q,d) between each document and query q is calculated. i The sorting process is performed to reorder the documents, ultimately returning k (default is 5) documents as the final retrieved knowledge set D. * :

[0226] D * ={d i |rank(d i )≤k} (25)

[0227] Where rank(·) represents the relevance score s(q,d) i Perform sorting operations;

[0228] Step 5: Iterative Calibration and Dynamic Optimization of Industrial Knowledge

[0229] Based on the retrieved knowledge set D * The generated content is iteratively optimized to establish a verifiable knowledge integration loop, specifically including:

[0230] Step 5.1 Construct an industrial-scenario-based calibration mechanism

[0231] Define the calibration function:

[0232] T * =G(P;η) (26)

[0233] Where T * G represents the calibrated text, G is the LLM's generation function, P represents the structured prompt information generated by the prompt template, which includes the question, retrieved knowledge, current text and correction instructions, and is the input carrier that triggers the model to perform calibration inference. η is a set of learnable parameters inside the LLM, which includes the model's weights and biases.

[0234] Step 5.2 Knowledge Integration

[0235] To achieve efficient knowledge fusion, a dual mechanism of contextual enhancement and prompting engineering is introduced; in the contextual semantic information enhancement stage, the retrieved knowledge set D... * It is transformed into linear context information, specifically in the following form:

[0236]

[0237] Where C represents the enhanced context, and [i] represents document d. i index tags, This indicates a concatenation operation. This means adding an index tag before each document, which preserves the original order of the documents and identifies the source of each segment through the index, making it easier for subsequent traceability verification;

[0238] Hint engineering constructs a complete hint message P, which combines the question q, the reinforcement context C, and the current text T. t Integrate into a logical hierarchical structure to form an input format that the model can recognize, ensuring that knowledge is effectively utilized during the generation process;

[0239] Step 5.3 Iterative optimization

[0240] Iterative optimization follows the incremental update principle. Let the text at time t be T. t Based on the prompt message P, the LLM is invoked to generate the correction amount Δ. t Update the generated text:

[0241]

[0242] Where the correction amount Δ t Defined as:

[0243] Δ t =arg max δ P(δ∣P t ;η) (29)

[0244] Where η is consistent with Equation 26, representing the parameters of LLM, P(δ|P t η) represents the generation probability of the correction amount; this means that at time point t, the system will select the correction content with the highest selection probability as the update amount based on the prompt information P and the model parameter η.

[0245] This iterative optimization process is implemented through a closed-loop mechanism: In the initial stage, T0 represents the original text generated by the model, and the initial prompt information P0 is constructed based on T0; in each iteration, T0 is first optimized... t Perform joint evaluation of token confidence to determine if there is any content that conflicts with knowledge; if a low-confidence token is detected, perform relevant knowledge retrieval and call the calibration function T. * Update the text to T t+1 Subsequently based on T t+1 Reconstruction prompt message P t+1 Then proceed to the next iteration, and this process continues until the termination condition is met.

[0246] Step 5.4 Iteration Termination Control

[0247] To balance calibration effectiveness and computational efficiency, a multi-condition collaborative termination strategy is adopted, which explicitly includes three triggering scenarios:

[0248] ① Natural termination: When a predefined end symbol ( <eos>) appears in the correction amount Δ t When the time is right, it indicates that the model has determined that the current text does not require further correction;

[0249] ② Length constraint: When the text length |T t |Exceeds the maximum threshold L max (Default is 64, can be adjusted flexibly) When the iteration is terminated, it is to avoid the accumulation of redundant information;

[0250] ③ Stable convergence: When the distance metric between two consecutive corrections is less than the threshold ∈, the following condition is satisfied:

[0251] d(Δ t ,Δ t-1 )<∈ (30)

[0252] Where d represents the distance metric function of the correction amount (here it is based on the number of tokens, but in practical applications it can be flexibly selected, such as based on edit distance or similarity), ∈ takes a value of 1-2, representing the difference of tokens, and its value depends on the selection of d, indicating that the text has reached a stable state, further iterations are unlikely to significantly improve the quality, and the calibration process tends to converge;

[0253] Step 6: Output the semantic command interaction results for the industrial vertical domain.

[0254] The final calibration text T at the end of the iteration final As the output.< / eos> < / eos>

Claims

1. A knowledge-guided retrieval enhancement generation method for data-scarce industrial verticals, characterized in that: The method includes the following steps: Step 1: Building an external knowledge base Step 1.1 Selection and Processing of Knowledge Corpus The general foundational knowledge corpus is based on 500,000 English Wikipedia text paragraphs; the industrial vertical knowledge corpus comes from scattered text content in the field, including technical manuals, operation guides, and failure cases. Step 1.2 BGE-M3 Model Encoding The BGE-M3 model is used to encode the corpora of the general basic layer and the industrial vertical layer, forming a hybrid knowledge representation system of "dense vectors + sparse vectors": Suppose the document set is D = {d1, d2, ..., dn} m }, where m is the total number of documents, and the sparse coding capability of the BGE-M3 model is used to encode each document d i Encoded as a sparse vector v d (i) : Where T is the global term set, that is, the unique token appearing in all documents, which is the smallest unit or basic element of the text to be processed, and w d (i,k) For document d i Chinese term t k The weights, learned by the BGE-M3 model, are used to determine the relationship between the term and document d. Non-zero values ​​indicate the relationship between the term and document d. i It makes a semantic contribution; the sparse vector set V corresponding to the document set D can be generated using Formula 1: In the intensive encoding stage, a complex nonlinear transformation is used to transform each document d in the document set D using the BGE-M3 model. i Embedded into a 1024-dimensional dense vector space: e di =f γ (d i ) (3) Where f γ The nonlinear transformation function representing the embedded model; e di Document d represents document d i via f γ The resulting 1024-dimensional dense vector; the dense vector set E corresponding to the document set D can be generated using Formula 3: The vector set E is directly imported into the FAISS database to build an efficient index. FAISS achieves fast subsequent retrieval by indexing E. The external knowledge base is constructed through the "feature complementarity" of two types of vectors, Equation 2 and Equation 4: the sparse vector represented by Equation 2 achieves key feature localization through non-zero weight terms; the dense vector represented by Equation 4 captures deep associations through 1024-dimensional semantic embedding; the two together constitute the core data of the knowledge base, and knowledge matching is achieved through the collaboration of sparse retrieval and dense retrieval in subsequent retrieval. Step 2: Joint Assessment of Token Confidence Step 2.1 Initial Generation Based on the input demo, question, and currently generated text, the initial generation is empty. The LLM is invoked to generate initial text content. Assuming the vocabulary is V, the initial content sequence generated by the LLM is T = {t1, t2, ..., t...}. n }, where t j This represents the token at position j in the sequence, and n is the maximum number of tokens that can be generated, which is usually 64. Step 2.2 Uncertainty Assessment For the sequence T = {t1, t2, ..., t n For each token in}, calculate the logarithmic entropy S. i Uncertainty generated by quantification: Where p i (v) represents the probability of generating token v at position i in the sequence; Step 2.3 Impact Assessment By leveraging the inherent self-attention mechanism of the Transformer architecture, an attention matrix is ​​constructed: Where Q is the query matrix, K represents the key matrix, and d k The dimension of the key vector is represented by , and Mask represents the masking function, which is applied to the scaled dot product matrix W = QK. T / √d k Specifically, when the number of rows in matrix W equals the number of columns, the masking function processes the elements of matrix W according to the following rules: Traverse the row index (i) and column index (j) of matrix W. When i < j, it indicates that the current position is in the subsequent position, and the matrix element W i,j is assigned -∞. This operation can mask the upper triangular part of the square matrix; when i ≥ j, corresponding to the historical or current position, the value of the matrix element W i,j remains unchanged. After the masking operation, apply the Softmax function to each row of the matrix to obtain the attention weight matrix A. Accordingly, for token t i , its maximum attention value a max(i) is determined by screening the maximum value of A i,j in all cases where i < j, thereby quantifying the influence of t i on the subsequent context: a max (i)=max i<j A j,i (8) Step 2.3 Token Confidence Score Calculation The spaCy tool is used to filter tokens with low semantic contribution, resulting in a key token set T. key Combining Formula 1 and Formula 4, the uncertainty and impact assessment defines the token confidence score C(t). i ): C(t i )=S i ·a max (i)·I(t i ∈T key ) (9) Where I is an indicator function, when t i Belongs to T key The value is 1 if it is true, and 0 otherwise. Step 2.4 External Knowledge Retrieval Judgment A threshold θ is set to quantify the critical level of token confidence. If the calculated token confidence score C(t) is... i When )>θ, it indicates that token t i If the confidence level is low, record the current position i and construct the query; if C(t) i )<θ, indicating that token t i There is a certain level of confidence, so there is no need to perform external knowledge retrieval; the result can be directly concatenated into the current result. The threshold θ is set between 0.5 and 1.

0. Step 3: Adaptive Query Generation Step 3.1 Extract the attention weights of the preceding tokens For judging token t with low confidence i Extract the preceding token sequence {t} from the attention matrix A in Formula 2. i-1 ,t i-2 The attention weights of {t1, ...,t2} are used to reflect the role of each preceding token in generating t. i The importance of process: A i ={a i,1 ,a i,2 ,…,a i,i-1 } (10) Among them, a i,j (j = 1, 2, ..., i-1) indicates that during the generation of token t i At that time, token t j For t i Attention weights, which are used to measure t j In t i The importance of the information provided during the generation process; Step 3.2 Token Filtering Sort the attention weights in descending order and select the n tokens with the highest weights to form a set: T n ={t k ∣rank(a i,k )≤n} (11) The value of n ranges from 20 to 40; Use the indicator function in Formula 5 to filter out the input key token set T. key Token T s : T s ={t k ∈T n ∣I(t k ∈T key )=1} (12) Step 3.3 Generation of the query term set Extracting T from the vocabulary s The words corresponding to the tokens are arranged in the order of their appearance in the original text to construct a query term set W that reflects the actual information needs of LLM. q ; Step 4: Integrated Retrieval Reordering Based on Knowledge Collaboration Step 4.1 Calculation of keyword density and semantic feature complexity Given a query Q = {q1, q2, ..., q...} n The spaCy model is used for word segmentation and part-of-speech tagging, and a vocabulary weight mapping table W is constructed. P Based on the lexical attributes, each word q i Weights are assigned to reflect the importance of different lexical attributes in semantic contribution; simultaneously, a custom terminology dictionary D is constructed. lex To assign higher weight to specialized terms and adapt to the needs of the field, the keyword density KD(Q) calculation formula is as follows: Where T(Q) is the set of effective query terms, that is, the set of terms selected from query Q using the SPacy tool that participate in the keyword density calculation. These terms can truly affect the semantic expression, specialization, and concentration of the query; |T(Q)| represents the number of terms contained in T(Q); w c (t) represents the custom term weight, a weight value specifically set for professional terms to reflect their importance in the semantic expression of a particular domain; w p (t) represents the lexical weight, which is assigned to words based on their part of speech, reflecting the importance of each word in its general semantic contribution; I lex (t) is the term matching indicator function, i.e., when t∈D lex Time I lex (t) = 1, at this time w is used c (t) serves as the weight of that word; conversely, I lex (t) = 0, at this time w is used p (t) represents the weight of the word; ε is a very small positive number used to prevent the denominator of the formula from being 0; the output value is obtained by max(w p Normalization maps to the interval [0,1]. Semantic feature complexity measures the degree of query expansion in the semantic space and the semantic changes between layers, including differences between two layers and semantic diffusion; given a query Q = {q1,q2,....,q...} n After vectorization and extraction of hierarchical semantic features by the BGE-M3 model, H is defined. (l) Let h be the hidden state matrix of the l-th layer. l =Flatten(H (l) The vectorized hidden state vector is represented by , and Flatten represents the dimensionality reduction function. The focus is on three feature layers: the input layer, the intermediate layer, and the output layer. Here, "layer" refers to the network layers of the pre-trained BGE-M3 model. The input layer represents the initial hidden layer (layer 0), receiving the text vector processed by the previous embedding layers as input and outputting the surface features of the text. The intermediate layers (layers 2, 4, 8, and 10) are responsible for progressively abstracting and processing the text features, outputting abstract features that incorporate contextual information, used to encode syntactic dependencies and phrase-level semantics. The output layer represents the final hidden layer (layer 12), outputting high-level semantic features processed by all layers, used to characterize logical structure and deep semantics. The semantic embedding of the text at each layer progressively abstracts from the surface to the deepest levels. Interlayer difference ID(Q) is defined as the squared mean of the JS divergence between adjacent layers, and is used to measure the degree of variation in semantic representations between different layers: Where L = 6 is the total number of floors, P i h represents the hidden state vector of the i-th layer. i After normalization, Q i h represents the hidden state vector of layer i+1. i+1 The values ​​are normalized; semantic diffusion is used to evaluate the breadth of semantic distribution in multidimensional space, and all selected layers are connected as E = [h l0 ,h l1 ,...h lL ], and PCA is applied for dimensionality reduction, where semantic diffusion is defined as the Euclidean distance between the maximum and minimum points in the reduced dimensionality space: PCA(E) i Let represent the i-th embedding vector after dimensionality reduction; the semantic feature complexity is defined as the weighted sum of inter-layer differences and semantic diffusion: SC(Q)=αLD(Q)+βSD(Q) (16) Where α and β are weight parameters, with default values ​​of 0.6 and 0.4; Step 4.2 External Knowledge Retrieval The sparse retrieval tool SR is built upon the sparse coding capabilities of the BGE-M3 model to achieve efficient information retrieval; for the generated query term set W... q For each query q in the algorithm, q is first sparsely encoded to obtain a sparse vector v. q : Its structure and parameters are consistent with the document vector in Formula 1, retaining only terms with non-zero weights; SR achieves retrieval by calculating the similarity score between the document and the query, and its core is the summation of the product of the weights of co-occurring terms: Where s(q,d) i ) indicates query q and document d i The similarity score between them, T q,i The set of co-occurring terms, i.e., terms that appear in both query q and document d. i The set of terms in w q (k) This represents the weight of term k in query q. A higher weight indicates that the term plays a more crucial role in expressing the core semantics of query q; w d (k) Document d represents document d i The weight of term k in document d; the higher the weight, the more relevant the term is to document d. i The closer the association, the better; the weighted product reflects the synergistic contribution of "query term importance" and "document term relevance", and the summation yields the overall matching degree; the documents are sorted in descending order of similarity score and 5-20 documents are selected as the final sparse search results; The dense retrieval tool (DR) is built upon the dense encoding capabilities of the BGE-M3 model. First, each query q is encoded and embedded into a 1024-dimensional dense vector space. A similarity score is calculated based on the cosine similarity between the vectors. Assuming the encoded document vector is 'a' and the query vector is 'b', the cosine similarity is expressed as: Where a i and b i Let i represent the i-th component of vectors a and b; sort the search results in descending order based on cosine similarity, and select 5-20 documents as the final sparse search results; Step 4.3 Dynamic Weight Adjustment Based on the keyword density KD(Q) and semantic feature complexity SFC(Q) calculated using Formulas 9 and 12, the weights of the sparse search engine SR and the dense search engine DR are dynamically adjusted: Where w S With w D w represents the weights of the sparse search engine SR and the dense search engine DR, respectively. b,S and w b,D This represents the initial weight, with a value range of [0,1]. The initial setting defaults to 0.5 and 0.5, which can be adjusted according to the actual application scenario. If the business scenario relies more on keyword matching-based retrieval, the initial weight w of the sparse search engine can be increased. b,S Increase the initial weight w of the dense search engine if the task focuses on precise semantic retrieval. b,D γ1 and γ2 represent weight parameters controlling the relative importance of KD and SFC, with a fixed value of 0.2, which is the optimal result obtained from experiments; then normalization is performed: Among them w' S and w' D These represent the weights of the normalized sparse search engine SR and the dense search engine DR, respectively, ensuring that the sum of the weights is 1; Step 4.4 Retrieval Result Fusion and Re-ranking Based on the weights of SR and DR obtained from Formula 19, a weighted inverse sorting fusion algorithm is further used to fuse the retrieval results of the retrieval machines SR and DR; assuming that the merged and deduplicated SR and DR retrieval results are a set D = {d1, d2, ..., d...} n }, where n represents the number of documents, for document d i Calculate the fusion score: Where c = 60 is the smoothing constant; rank S (d i ) indicates document d i Rank is the sorting position in the search results of the SR search engine. D (d i ) indicates document d i The sorting position in the DR search results; I di∈SR with I di∈DR The indicator function is set to 1 when the document belongs to the corresponding retrieval result, ensuring that only documents present in the corresponding retrieval result are included in the contribution calculation of that retrieval to the fusion score. After fusion, a cross-coding re-ranking model is constructed using a pre-trained ms-marco-MiniLM-L-6-v2 model to achieve deep semantic interaction between the query and the document. For query q and document d, a joint representation is constructed using a pre-trained language model M. in This indicates a concatenation operation, where [CLS] is the classification symbol and [SEP] is the delimiter. This architecture employs a multi-head cross-attention mechanism to accurately capture fine-grained interaction relationships. The relevance score s(q,d) is generated by the classification head C. Where dim(C) represents the dimension of the classifier head C; when dim(C) = 1, w and b represent the weight vector and bias scalar, respectively; when dim(C) > 1, W c and b c These correspond to the weight matrix and bias vector in the multidimensional case, respectively; softmax(·)[1] represents the positive class probability value in the softmax output; For the candidate document set fused by Formula 15, the relevance score s(q,d) between each document and query q is calculated. i The sorting process is performed to reorder the documents, ultimately returning k documents (k = 5), which serve as the final knowledge set D obtained from the retrieval. * : D * ={d i ∣rank(d i )≤k} (25) Where rank(·) represents the relevance score s(q,d) i Perform sorting operations; Step 5: Iterative Calibration and Dynamic Optimization of Industrial Knowledge Based on the retrieved knowledge set D * The generated content is iteratively optimized to establish a verifiable knowledge integration loop, specifically including: Step 5.1 Construct an industrial-scenario-based calibration mechanism Define the calibration function: T * =G(P;η) (26) Where T * G represents the calibrated text, G is the LLM's generation function, P represents the structured prompt information generated by the prompt template, which includes the question, retrieved knowledge, current text and correction instructions, and is the input carrier that triggers the model to perform calibration inference. η is a set of learnable parameters inside the LLM, which includes the model's weights and biases. Step 5.2 Knowledge Integration During the contextual semantic information enhancement stage, the retrieved knowledge set D * It is transformed into linear context information, specifically in the following form: Where C represents the enhanced context, and [i] represents document d. i index tags, This indicates a concatenation operation. This means adding an index tag before each document, which preserves the original order of the documents and identifies the source of each segment through the index, making it easier for subsequent traceability verification; Hint engineering constructs a complete hint message P, which combines the question q, the reinforcement context C, and the current text T. t Integrate into a logical hierarchical structure to form an input format that the model can recognize, ensuring that knowledge is effectively utilized during the generation process; Step 5.3 Iterative optimization Iterative optimization follows the incremental update principle; let the text at time t be T. t Based on the prompt message P, the LLM is invoked to generate the correction amount Δ. t Update the generated text: Where the correction amount Δ t Defined as: △ t =argmax δ P(δ∣P t ;η) (29) Where η is consistent with Equation 26, representing the parameters of LLM, P(δ|P t η) represents the generation probability of the correction amount; this means that at time point t, the correction content with the highest selection probability will be selected as the update amount based on the prompt information P and the model parameter η. This iterative optimization process is implemented through a closed-loop mechanism: In the initial stage, T0 represents the original text generated by the model, and the initial prompt information P0 is constructed based on T0; in each iteration, T0 is first optimized... t Perform joint evaluation of token confidence to determine if there is any content that conflicts with knowledge; if a low-confidence token is detected, perform relevant knowledge retrieval and call the calibration function T. * Update the text to T t+1 Subsequently based on T t+1 Reconstruction prompt message P t+1 Then proceed to the next iteration, and this process continues until the termination condition is met. Step 5.4 Iteration Termination Control To balance calibration effectiveness and computational efficiency, a multi-condition collaborative termination strategy is adopted, which explicitly includes three triggering scenarios: ① Natural termination: When a predefined end symbol appears in the correction amount Δ t When the time is right, it indicates that the model has determined that the current text does not require further correction; ② Length constraint: When the text length |T t |Exceeds the maximum threshold L max At that time, L max The iteration is terminated when the value is 64 to avoid the accumulation of redundant information; ③ Stable convergence: When the distance metric between two consecutive corrections is less than ∈, that is: Where d represents the distance metric function of the correction amount, taking a value of 1-2, representing the difference of the tokens. Its value depends on the selection of d, indicating that the text has reached a stable state and the calibration process tends to converge. Step 6: Output the semantic command interaction results for the industrial vertical domain. The final calibration text T at the end of the iteration final As the output.

Images