Intelligent question answering system based on large language model

Abstract

The invention discloses an intelligent question answering system based on a large language model, and relates to the technical field of natural language processing and information retrieval. According to the system, a retrieval enhancement generation (RAG) and knowledge reordering (Re-rankig) technology is utilized, and the knowledge processing capacity for mechanical manufacturing process design, industrial standards and related specifications is improved. The system is mainly composed of four parts, wherein the first part is used for constructing a knowledge base covering machine manufacturing knowledge; in the second part, a LoRA + fine-tuned Qwen2-7B model is used for guaranteeing high-precision requirements such as process parameter calculation; in the third part, the RAG technology is integrated to effectively retrieve and process domain knowledge; and 4, introducing a reordering model to optimize answers. The system is remarkably superior to a traditional large language model LLM in BLEU-4 and ROUGE text matching evaluation indexes, the reliability and speciality of application of the system in the mechanical manufacturing field are verified, an effective scheme is provided for intelligent question answering in the field, and the prospect is wide.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing and information retrieval, and more specifically, to an intelligent question-answering system based on a large language model. Background Art

[0002] The machinery manufacturing industry is the backbone of modern industry, covering complex processes such as product design, material processing and equipment assembly, providing support for key industries such as automobiles, aerospace and energy, and promoting technological innovation and economic development. As the manufacturing industry transforms towards intelligence and digitalization, engineers and managers are increasingly demanding timely access to accurate knowledge. However, knowledge in this field is highly specialized, complex and fragmented, and is distributed in technical documents, standards and specifications, and patent literature. Traditional knowledge acquisition methods such as manual query and document retrieval are time-consuming and labor-intensive, cannot meet real-time needs, and reduce work efficiency.

[0003] In order to improve the efficiency of knowledge acquisition, mechanical manufacturing companies urgently need question-answering systems that can quickly and accurately answer professional technical questions. Such systems must understand domain terminology and handle complex technical issues to provide high-quality answers to support design, manufacturing, and maintenance decisions. This has prompted the research on vertical domain question-answering systems based on advanced natural language processing technology. In this context, although LLM (large language model) performs well in open domain question answering, it faces challenges in accuracy and professional application in professional fields such as mechanical manufacturing.

[0004] Although closed-source models such as GPT-3 and ChatGPT have demonstrated excellent capabilities through pre-training and fine-tuning, their lack of transparency, controllability, and high resource requirements limit their application in high-security and customized environments. This study chose to use open-source models to address the shortcomings of closed-source models. Open-source models can be flexibly deployed and can be customized according to specific needs, thus better adapting to the question-answering needs of professional fields.

[0005] To make up for the lack of LLM in domain knowledge coverage, RAG technology came into being. RAG combines information retrieval and generation models, uses the retrieval system to extract relevant document fragments from external knowledge bases, and generates more accurate and context-related answers through LLM processing. This method improves the accuracy and relevance of answers to complex queries in professional fields, but still needs to be optimized in terms of retrieval and generation integration, real-time performance, and domain knowledge understanding. Summary of the invention

[0006] In view of the deficiencies in the prior art, the object of the present invention is to provide an intelligent question-answering system based on a large language model.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] An intelligent question-answering system based on a large language model, comprising a knowledge base for the field of mechanical manufacturing, the knowledge base covering mechanical manufacturing process design, industry standards and related specifications;

[0009] The system optimizes the accuracy and professionalism of answers by combining the retrieval enhancement generation (RAG) technology and the re-ranking (Re-ranking) technology. The steps are as follows:

[0010] Step 1: Build a knowledge base that contains diverse knowledge in the field of mechanical manufacturing, including process design, material properties, and equipment operation information, to support the answering of a wide range of field questions;

[0011] Step 2: Fine-tune a variety of open source large models and train them according to the complexity and high-precision requirements of the mechanical manufacturing field to ensure that the models have strong reasoning and computing capabilities. Among them, the Qwen2-7B model fine-tuned with LoRA+ has the best effect;

[0012] Step 3: Use RAG technology to effectively retrieve and process domain knowledge from the knowledge base, especially numerical tables and complex technical specifications, to obtain preliminary answers;

[0013] Step 4: Use Re-ranking technology to optimize the retrieved results to generate more professional and accurate answers, ensuring that the output content meets the requirements of the mechanical manufacturing field.

[0014] Preferably, in step 1, knowledge from mechanical manufacturing process design, industry standards and related specifications are integrated to form a structured knowledge base, which is convenient for the system to call during the answering process; data collection covers multiple important sources in the field of mechanical manufacturing process design, ensuring the advantages of the question-answering system in professionalism and accuracy.

[0015] Preferably, in step 2, the large model uses a specific data set for the mechanical manufacturing field for pre-training and supervised fine-tuning during the training process to improve the model's ability to understand and apply domain knowledge;

[0016] The Qwen2-7B model is an open source large model. After comparing multiple open source large models, it was found that the Qwen2-7B model using LoRA+ fine-tuning has the best effect in the field of mechanical manufacturing; the Qwen2-7B model has a large parameter scale and can handle complex tasks;

[0017] The core architecture of the large model is the Transformer-based autoregressive model. The autoregressive model is a sequence generation model that depends on the previously generated elements when generating each element of the sequence. It has the ability to predict the i-th token based on the previous i-1 tokens:

[0018] P(ω i ω1,ω2,…,ω i-1 ;θ)

[0019] The most traditional causal modeling method is used in the pre-training process, and the model is optimized in an unsupervised manner by minimizing the log-likelihood of the following formula:

[0020]

[0021] Among them, ω1, ω2, …, ω i-1 is an input sequence consisting of i words;

[0022] During the fine-tuning phase, each data sample includes an "instruction" field that describes the specific requirements of the task; an "input" field that provides the input content of the model; and an "output" field that contains the target output that the model should generate;

[0023] LoRA is a method for fine-tuning large models by adding a low-rank matrix called an adapter to the weights of a pre-trained model, rather than adjusting all model parameters; this method can effectively adapt to new tasks while keeping the computational cost low;

[0024] When the original LoRA processes a model with a large width, the fine-tuning effect is not good because the adapter matrices A and B are updated using the same learning rate; this is because in a large-width network, using the same learning rate is not conducive to effective feature learning; to solve this problem, the proposed new method LoRA+ sets different learning rates for the adapter matrices A and B and maintains a fixed ratio. It is recommended to set the learning rate of B to λ times the learning rate of A and λ>1, thereby improving the feature learning efficiency; through a large number of experiments, LoRA+ can improve performance and fine-tuning speed while maintaining the same computational cost as LoRA; using the LoRA+ fine-tuning method, and according to the rule of thumb, let λ=2 4 =16.

[0025] Preferably, in step 3, the RAG technology screens out the most relevant information from the knowledge base through a correlation analysis of the question input by the user, thereby providing a preliminary basis for the answer;

[0026] RAG technology involves the Embedding model and vector database. The Embedding model used is the BGE M3e model, which is an open source model that can handle input granularity at different levels, from "sentences" to "documents", and supports dense retrieval, vocabulary retrieval, and multi-vector retrieval.

[0027] Dense retrieval transforms the input query q into hidden states H based on the text encoder q , and use the normalized hidden state with a special tag “[CLS]” to represent the embedding representation of the query: q =norm(H q [0]); Similarly, we can get the embedding representation of paragraph p: p =norm(H p [0]); therefore, the relevance score between the query and the passage is represented by two embeddings e p and e q The inner product is measured as: s dense ← <e p ,e q >

[0028] The embedding output from the lexical search is used to evaluate the importance of each term for lexical level search; for each term t in the query, its weight is calculated as Among them, W lex ∈R d×1 is a matrix used to map hidden states to floating point numbers; if the same term t appears multiple times in the query, only its maximum weight is retained; the weights of terms in the paragraph are calculated in the same way; based on these term weights, the relevance score between the query and the paragraph is calculated by the joint importance of the co-occurring terms, denoted as q∩p:

[0029] On top of dense retrieval, the multi-vector approach uses full output embeddings to represent the query and the paragraph: Where W mul ∈R d×d is a trainable projection matrix; and late interactions are used to compute refined relevance scores: N and M represent the length of the query and paragraph, respectively;

[0030] The multifunctional nature of the BGE M3e model allows the retrieval process to be performed in a hybrid manner; first, candidate results can be retrieved using each method separately; then, the final retrieval results are re-ranked based on the combined relevance scores: rank ←s dense +s lex +s mul ;

[0031] The training goal of the embedding model is to distinguish positive samples from negative samples; for each retrieval method, it is expected to assign higher scores to positive samples of the query and lower scores to negative samples; therefore, the training process aims to minimize the InfoNCE loss, the general form of which is:

[0032]

[0033] where p * and p' represent the positive and negative samples of query q respectively; s(·) is {s dense (·),s lex (·),s mul (·)};

[0034] The training objectives of different retrieval methods may conflict with each other, and multi-objective training may be detrimental to the quality of embedding; therefore, based on the principle of ensemble learning, the training process is unified on the basis of self-knowledge distillation, combining the predictions of different retrieval methods to provide a more accurate relevance score; the simplest form is to sum up the different prediction scores:

[0035] s inter ←s dense +s lex +s mul

[0036] In previous studies, the training quality of embedding models can benefit from knowledge distillation, which utilizes fine-grained soft labels from another ranking model; the BGE M3e model uses an ensemble score s inter As a teacher, I modified the loss function of each retrieval method:

[0037] L' * ←-p(s inter )*logp(s * )

[0038] where p(·) is the softmax activation; s * Yes dense 、s lex and mul Any one of them; further integrate and normalize the modified loss function as follows:

[0039] L'←(L' dense +L' lex +L' mul ) / 3

[0040] Finally, the final loss function of self-knowledge distillation is derived through the linear combination of L and L':

[0041] L final ←L+L'

[0042] After generating high-quality embedding vectors, the Faiss vector database is used to achieve efficient similarity search. Faiss is designed for processing large-scale vector data sets and can quickly retrieve vectors that are most similar to the query vector. Its core technologies include product quantization PQ and inverted file system IVF.

[0043] PQ compresses the original vector into a cluster center ID sequence by dividing the vector into multiple sub-vectors and clustering them. During the retrieval process, the query vector is also divided into corresponding sub-vectors, and the distance with the vector in the library is quickly calculated through table lookup operation. Compared with traditional brute force calculation, the PQ method calculates the distance between sub-vectors through table lookup;

[0044] Although PQ has optimized the calculation of vector retrieval, when the number of vectors in the database is extremely large, it is still necessary to traverse the entire database to calculate the distance, which affects the retrieval efficiency. To this end, Faiss introduced IVF and further developed IVFPQ technology. The core idea of ​​IVFPQ is to lock the "region of interest" of the query vector by pre-segmenting the vector space to avoid global retrieval. In the specific implementation, Faiss first performs KMeans clustering on the vectors in the database to generate several cluster centers. When the query vector is input, it first calculates its distance to these cluster centers and only retrieves the vector subspace related to the nearest cluster center, thereby avoiding unnecessary full database calculations.

[0045] In addition, Faiss introduced residual optimization, that is, when calculating the distance between the query vector and the subspace vector, all vectors are converted into residuals with the cluster center, which further improves the accuracy of PQ calculation; through this method, IVFPQ reduces the order of magnitude of the vectors to be calculated and greatly improves the efficiency of vector retrieval.

[0046] Preferably, in step 4, the Re-ranking technology significantly improves the accuracy and professionalism of the final answer through multi-level information fusion and optimization, especially performs well in text matching evaluation indicators such as BLEU-4 and ROUGE;

[0047] The BGE M3e model converts the input text into a vector representation of fixed dimension to capture semantic information, and normalizes the vector to a length of 1 through L2 regularization so that the cosine similarity can be calculated through dot product;

[0048] The regularized vectors are stored in the Faiss vector database and quickly retrieved through an efficient indexing algorithm. In the initial retrieval stage, the cosine similarity of dense vectors and the keyword search of sparse vectors are combined to usually recall the top K texts most relevant to the query. However, the correct answer may be ranked low, causing LLM to process redundant information, reduce reasoning efficiency and increase the probability of error. Therefore, further screening is required through re-ranking to improve the accuracy of relevant texts.

[0049] The re-ranking uses the BERT model and linear classifier to judge text matching and capture more complex semantic details. Although the computational complexity of the classifier is high, combined with the cosine similarity calculation in the preliminary retrieval, the system achieves a balance between efficiency and accuracy. The system evaluates the quality of the generated answers through the BLEU and ROUGE indicators:

[0050]

[0051]

[0052] Among them, BP is used to punish candidate texts that are too short, p n Indicates the accuracy of n-gram, that is, the proportion of n-grams in the candidate text in the reference text; usually for all n values, the weight ω n Equal; n represents n-gram, i.e. gram n Size, Count match (gram n ) represents the maximum number of n-grams that co-occur in the generated answer and the reference answer set.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. This paper proposes an innovative cross-technology combination framework, which combines LLM, RAG and resequencer for the first time in the field of mechanical manufacturing, significantly improving the overall performance of the question-answering system.

[0055] 2. This paper forms a Mechanical Manufacturing Process Design (MMPD) dataset, which optimizes the performance of the system in professional scenarios by integrating and fine-tuning domain knowledge.

[0056] 3. The present invention proposes a collaborative mechanism for retrieval generation to ensure that the generation model can make full use of the retrieved relevant document information to improve the accuracy and contextual consistency of the answer; at the same time, it successfully integrates the re-ranker to re-rank the documents retrieved by RAG to further optimize the quality of the final answer.

[0057] 4. This invention expands the application of LLM in vertical field question-answering systems, and verifies the feasibility of combining multiple technologies in industrial applications through experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a process for optimizing answers in an intelligent question-answering system based on a large language model is proposed for the present invention;

[0059] Figure 2 A structural schematic diagram of an intelligent question-answering system based on a large language model is proposed for the present invention;

[0060] Figure 3 This is a schematic diagram of the construction method of the RAG proposed in the present invention;

[0061] Figure 4 Schematic diagram of fine-tuning the large model proposed in the present invention. DETAILED DESCRIPTION

[0062] Reference Figures 1 to 4 .

[0063] The embodiment further illustrates an intelligent question-answering system based on a large language model proposed by the present invention.

[0064] An intelligent question-answering system based on a large language model, comprising a knowledge base for the field of mechanical manufacturing, the knowledge base covering mechanical manufacturing process design, industry standards and related specifications;

[0065] The system optimizes the accuracy and professionalism of answers by combining the retrieval enhancement generation (RAG) technology and the re-ranking (Re-ranking) technology. The steps are as follows:

[0066] Step 1: Build a knowledge base that contains diverse knowledge in the field of mechanical manufacturing, including process design, material properties, and equipment operation information, to support the answering of a wide range of field questions;

[0067] Step 2: Fine-tune a variety of open source large models and train them according to the complexity and high-precision requirements of the mechanical manufacturing field to ensure that the models have strong reasoning and computing capabilities. Among them, the Qwen2-7B model fine-tuned with LoRA+ has the best effect;

[0068] Step 3: Use RAG technology to effectively retrieve and process domain knowledge from the knowledge base, especially numerical tables and complex technical specifications, to obtain preliminary answers;

[0069] Step 4: Use Re-ranking technology to optimize the retrieved results to generate more professional and accurate answers, ensuring that the output content meets the requirements of the mechanical manufacturing field.

[0070] In step 1, knowledge from mechanical manufacturing process design, industry standards and related specifications is integrated to form a structured knowledge base, which is convenient for the system to call during the solution process.

[0071] The data collection covers multiple important sources in the field of mechanical manufacturing process design, ensuring the advantages of the question-answering system in professionalism and accuracy; specific data sources include the following authoritative documents and standards: "Mechanical Manufacturing Process Design Manual", "Mechanical Manufacturing Technology", "Mechanical Manufacturing Technology Fundamentals", "Mechanical Engineering Materials", "GBT4863-2008 Basic Terms of Mechanical Manufacturing Process", "National Standard for Digital Simulation of Mechanical Manufacturing 1: General Requirements", "National Standard for Digital Simulation of Mechanical Manufacturing 2: Production Line Planning and Layout Simulation Requirements", "National Standard for Digital Simulation of Mechanical Manufacturing 3: Assembly Workshop Logistics Simulation Requirements", "National Standard for Digital Simulation of Mechanical Manufacturing 4: CNC Machining Process Simulation Requirements".

[0072] In step 2, the large model uses a specific data set for the mechanical manufacturing field for pre-training and supervised fine-tuning during the training process to improve the model's understanding and application capabilities of domain knowledge.

[0073] The Qwen2-7B model is an open source large model. After comparing multiple open source large models, it was found that the Qwen2-7B model using LoRA+ fine-tuning has the best effect in the field of mechanical manufacturing; the Qwen2-7B model has a large parameter scale (7 billion parameters) and can handle complex tasks.

[0074] The core architecture of the large model is the Transformer-based autoregressive model. The autoregressive model is a sequence generation model that depends on the previously generated elements when generating each element of the sequence. It has the ability to predict the i-th token based on the previous i-1 tokens.

[0075] P(ω i ω1,ω2,…,ω i-1 ;θ)

[0076] The most traditional causal modeling method is used in the pre-training process, and the model is optimized in an unsupervised manner by minimizing the log-likelihood of the following formula:

[0077]

[0078] Among them, ω1, ω2, …, ω i-1 is an input sequence consisting of i words.

[0079] During the fine-tuning phase, each data sample includes an "instruction" field that describes the specific requirements of the task; an "input" field that provides the input content of the model; and an "output" field that contains the target output that the model should generate. This structured format design can effectively guide the behavior of the model in specific tasks and ensure that the model can fully understand and generate high-quality responses when dealing with questions and answers in the field of mechanical manufacturing.

[0080] LoRA is a method for fine-tuning large models by adding a low-rank matrix (called an adapter) to the weights of a pre-trained model, rather than adjusting all model parameters; this approach is efficient for adapting to new tasks while keeping computational costs low.

[0081] When the original LoRA processes a model with a large width (embedding dimension), the fine-tuning effect is not good because the adapter matrices A and B are updated using the same learning rate; this is because in a large-width network, using the same learning rate is not conducive to effective feature learning; in order to solve this problem, the proposed new method LoRA+ sets different learning rates for the adapter matrices A and B and maintains a fixed ratio. It is recommended to set the learning rate of B to λ times the learning rate of A and λ>1, thereby improving the efficiency of feature learning; through a large number of experiments, LoRA+ can improve performance (1%-2% improvement) and fine-tuning speed (about 2 times the speed increase) while maintaining the same computational cost as LoRA; using the LoRA+ fine-tuning method, and according to the rule of thumb, let λ=2 4 =16.

[0082] In step 3, RAG technology screens out the most relevant information from the knowledge base through correlation analysis of the user input question, thereby providing a preliminary basis for the answer.

[0083] RAG technology involves Embedding models and vector databases; the Embedding model used is the BGE M3e (BGE M3-Embedding) model. This open source model can handle different levels of input granularity from "sentences" to "documents", and supports intensive retrieval, vocabulary retrieval and multi-vector retrieval.

[0084] Dense retrieval transforms the input query q into hidden states H based on the text encoder q , and use the normalized hidden state with a special tag “[CLS]” to represent the embedding representation of the query: q =norm(H q [0]); Similarly, we can get the embedding representation of paragraph p: p =norm(Hp [0]); therefore, the relevance score between the query and the passage is represented by two embeddings e p and e q The inner product is measured as: s dense ← <e p ,e q >.

[0085] The embeddings output from the lexical search are used to evaluate the importance of each term for lexical-level search; for each term t in the query (in this work, terms correspond to tokens), its weight is calculated as Among them, W lex ∈R d×1 is a matrix used to map hidden states to floating point numbers; if the same term t appears multiple times in the query, only its maximum weight is retained; the weights of terms in the paragraph are calculated in the same way; based on these term weights, the relevance score between the query and the paragraph is calculated by the joint importance of the co-occurring terms, denoted as q∩p:

[0086] On top of dense retrieval, the multi-vector approach uses full output embeddings to represent the query and the paragraph: Where W mul ∈R d×d is a trainable projection matrix; and late interactions are used to compute refined relevance scores: N and M represent the length of the query and paragraph respectively.

[0087] The multifunctional nature of the BGE M3e model allows the retrieval process to be performed in a hybrid manner; first, candidate results can be retrieved using each method separately; then, the final retrieval results are re-ranked based on the combined relevance scores: rank ←s dense +s lex +s mul .

[0088] The training goal of the embedding model is to distinguish positive samples from negative samples; for each retrieval method, it is expected to assign higher scores to positive samples of the query and lower scores to negative samples; therefore, the training process aims to minimize the InfoNCE loss, the general form of which is:

[0089]

[0090] where p * and p' represent the positive and negative samples of query q respectively; s(·) is {s dense (·),s lex (·),s mul(·)}.

[0091] The training objectives of different retrieval methods may conflict with each other, and multi-objective training may be detrimental to the quality of embedding; therefore, based on the principle of ensemble learning, the training process is unified on the basis of self-knowledge distillation, combining the predictions of different retrieval methods to provide a more accurate relevance score; the simplest form is to sum up the different prediction scores:

[0092] s inter ←s dense +s lex +s mul

[0093] In previous studies, the training quality of embedding models can benefit from knowledge distillation, which utilizes fine-grained soft labels from another ranking model; the BGE M3e model uses an ensemble score s inter As a teacher, I modified the loss function of each retrieval method:

[0094] L' * ←-p(s inter )*logp(s * )

[0095] where p(·) is the softmax activation; s * Yes dense 、s lex and mul Any one of them; further integrate and normalize the modified loss function as follows:

[0096] L'←(L' dense +L' lex +L' mul ) / 3

[0097] Finally, the final loss function of self-knowledge distillation is derived through the linear combination of L and L':

[0098] L final ←L+L'

[0099] After generating high-quality embedded vectors, the Faiss vector database is used to achieve efficient similarity search. Faiss is designed specifically for processing large-scale vector data sets and can quickly retrieve vectors that are most similar to the query vector. Its core technologies include product quantization PQ and inverted file system IVF.

[0100] PQ compresses the original vector into a sequence of cluster center IDs by splitting the vector into multiple sub-vectors and clustering them. This process reduces the dimension and retains the semantic information, greatly reducing the computational complexity. During the retrieval process, the query vector is also split into corresponding sub-vectors, and the distance to the vector in the library is quickly calculated through table lookup operations. Compared with traditional brute force calculations, the PQ method effectively reduces the calculation time by table lookup and distance calculation between sub-vectors.

[0101] Although PQ has optimized the calculation of vector retrieval, when the number of vectors in the database is extremely large, it is still necessary to traverse the entire database to calculate the distance, which affects the retrieval efficiency. For this reason, Faiss introduced IVF and further developed IVFPQ technology. The core idea of ​​IVFPQ is to lock the "region of interest" of the query vector by pre-segmenting the vector space to avoid global retrieval. In the specific implementation, Faiss first performs KMeans clustering on the vectors in the database to generate several cluster centers. When the query vector is input, the distance between it and these cluster centers is calculated first, and only the vector subspace related to the nearest cluster center is retrieved, thereby avoiding unnecessary full database calculations.

[0102] In addition, Faiss introduced residual optimization, that is, when calculating the distance between the query vector and the subspace vector, all vectors are converted into residuals with the cluster center, which further improves the accuracy of PQ calculation; through this method, IVFPQ reduces the order of magnitude of the vectors to be calculated and greatly improves the efficiency of vector retrieval.

[0103] In step 4, the Re-ranking technology significantly improves the accuracy and professionalism of the final answer through multi-level information fusion and optimization, especially in text matching evaluation indicators such as BLEU-4 and ROUGE.

[0104] The BGE M3e model captures semantic information by converting the input text into a vector representation of fixed dimension, and normalizes the vector to a length of 1 through L2 regularization, so that the cosine similarity can be calculated through dot product; the regularized vector is stored in the Faiss vector database and quickly retrieved through an efficient indexing algorithm; the preliminary retrieval stage combines the cosine similarity of dense vectors and the keyword retrieval of sparse vectors, usually recalling the top K texts most relevant to the query; however, in the preliminary retrieval results, the text containing the correct answer may be ranked lower, causing LLM to process a large amount of redundant information, reducing reasoning efficiency and increasing the probability of error; therefore, further screening is required through re-ranking to improve the accuracy of the most relevant text.

[0105] Reranking is done by processing the recalled texts through the BERT model and using a linear classifier to perform a binary classification task to determine text matching. Compared with matching methods that rely solely on cosine similarity, reranking can capture more complex semantic details. This is because BERT's attention mechanism and feedforward neural network can extract richer features, and the classifier further improves the accuracy of the results through parameter adjustment. Although the classifier has a high computational complexity, combined with the cosine similarity calculation in the preliminary retrieval, the system achieves a balance between efficiency and accuracy. By first using efficient cosine similarity for preliminary screening and then using reranking to improve accuracy, the overall performance of the system is significantly improved. The system evaluates the quality of generated answers using the BLEU and ROUGE indicators:

[0106]

[0107]

[0108] Among them, BP is used to punish candidate texts that are too short, p n Indicates the accuracy of n-gram, that is, the proportion of n-grams in the candidate text in the reference text; usually for all n values, the weight ω n Equal; n represents n-gram, i.e. gram n Size, Count match (gram n ) represents the maximum number of n-grams that co-occur in the generated answer and the reference answer set.

[0109] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An intelligent question-answering system based on a large language model, characterized in that: It includes a knowledge base for the field of mechanical manufacturing, which covers mechanical manufacturing process design, industry standards and related specifications; The system optimizes the accuracy and professionalism of answers by combining the retrieval enhancement generation (RAG) technology and the re-ranking technology. The steps are as follows: Step 1: Build a knowledge base that contains diverse knowledge in the field of mechanical manufacturing, including process design, material properties, and equipment operation information, to support the answering of a wide range of field questions; Step 2: Fine-tune a variety of open source large models and train them according to the complexity and high-precision requirements of the mechanical manufacturing field to ensure that the models have strong reasoning and computing capabilities. Among them, the Qwen2-7B model fine-tuned with LoRA+ has the best effect; Step 3: Use RAG technology to effectively retrieve and process domain knowledge from the knowledge base, especially numerical tables and complex technical specifications, to obtain preliminary answers; Step 4: Use Re-ranking technology to optimize the retrieved results to generate more professional and accurate answers, ensuring that the output content meets the requirements of the mechanical manufacturing field.

2. The intelligent question-answering system based on a large language model according to claim 1, characterized in that: In step 1, knowledge from mechanical manufacturing process design, industry standards and related specifications is integrated to form a structured knowledge base, which is convenient for the system to call during the answering process; data collection covers multiple important sources in the field of mechanical manufacturing process design, ensuring the advantages of the question-answering system in professionalism and accuracy.

3. The intelligent question-answering system based on a large language model according to claim 2, characterized in that: In step 2, the large model uses a specific data set for the mechanical manufacturing field for pre-training and supervised fine-tuning during the training process to improve the model's ability to understand and apply domain knowledge; The Qwen2-7B model is an open source large model. After comparing multiple open source large models, it was found that the Qwen2-7B model using LoRA+ fine-tuning has the best effect in the field of mechanical manufacturing; the Qwen2-7B model has a large parameter scale and can handle complex tasks; The core architecture of the large model is the Transformer-based autoregressive model. The autoregressive model is a sequence generation model that depends on the previously generated elements when generating each element of the sequence. It has the ability to predict the i-th token based on the previous i-1 tokens: P(ω i |ω1,ω2,…,ω i-1 (i) The most traditional causal modeling method is used in the pre-training process, and the model is optimized in an unsupervised manner by minimizing the log-likelihood of the following formula: Among them, ω1, ω2, …, ω i-1 is an input sequence consisting of i words; During the fine-tuning phase, each data sample includes an "instruction" field that describes the specific requirements of the task; an "input" field that provides the input content of the model; and an "output" field that contains the target output that the model should generate; LoRA is a method for fine-tuning large models by adding a low-rank matrix called an adapter to the weights of a pre-trained model, rather than adjusting all model parameters; this method can effectively adapt to new tasks while keeping the computational cost low; When the original LoRA processes a model with a large width, the fine-tuning effect is not good because the adapter matrices A and B are updated using the same learning rate; this is because in a large-width network, using the same learning rate is not conducive to effective feature learning; to solve this problem, the proposed new method LoRA+ sets different learning rates for the adapter matrices A and B and maintains a fixed ratio. It is recommended to set the learning rate of B to λ times the learning rate of A and λ>1, thereby improving the feature learning efficiency; through a large number of experiments, LoRA+ can improve performance and fine-tuning speed while maintaining the same computational cost as LoRA; using the LoRA+ fine-tuning method, and according to the rule of thumb, let λ=2 4 =16.

4. The intelligent question-answering system based on a large language model according to claim 3, characterized in that: In step 3, RAG technology screens out the most relevant information from the knowledge base through correlation analysis of the user input question, thereby providing a preliminary basis for answering; RAG technology involves the Embedding model and vector database. The Embedding model used is the BGE M3e model, which is an open source model that can handle input granularity at different levels, from "sentences" to "documents", and supports dense retrieval, vocabulary retrieval, and multi-vector retrieval. Dense retrieval transforms the input query q into hidden states H based on the text encoder q , and use the normalized hidden state with the special tag "[CLS]" to represent the embedding representation of the query: q =norm(H q [0]); Similarly, we can get the embedding representation of paragraph p: p =norm(H p [0]); therefore, the relevance score between the query and the passage is represented by two embeddings e p and e q The inner product is measured as: s dense ← <e p ,e q > The embedding output from the lexical search is used to evaluate the importance of each term for lexical level search; for each term t in the query, its weight is calculated as Among them, W lex ∈R d×1 is a matrix used to map hidden states to floating point numbers; if the same term t appears multiple times in the query, only its maximum weight is retained; the weights of terms in the paragraph are calculated in the same way; based on these term weights, the relevance score between the query and the paragraph is calculated by the joint importance of the co-occurring terms, denoted as q∩p: On top of dense retrieval, the multi-vector approach uses full output embeddings to represent the query and the paragraph: Where W mul ∈R d×d is a trainable projection matrix; and late interactions are used to compute refined relevance scores: N and M represent the length of the query and paragraph, respectively; The multifunctional nature of the BGE M3e model allows the retrieval process to be performed in a hybrid manner; first, candidate results can be retrieved using each method separately; then, the final retrieval results are re-ranked based on the combined relevance scores: rank ←s dense +s lex +s mul ; The training goal of the embedding model is to distinguish positive samples from negative samples; for each retrieval method, it is expected to assign higher scores to positive samples of the query and lower scores to negative samples; therefore, the training process aims to minimize the InfoNCE loss, the general form of which is: where p * and p' represent the positive and negative samples of query q respectively; s(·) is {s dense (·),s lex (·),s mul (·)}; The training objectives of different retrieval methods may conflict with each other, and multi-objective training may be detrimental to the quality of embedding; therefore, based on the principle of ensemble learning, the training process is unified on the basis of self-knowledge distillation, combining the predictions of different retrieval methods to provide a more accurate relevance score; the simplest form is to sum up the different prediction scores: s inter ←s dense +s lex +s mul In previous studies, the training quality of embedding models can benefit from knowledge distillation, which utilizes fine-grained soft labels from another ranking model; the BGE M3e model uses an ensemble score s inter As a teacher, I modified the loss function of each retrieval method: L' * ←-p(s inter )*logp(s * ) where p(·) is the softmax activation; s * Yes dense 、s lex and mul Any one of them; further integrate and normalize the modified loss function as follows: The ←(The dense +The lex +The mul ) / 3 Finally, the final loss function of self-knowledge distillation is derived through the linear combination of L and L': L final ←L+L' After generating high-quality embedding vectors, the Faiss vector database is used to achieve efficient similarity search. Faiss is designed for processing large-scale vector data sets and can quickly retrieve vectors that are most similar to the query vector. Its core technologies include product quantization PQ and inverted file system IVF. PQ compresses the original vector into a cluster center ID sequence by dividing the vector into multiple sub-vectors and clustering them. During the retrieval process, the query vector is also divided into corresponding sub-vectors, and the distance with the vector in the library is quickly calculated through table lookup operation. Compared with traditional brute force calculation, the PQ method calculates the distance between sub-vectors through table lookup; Although PQ has optimized the calculation of vector retrieval, when the number of vectors in the database is extremely large, it is still necessary to traverse the entire database to calculate the distance, which affects the retrieval efficiency. To this end, Faiss introduced IVF and further developed IVFPQ technology. The core idea of ​​IVFPQ is to lock the "region of interest" of the query vector by pre-segmenting the vector space to avoid global retrieval. In the specific implementation, Faiss first performs KMeans clustering on the vectors in the database to generate several cluster centers. When the query vector is input, it first calculates its distance to these cluster centers and only retrieves the vector subspace related to the nearest cluster center, thereby avoiding unnecessary full database calculations. In addition, Faiss introduced residual optimization, that is, when calculating the distance between the query vector and the subspace vector, all vectors are converted into residuals with the cluster center, which further improves the accuracy of PQ calculation; through this method, IVFPQ reduces the order of magnitude of the vectors to be calculated and greatly improves the efficiency of vector retrieval.

5. The intelligent question-answering system based on a large language model according to claim 4, characterized in that: In step 4, the Re-ranking technology significantly improves the accuracy and professionalism of the final answer through multi-level information fusion and optimization, especially in terms of text matching evaluation indicators such as BLEU-4 and ROUGE; The BGE M3e model converts the input text into a vector representation of fixed dimension to capture semantic information, and normalizes the vector to a length of 1 through L2 regularization so that the cosine similarity can be calculated through dot product; The regularized vectors are stored in the Faiss vector database and quickly retrieved through an efficient indexing algorithm. In the initial retrieval stage, the cosine similarity of dense vectors and the keyword search of sparse vectors are combined to usually recall the top K texts most relevant to the query. However, the correct answer may be ranked low, causing LLM to process redundant information, reduce reasoning efficiency and increase the probability of error. Therefore, further screening is required through re-ranking to improve the accuracy of relevant texts. The re-ranking uses the BERT model and linear classifier to judge text matching and capture more complex semantic details. Although the computational complexity of the classifier is high, combined with the cosine similarity calculation in the preliminary retrieval, the system achieves a balance between efficiency and accuracy. The system evaluates the quality of the generated answers through the BLEU and ROUGE indicators: Among them, BP is used to punish candidate texts that are too short, p n Indicates the accuracy of n-gram, that is, the proportion of n-grams in the candidate text in the reference text; usually for all n values, the weight ω n Equal; n represents n-gram, i.e. gram n Size, Count match (gram n ) represents the maximum number of n-grams that co-occur in the generated answer and the reference answer set.

Images