Government affair knowledge real-time dialogue method based on improved fine tuning and combined retrieval RAG and Qwen2.5-72B

Abstract

The invention relates to a government affair knowledge real-time dialogue method based on improved fine tuning, combined retrieval RAG and Qwen2.5-72B. The method comprises the steps that government affair dialogue data is collected and marked, and a supervision data set, namely a training set, is formed; fine adjustment is carried out on the Qwen2.5-72B large model by utilizing the supervision data set, so that the fine-adjusted Qwen2.5-72B large model can more accurately carry out government affair dialogue answering; related knowledge of government affairs is crawled, and knowledge question and answer vector data is constructed and stored in a vector memory of LangChain; and the fine-tuned Qwen2.5-72B large model generates an inquiry answer by using the first five blocks of the original text corresponding to the embedded vector retrieved by the user query question, the prompt prompt word and the combinatorial retrieval RAG. According to the scheme, a Qwen2.5-72B large model is adopted to construct a training set, fine tuning is improved, the fine-tuned model serves as an embedded model, RAG retrieval and Prompt cue word construction are combined, the training data quality can be improved, relationships and modes in data are understood more meticulously to help to reduce deviation, the dialogue intention of a user is fully understood, accurate retrieval is achieved, and the user experience is improved. Therefore, the dialogue response accuracy is obviously improved.

Description

Technical Field

[0001] The present invention belongs to the field of real-time dialogue of Qwen2.5, and in particular relates to a real-time dialogue method of government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B. Background Art

[0002] With the rapid development of information technology, the application of artificial intelligence technology in public management services is gradually increasing. As a key component of public management services, the accuracy and response speed of conversation data directly affect the quality of government services and public satisfaction. However, due to the complexity and diversity of government conversation data, traditional natural language processing technology often faces problems of insufficient accuracy and slow response when processing such conversations.

[0003] In current technical practice, the processing of government dialogue data mainly relies on rule-driven methods or classic machine learning models. The rule-driven method requires the manual creation of numerous rule templates to adapt to diverse dialogue scenarios. This method is not only labor-intensive and costly to maintain, but also difficult to cope with the complexity and variability of the government dialogue environment. Although machine learning models can improve the accuracy of dialogue processing to a certain extent, they have obvious limitations in practical applications because their training process relies on a large amount of labeled data and has high requirements for the quality and quantity of data.

[0004] Recently, large-scale models based on deep learning have made breakthrough progress in the field of natural language processing. Qwen2.5, a pre-trained language model based on the Transformer architecture, demonstrates excellent language understanding and generation capabilities. By pre-training on large-scale unlabeled text data, the model can absorb rich language knowledge and semantic information, and therefore performs well in various natural language processing tasks. However, for the specific field of government dialogue, the Qwen2.5 model without fine-tuning and retrieval optimization may not fully meet actual needs. Given the special context and expertise of government dialogue, targeted fine-tuning and precise retrieval of the Qwen2.5 model are the key to improving its accuracy and efficiency in government dialogue processing. Summary of the invention

[0005] In view of the above problems, the present invention provides a real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B, comprising the following steps:

[0006] Step 1: Collect and annotate government dialogue data to form a supervision data set, i.e., a training set;

[0007] Step 2: Use the supervised dataset to fine-tune the Qwen2.5-72B large model so that the fine-tuned Qwen2.5-72B large model can answer government affairs dialogues more accurately; this includes:

[0008] Using the training set, the Layerwise Importance Sampled AdamW (LISA) method was used to fine-tune the Qwen2.5-72B large model. During the fine-tuning process, the fine-tuning prompt was used to require the Qwen2.5-72B large model to be fine-tuned according to the prompt requirements.

[0009] Among them, when the LISA method fine-tunes the Qwen2.5-72B large model, it samples according to the importance of each layer and randomly freezes most of the intermediate layers, thereby improving the training speed and performance; and when LISA fine-tunes, it combines regularization to build a complex loss function to improve the model fine-tuning accuracy. The function L is as follows:

[0010]

[0011] In the formula:

[0012] is the predicted output of the model,

[0013] True value,

[0014] N is the sample size,

[0015] M is the number of model parameters,

[0016] and is the regularization coefficient, which controls the strength of L1 and L2 regularization.

[0017] is the L1 regularization term, which sums the absolute values ​​of the model parameters,

[0018] is the L2 regularization term, which sums the squares of the model parameters;

[0019] Among them, the construction of fine-tuning prompt words uses the prompt word logic structure and uses the markdown format. The prompt word content adopts Few-Shot learning to provide examples to more accurately guide the model fine-tuning training;

[0020] Step 3: crawl government affairs related knowledge, construct knowledge question and answer vector data and save it in LangChain’s vector storage;

[0021] Step 4: The fine-tuned Qwen2.5-72B large model generates the query answer using the user query, prompt word, and the first 5 blocks of the original text corresponding to the embedding vector retrieved by the combined retrieval RAG.

[0022] The step of collecting and labeling government dialogue data in step 1 to form a supervision data set, i.e., a training set, includes:

[0023] Obtain government knowledge base and government knowledge text from government business, use Qwen2.5-72B large model to mine all possible questions from the government knowledge base; use Qwen2.5-72B large model to understand the government knowledge text blocks, and mine questions from all blocks; use real user questions; use manually annotated questions; construct a supervision data set with the questions obtained by these four methods and the corresponding documents; screen and standardize the question and answer pairs of the supervision data set.

[0024] Step 3: crawl government-related knowledge, construct knowledge question and answer vector data and save it in the vector storage of LangChain, including:

[0025] LangChain’s vector storage uses the Chroma database.

[0026] Block method that retains more contextual information: First, the entire text is input into the long-context embedding model TCT-Colbert to generate a vector representation of each token, thereby capturing the contextual information of the entire text; after obtaining the token-level embedding of the entire text, the text is divided into multiple blocks. Each block contains a certain number of tokens, and there may be overlapping parts to retain the contextual information at the boundary; for each block, the mean pooling method is used to aggregate the vector representations of all tokens in the block into a block embedding, thereby retaining more contextual information.

[0027] A vector collection table vector-storage1 of the vector storage is generated by using a blocking method that retains more context information.

[0028] Block segmentation based on the Qwen2.5-72B large model: The text is understood through the fine-tuned Qwen2.5-72B large model and split into multiple vector blocks.

[0029] A vector set table vector-storage2 of the vector storage is generated based on the block method of the Qwen2.5-72B large model.

[0030] The step of generating the query answer by using the first five blocks of the original text corresponding to the embedding vector retrieved by the user query sentence, prompt word and combined search RAG after the fine-tuned Qwen2.5-72B large model described in step 4 includes:

[0031] The user inputs a query, and the combined retriever R retrieves the top K blocks that are most relevant to the query. The retrieval evaluator is then used to analyze the relevance of the K blocks to the question. If the block is evaluated as relevant, the cross encoder is used to reorder the K blocks, and the fine-tuned Qwen2.5-72B large model generates an accurate answer based on the top K blocks returned by the retrieval, the query, and the prompt word Prompt content. If the block is evaluated as irrelevant, the system returns the answer that could not be found.

[0032] Preferably, the combined retriever R is composed of the following four methods: HNSW (Hearest Neighbor Search in High Dimensional Space) method, dual encoder (BERT) method, density function method and cross encoder method.

[0033] The dual encoder method uses the sentence-transformers library to load the model multi-qa-MiniLM-L6-cos-v1 to initialize and generate the bi_encoder model, uses the bi_encoder model to encode the query into an embedding vector, and then uses the semantic_search function of the sentence-transformers library to calculate the similarity between the query embedding vector and each block in the vector set table vector-storage2, and obtains the top K / 3 most similar blocks;

[0034] The HNSW method uses the hnswlib library to declare the HNSW index, and then uses the init_index function of the library to initialize the index. The fine-tuned Qwen2.5-72B large model is used to encode the query into an embedded vector. The add_items function of the library is used to add indexes to the block data of the vector collection table vector-storage1. The knn_query function of the library is used to calculate the indexes of the K / 3 block neighbors of vector-storage1 closest to the query, and obtain the top K / 3 most similar blocks.

[0035] The density function method first uses the sklearn library to obtain the Gaussian Mixture Model (GMM) and the principal component analysis model PCA, and uses the fine-tuned Qwen2.5-72B large model to encode the query into an embedding vector. Then, PCA is used to reduce the dimensions of each block in the vector collection table vector-storage1 and the query embedding vector. The GMM model is used to calculate the log-likelihood score of each block of the query in vector-storage1, and the top K / 3 blocks with the highest scores are selected.

[0036] Furthermore, the K / 3 blocks obtained by the HNSW (nearest neighbor search in high dimensional space) method, the dual encoder (BERT) method and the density function method are respectively aggregated into a vector set table vector-storage3 of a vector storage of K blocks; the cross encoder method reorders vector-storage3 according to the relevance score, and returns the first 5 blocks of the original text corresponding to the embedding vector with the highest relevance score predicted by the cross encoder to the fine-tuned Qwen2.5-72B large model.

[0037] Preferably, the prompt word Prompt introduces a COT (Chain of Thought) prompt during tuning.

[0038] Preferably, the HNSW, before use, includes setting the distance calculation method, the dimension of the vector, the maximum number of elements that can be stored in the index structure, the trade-off between speed and accuracy during index construction, and the number of doubly linked lists created for each element during construction.

[0039] Preferably, the PCA includes setting the number of principal components to be retained, the type of SVD (Singular Value Decomposition) solver to be used, and the seed of the random number generator before use.

[0040] Preferably, before using the GMM, parameter settings include setting the number of Gaussian distributions, the threshold for judging the convergence criterion, the regularization term added in the covariance matrix, the maximum number of iterations and the type of the covariance matrix, and then using the merged data of each block in vector-storage1 after PCA dimensionality reduction and the query question embedding vector to complete the GMM model training.

[0041] Preferably, the cross encoder obtains CrossEncoder from the sentence_transformers library as a cross encoder before use, and then uses CrossEncoder to load the model ms-marco-MiniLM-L-6-v2 for initialization, and then pairs the query sentence with each block of the original text corresponding to the embedding vector corresponding to each block in the vector set table vector-storage3 to form an input pair; the cross encoder predicts a relevance score for each pair of inputs.

[0042] The present invention is a real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval of RAG and Qwen2.5-72B. The Qwen2.5-72B large model is used to build a training set, improved fine-tuning, the fine-tuned model is used as an embedding model, combined retrieval of RAG and construction of Prompt prompt words, which can improve the quality of training data, understand the relationships and patterns in the data more carefully to help reduce deviations, and fully understand the user's dialogue intentions and accurate retrieval, thereby significantly improving the accuracy of dialogue responses.

[0043] Other features and advantages of the present invention will be described in the following description, and partly become obvious from the description, or be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A flow chart of a real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B in an embodiment of the present invention is shown;

[0046] Figure 2 A schematic diagram of a model of a real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B in an embodiment of the present invention is shown;

[0047] Figure 3 A schematic diagram of fine-tuning training design of the Qwen2.5-72B large model in an embodiment of the present invention is shown;

[0048] Figure 4 A schematic diagram of a model of a density function method according to an embodiment of the present invention is shown; DETAILED DESCRIPTION

[0049] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0050] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware units or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0051] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0052] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the implementation examples of the application described herein are implemented in an order other than those illustrated or described herein.

[0053] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or sub-modules is not necessarily limited to those steps or sub-modules explicitly listed, but may include other steps or sub-modules not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0054] Example:

[0055] The present invention is achieved through the following technical solutions: Figure 1 As shown, the present invention proposes a real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B, comprising the following steps:

[0056] Step 1: Collect and annotate government dialogue data to form a supervision data set, i.e., a training set.

[0057] The prompt word Prompt is set, and the prompt word content adopts Few-Shot learning. By providing examples, the model Qwen2.5-72B large model is guided more accurately. The government affairs knowledge base and government affairs knowledge text are obtained from government affairs business, and the Qwen2.5-72B large model is used to mine all possible problems from the government affairs knowledge base; the prompt word Prompt is set, and the prompt word content adopts Few-Shot learning. By providing examples, the model Qwen2.5-72B large model is guided more accurately, and the Qwen2.5-72B large model is used to understand the government affairs knowledge text blocks and mine problems from all blocks; real user questions are used; manually annotated questions are used; the questions obtained by these four methods and the corresponding documents are constructed into a supervised data set; the question and answer pairs of the supervised data set are screened and standardized to form a supervised data set, i.e., a training set.

[0058] In this solution, the training set is constructed by the Qwen2.5-72B large model, the training set is constructed by blocks, the training set is constructed by real questions and answers, and the training set is constructed by manual annotation, so as to form a rich training set, which can improve the quality of training data. Then, the training set is used to fine-tune the Qwen2.5-72B large model, and the fine-tuned Qwen2.5-72B large model is used as an embedding model to understand the relationships and patterns in the data more carefully to help reduce bias and significantly improve the precision and recall of the model question and answer.

[0059] Step 2: Use the supervised dataset to fine-tune the Qwen2.5-72B large model so that the fine-tuned Qwen2.5-72B large model can answer government affairs dialogues more accurately. This includes:

[0060] Please see Figure 3 , using the training set, the Layerwise Importance Sampled AdamW (LISA) method is used to fine-tune the Qwen2.5-72B large model. During the fine-tuning process, the fine-tuning prompt is used to require the Qwen2.5-72B large model to be fine-tuned according to the prompt requirements.

[0061] Among them, when the LISA method fine-tunes the Qwen2.5-72B large model, it samples according to the importance of each layer and randomly freezes most of the intermediate layers, thereby improving the training speed and performance; and when LISA fine-tunes, it combines regularization to build a complex loss function to improve the model fine-tuning accuracy. The function L is as follows:

[0062]

[0063] In the formula:

[0064] is the predicted output of the model,

[0065] True value,

[0066] N is the sample size,

[0067] M is the number of model parameters,

[0068] and is the regularization coefficient, which controls the strength of L1 and L2 regularization.

[0069] is the L1 regularization term, which sums the absolute values ​​of the model parameters,

[0070] is the L2 regularization term, which sums the squares of the model parameters.

[0071] Among them, the construction of fine-tuning prompt words uses the prompt word logic structure and uses the markdown format. The prompt word content adopts Few-Shot learning to provide examples to more accurately guide the model fine-tuning training.

[0072] Step 3: Crawl government-related knowledge, construct knowledge question and answer vector data and save it in LangChain’s vector storage.

[0073] LangChain’s vector storage uses the Chroma database.

[0074] Block method that retains more contextual information: First, the entire text is input into the long-context embedding model TCT-Colbert to generate a vector representation of each token, thereby capturing the contextual information of the entire text; after obtaining the token-level embedding of the entire text, the text is divided into multiple blocks. Each block contains a certain number of tokens, and there may be overlapping parts to retain the contextual information at the boundary; for each block, the mean pooling method is used to aggregate the vector representations of all tokens in the block into a block embedding, thereby retaining more contextual information.

[0075] A vector collection table vector-storage1 of the vector storage is generated by using a blocking method that retains more context information.

[0076] Block segmentation based on the Qwen2.5-72B large model: The text is understood through the fine-tuned Qwen2.5-72B large model and split into multiple vector blocks.

[0077] A vector set table vector-storage2 of the vector storage is generated based on the block method of the Qwen2.5-72B large model.

[0078] Step 4: The fine-tuned Qwen2.5-72B large model generates the query answer using the user query, prompt word, and the first 5 blocks of the original text corresponding to the embedding vector retrieved by the combined retrieval RAG.

[0079] Please see Figure 2 , the user inputs a query, the combined retriever R retrieves the top K blocks that are most relevant to the query, and then uses the retrieval evaluator to analyze the relevance of the K blocks to the question; if the block is evaluated as relevant, the cross encoder is used to reorder the K blocks, and the fine-tuned Qwen2.5-72B large model generates an accurate answer based on the top K blocks returned by the retrieval, the query, and the prompt word Prompt content; if the block is evaluated as irrelevant, the system returns the answer that cannot be found.

[0080] The combined retriever R is composed of the following four methods: HNSW (Hearest Neighbor Search in High Dimensional Space) method, dual encoder (BERT) method, density function method and cross encoder method.

[0081] The dual encoder method uses the sentence-transformers library to load the model multi-qa-MiniLM-L6-cos-v1 to initialize and generate the bi_encoder model, uses the bi_encoder model to encode the query into an embedding vector, and then uses the semantic_search function of the sentence-transformers library to calculate the similarity between the query embedding vector and each block in the vector set table vector-storage2, and obtains the top K / 3 most similar blocks;

[0082] The HNSW method uses the hnswlib library to declare the HNSW index, and then uses the init_index function of the library to initialize the index. The fine-tuned Qwen2.5-72B large model is used to encode the query into an embedded vector. The add_items function of the library is used to add indexes to the block data of the vector collection table vector-storage1. The knn_query function of the library is used to calculate the indexes of the K / 3 block neighbors of vector-storage1 closest to the query, and obtain the top K / 3 most similar blocks.

[0083] Please see Figure 4 The density function method first uses the sklearn library to obtain the Gaussian Mixture Model (GMM) and the principal component analysis model PCA, and uses the fine-tuned Qwen2.5-72B large model to encode the query into an embedding vector. Then, PCA is used to reduce the dimensions of each block in the vector set table vector-storage1 and the query embedding vector. The GMM model is used to calculate the log-likelihood score of each block in vector-storage1, and the top K / 3 blocks with the highest scores are selected.

[0084] Furthermore, the K / 3 blocks obtained by the HNSW (nearest neighbor search in high dimensional space) method, the dual encoder (BERT) method and the density function method are respectively aggregated into a vector set table vector-storage3 of a vector storage of K blocks; the cross encoder method reorders vector-storage3 according to the relevance scores, and returns the blocks of the original text corresponding to the five embedding vectors with the highest relevance scores predicted by the cross encoder to the fine-tuned Qwen2.5-72B large model.

[0085] The prompt word Prompt introduces a COT (Chain of Thought) prompt during tuning.

[0086] Before using the HNSW, the method of distance calculation, the dimension of the vector, the maximum number of elements that can be stored in the index structure, the trade-off between speed and accuracy during index construction, and the number of doubly linked lists created for each element during construction are set.

[0087] Before using the PCA, the number of principal components that should be retained, the type of SVD (singular value decomposition) solver used, and the seed of the random number generator are set.

[0088] Before using the GMM, parameter settings include setting the number of Gaussian distributions, the threshold for judging the convergence standard, the regularization term added to the covariance matrix, the maximum number of iterations and the type of the covariance matrix. Then, the GMM model training is completed using the merged data of each block in vector-storage1 after PCA dimensionality reduction and the query question embedding vector.

[0089] Before use, the cross encoder obtains CrossEncoder from the sentence_transformers library as a cross encoder, and then uses CrossEncoder to load the model ms-marco-MiniLM-L-6-v2 for initialization, and then pairs the query sentence with each block of the original text corresponding to the embedding vector corresponding to each block of the vector set table vector-storage3 to form an input pair; the cross encoder predicts a relevance score for each pair of inputs.

[0090] In this solution, the loss function, prompt words, and LISA method are set to improve fine-tuning, so that the fine-tuned Qwen2.5-72B large model can fully understand the user's conversation intention and answer government affairs conversations more accurately. The combined retrieval RAG constructed using the designed HNSW (nearest neighbor search in high-dimensional space) method, dual encoder (BERT) method, density function method, and cross encoder method, combined with the constructed prompt words, can significantly improve the retrieval accuracy, thereby significantly improving the accuracy of conversation responses.

[0091] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B, characterized in that: The steps include: Step 1: Collect and annotate government dialogue data to form a supervision data set, i.e., a training set; Step 2: Use the supervised dataset to fine-tune the Qwen2.5-72B large model so that the fine-tuned Qwen2.5-72B large model can answer government affairs dialogues more accurately. These include: Using the training set, the LISA method is used to fine-tune the Qwen2.5-72B large model. During the fine-tuning process, the fine-tuning prompt is used to require the Qwen2.5-72B large model to be fine-tuned according to the prompt requirements; Among them, when the LISA method fine-tunes the Qwen2.5-72B large model, it samples according to the importance of each layer and randomly freezes most of the intermediate layers, thereby improving the training speed and performance; and when LISA fine-tunes, it combines regularization to build a complex loss function to improve the model fine-tuning accuracy. The function L is as follows: ； here is the predicted output of the model, True value, N is the number of samples, M is the number of model parameters, and is the regularization coefficient, which controls the strength of L1 and L2 regularization. is the L1 regularization term, which sums the absolute values ​​of the model parameters, is the L2 regularization term, which sums the squares of the model parameters; Among them, the construction of fine-tuning prompt words uses the prompt word logic structure and uses the markdown format. The prompt word content adopts Few-Shot learning to provide examples to more accurately guide the model fine-tuning training; Step 3: Crawl government-related knowledge, construct knowledge question and answer vector data and save it in LangChain’s vector storage; Step 4: The fine-tuned Qwen2.5-72B large model generates the query answer using the user query, prompt word, and the first 5 blocks of the original text corresponding to the embedding vector retrieved by the combined retrieval RAG.

2. According to claim 1, a real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B, characterized in that: The step of collecting and labeling government dialogue data in step 1 to form a supervision data set, i.e., a training set, includes: Obtain government knowledge base and government knowledge text from government business, use Qwen2.5-72B large model to mine all possible questions from the government knowledge base; use Qwen2.5-72B large model to understand the government knowledge text blocks, and mine questions from all blocks; use real user questions; use manually annotated questions; construct a supervision data set with the questions obtained by these four methods and the corresponding documents; screen and standardize the question and answer pairs of the supervision data set.

3. According to claim 1, a real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B, characterized in that: Step 3: crawl government-related knowledge, construct knowledge question and answer vector data and save it in the vector storage of LangChain, including: LangChain’s vector storage uses the Chroma database; Chunking method that retains more contextual information: First, the entire text is input into the long-context embedding model TCT-Colbert to generate a vector representation of each token, thereby capturing the contextual information of the entire text; after obtaining the token-level embedding of the entire text, the text is divided into multiple chunks; each chunk contains a certain number of tokens, and there may be overlapping parts to retain the contextual information at the boundaries; for each chunk, the mean pooling method is used to aggregate the vector representations of all tokens in the chunk into a chunk embedding, thereby retaining more contextual information; Generate the vector collection table vector-storage1 of the vector storage using a blocking method that retains more context information; Block segmentation based on the Qwen2.5-72B large model: The fine-tuned Qwen2.5-72B large model is used to understand the text and split the text into multiple vector blocks; A vector set table vector-storage2 of the vector storage is generated based on the block method of the Qwen2.5-72B large model.

4. According to claim 1, a real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B is characterized in that: The step of generating the query answer by using the first five blocks of the original text corresponding to the embedding vector retrieved by the user query sentence, prompt word and combined search RAG after the fine-tuned Qwen2.5-72B large model described in step 4 includes: The user inputs a query, and the combined retriever R retrieves the top K blocks that are most relevant to the query. The retrieval evaluator is then used to analyze the relevance of the K blocks to the question. If the block is evaluated as relevant, the cross encoder is used to reorder the K blocks, and the fine-tuned Qwen2.5-72B large model generates an accurate answer based on the top K blocks returned by the retrieval, the query, and the prompt word Prompt content. If the block is evaluated as irrelevant, the system returns the answer that could not be found.

5. According to claim 4, a real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B is characterized in that: The combined retriever R is composed of the following four methods: HNSW method, dual encoder method, density function method and cross encoder method; The dual encoder method uses the sentence-transformers library to load the model multi-qa-MiniLM-L6-cos-v1 to initialize and generate the bi_encoder model, uses the bi_encoder model to encode the query into an embedding vector, and then uses the semantic_search function of the sentence-transformers library to calculate the similarity between the query embedding vector and each block in the vector set table vector-storage2, and obtains the top K / 3 most similar blocks; The HNSW method uses the hnswlib library to declare the HNSW index, and then uses the init_index function of the library to initialize the index. The fine-tuned Qwen2.5-72B large model is used to encode the query into an embedded vector. The add_items function of the library is used to add indexes to the block data of the vector collection table vector-storage1. The knn_query function of the library is used to calculate the indexes of the K / 3 block neighbors of vector-storage1 closest to the query, and obtain the top K / 3 most similar blocks. The density function method first uses the sklearn library to obtain the Gaussian mixture model GMM and the principal component analysis model PCA, and uses the fine-tuned Qwen2.5-72B large model to encode the query into an embedding vector. Then, PCA is used to reduce the dimensions of each block in the vector set table vector-storage1 and the query embedding vector. The GMM model is used to calculate the log-likelihood score of each block in vector-storage1, and the top K / 3 blocks with the highest scores are selected. Furthermore, the K / 3 blocks obtained by the HNSW method, the dual encoder method and the density function method respectively are summarized into a vector set table vector-storage3 of a vector storage of K blocks; the cross encoder method reorders vector-storage3 according to the correlation scores, and returns the first 5 blocks of the original text corresponding to the embedding vector with the highest correlation score predicted by the cross encoder to the fine-tuned Qwen2.5-72B large model.

6. A real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B according to claim 4, characterized in that: The prompt word Prompt introduces a COT prompt during tuning.

7. A real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B according to claim 5, characterized in that: Before using the HNSW, the method of distance calculation, the dimension of the vector, the maximum number of elements that can be stored in the index structure, the trade-off between speed and accuracy during index construction, and the number of doubly linked lists created for each element during construction are set; Before using the PCA, the number of principal components to be retained, the type of SVD solver to be used, and the seed of the random number generator are set; Before using the GMM, parameter settings include setting the number of Gaussian distributions, the threshold for judging the convergence standard, the regularization term added to the covariance matrix, the maximum number of iterations and the type of the covariance matrix. Then, the GMM model training is completed using the merged data of each block in vector-storage1 after PCA dimensionality reduction and the query question embedding vector.

8. According to claim 5, a real-time dialogue method for government affairs knowledge based on improved fine-tuning, combined retrieval RAG and Qwen2.5-72B is characterized in that: Before use, the cross encoder obtains CrossEncoder from the sentence_transformers library as a cross encoder, and then uses CrossEncoder to load the model ms-marco-MiniLM-L-6-v2 for initialization, and then pairs the query sentence with each block of the original text corresponding to the embedding vector corresponding to each block of the vector set table vector-storage3 to form an input pair; the cross encoder predicts a relevance score for each pair of inputs.

Images