Domain-specific large model interaction method and device based on loRA fine-tuning and multi-dimensional semantic enhancement

Abstract

The application discloses a vertical domain large model interaction method and device based on LoRA fine tuning and multi-dimensional semantic enhancement, relates to the technical field of artificial intelligence and natural language processing, and comprises the following steps: in step S1, text data in a vertical domain knowledge base is acquired, and the text data is preprocessed to output training data; in step S2, LoRA fine tuning training is performed on a large language model based on the training data, the model learns domain professional knowledge and a response style, and a domain adaptive model is obtained. The application aims to realize efficient, reliable and humanized deployment of a large language model in a vertical domain.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a method and device for interaction of large vertical domain models based on LoRA fine-tuning and multidimensional semantic enhancement. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have demonstrated powerful capabilities in natural language understanding and generation, and have been widely applied in general scenarios such as dialogue systems, content generation, and knowledge question answering. However, when directly applying them to specific industries or vertical domains with specific business scenarios, the following key challenges still exist: (1) Insufficient depth of domain knowledge: Although the general large language model is trained on a wide range of corpora, it lacks structured knowledge and professional terminology understanding for specific domains, which can easily lead to factual errors or "illusions" and is not reliable enough in scenarios that require high accuracy.

[0003] (2) Fine-tuning cost and efficiency issues: Traditional full-parameter fine-tuning methods require updating billions or even hundreds of billions of parameters, which consumes huge computational resources, takes a long time to train, and is prone to overfitting. They are not economical or practical for industrial scenarios with limited data and frequent iterations.

[0004] (3) Diversity of user question expression: In practical applications, users often ask questions in a colloquial, fragmented, and multi-expression manner, and the same intention may correspond to dozens of different expressions. Traditional keyword matching or simple semantic models are difficult to cover this long-tail distribution, resulting in low recall and high interaction frequency.

[0005] (4) Poor real-time interactive experience: Most existing question-and-answer systems adopt a one-time response mechanism, which appears rigid and unnatural in complex questions or multi-turn dialogues. The streaming output capability is insufficient, and it is impossible to realize progressive answers and dynamic guidance, which affects users' willingness to continue using the system.

[0006] (5) Data scarcity and labeling difficulties: The cost of acquiring high-quality labeled data in vertical fields is high, and professional knowledge is updated quickly, making it difficult to continuously expand and maintain the traditional method of manually building knowledge bases.

[0007] To address the above issues, the industry has already made some attempts, such as adopting the following solutions: Search-enhanced generation: Although it can introduce external knowledge, it depends on the search quality, has poor real-time performance, and is prone to introducing noise.

[0008] Tip Engineering: A sophisticated prompt-guided model is designed, but it is sensitive to prompts and has weak generalization ability.

[0009] Lightweight fine-tuning techniques, such as LoRA and Adapter, reduce training costs, but there is still no mature solution for combining them with system-level capabilities such as semantic matching, multi-turn dialogue, and streaming output.

[0010] Therefore, there is an urgent need for a large model technology approach that balances domain adaptability, response accuracy, natural interaction, and system scalability. Summary of the Invention

[0011] In view of the above-mentioned shortcomings and deficiencies of the existing technology, the present invention provides a vertical domain large model interaction method and device based on LoRA fine-tuning and multi-dimensional semantic enhancement, which aims to realize the efficient, reliable and user-friendly deployment of large language models in vertical domains, and is particularly suitable for high-requirement scenarios such as equipment maintenance, technical support, and intelligent customer service.

[0012] To achieve the above objectives, the main technical solutions adopted by the present invention include: A method for interaction between large vertical models based on LoRA fine-tuning and multidimensional semantic enhancement includes the following steps: Step S1: Obtain text data from the vertical domain knowledge base, wherein the text data includes: Standard question-answer pair set ;in For the m-th standard problem, This represents the m-th standard answer, where N is the total number of question-answer pairs. Vertical domain technical documentation ; A collection of multiple user expression templates for each type of question. ,in As for the problem category, To express variants, This represents the j-th type of natural expression for the i-th type of problem; Preprocess the text data to generate training data from its output; Step S2: Based on the training data from Step S1, perform LoRA fine-tuning training on the large language model Qwen2.5-0.5B-instruct. This includes freezing all its original parameters, inserting trainable LoRA adapters into the fully connected layers or attention modules of the large language model, and correcting the forward propagation process using LoRA adapters so that the output features simultaneously contain the original knowledge representation and the vertical domain-adapted representation. Iteratively train the low-rank adapter using vertical domain data until the model's accuracy on the vertical domain test set reaches a preset threshold. This allows the model to learn domain-specific knowledge and response styles, resulting in a domain-adapted model. ; Step S3: Based on the Sentence-BERT architecture, construct a dual-tower semantic encoding model E(•). The model structure includes: an input layer for receiving question text, an embedding layer for converting text into word vectors, a bidirectional GRU layer for capturing contextual information, a pooling layer for generating sentence vectors, and an output layer for outputting semantic vectors of a preset dimension. Construct training samples using the user expression template set T: any two different expressions belonging to the same question category form a positive sample pair, and any two expressions belonging to different question categories form a negative sample pair; Using cosine similarity contrast loss as the loss function, a dual-tower semantic coding model is trained using training samples, which increases the similarity of positive sample pairs in the semantic vector space and decreases the similarity of negative sample pairs in the semantic vector space. Step S4: For each standard problem q, use the domain adaptation model. Generate several semantically identical but differently expressed user questions, add them to the user expression template set T, and expand the prompt word templates; for key concepts and entries in the knowledge base, use a domain adaptation model. Generate definitions, summaries, or related questions to form supplementary knowledge entries. Integrate with the original knowledge base to semantically expand the domain knowledge base; Step S5: Based on the acquired natural language query problem Using the semantic encoding model E(•) trained in step S3, Encoded as query vector ; Step S6, Calculation Compared with all standard question vectors in the knowledge base cosine similarity s i , If the maximum similarity , If the threshold is used for matching, then the matching problem is obtained directly. Execute step S7; if Then select the top three standard problems sorted from high to low similarity values. As a list of recommended questions, the system receives the target question selected by the user based on the list and uses that target question as the matching question. ; Step S7, The contextual information and related information are used to construct prompt words, which are then input into the domain adaptation model. Generate preliminary answer text .

[0013] Furthermore, it also includes... Perform chunking processing, splitting into sequences according to sentences or semantic units. , For the m-th sequence, output it block by block in sequence via a streaming protocol. A controllable delay is inserted between each block of answer text. During the output process, if the answer contains key points that can be followed up, an interactive prompt is inserted at the corresponding position.

[0014] Furthermore, in step S1, the vertical domain technical documents include textual corpora of equipment maintenance manuals, maintenance specifications, fault diagnosis procedures, technical bulletins, maintenance case libraries, parts manuals, and training materials.

[0015] Furthermore, in step S1, the preprocessing of text data includes cleaning, deduplication, format unification, special character filtering, long text segmentation, noise removal, and sensitive information desensitization of the original corpus data excluding question-answer pairs and user expression templates; verifying and classifying question-answer pairs; and classifying user expression templates according to question categories and expanding them based on synonym rewriting.

[0016] Furthermore, in step S1, the preprocessed data output fields include question ID, standard question, standard answer, and category.

[0017] Furthermore, in step S2, the forward propagation process is corrected using the LoRA adapter, and the correction is as follows: ; in, Here is the original frozen weight matrix of the model, and x is the input feature vector. Incremental weights introduced for LoRA Let A be a LoRA descending low-rank matrix. B is the LoRA up-row low-rank matrix, r is the LoRA low-rank dimension, satisfying r≪min(d,k), d is the input dimension of the original weight matrix, and k is the output dimension of the original weight matrix.

[0018] Furthermore, in step S4, the supplementary knowledge entries formed The rules for merging with the original knowledge base are as follows: the latest version of the same entry is retained, and new entries are directly inserted into the knowledge base.

[0019] A vertical-domain large-scale model interaction device based on LoRA fine-tuning and multi-dimensional semantic enhancement includes a knowledge base management module, a model training module, a semantic processing module, and an interactive reasoning module. The knowledge base management module is used to store text data and prompt word question-answer pair data expanded from the vertical domain knowledge base, and outputs the preprocessed training data and expanded prompt word question-answer pair data to the model training module and the interactive reasoning module. The model training module is equipped with a LoRA fine-tuning unit, which is used to load pre-trained weights and train a low-rank adapter matrix, and output fine-tuned model weights for a specific domain. The model training module receives training data from the knowledge base management module and outputs the trained domain-adaptive model weights to the streaming generation unit of the interactive reasoning module. The semantic processing module includes a vectorization unit and a matching calculation unit. The vectorization unit is used to convert text into vectors. It receives the query text and the standard question text from the knowledge base, and outputs the generated vector data to the matching calculation unit. The matching calculation unit is used to perform cosine similarity calculation and select the top three standard questions with high to low similarity values ​​as recommendation questions. The matching calculation unit receives the query vector and the question vector from the vectorization unit, and outputs the similarity calculation results and the top three standard questions with high to low similarity values ​​as recommendation questions to the decision control unit of the interactive reasoning module. The interactive reasoning module includes a decision control unit and a streaming generation unit. The decision control unit determines whether to execute an accurate answer or a recommended guidance based on a threshold. The decision control unit receives the similarity results and recommended questions output by the matching calculation unit, and inputs the matched standard questions or user-confirmed questions into the streaming generation unit. The streaming generation unit loads the fine-tuned model, executes reasoning, and outputs the results in streaming data format. The streaming generation unit receives the target question input by the decision control unit, and simultaneously loads the fine-tuned model weights output by the model training module and the question-answer pair data from the knowledge base management module to generate streaming answer results.

[0020] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned method for interaction of large vertical domain models based on LoRA fine-tuning and multidimensional semantic enhancement.

[0021] An electronic device includes at least one processor and at least one memory connected to the processor, the processor being configured to invoke program instructions in the memory to execute the vertical domain large model interaction method based on LoRA fine-tuning and multidimensional semantic enhancement.

[0022] The beneficial effects of this invention are: the vertical domain large model interaction method based on LoRA fine-tuning and multi-dimensional semantic enhancement adopts a technical system of lightweight fine-tuning, semantic matching, streaming interaction, and intelligent data expansion to achieve efficient, reliable, and user-friendly deployment of large language models in vertical domains.

[0023] This invention presents a vertical domain large-scale model interaction method based on LoRA fine-tuning and multi-dimensional semantic enhancement, which improves the adaptation accuracy of large language models in vertical domains, increases the hit rate of large language models' responses, enhances user experience and task completion rate, and expands the semantic understanding boundaries of the model. It is particularly suitable for high-requirement scenarios such as equipment maintenance, technical support, and intelligent customer service. Detailed Implementation

[0024] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below through specific embodiments.

[0025] This invention provides a method for interaction of large-scale vertical models based on LoRA fine-tuning and multi-dimensional semantic enhancement, which is particularly suitable for high-requirement scenarios such as equipment maintenance, technical support, and intelligent customer service. It includes the following steps: Step S1: Obtain text data from the vertical domain knowledge base, wherein the text data includes: Standard question-answer pair set ;in For the m-th standard problem, This represents the m-th standard answer, where N is the total number of question-answer pairs. Vertical domain technical documentation ; A collection of multiple user expression templates for each type of question. ,in As for the problem category, To express variants, This represents the j-th type of natural expression for the i-th type of problem; The text data is preprocessed to produce training data.

[0026] Specifically, this could involve obtaining structured and unstructured text data from maintenance knowledge bases, such as vehicle maintenance knowledge bases. Vertical domain technical documents. It can be a collection of various technical documents in the maintenance field, including equipment maintenance manuals, maintenance specifications, fault diagnosis procedures, technical bulletins, maintenance case libraries, parts manuals, training materials, and other textual materials.

[0027] Specifically, preprocessing the text data includes cleaning, deduplication, format standardization, special character filtering, long text segmentation, noise removal, and sensitive information anonymization of the raw corpus data (excluding question-answer pairs and user expression templates). Existing data processing methods can be used for these processes, as long as they can achieve the desired results. Question-answer pairs are manually verified and categorized to ensure data accuracy and standardization. User expression templates are categorized by question type and expanded manually and through model-assisted extensions based on synonym rewriting to form standardized training corpus.

[0028] The preprocessed data output format is as follows: standard question-answer pairs are stored in JSON format, with fields including question ID, standard question, standard answer, and category, i.e., question_id, standard_question, standard_answer, and category; domain documents are stored in TXT / Markdown format, split by chapter and labeled with their source.

[0029] Step S2: Based on the training data within the vertical domain prepared in Step S1, perform LoRA fine-tuning training on the large language model Qwen2.5-0.5B-instruct. This includes freezing all its original parameters, inserting trainable LoRA adapters into the fully connected layers or attention modules of the large language model, and correcting the forward propagation process through the LoRA adapters so that the output features simultaneously contain the original knowledge representation and the vertical domain-adapted representation. Iteratively train the low-rank adapter using vertical domain data until the model's accuracy on the vertical domain test set reaches a preset threshold. This allows the model to learn domain-specific knowledge and response styles, resulting in a domain-adapted model. .

[0030] Specifically, based on the training data within the vertical domain prepared in step S1 and the Qwen2.5-0.5B-instruct LLM (an open-source lightweight dialogue model with 0.5B parameters, supporting instruction following and question-answering generation), the model is fine-tuned using LoRA. The pre-trained large language model Qwen2.5-0.5B-instruct is loaded, and all its original parameters are frozen. A trainable LoRA adapter is inserted into the model's fully connected layers or attention modules, for the original weight matrix... Its forward propagation process is corrected as follows: ; in, Here is the original frozen weight matrix of the model, and x is the input feature vector. Incremental weights introduced for LoRA Let A be a LoRA descending low-rank matrix. B is the LoRA up-row low-rank matrix, r is the LoRA low-rank dimension, satisfying r≪min(d,k), d is the input dimension of the original weight matrix, and k is the output dimension of the original weight matrix.

[0031] Use step S1 We construct a fine-tuning dataset of instructions and train the LoRA parameters using the cross-entropy loss function, enabling the model to learn domain-specific knowledge and response styles, thus obtaining a domain-adaptive model. Training parameter settings: LoRA parameters are trained using the cross-entropy loss function. Training hyperparameters include: learning rate 1e⁻⁴, batch size 16, training epochs 10, AdamW optimizer selected, and weight decay 0.01. The training objective is to enable the model to learn domain-specific knowledge and response styles, resulting in a domain-adapted model. .

[0032] Step S3: Based on the Sentence-BERT architecture, construct a dual-tower semantic encoding model E(•). The model structure includes: an input layer for receiving question text, an embedding layer for converting text into word vectors, a bidirectional GRU layer for capturing contextual information, a pooling layer for generating sentence vectors, and an output layer for outputting semantic vectors of a preset dimension. Specifically, the output layer outputs semantic vectors with a dimension of 768.

[0033] Sample pair construction: Construct training samples using the user expression template set T: Construct positive sample pairs from any two different expressions belonging to the same question category. , Let l represent the l-th natural expression of the i-th type of problem, and form a negative sample pair by any two expressions belonging to different problem categories.

[0034] The training objective is to use cosine similarity contrast loss as the loss function and train a dual-tower semantic encoding model using training samples. This increases the similarity of positive sample pairs in the semantic vector space and decreases the similarity of negative sample pairs in the semantic vector space. This ensures that questions with the same intent are close in distance in the vector space, while questions with different intents are far apart.

[0035] Step S4: For each standard problem q, use the domain adaptation model. Generate several user questions with the same semantics but different expressions, add them to the user expression template set T, and expand the prompt word template. Specifically, this can be used for... Generate 10-15 user questions with the same semantics but different expressions, and add them to the template set. .

[0036] For key concepts and entries in the knowledge base, a domain adaptation model is used. Generate definitions, summaries, or related questions to form supplementary knowledge entries. It is integrated with the original knowledge base to semantically expand the domain knowledge base. Specifically, the resulting supplementary knowledge entries... The rules for merging with the original knowledge base are as follows: the latest version of the same entry is retained, and new entries are directly inserted into the knowledge base.

[0037] Step S5: Based on the acquired natural language query problem Using the semantic encoding model E(•) trained in step S3, Encoded as query vector .

[0038] Step S6, Calculation Compared with all standard question vectors in the knowledge base cosine similarity s i , If the maximum similarity , If the threshold is used for matching, then the matching problem is obtained directly. Execute step S7; if Then select the top three standard problems sorted from high to low similarity values. The top-3 standard questions are output as a list of recommended questions. After receiving the target question selected by the user based on the list of recommended questions, the target question is used as the matching question. .

[0039] Step S7, The contextual information and related information are used to construct prompt words, which are then input into the domain adaptation model. Generate preliminary answer text .

[0040] Specifically, it also includes the Perform chunking processing, splitting into sequences according to sentences or semantic units. , For the m-th sequence, output blocks sequentially via a streaming protocol, with controlled delays inserted between each block of answer text to simulate the rhythm of natural language output. During the output process, if the answer contains key points that can be followed up, interactive prompts can be inserted at the appropriate positions, allowing users to interrupt or ask further questions midway.

[0041] Specifically, a Suntech RM5220-F rack server can be selected, configured with ≥32 CPU cores, ≥256GB of memory, and ≥4TB of high-speed storage for data storage and model training in the above steps. It may also include a client for displaying information and providing users with options or confirmation of their intent.

[0042] The method of the present invention also includes updating the current dialogue state on the server and saving the query q. user Matching problem q match Interaction data, such as returned answers, is recorded in the dialogue log for subsequent model optimization and analysis.

[0043] The server runs a complete intelligent question-answering system, integrating the aforementioned knowledge base, model, preprocessing module, and question-answering module. The system records the user's selection of recommended questions, satisfaction rating of the generated answers, and subsequent follow-up questions as feedback signals.

[0044] At regular intervals, the LoRA adapter and semantic matching model are incrementally fine-tuned using newly collected high-quality interaction data to achieve continuous improvement in system performance.

[0045] This invention also provides a vertical domain large model interactive device based on LoRA fine-tuning and multi-dimensional semantic enhancement, including a knowledge base management module, a model training module, a semantic processing module, and an interactive reasoning module. Specifically, it can be deployed in a Shentai RM5220-F rack server, with communication connections between the modules, and the interactive device interacts with the user client for data exchange.

[0046] The knowledge base management module stores text data and expanded prompt word question-answer pair data from the vertical domain knowledge base. It outputs the preprocessed training data and expanded prompt word question-answer pair data to the model training module and the interactive reasoning module.

[0047] The model training module is equipped with a LoRA fine-tuning unit, which is used to load pre-trained weights and train a low-rank adapter matrix, and output fine-tuned model weights for a specific domain. The model training module receives training data from the knowledge base management module and outputs the trained domain-adaptive model weights to the streaming generation unit of the interactive inference module.

[0048] The semantic processing module includes a vectorization unit and a matching calculation unit. The vectorization unit is used to convert text into vectors. It receives the query text and the standard question text from the knowledge base, and outputs the generated vector data to the matching calculation unit. The matching calculation unit is used to perform cosine similarity calculation and select the top three standard questions with the highest to lowest similarity values ​​as recommendation questions. The matching calculation unit receives the query vector and the knowledge base question vector output by the vectorization unit, and outputs the similarity calculation results and the top three standard questions with the highest to lowest similarity values ​​as recommendation questions to the decision control unit of the interactive reasoning module.

[0049] The interactive reasoning module includes a decision control unit and a streaming generation unit. The decision control unit determines whether to execute an accurate answer or a recommended guidance based on a threshold. The decision control unit receives the similarity results and recommended questions output by the matching calculation unit, and inputs the matched standard questions or user-confirmed questions into the streaming generation unit. The streaming generation unit loads the fine-tuned model, executes reasoning, and outputs the results in streaming data format. The streaming generation unit receives the target question input by the decision control unit, and simultaneously loads the fine-tuned model weights output by the model training module and the question-answer pair data from the knowledge base management module to generate streaming answer results.

[0050] The user client can be an existing terminal device, including but not limited to industrial control touch screens, computers, mobile terminals, etc., and is not part of the hardware of this device. The vertical domain large model interactive device based on LoRA fine-tuning and multi-dimensional semantic enhancement communicates with the user client through a network, receives user queries uploaded by the client, and returns the inference results and streaming answers to the client for display.

[0051] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned method for interaction of large vertical domain models based on LoRA fine-tuning and multidimensional semantic enhancement.

[0052] The present invention also provides an electronic device, including at least one processor and at least one memory connected to the processor, wherein the processor is configured to call program instructions in the memory to execute the vertical domain large model interaction method based on LoRA fine-tuning and multidimensional semantic enhancement.

[0053] Compared with existing domain question answering systems based on general large models or simple retrieval, the vertical domain large model interaction method proposed in this invention, based on LoRA fine-tuning and multi-dimensional semantic enhancement, achieves significant improvements in several key performance indicators, as follows: (1) The accuracy of domain adaptation has been significantly improved, and the accuracy of manual evaluation has been improved.

[0054] The experimental setup is as follows: Test set: An internal test set for equipment maintenance, containing 1200 labeled samples, covering typical scenarios such as professional terminology explanation, fault diagnosis, operation procedure instructions, and parts replacement. Among them, there are 300 professional terminology samples, 400 fault diagnosis samples, 300 operation procedure samples, and 200 other samples.

[0055] Comparison with benchmark: Baseline 1: Zero-shot question answering is performed directly using the open-source general-purpose model Qwen2.5-0.5B-instruct without any domain-specific fine-tuning.

[0056] Baseline 2: A large-scale domain model of the same size fine-tuned using LORA (based on Qwen2.5-0.5B-instruct, with full parameter updates using the same training data).

[0057] Evaluation metric: Response accuracy (human evaluation; answers that are accurate, complete, and conform to domain standards are marked as correct).

[0058] Experimental results: On the internal test set in the field of equipment maintenance, compared with directly using the general large model Qwen2.5-0.5B-instruct for zero-shot question answering, the accuracy of the answer in scenarios involving professional terminology, fault diagnosis, and operation procedures increased from 68.2% to 84.7%, a relative improvement of more than 16.5%. Among them, the accuracy improvement in the fault diagnosis scenario was the most significant, from 59.3% to 82.1%, a relative improvement of 22.8%.

[0059] (2) The semantic matching module significantly improves the hit rate of the first round of answers.

[0060] The experimental setup is as follows: Test set: Contains 800 real user queries, covering common issues in the vehicle repair field, including 400 queries that completely match the standard question database and 400 queries that are semantically similar but have different expressions.

[0061] Benchmark: Traditional keyword matching methods (based on TF-IDF and BM25 algorithms).

[0062] Evaluation metrics: First-round match hit rate (the proportion of user queries that directly match the standard question), Top-3 recommendation hit rate (the proportion of the Top-3 recommendation list that contains the question that reflects the user's true intent).

[0063] Experimental results: By introducing Sentence-BERT-based semantic similarity matching and Top-3 recommendation mechanism, the system achieved an 89.5% first-round matching hit rate between user queries and the standard question database, which is 17.4% higher than the traditional keyword matching method.

[0064] Even when the query is not directly matched, the Top-3 recommendation list contains 94.2% of questions that reflect the user's true intent, significantly reducing the need for users to ask repeated questions or rewrite their queries and improving interaction efficiency.

[0065] (3) The streaming return mechanism significantly improves user experience and task completion rate.

[0066] The experimental setup is as follows: Experimental method: A / B testing, users were randomly divided into two groups. The control group was given a one-time return of the whole answer, while the experimental group was given a streaming return mechanism.

[0067] Sample size: 5,000 users in each group, covering scenarios such as vehicle repair consultation and operation guidance, with long answers and multi-step instructions accounting for 40%.

[0068] Evaluation metrics: average number of dialogue rounds, user task completion rate (the percentage of users who explicitly stated that their problem was solved), user satisfaction rating (out of 1-5 points), and operation step compliance rate (the percentage of users who completed the operation according to the answer steps).

[0069] Experimental results: A / B testing showed that using a streaming response method reduced the average number of conversation rounds by 23% compared to returning the complete answer all at once, increased the user task completion rate by 18%, and improved the average satisfaction rating from 3.8 / 5 to 4.5 / 5.

[0070] In scenarios involving long answers and multi-step instructions, streaming output combined with an interactive mode that allows users to ask questions midway through the process can increase the proportion of users who follow the correct steps by 34%.

[0071] (4) Data augmentation strategies effectively expand the semantic understanding boundary of the model.

[0072] The experimental setup is as follows: Test set: Contains 600 user-generated, colloquial, abbreviated queries containing typos, covering common issues in the vehicle repair field.

[0073] Baseline model: The baseline model without data augmentation (trained using only the original 500 labeled question-answer pairs).

[0074] Evaluation metrics: Expression coverage (the proportion of user expressions that the model can correctly understand) and recall (the proportion of correct answers recalled by the model in a cold start scenario).

[0075] Experimental results: By expanding the prompt word templates and semantically expanding the domain knowledge base, the model's coverage of different expressions of the same problem increased from 42% to 88%, significantly enhancing its generalization ability to colloquialisms, abbreviations, and misspellings.

[0076] In a cold start scenario with only 500 labeled question-answer pairs, the enhanced model showed a 29.6% improvement in recall on the test set compared to the unenhanced version, demonstrating its ability to quickly adapt to a small sample size.

[0077] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Any modifications, alterations, substitutions, and variations made by those skilled in the art to the above embodiments are within the scope of the present invention.

Claims

1. A method for interaction between large vertical models based on LoRA fine-tuning and multidimensional semantic enhancement, characterized in that, Includes the following steps: Step S1: Obtain text data from the vertical domain knowledge base, the text data including: Standard question-answer pair set ;in For the m-th standard problem, This represents the m-th standard answer, where N is the total number of question-answer pairs. Vertical domain technical documentation ; A collection of multiple user expression templates for each type of question. ,in As for the problem category, To express variants, This represents the j-th type of natural expression for the i-th type of problem; Preprocess the text data to generate training data from its output; Step S2: Based on the training data from Step S1, perform LoRA fine-tuning training on the large language model Qwen2.5-0.5B-instruct. This includes freezing all its original parameters, inserting trainable LoRA adapters into the fully connected layers or attention modules of the large language model, and correcting the forward propagation process using LoRA adapters so that the output features simultaneously contain the original knowledge representation and the vertical domain-adapted representation. Iteratively train the low-rank adapter using vertical domain data until the model's accuracy on the vertical domain test set reaches a preset threshold. This allows the model to learn domain-specific knowledge and response styles, resulting in a domain-adapted model. ; Step S3: Based on the Sentence-BERT architecture, construct a dual-tower semantic encoding model E(•). The model structure includes: an input layer for receiving question text, an embedding layer for converting text into word vectors, a bidirectional GRU layer for capturing contextual information, a pooling layer for generating sentence vectors, and an output layer for outputting semantic vectors of a preset dimension. Construct training samples using the user expression template set T: any two different expressions belonging to the same question category form a positive sample pair, and any two expressions belonging to different question categories form a negative sample pair; Using cosine similarity contrast loss as the loss function, a dual-tower semantic coding model is trained using training samples, which increases the similarity of positive sample pairs in the semantic vector space and decreases the similarity of negative sample pairs in the semantic vector space. Step S4: For each standard problem q, use the domain adaptation model. Generate several semantically identical but differently expressed user questions, add them to the user expression template set T, and expand the prompt word templates; for key concepts and entries in the knowledge base, use a domain adaptation model. Generate definitions, summaries, or related questions to form supplementary knowledge entries. Integrate with the original knowledge base to semantically expand the domain knowledge base; Step S5: Based on the acquired natural language query problem Using the semantic encoding model E(•) trained in step S3, Encoded as query vector ; Step S6, Calculation With all standard question vectors in the knowledge base cosine similarity s i , If the maximum similarity , If the threshold is used for matching, then the matching problem is obtained directly. Execute step S7; if Then select the top three standard problems sorted from high to low similarity values. As a list of recommended questions, the system receives the target question selected by the user based on the list and uses that target question as the matching question. ; Step S7, The contextual information and related information are used to construct prompt words, which are then input into the domain adaptation model. Generate preliminary answer text .

2. The vertical domain large model interaction method based on LoRA fine-tuning and multidimensional semantic enhancement according to claim 1, characterized in that: Also includes the Perform chunking processing, splitting into sequences according to sentences or semantic units. , For the m-th sequence, output it block by block in sequence via a streaming protocol. A controllable delay is inserted between each block of answer text. During the output process, if the answer contains key points that can be followed up, an interactive prompt is inserted at the corresponding position.

3. The vertical domain large model interaction method based on LoRA fine-tuning and multidimensional semantic enhancement according to claim 1, characterized in that: In step S1, the vertical domain technical documents include textual corpora of equipment maintenance manuals, maintenance specifications, fault diagnosis procedures, technical bulletins, maintenance case libraries, parts manuals, and training materials.

4. The vertical domain large model interaction method based on LoRA fine-tuning and multidimensional semantic enhancement according to claim 1, characterized in that: In step S1, the preprocessing of text data includes cleaning, deduplication, format unification, special character filtering, long text segmentation, noise removal, and sensitive information desensitization of the original corpus data excluding question-and-answer pairs and user expression templates. Validate and categorize the question-and-answer pairs; User expression templates are categorized by question type and expanded based on synonym rewriting.

5. The vertical domain large model interaction method based on LoRA fine-tuning and multidimensional semantic enhancement according to claim 1, characterized in that: In step S1, the preprocessed data output fields include question ID, standard question, standard answer, and category.

6. The vertical domain large model interaction method based on LoRA fine-tuning and multidimensional semantic enhancement according to claim 1, characterized in that: In step S2, the forward propagation process is corrected using the LoRA adapter, and the correction is as follows: ； in, Here is the original frozen weight matrix of the model, and x is the input feature vector. Incremental weights introduced for LoRA Let A be a LoRA descending low-rank matrix. B is the LoRA up-row low-rank matrix, r is the LoRA low-rank dimension, satisfying r≪min(d,k), d is the input dimension of the original weight matrix, and k is the output dimension of the original weight matrix.

7. The vertical domain large model interaction method based on LoRA fine-tuning and multidimensional semantic enhancement according to claim 1, characterized in that: The supplementary knowledge entries formed in step S4 The rules for merging with the original knowledge base are as follows: the latest version of the same entry is retained, and new entries are directly inserted into the knowledge base.

8. A vertical domain large-scale model interaction device based on LoRA fine-tuning and multi-dimensional semantic enhancement, characterized in that: It includes a knowledge base management module, a model training module, a semantic processing module, and an interactive reasoning module. The knowledge base management module is used to store text data and prompt word question-answer pair data expanded from the vertical domain knowledge base, and outputs the preprocessed training data and expanded prompt word question-answer pair data to the model training module and the interactive reasoning module. The model training module is equipped with a LoRA fine-tuning unit, which is used to load pre-trained weights and train a low-rank adapter matrix, and output fine-tuned model weights for a specific domain. The model training module receives training data from the knowledge base management module and outputs the trained domain-adaptive model weights to the streaming generation unit of the interactive reasoning module. The semantic processing module includes a vectorization unit and a matching calculation unit. The vectorization unit is used to convert text into vectors. It receives the query text and the standard question text from the knowledge base, and outputs the generated vector data to the matching calculation unit. The matching calculation unit is used to perform cosine similarity calculation and select the top three standard questions with high to low similarity values ​​as recommendation questions. The matching calculation unit receives the query vector and the question vector from the vectorization unit, and outputs the similarity calculation results and the top three standard questions with high to low similarity values ​​as recommendation questions to the decision control unit of the interactive reasoning module. The interactive reasoning module includes a decision control unit and a streaming generation unit. The decision control unit is used to determine whether to execute an accurate answer or a recommendation based on a threshold. The decision control unit receives the similarity results and recommended questions output by the matching calculation unit, and inputs the matched standard questions or questions confirmed by the user into the streaming generation unit. The streaming generation unit is used to load the fine-tuned model, perform inference, and output the results in streaming data format. The streaming generation unit receives the target question input by the decision control unit, and loads the fine-tuned model weights output by the model training module and the question-answer pair data in the knowledge base management module to generate streaming answer results output.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the vertical domain large model interaction method based on LoRA fine-tuning and multidimensional semantic enhancement as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes at least one processor and at least one memory connected to the processor, the processor being used to invoke program instructions in the memory to execute the vertical domain large model interaction method based on LoRA fine-tuning and multidimensional semantic enhancement as described in any one of claims 1 to 7.

Images