LLM capability enhancement method of hybrid enhancement strategy

Abstract

The invention discloses an LLM capability enhancement method of a hybrid enhancement strategy, and particularly relates to the technical field of natural language processing. The method comprises the following steps: obtaining corpus data and external literature data of a target application field, carrying out denoising, word segmentation and semantic indexing processing, and establishing a searchable heterogeneous knowledge base; secondly, classifying the corpus data by adopting a topic analysis and word vector clustering method to form a basic corpus set; performing multi-round parameter updating on the large language model by using a domain fine tuning algorithm based on the basic corpus set, and setting a fusion threshold value between a new parameter and an original parameter; and retrieving related literature fragments from the heterogeneous knowledge base through a matching algorithm, screening content conforming to field correlation by utilizing semantic similarity evaluation, fusing the content with the fine-tuned model, processing a fusion result by adopting a context fusion algorithm and an entity linking method, completing syntactic analysis, and generating a precise solution oriented to the target application field.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and more specifically, to an LLM capability enhancement method of a hybrid enhancement strategy. Background Art

[0002] The widespread application of large language models (LLMs) in the field of natural language processing has significantly improved language understanding and generation capabilities. However, in terms of knowledge enhancement in specific fields, due to the diversity and heterogeneity of external knowledge sources, existing technologies usually rely on retrieval enhancement generation and domain adaptability fine-tuning to optimize model capabilities. These technical methods obtain relevant information from external knowledge sources and combine them with the model's existing knowledge to improve the adaptability of language models to specific fields. Usually, information in external knowledge sources often exists in fragmented form and lacks a unified structure and semantic organization.

[0003] In the existing technology, when integrating external knowledge sources with existing knowledge of language models, it is difficult to efficiently merge the knowledge fragments due to the lack of semantic coherence and consistency. This will cause the generated content to be logically confusing or lack domain specificity, limiting the practical application of language models in complex problem analysis and reasoning. Summary of the invention

[0004] In order to overcome the above defects of the prior art, an embodiment of the present invention provides an LLM capability enhancement method of a hybrid enhancement strategy to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: A LLM capability enhancement method of a hybrid enhancement strategy comprises the following steps: Acquire corpus data and external literature data in the target application field, perform denoising and word segmentation on the corpus data and semantic indexing on the external literature data to establish a searchable heterogeneous knowledge base; The topic analysis algorithm is used to classify the corpus data, and the word vector clustering method is used to divide the corpus data into multiple semantic subdomains to form a basic corpus set; For the large language model that has been modeled, the domain fine-tuning algorithm is used to perform multiple rounds of parameter updates on the basic corpus set, and the fusion threshold between the new parameters and the original parameters is set during the parameter update process; Based on the matching algorithm, the document fragments related to the current user request are retrieved from the heterogeneous knowledge base, and the information irrelevant to the user request is eliminated by the semantic similarity evaluation method to obtain the document fragments that meet the domain relevance; The literature fragments that meet the domain relevance are fused with the large language model processed by the domain fine-tuning algorithm, and the fusion results are syntactically analyzed using the context fusion algorithm and entity linking method to generate answers to questions oriented to the target application field.

[0006] In a preferred embodiment, corpus data and external document data of the target application field are obtained, denoising and word segmentation are performed on the corpus data, and semantic indexing is performed on the external document data to establish a searchable heterogeneous knowledge base, including: Acquire corpus data of the target application domain, the corpus data including a set of text data generated in the target application domain; Perform noise removal on the corpus data, which includes deleting invalid characters, repeated content, and irrelevant text in the corpus data; The corpus data after noise removal is subjected to word segmentation processing, which includes segmenting the text content in the corpus data according to domain-specific terms using a word segmentation algorithm based on a domain vocabulary library, and retaining the integrity of domain-related terms; Obtain external literature data in the target application field, including literature from academic paper libraries, industry report libraries, and authoritative knowledge sources; Perform semantic indexing on external document data. The semantic indexing includes extracting semantic features from external document data using a semantic analysis algorithm and indexing the extracted semantic features based on a topic model of the target application domain. Establish a searchable heterogeneous knowledge base, which includes corpus data that has been processed by denoising and word segmentation and external document data that has been processed by semantic indexing, and classify and store the data in the heterogeneous knowledge base according to the classification rules of the target application field.

[0007] In a preferred embodiment, the corpus data is classified using a topic analysis algorithm, and the corpus data is divided into multiple semantic subdomains using a word vector clustering method to form a basic corpus set, including: The topic feature extraction is performed on the corpus data after denoising and word segmentation. The topic feature extraction includes a topic analysis algorithm based on the latent Dirichlet allocation model, which generates a topic probability distribution by performing multi-dimensional semantic feature analysis on the text content of the corpus data; The results of topic analysis are classified into different topic sets according to the text content of the corpus data. The classification process is to classify each text in the corpus into the topic set with the highest correlation based on the correlation between each text segment and the topic probability distribution. The text data in the classified topic set is processed by word vector representation. The word vector representation processing is to convert the words in the text data into high-dimensional vector representation using an embedding model trained based on the target domain vocabulary library; A text segmentation method based on a clustering algorithm is used to semantically cluster text data. The semantic clustering method includes using a K-means algorithm based on cosine similarity to cluster texts with high similarity into the same semantic subdomain by comparing the semantic similarity between word vectors. The clustered text collection is grouped according to semantic subdomains, and the text data and semantic features contained in each semantic subdomain are recorded to form a basic corpus collection.

[0008] In a preferred embodiment, the domain fine-tuning algorithm is used to perform multiple rounds of parameter updates on the basic corpus set for the large language model that has been modeled, and the fusion threshold between the new parameters and the original parameters is set during the parameter update process, including: Obtain a basic corpus set, which is organized according to the classification results of semantic subdomains; Initialize the parameter configuration of the large language model, which includes the original weight parameters of the pre-trained model and the target parameters that need to be updated during domain fine-tuning; The domain fine-tuning algorithm is used to optimize the parameters of large language models. The parameter optimization process includes training the model batch by batch through the basic corpus set, and calculating the update gradient of the model parameters according to the characteristics of the input data in each batch training; In the process of parameter updating, a fusion threshold is set, which is used to control the weight distribution ratio between the new parameter and the original parameter; The large language model is gradually optimized through multiple rounds of parameter updates, and the domain adaptation capability of the large language model is verified after each round of training.

[0009] In a preferred embodiment, the fusion threshold is used to control the weight distribution ratio between the new parameter and the original parameter, specifically: when the gradient update amplitude of the new parameter is lower than the fusion threshold, the weight of the original parameter is kept unchanged; when the gradient update amplitude of the new parameter is higher than the fusion threshold, the weight of the new parameter is proportionally distributed to the original parameter to achieve progressive fusion of domain knowledge.

[0010] In a preferred embodiment, based on a matching algorithm, document fragments related to the current user request are retrieved from a heterogeneous knowledge base, and information irrelevant to the user request is eliminated using a semantic similarity evaluation method to obtain document fragments that meet the domain relevance, including: Receive the current user's request and pre-process the user request; Loading document fragment data from a heterogeneous knowledge base, the document fragment data includes text content processed by semantic indexing and its corresponding semantic features; Perform preliminary relevance matching between user requests and document fragments in heterogeneous knowledge bases based on matching algorithms. The matching algorithms include keyword matching and topic matching. By calculating the intersection ratio between the user request keyword set and the keyword set in the document fragment, the document fragments related to the user request are screened out. The semantic similarity evaluation method is used to calculate the relevance of the document fragments related to the user request. The semantic similarity evaluation method includes converting the user request and the document fragment into a high-dimensional semantic vector using a text embedding model and measuring the semantic relevance between the user request and the document fragment according to the cosine similarity value of the semantic vector; The relevance threshold is set according to the calculation results of semantic similarity, and the document fragments with semantic similarity lower than the relevance threshold are eliminated, and the document fragments that meet the field relevance are retained.

[0011] In a preferred embodiment, document fragments that meet the domain relevance are fused with a large language model processed based on a domain fine-tuning algorithm, and the fusion results are syntactically analyzed using a context fusion algorithm and an entity linking method to generate answers to questions oriented to the target application domain, including: Obtain a collection of literature fragments that are relevant to the domain and call a large language model based on a domain-fine-tuned algorithm; The semantic features of a collection of domain-relevant document fragments are fused with the internal knowledge representation of a large language model. The fusion is accomplished through a contextual fusion algorithm. Perform entity linking on the fused results, extract named entities from the document fragments and fusion results, compare the extracted named entities with the entities in the domain knowledge base, identify the semantic associations between entities, and supplement and correct the contextual relationships of the entities based on the comparison results; Perform syntactic analysis on the results after context fusion and entity linking. The syntactic analysis includes parsing the syntactic structure, semantic dependency, and lexical consistency of the generated content. Based on the results of syntactic analysis, question answers oriented to the target application field are generated. The question answers are output in the form of structured text and retain semantic context information and entity link relationships.

[0012] In a preferred embodiment, the semantic features of a set of document fragments that meet the domain relevance are fused with the internal knowledge representation of a large language model, specifically: The text content of the document fragment and the context information of the user request are spliced ​​into a complete input sequence; The integrated context input sequence is vectorized to generate a context semantic vector; the semantic vector is mapped through an embedding model to capture the semantic association information in the context; The knowledge representation inside the large language model is adjusted based on the contextual semantic vector; the adjustment method is dynamic weighted fusion, that is, by calculating the weight ratio between the contextual semantic vector and the original knowledge vector of the model.

[0013] The technical effects and advantages of the LLM capability enhancement method of the present invention using a hybrid enhancement strategy are as follows: 1. By denoising, segmenting and subject-classifying the corpus data, and using the word vector clustering method to form multiple semantic subdomains, the efficient construction of the basic corpus collection is achieved. The establishment of a heterogeneous knowledge base can not only integrate data from a variety of external documents, but also improve retrieval efficiency through semantic indexing, providing high-quality knowledge input for subsequent retrieval enhancement generation. This method effectively solves the data integration problem caused by the fragmentation and heterogeneity of traditional external knowledge sources, and lays a solid foundation for knowledge adaptation in specific fields.

[0014] 2. Through domain fine-tuning and fusion strategies, large language models retain their original reasoning capabilities while absorbing external knowledge. Multiple rounds of parameter updates combined with the setting of fusion thresholds achieve a dynamic balance between new and old parameters, avoiding the problem of model overfitting or knowledge loss during domain fine-tuning. At the same time, through the application of context fusion algorithms and entity linking methods, the generated content has been significantly improved in terms of semantic coherence and logical consistency, and can respond to user needs more accurately. In addition, the combination of semantic similarity evaluation and syntactic analysis further ensures the accuracy and domain specificity of the generated content. The technical solution of the present invention has demonstrated strong practicality and adaptability in complex problem analysis and reasoning applications, effectively expanding the scope of application of large language models in specific fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of an LLM capability enhancement method of a hybrid enhancement strategy of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] Example: Figure 1 The present invention provides a LLM capability enhancement method of a hybrid enhancement strategy, which comprises the following steps: Acquire the corpus data and external literature data of the target application field, perform denoising and word segmentation on the corpus data and semantic indexing on the external literature data to establish a searchable heterogeneous knowledge base.

[0018] The topic analysis algorithm is used to classify the corpus data, and the word vector clustering method is used to divide the corpus data into multiple semantic subdomains to form a basic corpus set.

[0019] The domain fine-tuning algorithm is used to perform multiple rounds of parameter updates on the basic corpus set for the large language model that has been modeled, and the fusion threshold between the new parameters and the original parameters is set during the parameter update process.

[0020] Based on the matching algorithm, the document fragments related to the current user request are retrieved from the heterogeneous knowledge base, and the semantic similarity evaluation method is used to eliminate the information irrelevant to the user request to obtain the document fragments that meet the domain relevance.

[0021] The literature fragments that meet the domain relevance are fused with the large language model processed by the domain fine-tuning algorithm, and the fusion results are syntactically analyzed using the context fusion algorithm and entity linking method to generate answers to questions oriented to the target application field.

[0022] Acquire corpus data and external literature data in the target application field, perform denoising and word segmentation on the corpus data and semantic indexing on the external literature data, and establish a searchable heterogeneous knowledge base, including: Get the corpus data of the target application domain, which includes the text data set generated in the target application domain: According to the needs of the target application field, corpus data is obtained from the text data set generated in the field. The corpus data includes technical documents generated within the industry, research reports, user feedback data, online discussion forum content, and other text resources related to the target field.

[0023] Specifically, these data can be obtained through domain-specific data collection platforms or web crawler tools, and the authority and domain relevance of the data source must be ensured. The acquired corpus data needs to be preliminarily classified according to the source to facilitate targeted optimization in the subsequent processing stage.

[0024] Perform noise removal on the corpus data. Noise removal includes deleting invalid characters, repeated content, and irrelevant text in the corpus data: Invalid character removal: Clean up spaces, HTML tags, special symbols, and other non-content characters in the text.

[0025] Duplicate content removal: Duplicate detection is performed on text data, and duplicate paragraphs or documents are deleted through hash algorithms or similarity analysis.

[0026] Irrelevant text filtering: Eliminate text data irrelevant to the target field, such as advertising content and invalid messages, through keyword matching or semantic classification models.

[0027] The corpus data after noise removal is segmented. The segmentation process includes segmenting the text content in the corpus data according to domain-specific terms using a segmentation algorithm based on a domain vocabulary library, and retaining the integrity of domain-related terms: The corpus data after noise removal is segmented based on the domain vocabulary to ensure that the segmentation results can retain the integrity of domain-related terms to the greatest extent.

[0028] Vocabulary construction: Based on the existing technical terminology library in the target field, a domain-specific vocabulary library is generated in combination with commonly used word segmentation algorithms.

[0029] Word segmentation algorithm selection: A word segmentation algorithm based on a combination of statistics and rules is used to prioritize the integrity of domain-specific terms while avoiding ambiguity in word meaning.

[0030] Word segmentation result check: perform sampling verification on the word segmentation results to ensure that the domain terms are not split incorrectly, and optimize the word segmentation model by adjusting the rules.

[0031] Obtain external literature data in the target application field. External literature data includes literature from academic paper libraries, industry report libraries, and other authoritative knowledge sources: The sources of external literature data include but are not limited to academic paper databases (such as PubMed, IEEE Xplore), industry research report libraries (such as Gartner reports), and other professional and authoritative literature databases. Downloading and preprocessing is to obtain literature data through API interfaces or batch download tools, and perform unified format conversion for literature data formats (PDF, HTML, text) from different sources to ensure the compatibility of subsequent semantic processing.

[0032] The acquired external document data is stored separately from the corpus data and preliminarily classified to facilitate subsequent semantic indexing processing.

[0033] Perform semantic indexing on external document data. Semantic indexing includes using semantic analysis algorithms to extract semantic features from external document data and indexing the extracted semantic features based on the topic model of the target application field: Semantic feature extraction: Use pre-trained natural language processing models (such as BERT or GPT) to perform deep semantic analysis on document data to extract core concepts, keywords, and contextual semantic relationships in the document.

[0034] Topic model classification: Based on the topic model of the target field (such as LDA or PLSA), the extracted semantic features are classified into topics to generate multi-dimensional labels for the document content.

[0035] Indexing processing: Map the extracted semantic features and document content into a structured index, using an inverted index structure or knowledge graph storage to ensure that the document content can be efficiently retrieved through keywords and semantic relationships.

[0036] Establish a searchable heterogeneous knowledge base, which includes corpus data that has been processed by denoising and word segmentation and external document data that has been processed by semantic indexing. The data in the heterogeneous knowledge base is classified and stored according to the classification rules of the target application field: Integrate the processed corpus data with external literature data according to the classification rules of the topic model to ensure the semantic consistency and comprehensive coverage of the knowledge base content. According to the classification rules of the target field, divide the knowledge base data into multiple semantic subdomains and store them in a database that supports efficient retrieval (such as Elasticsearch or Neo4j). Embed a semantic retrieval module in the knowledge base to support multi-dimensional retrieval based on keywords, semantic similarity and topic models, and provide services for subsequent application scenarios.

[0037] The topic analysis algorithm is used to classify the corpus data, and the word vector clustering method is used to divide the corpus data into multiple semantic subdomains to form a basic corpus set, including: The topic feature extraction is performed on the corpus data after denoising and word segmentation. The topic feature extraction includes a topic analysis algorithm based on the latent Dirichlet allocation model. By performing multi-dimensional semantic feature analysis on the text content of the corpus data, the topic probability distribution is generated: Assume that the corpus data contains multiple documents, each document is represented as , which contains the number of words . Assume that the words in each document belong to There are many latent topics, and the latent Dirichlet allocation model generates topic distribution through the following steps: For each document , generate topic distribution , which follows the Dirichlet distribution: ;in, represents the hyperparameter of the Dirichlet distribution, which is used to control the sparsity of the topic distribution.

[0038] For each topic, generate the word distribution , which follows the Dirichlet distribution: ;in, A sparsity parameter representing the vocabulary distribution.

[0039] For Documents Each word in : From the topic distribution of documents Extract a topic from : ; Extract words from the word distribution of the chosen topic: ;in, The distribution of words representing the selected topics.

[0040] A number used to represent a document or text, used to distinguish different documents in the corpus data; A number representing the position of a word in a document or text; Indicates the number of the topic.

[0041] Through the above process, the topic probability distribution matrix of each document is generated , where each row represents the probability of a document on different topics.

[0042] The results of topic analysis are classified into different topic sets according to the text content of the corpus data. The classification process classifies each text in the corpus into the topic set with the highest correlation based on the correlation between each text segment and the topic probability distribution: According to the generated topic probability distribution matrix , the text content in the corpus data is classified into topics. Each text segment belongs to the most relevant topic set according to its topic distribution probability value.

[0043] For the first The text of this paragraph belongs to The probability of a topic is given by Indicates. Segments of text are grouped into topics, where: .

[0044] The classification process is implemented using a clustering algorithm to ensure that each text is uniquely attributed to a topic set. For multi-topic texts with similar probability values, a secondary classification model can be used to further optimize the classification results. The classified text is divided into multiple topic sets to form corpus data organized by topic for subsequent word vector representation processing.

[0045] The text data in the classified topic set is processed by word vector representation. The word vector representation processing uses an embedding model trained based on the target domain vocabulary library to convert the words in the text data into high-dimensional vector representations: When performing word vector representation processing on the text data in the classified topic set, firstly, according to the characteristics of the target field, a vocabulary library designed specifically for the field is selected, and an embedding model is trained based on it. The embedding model can represent each word in the text as a multi-dimensional vector to capture the semantic features and contextual information of the word. In actual operation, the text data will be input into the embedding model one by one, and the model will map the words into the vector space through the semantic rules in the vocabulary library, and each word will generate a vector of fixed length. These vectors can not only retain the semantic meaning of the word, but also reflect its relationship with other words according to the position and frequency of use of the word in the context. In the processing process, in order to improve the accuracy of semantic expression, the integrity and semantic features of the domain terms are specially considered, and they are retained in the high-dimensional vector without being weakened or split. After completion, the text data in each topic set will be converted into a semantic structure represented by a high-dimensional vector, providing a basis for subsequent clustering processing.

[0046] The text data is semantically clustered using a text segmentation method based on a clustering algorithm. The semantic clustering method includes using a K-means algorithm based on cosine similarity to cluster texts with high similarity into the same semantic subdomain by comparing the semantic similarity between word vectors: When semantic clustering is performed on text data represented by word vectors, the text data with similar semantics are classified into the same semantic subdomain through the clustering algorithm. In operation, a clustering method based on distance measurement is used to evaluate the semantic similarity between each text vector. In the actual clustering process, the text data vectors are organized into a multidimensional data set and classified according to the distribution pattern and semantic distance of the vectors. By comparing the proximity of the text vectors in the semantic space, texts with high semantic similarity are grouped into the same subdomain. The clustering algorithm randomly assigns cluster centers in the initial stage, and then gradually improves the accuracy of classification by repeatedly adjusting the classification results and recalculating the position of the center points. Ultimately, the text data in each semantic subdomain will share certain core semantic features, ensuring that the clustering results can not only reflect the internal consistency of the text, but also provide a more accurate corpus organization method for the subsequent domain fine-tuning steps.

[0047] Among them, whether the semantic similarity is high is achieved by professional and technical personnel in this field setting a preset similarity threshold according to actual conditions and comparing the semantic similarity with the preset similarity threshold. When the semantic similarity is greater than the preset similarity threshold, the semantic similarity is determined to be high.

[0048] The clustered text collection is grouped by semantic subdomains, and the text data and semantic features contained in each semantic subdomain are recorded to form a basic corpus collection: The clustering results are grouped according to semantic subdomains, and the text content and semantic features of each semantic subdomain are recorded. The text data in each semantic subdomain and its corresponding topic probability distribution and word vector representation are recorded and stored in a structured format. The basic corpus set is formed through the grouped corpus data to clarify the scope and content of each semantic subdomain.

[0049] The domain fine-tuning algorithm is used to perform multiple rounds of parameter updates on the basic corpus set for the large language model that has been modeled. During the parameter update process, the fusion threshold between the new parameters and the original parameters is set, including: Get the basic corpus set, which is organized according to the classification results of semantic subdomains: The basic corpus set is a data set organized according to the classification results of semantic subdomains. Each subdomain in the basic corpus set contains text data related to a specific topic and its semantic features, which are used to provide hierarchical training input for the model during domain fine-tuning. During the acquisition process, it is necessary to ensure that the corpus data of each semantic subdomain is complete and structured, such as adding corresponding topic labels and feature vectors to each text. These features provide data support for subsequent model fine-tuning, making the training more targeted.

[0050] Initialize the parameter configuration of the large language model, which includes the original weight parameters of the pre-trained model and the target parameters that need to be updated during domain fine-tuning: Load the pre-trained large language model into the training environment and initialize the model's parameter configuration. The parameter configuration includes the model's original weight parameters and the target parameters that need to be updated during domain fine-tuning. During initialization: Original weight parameters: represent the weights of the general knowledge that the model has learned during the pre-training phase. These parameters are the basis of the model’s performance.

[0051] Target parameters: represent the parameters that need to be optimized based on the base corpus during domain fine-tuning, such as the weights and biases of a specific layer.

[0052] During initialization, ensure that the model can match the input format of the base corpus set. For example, the input dimension, feature format, and batch size must be consistent with the model structure.

[0053] The domain fine-tuning algorithm is used to optimize the parameters of large language models. The parameter optimization process includes training the model batch by batch through the basic corpus set, and calculating the update gradient of the model parameters according to the characteristics of the input data in each batch of training: Batch training: The basic corpus set is divided into multiple training batches according to a fixed size. The model processes part of the corpus data in each batch and calculates the corresponding loss value.

[0054] Gradient calculation: In each batch, the gradient update value of the model parameters is calculated according to the characteristics of the input data and the loss value of the model output to ensure that the optimization direction is consistent with the domain goal.

[0055] Weight adjustment: Adjust the target parameters according to the optimization algorithm (such as Adam or SGD) to gradually reduce the loss value of the model on the basic corpus set and improve the adaptability of the model to the domain data.

[0056] The entire optimization process is based on the semantic features of the basic corpus set. The entire data set is traversed through batch training, and the optimization results of each batch are recorded.

[0057] The fusion threshold is set during the parameter update process. The fusion threshold is used to control the weight distribution ratio between the new parameters and the original parameters: The fusion threshold is used to control the weight distribution ratio between the new parameters and the original parameters. Specifically, when the gradient update amplitude of the new parameters is lower than the fusion threshold, the weight of the original parameters is kept unchanged to prevent interference with the pre-trained knowledge; when the gradient update amplitude of the new parameters is higher than the fusion threshold, the weight of the new parameters is proportionally distributed to the original parameters to achieve progressive fusion of domain knowledge. For example, the weight distribution can be dynamically adjusted according to the normalized value of the gradient, thereby gradually injecting new knowledge into the model.

[0058] The setting of the fusion threshold is based on the specific requirements of domain fine-tuning. For example, in highly professional fields (such as medicine or law), a higher fusion threshold can be set to retain more pre-trained knowledge.

[0059] The large language model is gradually optimized through multiple rounds of parameter updates. After each round of training, the domain adaptation capability of the large language model is verified to ensure that the model can accurately understand the semantic subdomain content in the basic corpus set: During the domain fine-tuning process, the large language model is gradually optimized by using multiple rounds of parameter updates. Each round of training takes the full basic corpus set as input to ensure that all semantic subdomains are fully covered.

[0060] After each round of training, the model’s domain adaptability is verified. The verification content includes: Semantic subdomain understanding ability: Verify the model's understanding depth of different semantic subdomains in the basic corpus, such as the accuracy of processing domain terms and topic context; Classification consistency: Check whether the model's performance in domain classification tasks is consistent, for example, whether the model can correctly classify domain data into corresponding semantic subdomains.

[0061] The optimization results of each round of training are used to adjust the training parameters of the next round, such as adjusting the learning rate or optimizing the hyperparameters of the algorithm, to improve the training efficiency of the model.

[0062] After multiple rounds of parameter updates are completed, the model is saved as a domain fine-tuning model. The domain fine-tuning model can not only retain the general knowledge of the original pre-trained model, but also accurately adapt to the domain features in the basic corpus set, serving as the core component of the subsequent knowledge fusion and question answering generation steps.

[0063] Based on the matching algorithm, the document fragments related to the current user request are retrieved from the heterogeneous knowledge base, and the information irrelevant to the user request is eliminated using the semantic similarity evaluation method to obtain the document fragments that meet the domain relevance, including: Receive the current user's request, which contains the user's query keywords or specific question description, and pre-process the user's request: Segment the text content in the user request, identify the keywords entered by the user according to the special vocabulary of the target application field, and ensure the integrity of the field-related terms. Clean up irrelevant stop words in the user request, such as prepositions, conjunctions and other common high-frequency words, to reduce the interference of irrelevant words on the matching results. Based on the vocabulary and synonym relationships of the target field, expand the core vocabulary in the user query to a set of synonyms and related words to cover possible semantic expressions.

[0064] The preprocessed user requests are stored in a standardized structured data format for use by subsequent matching algorithms.

[0065] Load document fragment data from a heterogeneous knowledge base. The document fragment data contains text content processed by semantic indexing and its corresponding semantic features: According to the target field requested by the user, a collection of literature fragments related to the field is loaded from the heterogeneous knowledge base. The literature fragments can come from different data sources, including industry reports, academic papers or professional databases. The semantic index is extracted from the loaded literature fragment data. The semantic index includes the subject tags, keyword sets and semantic feature vectors generated by the semantic embedding model of the literature, ensuring efficient processing and screening in the subsequent matching process.

[0066] Based on the matching algorithm, the user request and the document fragments in the heterogeneous knowledge base are preliminarily matched for relevance. The matching algorithm includes keyword matching and subject matching. By calculating the intersection ratio of the user request keyword set and the keyword set in the document fragment, the document fragments related to the user request are screened out: After the document fragments are loaded, a matching algorithm is used to perform preliminary relevance matching between user requests and document fragments in the heterogeneous knowledge base.

[0067] By matching the keyword set in the user request with the keyword set in the document fragment, the intersection ratio between them is calculated. The matching result reflects the basic relevance between the document fragment and the user request. The higher the ratio, the stronger the relevance.

[0068] Based on the similarity between the subject tags of the document fragments and the extended keyword set requested by the user, the subject relevance of the document fragments is further evaluated to ensure that the matching results not only consider the overlap of keywords, but also take into account the consistency of the overall semantics.

[0069] Document fragments with high relevance to user requests are retained, and document fragments with low relevance are eliminated to reduce the burden of subsequent semantic similarity calculations.

[0070] The semantic similarity evaluation method is used to calculate the relevance of the document fragments related to the user request. The semantic similarity evaluation method includes using a text embedding model to convert the user request and the document fragment into a high-dimensional semantic vector and measuring the semantic relevance between the user request and the document fragment according to the cosine similarity value of the semantic vector: The pre-trained semantic embedding model is used to map the text content of user requests and document fragments into high-dimensional semantic vectors, which can capture the contextual information and semantic relationships of the text.

[0071] The semantic relevance between the user request vector and the document fragment vector is quantified by calculating the cosine similarity between the two. The value of cosine similarity ranges from 0 to 1, and the closer the value is to 1, the stronger the semantic relevance is.

[0072] The document fragments are sorted according to the calculated semantic similarity values, and the document fragments with higher semantic similarity values ​​are retained first.

[0073] According to the calculation results of semantic similarity, a relevance threshold is set, and document fragments with semantic similarity lower than the relevance threshold are eliminated, and document fragments that meet the field relevance are retained: To ensure the domain relevance of the final results, a relevance threshold for semantic similarity is set. The specific value of the relevance threshold is determined by the domain requirements. For example, in highly technical fields (such as medicine or engineering), a higher threshold can be set to eliminate low-relevance document fragments.

[0074] The sorted document fragments are evaluated one by one, and the document fragments with semantic similarity below the relevance threshold are eliminated. Only the document fragment set with semantic similarity above the relevance threshold is retained to provide high-quality input data for the subsequent knowledge fusion step.

[0075] The literature fragments that meet the domain relevance are fused with the large language model processed by the domain fine-tuning algorithm. The fusion results are syntactically analyzed using the context fusion algorithm and entity linking method to generate answers to questions oriented to the target application domain, including: Get a collection of document fragments that are relevant to the domain and call a large language model based on the domain fine-tuning algorithm: A collection of document fragments that meet domain relevance is screened from the previous steps. The collection contains text content, topic tags, and semantic features related to user requests. The semantic features of the document fragment collection include high-dimensional vector representations obtained through semantic indexing and semantic similarity calculations, ensuring that the semantic information of each fragment is complete and relevant to domain requirements. At the same time, a large language model based on a domain fine-tuning algorithm is called. This model has undergone multiple rounds of parameter updates on the basic corpus set and has the ability to adapt to the target domain, providing support for subsequent knowledge fusion.

[0076] The document fragment collection and the large language model are the inputs of this step, and both will jointly participate in the knowledge fusion operation.

[0077] The semantic features of a collection of domain-relevant document fragments are fused with the internal knowledge representation of a large language model. The fusion is accomplished through a contextual fusion algorithm: The text content of the document fragment and the context information of the user request are spliced ​​into a complete input sequence to ensure that the semantic features of the document fragment are consistent with the semantic requirements of the user request.

[0078] The integrated context input sequence is vectorized to generate a context semantic vector. The dimension of the semantic vector is set according to the embedding model, and the common value is 512 or 1024. The semantic vector is mapped by the embedding model to capture the semantic association information in the context.

[0079] The knowledge representation inside the large language model is adjusted based on the contextual semantic vector. The adjustment method is dynamic weighted fusion, that is, by calculating the weight ratio between the contextual semantic vector and the original knowledge vector of the model, the original knowledge of the model is ensured not to be overwritten, while new domain information is injected.

[0080] Perform entity linking on the fused results, extract named entities from the document fragments and fusion results, compare the extracted named entities with the entities in the domain knowledge base, identify the semantic associations between entities, and supplement and correct the contextual relationships of the entities based on the comparison results to ensure that the entity references in the generated content are accurate: Extract named entities from the fusion results. Named entities include proper nouns, terms, or key concepts specific to a domain. Named entity extraction is achieved through a named entity recognition algorithm, which uses a domain pre-trained model to annotate text content and output an entity list.

[0081] The extracted named entities are semantically compared with the entities in the domain knowledge base. The semantic comparison uses a vector matching method to identify the semantic associations between entities by calculating the cosine similarity of entity vectors.

[0082] The contextual relationship of the entity is supplemented or corrected based on the comparison results. For example, when there is a semantic conflict between the extracted entity and an entity in the knowledge base, the entity in the knowledge base is given priority and the relevant description in the context is adjusted to be consistent with the domain standard.

[0083] Perform syntactic analysis on the results after context fusion and entity linking. Syntactic analysis includes parsing the syntactic structure, semantic dependency, and lexical consistency of the generated content: Parse the syntactic structure of the generated text, including subject-verb-object structure, modifiers, and clause relationships. Syntactic structure parsing is implemented using a syntactic analyzer, which displays the grammatical dependencies of the text through a dependency tree.

[0084] Analyze the semantic dependencies in the generated text, focusing on whether the semantic connections between core words are logical. For example, for sentences describing cause and effect relationships, ensure that the logical order of cause and effect relationships is correct.

[0085] Check the consistency of the vocabulary in the text to ensure that the use of vocabulary meets the domain standards and is consistent with the context information. For example, the expression of the same domain term in different locations should not cause ambiguity or inconsistency.

[0086] Generate question answers for the target application domain based on the results of syntactic analysis. The question answers are output in the form of structured text and retain semantic context information and entity link relationships: Based on the structured text information output by syntactic analysis, the answer content is organized according to the semantic requirements of the user's request. The answer content includes direct answers to questions, supplementary relevant background knowledge, and contextual reasoning results.

[0087] The generated answer content is output in the form of structured text. The format of the structured text includes hierarchical titles, paragraph numbers and annotations to ensure that the content is clear and easy to read.

[0088] Keep semantic context information in the answer content to ensure the logical coherence of the answer. For example, for complex questions, the answer content should indicate each step of the reasoning process and its basis.

[0089] Preserve entity link relationships in the answer content, such as providing explanations for the mentioned named entities or hyperlinking to related entries in the knowledge base to facilitate subsequent references and expansions.

[0090] The final generated question answers are provided in a structured form, which not only meets the needs of user requests, but also retains semantic association information and entity link relationships, providing high-quality answers to specific problems in the target application field.

[0091] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0092] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or may be transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0093] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0094] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0095] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0096] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, and may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0097] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0098] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0099] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0100] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A LLM capability enhancement method with a hybrid enhancement strategy, characterized in that: The steps include: Acquire corpus data and external literature data in the target application field, perform denoising and word segmentation on the corpus data and semantic indexing on the external literature data to establish a searchable heterogeneous knowledge base; The topic analysis algorithm is used to classify the corpus data, and the word vector clustering method is used to divide the corpus data into multiple semantic subdomains to form a basic corpus set; For the large language model that has been modeled, the domain fine-tuning algorithm is used to perform multiple rounds of parameter updates on the basic corpus set, and the fusion threshold between the new parameters and the original parameters is set during the parameter update process; Based on the matching algorithm, the document fragments related to the current user request are retrieved from the heterogeneous knowledge base, and the information irrelevant to the user request is eliminated by the semantic similarity evaluation method to obtain the document fragments that meet the domain relevance; The literature fragments that meet the domain relevance are fused with the large language model processed by the domain fine-tuning algorithm, and the fusion results are syntactically analyzed using the context fusion algorithm and entity linking method to generate answers to questions oriented to the target application field.

2. The LLM capability enhancement method of a hybrid enhancement strategy according to claim 1 is characterized in that: Acquire corpus data and external literature data in the target application field, perform denoising and word segmentation on the corpus data and semantic indexing on the external literature data, and establish a searchable heterogeneous knowledge base, including: Acquire corpus data of the target application domain, the corpus data including a set of text data generated in the target application domain; Perform noise removal on the corpus data, which includes deleting invalid characters, repeated content, and irrelevant text in the corpus data; The corpus data after noise removal is subjected to word segmentation processing, which includes segmenting the text content in the corpus data according to domain-specific terms using a word segmentation algorithm based on a domain vocabulary library, and retaining the integrity of domain-related terms; Obtain external literature data in the target application field, including literature from academic paper libraries, industry report libraries, and authoritative knowledge sources; Perform semantic indexing on external document data. The semantic indexing includes extracting semantic features from external document data using a semantic analysis algorithm and indexing the extracted semantic features based on a topic model of the target application domain. Establish a searchable heterogeneous knowledge base, which includes corpus data that has been processed by denoising and word segmentation and external document data that has been processed by semantic indexing, and classify and store the data in the heterogeneous knowledge base according to the classification rules of the target application field.

3. The LLM capability enhancement method of a hybrid enhancement strategy according to claim 1, characterized in that: The topic analysis algorithm is used to classify the corpus data, and the word vector clustering method is used to divide the corpus data into multiple semantic subdomains to form a basic corpus set, including: The topic feature extraction is performed on the corpus data after denoising and word segmentation. The topic feature extraction includes a topic analysis algorithm based on the latent Dirichlet allocation model, which generates a topic probability distribution by performing multi-dimensional semantic feature analysis on the text content of the corpus data; The results of topic analysis are classified into different topic sets according to the text content of the corpus data. The classification process is to classify each text in the corpus into the topic set with the highest correlation based on the correlation between each text segment and the topic probability distribution. The text data in the classified topic set is processed by word vector representation. The word vector representation processing is to convert the words in the text data into high-dimensional vector representation using an embedding model trained based on the target domain vocabulary library; A text segmentation method based on a clustering algorithm is used to semantically cluster text data. The semantic clustering method includes using a K-means algorithm based on cosine similarity to cluster texts with high similarity into the same semantic subdomain by comparing the semantic similarity between word vectors. The clustered text collection is grouped according to semantic subdomains, and the text data and semantic features contained in each semantic subdomain are recorded to form a basic corpus collection.

4. The LLM capability enhancement method of a hybrid enhancement strategy according to claim 1, characterized in that: The domain fine-tuning algorithm is used to perform multiple rounds of parameter updates on the basic corpus set for the large language model that has been modeled. During the parameter update process, the fusion threshold between the new parameters and the original parameters is set, including: Obtain a basic corpus set, which is organized according to the classification results of semantic subdomains; Initialize the parameter configuration of the large language model, which includes the original weight parameters of the pre-trained model and the target parameters that need to be updated during domain fine-tuning; The domain fine-tuning algorithm is used to optimize the parameters of large language models. The parameter optimization process includes training the model batch by batch through the basic corpus set, and calculating the update gradient of the model parameters according to the characteristics of the input data in each batch training; In the process of parameter updating, a fusion threshold is set, which is used to control the weight distribution ratio between the new parameter and the original parameter; The large language model is gradually optimized through multiple rounds of parameter updates, and the domain adaptation capability of the large language model is verified after each round of training.

5. The LLM capability enhancement method of a hybrid enhancement strategy according to claim 4 is characterized in that: The fusion threshold is used to control the weight distribution ratio between the new parameters and the original parameters. Specifically, when the gradient update amplitude of the new parameters is lower than the fusion threshold, the weight of the original parameters is kept unchanged; when the gradient update amplitude of the new parameters is higher than the fusion threshold, the weight of the new parameters is proportionally distributed to the original parameters to achieve progressive fusion of domain knowledge.

6. The LLM capability enhancement method of a hybrid enhancement strategy according to claim 1, characterized in that: Based on the matching algorithm, the document fragments related to the current user request are retrieved from the heterogeneous knowledge base, and the information irrelevant to the user request is eliminated using the semantic similarity evaluation method to obtain the document fragments that meet the domain relevance, including: Receive the current user's request and pre-process the user request; Loading document fragment data from a heterogeneous knowledge base, the document fragment data includes text content processed by semantic indexing and its corresponding semantic features; Perform preliminary relevance matching between user requests and document fragments in heterogeneous knowledge bases based on matching algorithms. The matching algorithms include keyword matching and topic matching. By calculating the intersection ratio between the user request keyword set and the keyword set in the document fragment, the document fragments related to the user request are screened out. The semantic similarity evaluation method is used to calculate the relevance of the document fragments related to the user request. The semantic similarity evaluation method includes converting the user request and the document fragment into a high-dimensional semantic vector using a text embedding model and measuring the semantic relevance between the user request and the document fragment according to the cosine similarity value of the semantic vector; The relevance threshold is set according to the calculation results of semantic similarity, and the document fragments with semantic similarity lower than the relevance threshold are eliminated, and the document fragments that meet the field relevance are retained.

7. The LLM capability enhancement method of a hybrid enhancement strategy according to claim 1, characterized in that: The literature fragments that meet the domain relevance are fused with the large language model processed by the domain fine-tuning algorithm. The fusion results are syntactically analyzed using the context fusion algorithm and entity linking method to generate answers to questions oriented to the target application domain, including: Obtain a collection of literature fragments that are relevant to the domain and call a large language model based on a domain-fine-tuned algorithm; The semantic features of a collection of domain-relevant document fragments are fused with the internal knowledge representation of a large language model through a contextual fusion algorithm. Perform entity linking on the fused results, extract named entities from the document fragments and fusion results, compare the extracted named entities with the entities in the domain knowledge base, identify the semantic associations between entities, and supplement and correct the contextual relationships of the entities based on the comparison results; Perform syntactic analysis on the results after context fusion and entity linking. The syntactic analysis includes parsing the syntactic structure, semantic dependency, and lexical consistency of the generated content. Based on the results of syntactic analysis, question answers oriented to the target application field are generated. The question answers are output in the form of structured text and retain semantic context information and entity link relationships.

8. The LLM capability enhancement method of a hybrid enhancement strategy according to claim 7 is characterized in that: The semantic features of a collection of domain-relevant literature fragments are fused with the internal knowledge representation of a large language model, specifically: The text content of the document fragment and the context information of the user request are spliced ​​into a complete input sequence; The integrated context input sequence is vectorized to generate a context semantic vector; the semantic vector is mapped through an embedding model to capture the semantic association information in the context; The knowledge representation inside the large language model is adjusted based on the contextual semantic vector; the adjustment method is dynamic weighted fusion, that is, by calculating the weight ratio between the contextual semantic vector and the original knowledge vector of the model.

Images