Intranet and extranet knowledge association retrieval method based on RAG technology

Abstract

The invention relates to the technical field of association retrieval, and discloses an internal and external network knowledge association retrieval method based on an RAG technology. The method comprises the following steps: S1, based on query content input by a user, retrieving matched entries from an enterprise internal knowledge base, and reducing a retrieval range through metadata filtering; and S2, converting internal retrieval results into vector representation through an embedded model, and sorting the results according to correlation on the basis of cosine similarity. By combining an enterprise internal knowledge base and external network search, the limitation that a traditional method only depends on a single data source is broken through, the coverage range of knowledge is remarkably expanded, professional and authoritative domain knowledge is provided for the enterprise internal knowledge base, and latest industry dynamic and open information is supplemented by external network search, so that the enterprise internal knowledge base and the external network search are combined, and the enterprise internal knowledge base and the external network search are combined. Through dynamic weight adjustment and a unified sorting algorithm, the most relevant data source can be preferentially selected according to user requirements, so that balance is achieved between content depth and breadth.

Description

Technical Field

[0001] The present invention relates to the technical field of association retrieval, and in particular to an internal and external network knowledge association retrieval method based on RAG technology. Background Art

[0002] With the rapid development of information technology, enterprises and individuals have an increasing demand for fast and accurate information. Although traditional search engines can provide a large amount of information, they have certain limitations in terms of professionalism and accuracy in specific fields. In recent years, the progress of deep learning and natural language processing technology has greatly promoted the development of intelligent question-answering systems, especially in the areas of internal knowledge management and customer service. However, most existing intelligent question-answering systems rely on pre-trained large-scale language models, which often lack the understanding and application of internal enterprise knowledge when generating answers, resulting in the need to improve the quality and applicability of answers.

[0003] Most search engines are based on broad general knowledge and cannot fully understand and process specialized terms and in-depth information in specific fields, making it difficult to meet the high standards of corporate users in terms of professionalism and accuracy. Insufficient use of internal knowledge bases: Enterprises usually have rich internal knowledge resources, including technical documents, operation manuals, and internal reports. These resources often exist in isolation in existing retrieval methods and are difficult to effectively integrate with external public information, resulting in the failure to fully utilize the value of internal knowledge bases; Traditional retrieval-based question-answering systems usually only provide a list of information rather than an integrated and organized answer. Although existing intelligent question-answering systems have introduced pre-trained language models, these models have limited understanding of corporate internal knowledge, and the answers they generate are still insufficient in accuracy and pertinence. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides an internal and external network knowledge association retrieval method based on RAG technology, which solves the problems of difficult to efficiently integrate the internal knowledge base of an enterprise with external network information, insufficient relevance and professionalism of retrieval results, and low quality of intelligent answers.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: an internal and external network knowledge association retrieval method based on RAG technology, comprising the following steps: S1. Based on the query content entered by the user, matching entries are retrieved from the enterprise internal knowledge base, and the search scope is narrowed by filtering through metadata; S2. The results of internal retrieval are converted into vector representations through the embedding model, and the results are sorted by relevance based on cosine similarity; S3, calling the external search engine interface to perform network search, and converting the external search results into vector representation; S4, integrate the vectorized results of the internal knowledge base and external search, and perform unified sorting based on relevance; S5. Append the top-ranked multi-source search results to the prompt information, and generate the final answer in combination with the large language model.

[0006] Preferably, the retrieval of the enterprise internal knowledge base in step S1 specifically includes the following steps: S1.1. Token-based authentication mechanism, using JSON Web Tokens (JWT) to verify user identity; S1.2. The system dynamically generates query keywords based on user input, and selects highly relevant keywords based on semantic similarity to ensure more accurate search results; S1.3, role-based access control, different user roles have different knowledge base access rights; S1.4. Verify the file type and size of the uploaded file and parse the content, clean up irrelevant characters, extract the core content of the file for knowledge base retrieval, and prevent malicious file uploads.

[0007] Preferably, the metadata filtering formula in step S1 is: ; in; D Represents a collection of documents in an enterprise knowledge base; M Represents metadata conditions; R Represents a filtered collection of documents.

[0008] Preferably, the embedding model in step S2 vectorizes the text using the following formula: ; in; T To retrieve text; V is the generated vector; function This is achieved by a pre-trained embedding model; The system combines keyword weights for matching and calculates relevance scores based on word frequency and position weights to ensure more accurate search results.

[0009] Preferably, the correlation ranking in step S2 is calculated based on cosine similarity, and the formula is: ; in; V 1 and V 2 are the user query vector and the knowledge base entry vector respectively.

[0010] Preferably, step S3 specifically includes: S3.1. The system simulates the behavior of search engines through web crawlers, crawls web page content in real time, and extracts highly relevant text data; S3.2, pre-processing the text content in the web search results, including removing irrelevant characters, formatting the text content, and removing duplicates; S3.3, use the embedding model to convert the preprocessed web search results into vector representation; S3.4. The system has a built-in exception handling mechanism that supports automatic retry of network requests and records exception logs to ensure that the retrieval process is stable and reliable.

[0011] Preferably, the result of the external search in step S3 is vectorized by the following formula: ; in; S Search document content for external purposes; V ext is the vectorized result; function Implemented by semantic embedding model.

[0012] Preferably, the unified sorting in step S4 is based on the following formula: ; in; R is the ranking score; Sim internal and Sim external are the relevance scores and weight coefficients of the internal knowledge base and external search results respectively. α and β Can be adjusted according to actual scenario.

[0013] Preferably, the large language model in step S5 generates an answer based on the following process: S5.1. The system supports context management, dynamically splicing user historical input, search results and new input content to build a complete prompt, and improving the coherence and accuracy of the answers generated by the model; S5.2. The system supports streaming response mode, gradually returning the content generated by the model, reducing waiting time and improving user experience; S5.3. Use prompt input to include context and multi-source search results; S5.4. Dynamically expand the prompt content based on the context.

[0014] Preferably, the large language model is a GPT series model, which generates the final answer in combination with the following context optimization method: Embed multi-source retrieval results into language models in segments; The final generated answers are further checked for relevance.

[0015] The present invention provides an internal and external network knowledge association retrieval method based on RAG technology. It has the following beneficial effects: 1. The present invention breaks through the limitation of traditional methods that rely only on a single data source by combining the internal knowledge base of the enterprise and external network search, significantly improves the coverage of knowledge, and provides professional and authoritative domain knowledge for the internal knowledge base of the enterprise, while the external network search supplements the latest industry trends and open information. Through dynamic weight adjustment and unified sorting algorithm, the present invention can give priority to the most relevant data source according to user needs, thereby achieving a balance between content depth and breadth.

[0016] 2. The present invention adopts a semantic embedding model to vectorize internal and external data, and realizes accurate evaluation of text relevance through quantitative methods such as cosine similarity. At the same time, combined with a dynamic weight adjustment strategy, the system can adapt to different user roles and query scenarios, and give priority to returning search items that best match the user's query content, overcoming the problem of result deviation caused by semantic ambiguity in traditional keyword matching, and significantly improving retrieval accuracy and relevance.

[0017] 3. By introducing a large language model, the present invention can dynamically construct prompts based on multi-source retrieval results and generate high-quality natural language answers in combination with contextual information. The answers generated by the model of the present invention are more context-relevant and can dynamically expand prompts to respond to user follow-up questions, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings of the present invention specification to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Please refer to the attached Figure 1 , the embodiment of the present invention provides an internal and external network knowledge association retrieval method based on RAG technology, characterized in that it includes the following steps; S1. Based on the query content entered by the user, matching entries are retrieved from the enterprise internal knowledge base, and the search scope is narrowed by filtering through metadata; S2. The results of internal retrieval are converted into vector representations through the embedding model, and the results are sorted by relevance based on cosine similarity; S3, calling the external search engine interface to perform network search, and converting the external search results into vector representation; S4, integrate the vectorized results of the internal knowledge base and external search, and perform unified sorting based on relevance; S5. Append the top-ranked multi-source search results to the prompt information, and generate the final answer in combination with the large language model.

[0021] Specific; In this embodiment, the purpose of step S1 is to quickly retrieve items that meet the query conditions from the enterprise internal knowledge base after receiving the query input by the user, and effectively filter the search scope through certain technical means to improve the search efficiency and accuracy of the system; In this embodiment, the construction and retrieval of the enterprise internal knowledge base: The internal knowledge base of an enterprise consists of structured and unstructured data, such as table data, documents, emails, and real-time updated data streams. To ensure the uniformity of data, the knowledge base adopts a standardized storage solution and annotates all data with metadata; In order to improve the accuracy of search results, after the user enters a query, the system will dynamically generate query keywords and filter out highly relevant keywords by combining semantic similarity calculation. This process uses the embedding model to vectorize and compare the user's query content with the entries in the knowledge base, generate a keyword list, and give priority to keywords with high matching as search conditions; Metadata includes, but is not limited to, attributes such as creation time, document type, content subject, and access rights; Maintain the data integrity and timeliness of the knowledge base through periodic data update and cleanup mechanisms; Specifically, after the user enters a query, the system will first perform a preliminary search of the knowledge base content. This process mainly relies on metadata filtering technology, and the filtering formula is as follows: ; in: D is the collection of all documents in the knowledge base; M(d) Represents a collection of documents that meet certain metadata conditions, which may include document type, last updated time, user role permissions, etc. is the filtered document collection; The conditions for metadata filtering can be dynamically adjusted based on user identity. For example, when an administrator role accesses the system, all documents can be accessed, while an ordinary employee role can only retrieve a subset related to their work content; User authentication and security control: To ensure the security of internal enterprise data, knowledge base retrieval has introduced multiple layers of security control measures.

[0022] As an option, user authentication uses JSON Web Tokens (JWT) to verify user identity through encrypted tokens; The JWT authentication process includes: When a user logs in for the first time, the server generates an encrypted token containing the user's identity information and returns it to the user; During each query, the system verifies whether the token submitted by the user is legitimate; If the token verification fails, the user is denied access to the knowledge base; Role-Based Access Control (RBAC) is applied to knowledge base permission management; The specific rules of RBAC are as follows: The administrator role can access all knowledge base content; Regular employee roles can only access data relevant to their job permissions, such as technical documentation for a specific project; The Customer role can only access public knowledge base entries; After verifying the type and size of the files uploaded by users, the files will be parsed to support multiple formats, including removing irrelevant characters, formatting text content and extracting core content. The extracted content is stored in the knowledge base for subsequent retrieval, thus preventing malicious files from affecting system performance and data security. The query efficiency of the knowledge base is improved through multi-threaded retrieval acceleration technology. Specifically, the knowledge base content is stored in fragments, and each fragment is processed by an independent thread. After the user query is split, multiple threads perform metadata filtering in parallel, and finally merge the results of all fragments into a filtered document set. ; To ensure the accuracy of the search results, the system will automatically remove duplicate entries when merging the results. For example, if the same contract document is retrieved repeatedly by multiple threads, only one entry will be retained when merging.

[0023] In this embodiment, step S2 is used to vectorize the items retrieved from the enterprise internal knowledge base and select the items that best match the user's query content by relevance sorting. The main function of step S2 is to convert unstructured text information into numerical vectors so as to realize the computer's quantitative evaluation of text similarity; Embedding model selection and vectorization processing: This embodiment uses a pre-trained semantic embedding model to vectorize the retrieved text entries. It should be noted that the role of the embedding model is to represent natural language text as high-dimensional dense vectors, which can reflect the semantic similarity between texts in the semantic space; Specifically, the vectorization formula for text is as follows;

[0024] in: T The text to be processed, such as user query or retrieved document content; V is the vectorization result of the text; It is an embedding model, which includes semantic feature extraction and vector mapping functions; The pre-training data of the embedding model can include large-scale public corpora, such as general natural language processing corpora. To enhance the domain adaptability of the embedding model, the model can be fine-tuned on the internal data of the enterprise through transfer learning. The system calculates the matching degree based on the keyword weights and calculates the relevance score of the search terms using the following formula;

[0025] in: is the weight of the keyword; is the frequency of the keyword in the document; The weight is dynamically adjusted based on the position and semantic importance of the keyword; Application of correlation calculation formula: After the vectorization is completed, the present invention evaluates the similarity between the user query vector and the search item vector through a correlation calculation formula. This embodiment uses cosine similarity as a similarity measurement method, and its formula is as follows:

[0026] in: A vector representation of the user query text; A vector representation representing a knowledge base entry; •Represents vector dot product operation; and are the magnitudes of the vectors respectively; The system searches for all search terms Computing and Querying Vectors The cosine similarity of the query is used to generate a similarity list. Then, the entries are sorted in descending order according to the similarity score, ensuring that the entries most relevant to the user's query are ranked first; The calculation of cosine similarity can effectively avoid the influence of text length differences on similarity evaluation. Therefore, even if the user's query text is short, the system can still accurately evaluate its semantic relevance to the long document; Sorting optimization techniques: In some embodiments, to improve the reliability of relevance ranking, the present invention combines a weighted ranking strategy to incorporate user priority requirements and metadata weights into the ranking score. Specifically, the comprehensive ranking formula is as follows:

[0027] in: R represents the ranking score; α and β is the weight coefficient, which is used to balance the influence of cosine similarity and metadata weight; W m Indicates metadata weight, such as document update time, access frequency, and other indicators; The system can be dynamically adjusted according to user roles α and β For example, for the administrator role, the system tends to return entries with a more recent update time first, increasing β For ordinary employee roles, more emphasis is placed on the content relevance of the items, increasing α The weight of Ranking optimization technology not only improves the quality of search items, but also avoids the ranking errors that may be caused by over-reliance on a single similarity indicator to a certain extent; In this embodiment, step S2 realizes the quantitative evaluation of text similarity by combining the embedding model with the sorting optimization technology, providing efficient technical support for the subsequent integration of multi-source information. At the same time, the dynamic adjustment method combined with different scenarios improves the applicability and reliability of the system. It should be noted that the specific implementation of the embedding model and the sorting strategy can be flexibly configured according to different application requirements to meet more diverse retrieval requirements.

[0028] In this embodiment, step S3 obtains relevant network information through an external search engine interface based on the internal knowledge base search results, and pre-processes and vectorizes the obtained external data to provide external knowledge support for subsequent unified sorting and comprehensive answer generation; Acquisition of external search data; In this embodiment, the acquisition of external search data: The system simulates search engine behavior through web crawlers, uses the Crawler class to implement web crawlers, simulates search engines to crawl web page content, supports dynamic URL splicing and delay mechanisms, extracts the main content of search results, crawls web page content in real time, and extracts highly relevant text data. This process combines user query content to generate search parameters to ensure that the network information obtained is highly relevant to user needs; The system optimizes external search calls in the following ways: Limit the number of results returned, for example, returning the first five most relevant documents to the user's query by default to reduce network bandwidth consumption; Set the search language, for example, in scenarios with multi-language support, you can give priority to search results in the same language as the user interface; Specify a time range, such as searching for content updated within the last year; Preprocessing of external search results; In this embodiment, in order to ensure the quality and availability of external search results, the present invention performs the following preprocessing steps on the acquired external data: Text cleaning: Remove HTML tags, special characters, and irrelevant content (such as advertisements and navigation bar information).

[0029] Remove duplicate content, such as multiple occurrences of the same document returned by different search engines.

[0030] Text formatting: Merge document title, abstract and main text into a standardized text structure;

[0031] in, T ext is the preprocessed external document text; Deduplication and prioritization: Compare the title and abstract of the document. If the similarity is higher than the set threshold, keep the more relevant document. Prioritize documents based on publication time, citation count, and other indicators to increase the weight of the latest and most authoritative content; The preprocessed document collection will be used as vectorized input to ensure the efficiency and accuracy of subsequent processing; In this embodiment, the preprocessed external search document T ext It is further converted into a vector representation. The vectorization process uses a semantic embedding model consistent with the internal knowledge base to ensure that the internal and external data are compared in the same vector space; Specifically, the vectorization formula for external documents is as follows;

[0032] in: T ext Search document text for external purposes; V ext is the vector representation of the external document; is the embedding function; The embedding model can be further optimized through joint training methods. For example, fine-tuning the internal knowledge base data and the external open corpus can be combined to enable the embedding model to better capture the semantic relationship between the two data sources. Exception handling mechanism: The system has a built-in exception handling module that supports automatic exponential backoff and retry mechanism for network requests, thus enhancing the stability of network requests; Exponential backoff mechanism: Backoff waiting time;

[0033] The waiting time for the current retry; is the basic backoff time; is the current number of retries; Add random jitter;

[0034] is the jitter factor, ranging from [-0.1, 0.1]; The abnormal information is recorded in the log. This mechanism improves the stability and reliability of network search and ensures that high-quality data can still be provided under unstable network conditions. In this embodiment, step S3 provides users with rich and relevant external knowledge by efficiently acquiring and processing external search data, ensuring the expansibility of the system in terms of knowledge coverage. All technical aspects of this embodiment have been fully disclosed, and combined with multiple optimization strategies and extended applications, a complete basis for those skilled in the art to reproduce the technical solution of the present invention is provided.

[0035] In this embodiment, step S4 realizes the key link of knowledge fusion and ranking optimization. Its core task is to uniformly sort the vectorized results of the internal knowledge base and external search. Through comprehensive evaluation of multi-source data, step S4 provides highly relevant and high-quality candidate search results for the final answer generation. In this embodiment, step S4 specifically includes a comprehensive ranking formula of multi-source vectors, a weight dynamic adjustment strategy, and a detailed implementation of an extended optimization technology, which are specifically described as follows; Comprehensive ranking of vectors from multiple sources: In this embodiment, a weight-based comprehensive ranking method is used to integrate the relevance scores of the internal knowledge base and external search into a unified ranking framework. The scoring formula for the comprehensive ranking is as follows:

[0036] in: R is the comprehensive ranking score; and Score the relevance of internal knowledge base results and external search results respectively; α and β is the weight coefficient, satisfying α+β=1 , used to balance the importance of two types of data; The weight coefficient setting can be dynamically adjusted according to the user's specific needs or query type. For example, for professional queries within an enterprise that require high accuracy, the system can increase α For issues that require extensive external information support, the weight can be increased. β The weight of Specifically, the relevance score and The calculations are based on cosine similarity, and the formulas are as follows;

[0037] in: is the vector representation of the user query; is a vector representation of an internal or external data entry; Unified sorting ensures the overall quality of search results by comprehensively calculating the relevance scores of data from two sources, avoiding the one-sidedness that may be caused by a single data source.

[0038] Dynamic weight adjustment strategy: This embodiment designs a strategy for dynamically adjusting weights to adapt to different query scenarios and user needs; Based on data availability: When there are fewer relevant results in the internal knowledge base, the system automatically increases β value, use external search results with higher weight; On the contrary, when the search results of the internal knowledge base are richer, the system tends to improve α The weight of .

[0039] Based on user role: Ordinary employees may be more interested in internal business data, while administrators or decision makers may be more interested in external information of the industry. Therefore, the system will adjust the weight distribution for different user roles. The dynamic adjustment strategy can significantly improve the system's adaptability and user experience, ensuring that the comprehensive ranking results meet the actual needs of users; Sorting optimization techniques: In order to further improve the efficiency and accuracy of unified sorting, this embodiment combines the following optimization techniques: Confidence-based candidate screening: The system calculates a confidence score for each data entry before comprehensive ranking;

[0040] in, and is the confidence weight parameter.

[0041] Entries with confidence levels lower than the set threshold will be filtered to ensure the overall reliability of candidate results; Sorting adjustment of time factors: The scoring of the items is adjusted by adding a time decay factor. The specific formula is as follows;

[0042] in, is the time-adjusted comprehensive ranking score, is the difference between the document publishing time and the current time, The time weight factor is particularly suitable for queries with high timeliness requirements, and can prioritize recently updated items. In this embodiment, through the comprehensive application of unified sorting formula, dynamic adjustment strategy and sorting optimization technology, step S4 realizes the effective integration of multi-source data, providing a high-quality input basis for the subsequent large language model to generate the final answer. All technical links have been fully disclosed to ensure that those skilled in the art can fully reproduce the technical solution of the present invention.

[0043] In this embodiment, step S5 generates a comprehensive answer using a large language model. This step is based on the sorting result of the aforementioned step S4, dynamically constructs prompts, combines the user query context, and achieves deep integration with multi-source information, thereby generating a high-quality natural language answer that meets the user's needs; Prompt build: Prompt is used to pass user query content and multi-source retrieval results to the large language model to generate answers that are semantically and contextually consistent; The construction logic of Prompt is as follows;

[0044] in: The user's query content; The front after sorting in step S4 Search results; Context represents the contextual integration of multiple search results, such as merging the content into a continuous text in the form of a summary; Streaming response mode: In order to reduce user waiting time, the system adopts a streaming response mode, gradually returning results while the model generates content. This mechanism improves the user experience and is particularly suitable for scenarios with long answers; Generation optimization and verification: Control the diversity of answers through temperature coefficients, and verify the relevance of generated content based on user feedback to ensure the accuracy of the final answers; Dynamic expansion of context: This embodiment designs a context dynamic expansion mechanism to adjust the organization of the prompt content according to the user's query intention. The main steps of context expansion include: Information filtering: When the number of search results is too large, the system will filter out the most representative entries based on ranking scores and semantic coverage to avoid information redundancy when generating answers.

[0045] Abstract: Generate a summary of the filtered search results to reduce the prompt length.

[0046] Dynamic Q&A adjustments: For the supplementary questions input by the user later, the system will automatically expand the prompt content; Application of Generative Optimization and Large Language Models: In this embodiment, the accuracy and coherence of the answer are improved by optimizing the generation parameters of the large language model. The specific parameters for generation optimization include: Temperature coefficient: To control the diversity of responses, the formula is as follows;

[0047] Where T is the temperature coefficient; For questions that require precise answers, the system sets T to a lower value, while for open-ended questions, a higher value can be set; Multi-round interaction and feedback mechanism: In this embodiment, multiple rounds of interaction between the user and the system are supported to continuously optimize the generated answers. The implementation logic of multiple rounds of interaction is as follows: Answer verification: The system will verify the accuracy of the generated answer in conjunction with the user's subsequent input; User Ratings and Improvements: Users can rate the generated answers, and the system adjusts the prompt construction logic based on the rating results; Multiple rounds of interaction not only enhance the system's responsiveness, but also improve the personalization of the user experience; In step S5 of this embodiment, through the combination of prompt construction, context expansion, generation optimization and multiple rounds of interaction, deep integration of internal and external network knowledge and accurate response to user needs are achieved. All technical aspects of this embodiment have been fully disclosed, providing clear guidance for those skilled in the art to reproduce the technical solution of the present invention.

[0048] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An internal and external network knowledge association retrieval method based on RAG technology, characterized in that: The steps include: S1. Based on the query content entered by the user, matching entries are retrieved from the enterprise internal knowledge base, and the search scope is narrowed by filtering through metadata; S2. The results of internal retrieval are converted into vector representations through the embedding model, and the results are sorted by relevance based on cosine similarity; S3, calling the external search engine interface to perform network search, and converting the external search results into vector representation; S4, integrate the vectorized results of the internal knowledge base and external search, and perform unified sorting based on relevance; S5. Append the top-ranked multi-source search results to the prompt information, and generate the final answer in combination with the large language model.

2. The RAG-based internal and external network knowledge association retrieval method according to claim 1 is characterized in that: The retrieval of the enterprise internal knowledge base in step S1 specifically includes: S1.

1. Token-based authentication mechanism, using JSON Web Tokens (JWT) to verify user identity; S1.

2. The system dynamically generates query keywords based on user input, and selects highly relevant keywords based on semantic similarity to ensure more accurate search results; S1.3, role-based access control, different user roles have different knowledge base access rights; S1.

4. Verify the file type and size of the uploaded file and parse the content, clean up irrelevant characters, extract the core content of the file for knowledge base retrieval, and prevent malicious file uploads.

3. The RAG-based internal and external network knowledge association retrieval method according to claim 1 is characterized in that: The metadata filtering formula in step S1 is: ； in; D Represents a collection of documents in an enterprise knowledge base; M Represents metadata conditions; R Represents a filtered collection of documents.

4. The RAG-based internal and external network knowledge association retrieval method according to claim 1 is characterized in that: The embedding model in step S2 vectorizes the text using the following formula: ； in; T To retrieve text; V is the generated vector; function This is achieved by a pre-trained embedding model; The system combines keyword weights for matching and calculates relevance scores based on word frequency and position weights to ensure more accurate search results.

5. The method for retrieving knowledge association between internal and external networks based on RAG technology according to claim 1 is characterized in that: The relevance ranking in step S2 is based on cosine similarity calculation, and the formula is: ； in; V 1 and V 2 are the user query vector and the knowledge base entry vector respectively.

6. The RAG-based internal and external network knowledge association retrieval method according to claim 1 is characterized in that: Step S3 specifically includes: S3.

1. The system simulates the behavior of search engines through web crawlers, crawls web page content in real time, and extracts highly relevant text data; S3.2, pre-processing the text content in the web search results, including removing irrelevant characters, formatting the text content, and removing duplicates; S3.3, use the embedding model to convert the preprocessed web search results into vector representation; S3.

4. The system has a built-in exception handling mechanism that supports automatic retry of network requests and records exception logs to ensure that the retrieval process is stable and reliable.

7. The method for retrieving knowledge association between internal and external networks based on RAG technology according to claim 1 is characterized in that: The result of the external search in step S3 is vectorized using the following formula: ； in; S Search document content for external purposes; V ext is the vectorized result; function Implemented by semantic embedding model.

8. The RAG-based internal and external network knowledge association retrieval method according to claim 1 is characterized in that: The unified sorting in step S4 is based on the following formula: ； in; R is the ranking score; Sim internal and Sim external are the relevance scores and weight coefficients of the internal knowledge base and external search results respectively. α and β Can be adjusted according to actual scenario.

9. The method for retrieving knowledge association between internal and external networks based on RAG technology according to claim 1 is characterized in that: The large language model described in step S5 generates an answer based on the following process: S5.

1. The system supports context management, dynamically splicing user historical input, search results and new input content to build a complete prompt, and improving the coherence and accuracy of the answers generated by the model; S5.

2. The system supports streaming response mode, gradually returning the content generated by the model, reducing waiting time and improving user experience; S5.

3. Use prompt input to include context and multi-source search results; S5.

4. Dynamically expand the prompt content based on the context.

10. The RAG-based internal and external network knowledge association retrieval method according to claim 1 is characterized in that: The large language model is a GPT series model, which generates the final answer by combining the following context optimization methods: Embed multi-source retrieval results into language models in segments; The final generated answers are further checked for relevance.

Images