Retrieval enhancement generation method and system

Abstract

The invention provides a retrieval enhancement generation method and system. The method comprises the following steps that 1, a user asks questions and inputs keywords and specific questions; step 2, an Internet of Things identification module which combines big data and network identification; 3, the RAG system carries out networking retrieval; 4, performing language, picture and video generation on the data obtained by the RAG system; 5, obtaining an answer to the asked question; according to the method and the system, a special RAG system is superposed in an original retrieval generation method and system for use, so that semantic understanding retrieval is performed on user questions through online and offline data processing vectorization of the RAG system, and then keyword enhancement is performed on obtained related document data for further enhanced retrieval; according to the method and the system provided by the invention, the range can be expanded for retrieval, and questions and answers and other related reference data are obtained, so that the retrieval generation method and system can accord with the characteristics of enhanced generation, and the accuracy of answering the questions of the user and the carrying of related reference literatures are further improved.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, and more specifically to a retrieval enhancement generation method and system. Background Technology

[0002] Search can be likened to a "huge library" that can find relevant information based on systematic keywords or concise descriptions of the user's questions and needs. After systematic searching, it can display the data most relevant to the question, allowing the user to identify and browse to find the content they need. In summary, the inventors have found that existing retrieval generation methods and systems mainly suffer from the following defects: As described above, current retrieval methods and systems cannot understand the meaning of the question raised; they merely rigidly match the keywords of the question to search for information. This results in limited information retrieved, which in turn prevents a deeper understanding of the question and the matching of enhanced generation features, thus reducing the intelligence of current retrieval generation methods and systems. Summary of the Invention

[0003] The technical solution adopted by this invention to achieve its technical objective is: a retrieval enhancement generation method and system, comprising the following steps: Step 1: User asks a question, entering keywords and specific questions; Step 2: IoT identification module, combining big data and network identification; Step 3: The RAG system performs an online search; Step 4: Generate audio, images, and videos from the data obtained from the RAG system; Step 5: Obtain the answers to the questions you asked.

[0004] As a further improvement of the present invention, the RAG system in step three specifically includes: s1: Online query processing and vectorization; Offline preprocessing of document slicing and vectorization is performed simultaneously. s2: Vector knowledge base, used to query relevant information and data for problem points; s3: Obtain relevant documents and materials; s4: Combine and enhance keywords, focusing on specific keywords in the question; s5: Further enhance the search by combining keywords; s6: Obtain specific documents and related materials related to the keywords.

[0005] As a further improvement of the present invention, the online query processing and vectorization of s1 includes: Query processing: The system uses the same text embedding model to transform the question into a query vector; Semantic retrieval: The system performs similarity extraction in the vector knowledge base, and then extracts the most similar text block vectors, which can be used as reference information; Enhanced suggestions: The system combines relevant searched text with the user's question to create an enhanced suggestion word; Answer generation: The enhanced prompts are sent to the large language model to generate the answer based on authoritative background information.

[0006] As a further improvement of the present invention, the offline preprocessing document slicing and vectorization of s1 includes: Data loading: Collect documents from sources such as PDF, Word, databases, and APIs; Document segmentation: dividing long documents into finely segmented and semantically complete modules; Vectorization: Each text block is transformed into a high-dimensional numerical vector using a text embedding model; Component vector traction: Store all text block vectors in a vector database to establish an efficient traction structure for easy and fast subsequent retrieval.

[0007] As a further improvement of the present invention, the RAG system in step three further includes: Data source: The knowledge sources required by the system, which can be internal company documents, product manuals, laws and regulations, the latest news, database records, etc. Retrieval Unit: Text Embedding Model: Responsible for converting text into vectors (such as OpenAI's text-embedding-ada-002, or open-source BGE and M3E models). Vector database: A database specifically designed for storing and quickly querying vectors; Generator: Large speech model: responsible for final understanding and text generation; Orchestration framework: The software layer that connects the above components and is responsible for handling the entire workflow.

[0008] Compared with the prior art, the present invention has the following beneficial effects: This invention utilizes a dedicated RAG system overlaid on existing retrieval generation methods and systems. Through online and offline data processing and vectorization using the RAG system, user questions are semantically understood and retrieved. The resulting relevant documents are then enhanced with keywords for further enhanced retrieval, expanding the search scope to provide answers and other relevant references. This allows the retrieval generation methods and systems to conform to the characteristics of enhanced generation, further improving the accuracy of answering user questions and including relevant references. Attached Figure Description

[0009] Figure 1 This is a structural diagram of a retrieval enhancement generation method and system.

[0010] Figure 2 A structural diagram of the RAG process method. Detailed Implementation

[0011] The present invention will be further described below with reference to the accompanying drawings: Example: Figures 1 to 2 As shown: This invention provides a retrieval enhancement generation method and system, comprising the following steps: Step 1: User asks a question, entering keywords and specific questions; Step 2: IoT identification module, combining big data and network identification; Step 3: The RAG system performs an online search; Step 4: Generate audio, images, and videos from the data obtained from the RAG system; Step 5: Obtain the answers to the questions you asked.

[0012] Specifically, the RAG system in step three also includes: s1: Online query processing and vectorization; Offline preprocessing of document slicing and vectorization is performed simultaneously. s2: Vector knowledge base, used to query relevant information and data for problem points; s3: Obtain relevant documents and materials; s4: Combine and enhance keywords, focusing on specific keywords in the question; s5: Further enhance the search by combining keywords; s6: Obtain specific documents and related materials related to the keywords.

[0013] The online query processing and vectorization of s1 includes: Query processing: The system uses the same text embedding model to transform the question into a query vector; Semantic retrieval: The system performs similarity extraction in the vector knowledge base, and then extracts the most similar text block vectors, which can be used as reference information; Enhanced suggestions: The system combines relevant searched text with the user's question to create an enhanced suggestion word; Answer generation: The enhanced prompts are sent to the large language model to generate the answer based on authoritative background information.

[0014] The offline preprocessing of document s1, including document slicing and vectorization, includes: Data loading: Collect documents from sources such as PDF, Word, databases, and APIs; Document segmentation: dividing long documents into finely segmented and semantically complete modules; Vectorization: Each text block is transformed into a high-dimensional numerical vector using a text embedding model; Component vector traction: Store all text block vectors in a vector database to establish an efficient traction structure for easy and fast subsequent retrieval.

[0015] The RAG system in step three further includes: Data source: The knowledge sources required by the system, which can be internal company documents, product manuals, laws and regulations, the latest news, database records, etc. Retrieval Unit: Text Embedding Model: Responsible for converting text into vectors (such as OpenAI's text-embedding-ada-002, or open-source BGE and M3E models). Vector database: A database specifically designed for storing and quickly querying vectors; Generator: Large speech model: responsible for final understanding and text generation; Orchestration framework: The software layer that connects the above components and is responsible for handling the entire workflow.

[0016] The specific functions and operation procedures of this embodiment are as follows: In this invention, the enhanced retrieval generation method and system receive user-asked questions, analyze them through an IoT identification module, and then connect the IoT identification module to the RAG system to achieve a wide-ranging online retrieval effect. The RAG system then generates relevant and accurate data from the enhanced retrieval, which is presented in three formats: language, images, and videos. This allows users to obtain accurate answers and relevant reference materials. Furthermore, the online query processing and vectorization of the RAG system can be performed simultaneously with offline preprocessing document slicing and vectorization. The offline preprocessing extracts relevant documents from the vector knowledge base, followed by semantic recognition and understanding retrieval to obtain relevant documents and reference materials. The system then combines the extracted data into new prompts, enabling further retrieval to yield accurate answers. This, combined with the enhanced retrieval features of the RAG system, improves the accuracy and effectiveness of the original retrieval generation method.

[0017] Any technical solution that achieves the above-mentioned technical effects by utilizing the technical solutions described in this invention, or by designing similar technical solutions by those skilled in the art under the inspiration of the technical solutions described in this invention, falls within the protection scope of this invention.

Claims

1. A retrieval enhancement generation method and system, characterized in that: Includes the following steps: Step 1: User asks a question, entering keywords and specific questions; Step 2: IoT identification module, combining big data and network identification; Step 3: The RAG system performs an online search; Step 4: Generate audio, images, and videos from the data obtained from the RAG system; Step 5: Obtain the answers to the questions you asked.

2. The retrieval enhancement generation method and system according to claim 1, characterized in that: The RAG system in step three specifically also includes: s1: Online query processing and vectorization; Offline preprocessing of document slicing and vectorization is performed simultaneously. s2: Vector knowledge base, used to query relevant information and data for problem points; s3: Obtain relevant documents and materials; s4: Combine and enhance keywords, focusing on specific keywords in the question; s5: Further enhance the search by combining keywords; s6: Obtain specific documents and related materials related to the keywords.

3. The retrieval enhancement generation method and system according to claim 2, characterized in that: The online query processing and vectorization of s1 includes: Query processing: The system uses the same text embedding model to transform the question into a query vector; Semantic retrieval: The system performs similarity extraction in the vector knowledge base, and then extracts the most similar text block vectors, which can be used as reference information; Enhanced suggestions: The system combines relevant searched text with the user's question to create an enhanced suggestion word; Answer generation: The enhanced prompts are sent to the large language model to generate the answer based on authoritative background information.

4. The retrieval enhancement generation method and system according to claim 2, characterized in that: The offline preprocessing of document s1, including document slicing and vectorization, includes: Data loading: Collect documents from sources such as PDF, Word, databases, and APIs; Document segmentation: dividing long documents into finely segmented and semantically complete modules; Vectorization: Each text block is transformed into a high-dimensional numerical vector using a text embedding model; Component vector traction: Store all text block vectors in a vector database to establish an efficient traction structure for easy and fast subsequent retrieval.

5. The retrieval enhancement generation method and system according to claim 1, characterized in that: The RAG system in step three also includes: Data source: The knowledge sources required by the system, which can be internal company documents, product manuals, laws and regulations, the latest news, database records, etc. Retrieval Unit: Text Embedding Model: Responsible for converting text into vectors (such as OpenAI's text-embedding-ada-002, or open-source BGE and M3E models). Vector database: A database specifically designed for storing and quickly querying vectors; Generator: Large speech model: responsible for final understanding and text generation; Orchestration framework: The software layer that connects the above components and is responsible for handling the entire workflow.

Images