RAG retrieval optimization method and system based on deep learning and electronic equipment

Abstract

The invention discloses an RAG retrieval optimization method and system based on deep learning and electronic equipment. The method comprises the following steps: preprocessing various documents stored in knowledge bases of different scenes; performing word segmentation processing on the preprocessed text blocks to obtain corpora after word segmentation processing of the text blocks; obtaining a to-be-retrieved user question, and vectorizing the user question and a text block in the document; constructing a vector search library for the vectorized text blocks through a python Annoy framework approximate search algorithm to obtain similar vectors of problem vectors; extracting keywords of the text blocks based on the corpora after word segmentation of the text blocks by using a TF-IDF (Term Frequency-Inverse Document Frequency) algorithm; carrying out model training by utilizing keywords of the text blocks and the similar vectors; and on the basis of the trained TF-IDF model, extracting the similarity of the keyword sets of the input questions and the text blocks, and carrying out classified recall on the target knowledge base. According to the scheme, knowledge base classification coarse recall is achieved through deep learning, and the defect that time consumption is long due to the fact that previous full text block recall is adopted can be overcome.

Description

Technical Field

[0001] The present invention relates to the field of retrieval enhancement generation technology, and specifically to a RAG retrieval optimization method, system and electronic device based on deep learning. Background Art

[0002] In the field of RAG (retrieval-augmented generation) retrieval, when there are many documents in the knowledge base, the number of text blocks is proportionally large, which can reach millions. At this time, vectorizing the user's questions and calculating and recalling several text blocks from the millions of text block indexes will result in low efficiency and slow recall speed, affecting the user experience.

[0003] RAG's existing methods and implementation principles divide the text blocks in the document into blocks, vectorize and store them to build an index, then vectorize the questions and find similarities with the text blocks in the knowledge base, recall the Topk text blocks and return them to the big model to generate answers. Summary of the invention

[0004] The purpose of the present invention is to provide a RAG retrieval optimization method, system and electronic device based on deep learning to solve the problems raised in the above background technology.

[0005] By adopting the above technical solution, a rough recall of knowledge base classification is achieved, which can make up for the time-consuming defect caused by the previous full text block recall.

[0006] The purpose of the present invention is achieved by adopting the following technical solutions:

[0007] A first aspect of the present invention provides a RAG retrieval optimization method based on deep learning, the method comprising:

[0008] Preprocess various documents stored in the knowledge base of different scenarios;

[0009] Perform word segmentation on the preprocessed text block to obtain the corpus after word segmentation of the text block;

[0010] Get the user questions to be retrieved, and vectorize the user questions and the text blocks in the document. Use the vectorized text blocks through the python Annoy framework approximate search algorithm to build a vector search library to obtain the similarity vector of the question vector.

[0011] Using the TF-IDF algorithm, based on the corpus after the text block is segmented, the keywords of the text block are extracted; the keywords of the text block and the similarity vector are used for model training;

[0012] Based on the trained TF-IDF model, the similarity of the keyword set between the input question and the text block is extracted, and the target knowledge base is classified and recalled.

[0013] Optionally, the preprocessing of various documents stored in the knowledge base of different scenarios includes:

[0014] Get all documents in the document collection, divide them into different scenarios according to their different contents, and store them in different scenario knowledge bases respectively;

[0015] For each document type in the scenario knowledge base, based on the semantic segmentation algorithm, the documents in different scenario knowledge bases are divided into several text blocks according to the set word count;

[0016] The data of the text block is used as the text to be processed; the text to be processed is subjected to abnormal processing such as word segmentation, stop word removal, and special symbol removal, and the text data after abnormal processing is filtered; and the text block with invalid semantics filtered is obtained.

[0017] Optionally, obtaining a pre-trained TF-IDF model includes:

[0018] Obtaining keywords and the similarity vectors of the document based on the text block to generate a semantic space;

[0019] Performing enhancement processing on the semantic space to obtain granularity information in the semantic space and context information of the semantic space;

[0020] Based on the encoder, learn the granularity information in the semantic space and the context information in the semantic space; obtain the correlation between the data in the semantic space and obtain the data correlation representation in the semantic space;

[0021] The method of learning the granularity information in the semantic space and the context information in the semantic space based on the encoder to obtain the correlation between the data in the semantic space, and obtaining the data correlation representation in the semantic space includes: when the original semantic space is input as an input vector into the encoder for encoding, merging the data correlation representation of each layer of the original semantic space, and obtaining the final data correlation representation in the semantic space by combining the data correlation representation in the semantic space of all layers;

[0022] By comparing the relevance between the data in the original semantic space and calculating the relevance scores between the data in the semantic space, the association relationship between different document types is obtained;

[0023] A sample selector is used to select a preset number of associated data from the results of descending sorting of the correlation scores based on the association relationship to form a positive sample set, and the remaining data form a negative sample set, and a loss function for comparative learning between the positive sample set and the negative sample set is defined;

[0024] The model is trained with the goal of minimizing the loss until a TF-IDF model that meets the goal is obtained after training.

[0025] Optionally, extracting the similarity of the keyword set of the input question and the text block based on the trained TF-IDF model, and classifying and recalling the target knowledge base includes: extracting the keyword set of the input question and the text block through the trained TF-IDF model, sorting the keyword set of the input question and the text block from high to low according to the score with the question query, and selecting a preset number of text block codes in turn;

[0026] By vectorizing similarities and keywords, using TF-IDF to extract keywords and combining the BM25 algorithm to recall the target scenario knowledge base, the input question is classified into the knowledge base to which it belongs.

[0027] In a second aspect, the present invention provides a RAG retrieval optimization system based on deep learning, the system comprising:

[0028] The preprocessing module is used to preprocess various documents stored in the knowledge base in different scenarios;

[0029] A word segmentation processing module is used to perform word segmentation processing on the preprocessed text block to obtain the corpus after the word segmentation processing of the text block;

[0030] The similarity calculation module is used to obtain the user questions to be retrieved and vectorize the user questions and the text blocks in the document. The vectorized text blocks are constructed into a vector search library through the python Annoy framework approximate search algorithm to obtain the similarity vector of the question vector.

[0031] A training module is used to extract keywords of a text block based on the corpus after the text block is segmented by using the TF-IDF algorithm; and to perform model training using the keywords of the text block and the similarity vector;

[0032] The recall module is used to extract the similarity of the keyword set between the input question and the text block based on the trained TF-IDF model, and classify and recall the target knowledge base.

[0033] Optionally, the preprocessing module includes:

[0034] An acquisition unit is used to acquire all documents in a document collection, divide them into different scenarios according to different contents of the documents, and store them in different scenario knowledge bases respectively;

[0035] The segmentation unit is used to divide each document type in the scene knowledge base into several text blocks based on the semantic segmentation algorithm and the set word count cutoff according to each document type in the scene knowledge base;

[0036] The screening unit is used to take the data of the text block as the text to be processed; perform abnormal processing such as word segmentation, stop word removal, and special symbol removal on the text to be processed, and filter the text data after abnormal processing; and obtain the text block with invalid filtered semantics.

[0037] Optionally, the training module includes:

[0038] A generating unit, used for acquiring keywords and the similarity vector of the document based on the text block, and generating a semantic space;

[0039] A processing unit, configured to perform enhancement processing on the semantic space to obtain granularity information in the semantic space and context information of the semantic space;

[0040] A learning unit is used to learn granularity information and context information in the semantic space based on the encoder; obtain the correlation between data in the semantic space and obtain the data correlation representation in the semantic space;

[0041] The method of learning the granularity information in the semantic space and the context information in the semantic space based on the encoder to obtain the correlation between the data in the semantic space, and obtaining the data correlation representation in the semantic space includes: when the original semantic space is input as an input vector into the encoder for encoding, merging the data correlation representation of each layer of the original semantic space, and obtaining the final data correlation representation in the semantic space by combining the data correlation representation in the semantic space of all layers;

[0042] A comparison unit is used to compare the relevance between the data in the original semantic space and obtain the association relationship between different document types by calculating the relevance score between the data in the semantic space;

[0043] A definition unit is used to use a sample selector to select a preset number of related data from the results of descending sorting of the relevance scores based on the association relationship to form a positive sample set, and the remaining data to form a negative sample set, and define a loss function for comparative learning between the positive sample set and the negative sample set;

[0044] The training unit is used to perform model training with the goal of minimizing the loss until a TF-IDF model that meets the goal after training is obtained.

[0045] Optionally, the recall module includes: a selection unit, configured to extract through a trained TF-IDF model, input a keyword set of the question and the text block, sort from high to low according to the score with the question query, and select a preset number of text block codes in sequence;

[0046] The retrieval unit is used to vectorize similarity and keywords, extract keywords using TF-IDF, and recall the target scenario knowledge base in combination with the BM25 algorithm to classify the input question into the knowledge base to which it belongs.

[0047] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0048] at least one processor; and,

[0049] a memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform any one of the methods according to the first aspect.

[0051] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement any one of the methods in the first aspect when executed.

[0052] Compared with the prior art, the invention has the following beneficial effects: the RAG retrieval optimization method, system and electronic device based on deep learning pre-processes various documents stored in the knowledge base in different scenarios; performs word segmentation on the pre-processed text block to obtain the text block word segmentation corpus; obtains the user question to be retrieved, and vectorizes the user question and the text block in the document; constructs a vector search library for the vectorized text block through the python Annoy framework approximate search algorithm to obtain the similarity vector of the question vector; uses the TF-IDF algorithm to extract the keywords of the text block based on the text block word segmentation corpus; uses the keywords of the text block and the similarity vector to train the model; based on the trained TF-IDF model, extracts the similarity between the input question and the keyword set of the text block, and classifies and recalls the target knowledge base. Using deep learning to achieve knowledge base classification rough recall can make up for the time-consuming defect caused by the previous full text block recall.

[0053] In the present invention, firstly, the semantic segmentation model is used to divide the document into blocks. After preprocessing, the vector is combined with the nearest neighbor retrieval framework Annoy to realize the similarity recall of questions and text blocks; secondly, the TF-IDF is used to extract keywords combined with the BM25 algorithm. The above two algorithms are integrated, which not only has the concept of semantics but also the supplement of keyword algorithm; finally, the recalled text blocks are mapped to the target scene knowledge base to realize the first stage of knowledge base scene classification. It is conducive to greatly improving its retrieval efficiency and accuracy, especially when the number of knowledge bases, documents and text blocks is larger. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A general flow chart of a RAG retrieval optimization method based on deep learning provided by the present invention;

[0055] Figure 2 A working principle diagram of the RAG retrieval optimization method based on deep learning provided by the present invention;

[0056] Figure 3 The present invention provides a flowchart of using TF-IDF to extract keywords combined with the BM25 algorithm.

[0057] Figure 4 A structural block diagram of a RAG retrieval optimization system based on deep learning provided in one embodiment;

[0058] Figure 5 The figure is a diagram of the internal structure of an electronic device provided in one embodiment. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] See also Figure 1 The present invention provides a technical solution: a RAG retrieval optimization method based on deep learning, comprising:

[0061] Step S101, pre-processing various documents stored in the knowledge base of different scenarios;

[0062] Step S102, performing word segmentation processing on the preprocessed text block to obtain a corpus after word segmentation processing of the text block;

[0063] Step S103, obtaining the user question to be retrieved, and vectorizing the user question and the text block in the document; constructing a vector search library for the vectorized text block through the python Annoy framework approximate search algorithm to obtain a similar vector to the question vector;

[0064] Step S104, using the TF-IDF algorithm, based on the corpus after the text block segmentation, extract the keywords of the text block; and use the keywords of the text block and the similarity vector to perform model training;

[0065] Step S105: Based on the trained TF-IDF model, extract the similarity between the input question and the keyword set of the text block, and classify and recall the target knowledge base.

[0066] In the above embodiment, the preprocessing of various documents stored in the knowledge base of different scenarios in step S101 includes:

[0067] Get all documents in the document collection, divide them into different scenarios according to their different contents, and store them in different scenario knowledge bases respectively;

[0068] For each document type in the scenario knowledge base, based on the semantic segmentation algorithm, the documents in different scenario knowledge bases are divided into several text blocks according to the set word count;

[0069] The data of the text block is used as the text to be processed; the text to be processed is subjected to abnormal processing such as word segmentation, stop word removal, and special symbol removal, and the text data after abnormal processing is filtered; and the text block with invalid semantics filtered is obtained.

[0070] In the above embodiment, the step S104 of obtaining the pre-trained TF-IDF model includes:

[0071] Obtaining keywords and the similarity vectors of the document based on the text block to generate a semantic space;

[0072] Performing enhancement processing on the semantic space to obtain granularity information in the semantic space and context information of the semantic space;

[0073] Based on the encoder, learn the granularity information in the semantic space and the context information in the semantic space; obtain the correlation between the data in the semantic space and obtain the data correlation representation in the semantic space;

[0074] The method of learning the granularity information in the semantic space and the context information in the semantic space based on the encoder to obtain the correlation between the data in the semantic space, and obtaining the data correlation representation in the semantic space includes: when the original semantic space is input as an input vector into the encoder for encoding, merging the data correlation representation of each layer of the original semantic space, and obtaining the final data correlation representation in the semantic space by combining the data correlation representation in the semantic space of all layers;

[0075] By comparing the relevance between the data in the original semantic space and calculating the relevance scores between the data in the semantic space, the association relationship between different document types is obtained;

[0076] A sample selector is used to select a preset number of associated data from the results of descending sorting of the correlation scores based on the association relationship to form a positive sample set, and the remaining data form a negative sample set, and a loss function for comparative learning between the positive sample set and the negative sample set is defined;

[0077] The model is trained with the goal of minimizing the loss until a TF-IDF model that meets the goal is obtained after training.

[0078] In the above embodiment, contrastive learning aims to improve the performance of the model by comparing the differences between different samples. In the model, the goal of contrastive learning is to shorten the distance between the query and related documents, while increasing the distance between the query and irrelevant documents, which is achieved through the InfoNCE loss function. NCE stands for noise contrast estimation, and the formula is:

[0079]

[0080] In the formula, c is the context, s is the sentence, and f k (s t+k ,c t ) is the score function.

[0081] In the above embodiment, the loss function is transferred to the student model through the knowledge of the teacher model, so that the student model has high-quality performance. Its loss function usually consists of two parts: one is a conventional loss function, such as cross entropy; the other is a distillation loss, the formula is:

[0082] L=(1-T)*L CE +T*L KD

[0083] Where L CE is the conventional loss function; L KD Represents the distance between the probability distributions output by the two models; T is used to balance the importance of the losses in the two parts.

[0084] In the model, the similarity scores of the three different methods are weighted and summed to get a teacher score. Then, the three similarity scores of the student model will learn the information of this teacher score, thereby obtaining three losses, and the loss is calculated using the mean square error (MSE).

[0085] Construction of vector data index library: This paper uses the Faiss open source library developed by Facebook AI Research to support fast retrieval of text representation vectors similar to RAG's query vector.

[0086] In the above embodiment, the step S105 extracts the similarity of the keyword set of the input question and the text block based on the trained TF-IDF model, and classifies and recalls the target knowledge base, including: extracting the keyword set of the input question and the text block through the trained TF-IDF model, sorting the keyword set from high to low according to the score with the question query, and selecting a preset number of text block codes in turn;

[0087] By vectorizing similarities and keywords, using TF-IDF to extract keywords and combining the BM25 algorithm to recall the target scenario knowledge base, the input question is classified into the knowledge base to which it belongs.

[0088] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0089] Based on the above content, the following implementation process is proposed in combination with specific application scenarios. The flowchart of the entire technical solution is as follows: Figure 2 shown.

[0090] Document scenario classification: First, all documents in the document collection are classified into different scenario knowledge bases, for example, document A is classified as "administration", document B is classified as "marketing", document C is classified as "finance", document D is classified as "human resources", etc. That is, the documents are divided into different scenarios according to their different contents and stored in knowledge bases of different scenarios.

[0091] Core algorithm: Figure 2 , which means Figure 1 The core algorithm in the dashed box can be summarized in one sentence: it roughly recalls possible target scene knowledge bases from a large number of documents / knowledge bases to narrow the scope of subsequent retrieval. Rough recall only takes 40-50ms, and the recall rate of the TOP5 target knowledge bases can reach 98%+, which can greatly improve the retrieval speed and generation effect of RAG.

[0092] Text segmentation: In this method, instead of using the common segmentation method, such as cutting off the text according to a set number of words, a semantic segmentation algorithm is used. The bert_document-segmentation_chinese model is used to divide the document into several text blocks.

[0093] Text preprocessing: The document is segmented, stop words are removed, special symbols are removed, and some text blocks without semantics and meaning, such as directory styles, are filtered out. The basic data style at this time is shown in Table 1.

[0094] id is the id of the text block, dataset_id is the knowledge base id, document_id is the document id, dataset_name is the knowledge base name, and document_name is the document name.

[0095] Table 1

[0096]

[0097] Vectorization: The user's question and the document chunks in the document are vectorized

[0098] The nlp_corom_sentence-embedding_chinese-base model is vectorized with a dimension of 768.

[0099] Text similarity: The vectorized text blocks are passed through the python Annoy framework approximate nearest neighbor search algorithm ANNOY (APPROXIMATE NEAREST NEIGHBORS OH YEAH) to build a vector search library, and the get_nns_by_vector method is used to obtain the vector most similar to the question vector. Here, the TopN text block similarity is set to 10.

[0100] Keywords: Due to the inconsistency of the semantic space between the question and the text block, in layman's terms, the question may only match one or two sentences in a text block. At this time, the similarity algorithm of the question-text block set alone cannot fully and effectively recall the words. It is necessary to increase keyword extraction. In the preprocessing stage, the corpus of each text block has been obtained after word segmentation, and the TF-IDF algorithm (term frequency-inverse document frequency) is used to extract the keywords of the text block. At this time, no more than 10 keywords will be extracted from each text block.

[0101] BM25: BM25 (Best Matching 25) is a classic information retrieval algorithm. It uses the keywords of each text block extracted in the previous stage. It also extracts the question using the TF-IDF model trained in the previous stage. It inputs the keyword set of the question and the text block to get the 10 text blocks with the highest scores for the question query. The code is as follows.

[0102] best document=bm25_model.get_top_n(query_keywords, chunks_keywords, n=10)

[0103] In the above, by combining vectorized similarity and keywords with the TOP20 text blocks recalled by BM25, according to Table 1, the target scene knowledge base can be roughly recalled through mapping, which is equivalent to classifying the knowledge base for the question.

[0104] Based on the same inventive concept, the embodiment of the present application also provides a deep learning-based RAG retrieval optimization system for implementing the above-mentioned deep learning-based RAG retrieval optimization method. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more deep learning-based RAG retrieval optimization systems provided below can be found in the above-mentioned limitations on the deep learning-based RAG retrieval optimization system method, which will not be repeated here.

[0105] In one embodiment, Figure 4 As shown, a RAG retrieval optimization system based on deep learning is provided, including: a preprocessing module 210, a word segmentation processing module 220, a similarity calculation module 230, a training module 240 and a recall module 250, wherein:

[0106] A preprocessing module 210 is used to preprocess various documents stored in the knowledge base of different scenarios;

[0107] The word segmentation processing module 220 is used to perform word segmentation processing on the preprocessed text block to obtain the corpus after the word segmentation processing of the text block;

[0108] The similarity calculation module 230 is used to obtain the user questions to be retrieved, and vectorize the user questions and the text blocks in the document; construct a vector search library for the vectorized text blocks through the python Annoy framework approximate search algorithm to obtain a similar vector of the question vector;

[0109] The training module 240 is used to extract keywords of the text block based on the corpus after the text block is segmented by using the TF-IDF algorithm; and perform model training using the keywords of the text block and the similarity vector;

[0110] The recall module 250 is used to extract the similarity between the input question and the keyword set of the text block based on the trained TF-IDF model, and classify and recall the target knowledge base.

[0111] Optionally, the preprocessing module 210 includes:

[0112] An acquisition unit is used to acquire all documents in a document collection, divide them into different scenarios according to different contents of the documents, and store them in different scenario knowledge bases respectively;

[0113] The segmentation unit is used to divide each document type in the scene knowledge base into several text blocks based on the semantic segmentation algorithm and the set word count cutoff according to each document type in the scene knowledge base;

[0114] The screening unit is used to take the data of the text block as the text to be processed; perform abnormal processing such as word segmentation, stop word removal, and special symbol removal on the text to be processed, and filter the text data after abnormal processing; and obtain the text block with invalid filtered semantics.

[0115] Optionally, the training module 240 includes:

[0116] A generating unit, used for acquiring keywords and the similarity vector of the document based on the text block, and generating a semantic space;

[0117] A processing unit, configured to perform enhancement processing on the semantic space to obtain granularity information in the semantic space and context information of the semantic space;

[0118] A learning unit is used to learn granularity information and context information in the semantic space based on the encoder; obtain the correlation between data in the semantic space and obtain the data correlation representation in the semantic space;

[0119] The method of learning the granularity information in the semantic space and the context information in the semantic space based on the encoder to obtain the correlation between the data in the semantic space, and obtaining the data correlation representation in the semantic space includes: when the original semantic space is input as an input vector into the encoder for encoding, merging the data correlation representation of each layer of the original semantic space, and obtaining the final data correlation representation in the semantic space by combining the data correlation representation in the semantic space of all layers;

[0120] A comparison unit is used to compare the relevance between the data in the original semantic space and obtain the association relationship between different document types by calculating the relevance score between the data in the semantic space;

[0121] A definition unit is used to use a sample selector to select a preset number of related data from the results of descending sorting of the relevance scores based on the association relationship to form a positive sample set, and the remaining data to form a negative sample set, and define a loss function for comparative learning between the positive sample set and the negative sample set;

[0122] The training unit is used to perform model training with the goal of minimizing the loss until a TF-IDF model that meets the goal after training is obtained.

[0123] Optionally, the recall module 250 includes: a selection unit, configured to extract through a trained TF-IDF model, input a keyword set of the question and the text block, sort from high to low according to the score with the question query, and select a preset number of text block codes in sequence;

[0124] The retrieval unit is used to vectorize similarity and keywords, extract keywords using TF-IDF, and recall the target scenario knowledge base in combination with the BM25 algorithm to classify the input question into the knowledge base to which it belongs.

[0125] Figure 5 A structural block diagram of an electronic device of a RAG retrieval optimization method based on deep learning provided by one or more embodiments of the present invention.

[0126] like Figure 5 As shown, the present application provides an electronic device, including: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0127] A computer program is stored in the memory. When the computer program is executed by the processor, the processor performs the steps of a RAG retrieval optimization method based on deep learning.

[0128] The present application also provides a computer-readable storage medium, which stores a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of a RAG retrieval optimization method based on deep learning.

[0129] The present application also provides a data platform, including:

[0130] An electronic device for implementing the steps of a RAG retrieval optimization method based on deep learning;

[0131] A processor, the processor runs a program, and when the program runs, the steps of the RAG retrieval optimization method based on deep learning are output from the electronic device;

[0132] A storage medium is used to store a program, which, when running, performs the steps of a deep learning-based RAG retrieval optimization method output from an electronic device.

[0133] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0134] The electronic device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory. The operating system can be any one or more computer operating systems that implement electronic device control through a process, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. In addition, in an embodiment of the present invention, the electronic device can be a handheld device such as a smart phone or a tablet computer, or can be an electronic device such as a desktop computer or a portable computer, which is not particularly limited in the embodiment of the present invention.

[0135] The execution subject of the electronic device control in the embodiment of the present invention may be an electronic device, or a functional module in the electronic device that can call and execute a program. The electronic device may obtain the firmware corresponding to the storage medium. The firmware corresponding to the storage medium is provided by the supplier. The firmware corresponding to different storage media may be the same or different, which is not limited here. After the electronic device obtains the firmware corresponding to the storage medium, the firmware corresponding to the storage medium may be written into the storage medium, specifically, the firmware corresponding to the storage medium may be burned into the storage medium. The process of burning the firmware into the storage medium may be implemented using existing technology, which will not be described in detail in the embodiment of the present invention.

[0136] The electronic device may also obtain a reset command corresponding to the storage medium. The reset command corresponding to the storage medium is provided by the supplier. The reset commands corresponding to different storage media may be the same or different, and are not limited here.

[0137] At this time, the storage medium of the electronic device is a storage medium in which the corresponding firmware is written, and the electronic device can respond to the reset command corresponding to the storage medium in the storage medium in which the corresponding firmware is written, so that the electronic device resets the storage medium in which the corresponding firmware is written according to the reset command corresponding to the storage medium. The process of resetting the storage medium according to the reset command can be implemented by the existing technology and will not be described in detail in the embodiments of the present invention.

[0138] For the convenience of description, the above devices are described in terms of functions and are divided into various units and modules. Of course, when implementing the present application, the functions of each unit and module can be implemented in the same or multiple software and / or hardware.

[0139] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0140] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0141] It can be known from the description of the above implementation modes that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute the methods described in the various implementation modes of the present application or certain parts of the implementation modes.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A RAG retrieval optimization method based on deep learning, characterized in that: The method comprises: Preprocess various documents stored in the knowledge base of different scenarios; Perform word segmentation on the preprocessed text block to obtain the corpus after word segmentation of the text block; Get the user questions to be retrieved, and vectorize the user questions and the text blocks in the document. Use the python Annoy framework approximate search algorithm to build a vector search library for the vectorized text blocks, and obtain the similarity vector of the question vector. Using the TF-IDF algorithm, based on the corpus after the text block is segmented, the keywords of the text block are extracted; the keywords of the text block and the similarity vector are used for model training; Based on the trained TF-IDF model, the similarity of the keyword set between the input question and the text block is extracted, and the target knowledge base is classified and recalled.

2. The RAG search optimization method according to claim 1, characterized in that: The preprocessing of various documents stored in the knowledge base of different scenarios includes: Get all documents in the document collection, divide them into different scenarios according to their different contents, and store them in different scenario knowledge bases respectively; For each document type in the scenario knowledge base, based on the semantic segmentation algorithm, the documents in different scenario knowledge bases are divided into several text blocks according to the set word count; The data of the text block is used as the text to be processed; the text to be processed is subjected to abnormal processing such as word segmentation, stop word removal, and special symbol removal, and the text data after abnormal processing is filtered; and the text block with invalid semantics filtered is obtained.

3. The RAG search optimization method according to claim 1, characterized in that: The step of obtaining a pre-trained TF-IDF model includes: Obtaining keywords and the similarity vectors of the document based on the text block to generate a semantic space; Performing enhancement processing on the semantic space to obtain granularity information in the semantic space and context information of the semantic space; Based on the encoder, learn the granularity information in the semantic space and the context information in the semantic space; obtain the correlation between the data in the semantic space and obtain the data correlation representation in the semantic space; The method of learning the granularity information in the semantic space and the context information in the semantic space based on the encoder to obtain the correlation between the data in the semantic space, and obtaining the data correlation representation in the semantic space includes: when the original semantic space is input as an input vector into the encoder for encoding, merging the data correlation representation of each layer of the original semantic space, and obtaining the final data correlation representation in the semantic space by combining the data correlation representation in the semantic space of all layers; By comparing the relevance between the data in the original semantic space and calculating the relevance scores between the data in the semantic space, the association relationship between different document types is obtained; A sample selector is used to select a preset number of associated data from the results of descending sorting of the correlation scores based on the association relationship to form a positive sample set, and the remaining data form a negative sample set, and a loss function for comparative learning between the positive sample set and the negative sample set is defined; The model is trained with the goal of minimizing the loss until a TF-IDF model that meets the goal is obtained after training.

4. The RAG search optimization method according to claim 1, characterized in that: The method of extracting the similarity of the keyword set of the input question and the text block based on the trained TF-IDF model and classifying and recalling the target knowledge base includes: extracting the keyword set of the input question and the text block through the trained TF-IDF model, sorting the keyword set of the input question and the text block from high to low according to the score with the question query, and selecting a preset number of text block codes in turn; By vectorizing similarities and keywords, using TF-IDF to extract keywords and combining the BM25 algorithm to recall the target scenario knowledge base, the input question is classified into the knowledge base to which it belongs.

5. A RAG retrieval optimization system based on deep learning, characterized in that: The system comprises: The preprocessing module is used to preprocess various documents stored in the knowledge base in different scenarios; A word segmentation processing module is used to perform word segmentation processing on the preprocessed text block to obtain the corpus after the word segmentation processing of the text block; The similarity calculation module is used to obtain the user questions to be retrieved and vectorize the user questions and the text blocks in the document. The vectorized text blocks are constructed into a vector search library through the python Annoy framework approximate search algorithm to obtain the similarity vector of the question vector. A training module is used to extract keywords of a text block based on the corpus after the text block is segmented by using the TF-IDF algorithm; and to perform model training using the keywords of the text block and the similarity vector; The recall module is used to extract the similarity of the keyword set between the input question and the text block based on the trained TF-IDF model, and classify and recall the target knowledge base.

6. The RAG search optimization system according to claim 5, characterized in that: The preprocessing module comprises: An acquisition unit is used to acquire all documents in a document collection, divide them into different scenarios according to different contents of the documents, and store them in different scenario knowledge bases respectively; The segmentation unit is used to divide each document type in the scene knowledge base into several text blocks based on the semantic segmentation algorithm and the set word count cutoff according to each document type in the scene knowledge base; The screening unit is used to take the data of the text block as the text to be processed; perform abnormal processing such as word segmentation, stop word removal, and special symbol removal on the text to be processed, and filter the text data after abnormal processing; and obtain the text block with invalid filtered semantics.

7. The RAG search optimization system according to claim 5, characterized in that: The training module includes: A generating unit, used for acquiring keywords and the similarity vector of the document based on the text block, and generating a semantic space; A processing unit, configured to perform enhancement processing on the semantic space to obtain granularity information in the semantic space and context information of the semantic space; A learning unit is used to learn granularity information and context information in the semantic space based on the encoder; obtain the correlation between data in the semantic space and obtain the data correlation representation in the semantic space; The method of learning the granularity information in the semantic space and the context information in the semantic space based on the encoder to obtain the correlation between the data in the semantic space, and obtaining the data correlation representation in the semantic space includes: when the original semantic space is input as an input vector into the encoder for encoding, merging the data correlation representation of each layer of the original semantic space, and obtaining the final data correlation representation in the semantic space by combining the data correlation representation in the semantic space of all layers; A comparison unit is used to compare the relevance between the data in the original semantic space and obtain the association relationship between different document types by calculating the relevance score between the data in the semantic space; A definition unit is used to use a sample selector to select a preset number of related data from the results of descending sorting of the relevance scores based on the association relationship to form a positive sample set, and the remaining data to form a negative sample set, and define a loss function for comparative learning between the positive sample set and the negative sample set; The training unit is used to perform model training with the goal of minimizing the loss until a TF-IDF model that meets the goal after training is obtained.

8. The RAG search optimization system according to claim 7, characterized in that: The recall module includes: a selection unit, which is used to extract through a trained TF-IDF model, input a keyword set of a question and a text block, sort them from high to low according to the question query score, and select a preset number of text block codes in sequence; The retrieval unit is used to vectorize similarity and keywords, extract keywords using TF-IDF, and recall the target scenario knowledge base in combination with the BM25 algorithm to classify the input question into the knowledge base to which it belongs.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 4 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 4 are implemented.

Images