Multi-mode RAG based on large model, device, equipment and storage medium

Abstract

The invention discloses a multi-modal RAG based on a large model, a device, equipment and a storage medium. The method comprises the following steps: preprocessing text information in a large-scale corpus by using a multi-modal technology; constructing courseware content fragments according to the preprocessed text information, and storing the courseware content fragments into the created index; based on the preprocessed keywords in the question issued by the user, recalling courseware content fragments related to the question in combination with the index to generate a first recall result; according to the result of the first recall, content expansion is carried out according to a preset rule, and second recall is carried out; performing multi-dimensional sorting on the result of the secondary recall; the sorted results are integrated with user questions, correlation judgment is carried out through a large model, and search results are further screened. Compared with the prior art, the method has the advantages that the accuracy of recalled contents is greatly improved by introducing a multi-modal technology, a secondary recall strategy and using a large model, and the generation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal RAG, device, equipment and storage medium based on a large model. Background Art

[0002] In the field of artificial intelligence, large-scale model RAG (retrieval-augmented generation technology) has become a hot topic of research in recent years. It combines the two key technologies of retrieval and generation, bringing revolutionary progress to natural language processing tasks. RAG uses a large-scale corpus for information retrieval and provides rich background knowledge and contextual information for the generation process, thereby improving the accuracy and diversity of the generated results. RAG is widely used in text generation, dialogue systems, question-answering systems and other fields.

[0003] The main RAG workflow includes:

[0004] 1. Preprocessing: First, preprocess the large-scale corpus, including word segmentation, stop word removal, vocabulary building, etc. These preprocessing operations help extract effective information from the text and lay the foundation for subsequent retrieval and generation processes.

[0005] 2. Retrieval: During the generation process, RAG technology will retrieve relevant text fragments in the corpus based on the current context information. This retrieval process is usually based on some similarity measurement method, such as cosine similarity F, etc. The retrieval results will serve as a reference and supplement to the generation process.

[0006] 3. Generation: After obtaining the search results, RAG technology will use the generative model (such as Transformer, GPT, etc.) to generate new text. The generation process will comprehensively consider the current context information, search results, and the knowledge base of the generative model itself to generate more accurate and diverse text.

[0007] 4. Post-processing: Finally, the generated text is post-processed, including steps such as removing duplications and correcting grammatical errors. These post-processing operations help improve the quality of the generated results.

[0008] However, there are still some problems in the above workflow: (1) The preprocessing stage usually supports simple text parsing and cannot fully and accurately extract text information; (2) Through simple recall, it is impossible to accurately retrieve text fragments related to the question across regions. In summary, the limitations of the preprocessing stage and the shortcomings of the retrieval process will have a negative impact on the entire RAG workflow, which not only affects the retrieval accuracy, but also makes the generation efficiency low. Summary of the invention

[0009] The purpose of the present invention is to provide a multimodal RAG, device, equipment and storage medium based on a large model, aiming to solve the technical problems that the limitations of the preprocessing stage of the existing RAG processing flow and the deficiencies in the retrieval process will affect the accuracy of the entire workflow retrieval and cause low generation efficiency.

[0010] In order to solve the above technical problems, the purpose of the present invention is achieved through the following technical solutions:

[0011] A first aspect of the present application provides a multimodal RAG based on a large model, comprising the following steps:

[0012] Use multimodal technology to preprocess text information in large-scale corpora;

[0013] Construct courseware content segments based on preprocessed text information and store them in the created index;

[0014] Based on the pre-processed keywords in the questions sent by users, the courseware content segments related to the questions are recalled in combination with the index to generate the first recall results;

[0015] Based on the results of the first recall, the content is expanded according to the preset rules and a second recall is conducted;

[0016] Sorting the results of the secondary recall in multiple dimensions;

[0017] Integrate the sorted results with user questions, make relevance judgments through a large model, and further filter the search results.

[0018] In a possible implementation, the step of preprocessing text information in a large-scale corpus using multimodal technology includes:

[0019] Using training courseware from different platforms as a corpus, the training courseware includes documents in different formats and audio and video;

[0020] Effectively extracting text from a corpus, the extraction method includes using a variety of different libraries and tools to extract.

[0021] In a possible implementation, the step of constructing courseware content segments according to the preprocessed text information and storing them in the created index includes:

[0022] Define the mapping of the index;

[0023] Convert courseware content slices into vector representations.

[0024] In a possible implementation, the step of converting the courseware content segments into vector representations includes:

[0025] Load a pre-trained word embedding model to convert each word in the text into its corresponding vector representation;

[0026] storing the vector representation in an index using a vector field;

[0027] Defines the type, dimension, and distance measure of a vector field.

[0028] In a possible implementation, the step of generating a first recall result by combining the index to recall the courseware content segments related to the question based on the preprocessed keywords in the question sent by the user includes:

[0029] Preprocess keywords by removing stop words, stemming, judging intent, extracting keywords, and weighting keywords;

[0030] Use the inverted index to query the processed keywords;

[0031] Calculate the similarity between the query question vector and the courseware content segment vector;

[0032] Recall the courseware content segments related to the question based on similarity.

[0033] In a possible implementation, the step of expanding the content according to a preset rule includes:

[0034] Set to expand forward by N pages and backward by M pages;

[0035] Adjust the values ​​of N and M according to actual business needs and the characteristics of the courseware content.

[0036] In a possible implementation, the results of the secondary recall are sorted in multiple dimensions, wherein the multiple dimensions include: intent recognition, word aggregation density, keyword weight, and fuzzy matching degree.

[0037] A second aspect of the present application provides a multimodal RAG device based on a large model, the device comprising:

[0038] A preprocessing unit, used to preprocess text information in a large-scale corpus using multimodal technology;

[0039] An index building unit, used to build courseware content segments based on preprocessed text information and store them in the created index;

[0040] The first recall unit is used to recall the courseware content segments related to the question based on the pre-processed keywords in the question sent by the user and in combination with the index to generate the first recall result;

[0041] A secondary recall unit is used to expand the content and conduct a secondary recall according to the result of the first recall and the preset rules;

[0042] A sorting unit, used to sort the results of the secondary recall in multiple dimensions;

[0043] The text generation unit is used to integrate the sorted results with the user questions, make relevance judgments through the large model, and further filter the search results.

[0044] A third aspect of the present application provides a computer device, characterized in that the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0045] A fourth aspect of the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method described in the first aspect above can be implemented.

[0046] The beneficial effects of the present invention compared with the prior art are: the present invention provides a multimodal RAG, device, equipment and storage medium based on a large model, including the following steps: preprocessing text information in a large-scale corpus using multimodal technology; constructing courseware content segments based on the preprocessed text information and storing them in the created index; based on the preprocessed keywords in the questions sent by the user, recalling the courseware content segments related to the questions in combination with the index to generate the first recall results; based on the results of the first recall, expanding the content according to preset rules, and performing a second recall; sorting the results of the second recall in multiple dimensions; integrating the sorted results with the user questions, judging the relevance through the large model, and further screening the search results. Compared with the prior art, the present invention greatly increases the accuracy of the recalled content and improves the generation efficiency by introducing multimodal technology, a second recall strategy and the use of a large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0048] Figure 1 A schematic diagram of an application environment of a multi-modal RAG based on a large model provided by an embodiment of the present invention;

[0049] Figure 2A schematic diagram of a multi-modal RAG process based on a large model provided by an embodiment of the present invention Figure 1 ;

[0050] Figure 3 A schematic diagram of a multi-modal RAG process based on a large model provided by an embodiment of the present invention Figure 2 ;

[0051] Figure 4 A schematic diagram of a multi-modal RAG process based on a large model provided by an embodiment of the present invention Figure 3 ;

[0052] Figure 5 A schematic diagram of a multi-modal RAG process based on a large model provided by an embodiment of the present invention Figure 4 ;

[0053] Figure 6 A schematic diagram of a multi-modal RAG process based on a large model provided by an embodiment of the present invention Figure 5 ;

[0054] Figure 7 A schematic diagram of a multi-modal RAG process based on a large model provided by an embodiment of the present invention Figure 6 ;

[0055] Figure 8 A schematic block diagram of a multi-modal RAG device based on a large model provided by an embodiment of the present invention;

[0056] Fig. 9 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0059] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0060] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0061] The embodiment of the present invention provides a multi-modal RAG based on a large model, which can be applied in Figure 1 In the application environment, the client communicates with the server through the network. The server can pre-process the text information in the large-scale corpus by using multimodal technology through the client; construct the courseware content fragments according to the pre-processed text information and store them in the created index; based on the pre-processed keywords in the questions issued by the user, combined with the index and query algorithm, recall the courseware content fragments related to the question to generate the first recall result; according to the result of the first recall, expand the content according to the preset rules, and perform a second recall; sort the results of the second recall in multiple dimensions; integrate the sorted results with the user's questions, judge the relevance through the big model, further filter the search results, and feed the generated text back to the client. In the present invention, the accuracy of the recalled content is greatly increased by introducing multimodal technology, the second recall strategy and the use of the big model, and the generation efficiency is improved. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0062] An embodiment of the present application provides a multimodal RAG based on a large model.

[0063] Figure 2 This is a schematic diagram of a multi-modal RAG process based on a large model provided in an embodiment of the present application. Figure 1 .like Figure 2 As shown, the method includes the following steps S110-S160.

[0064] S110, using multimodal technology to pre-process text information in a large-scale corpus;

[0065] Specifically, the multimodal technology in this embodiment includes combining image recognition, speech recognition, natural language processing and other technologies to comprehensively process multimedia information such as text, pictures, and audio in the corpus. Preprocessing includes steps such as word segmentation, stop word removal, part-of-speech tagging, entity recognition, and semantic understanding to improve the efficiency and accuracy of subsequent processing.

[0066] S120, constructing courseware content fragments according to the preprocessed text information and storing them in the created index;

[0067] Specifically, since ES is a distributed search engine, in order to process a large amount of data and achieve horizontal expansion, the index is usually decomposed into multiple parts, which are shards. Shards can be distributed on different nodes to achieve parallel processing and query of data. Each shard is an independent, self-contained index unit with its own inverted index, documents, mappings and settings. This embodiment divides the pre-processed text information into smaller units according to certain rules (such as paragraphs, chapters, etc.), and uses the distributed storage and indexing functions of ES to store the courseware content in shards and create corresponding indexes.

[0068] ES refers to Elasticsearch, which is an open source search engine based on the Lucene library and provides real-time search and analysis functions. Elasticsearch itself has the characteristics of distributed storage and fast retrieval speed, so it is often used to implement full-text retrieval functions. In addition, Elasticsearch also has natural language processing (NLP) functions, such as word segmentation, part-of-speech tagging, named entity recognition, sentiment analysis and text summarization, which make Elasticsearch more intelligent and efficient in processing text information.

[0069] S130, based on the pre-processed keywords in the question sent by the user, recall the courseware content segments related to the question in combination with the index to generate the first recall result;

[0070] Specifically, in the first recall phase, the system preprocesses the questions sent by users, such as word segmentation and stop word removal; then uses ES's query DSL (Domain Specific Language) to take the preprocessed keywords as query conditions, and combines the KNN (K-Nearest Neighbors) query algorithm to build query statements to recall the courseware content fragments related to the questions. The KNN algorithm can find the K vectors closest to the query vector based on the distance between the vectors. This step is mainly based on the keyword release es index, supplemented by the KNN semantic release query, to expand the recalled relevance fragments.

[0071] It should be understood that "recall" here refers to the process of retrieving or extracting data items that match or are related to the query conditions from the stored data based on specific query conditions or algorithms.

[0072] S140, based on the result of the first recall, expand the content according to the preset rules and conduct a second recall;

[0073] Specifically, through the courseware ID and the first hit page number, as well as actual business, such as the need to cross documents and expand pages, the dimensions of page number expansion before and after are determined, and a second recall is performed.

[0074] S150, sorting the results of the secondary recall in multiple dimensions;

[0075] S160, integrating the sorted results with the user questions, making relevance judgments through a large model, and further filtering the search results.

[0076] like Figure 3 As shown, in a more specific embodiment, executing step S110 further specifically includes executing steps S111 - S112 .

[0077] S111, using training courseware from different platforms as a corpus, wherein the training courseware includes documents in different formats and audio and video;

[0078] S112. Effectively extracting text from the corpus, wherein the extraction method includes using a variety of different libraries and tools for extraction.

[0079] Specifically, the training courseware in this embodiment is selected for the Zhiniao platform. When the format type of the training courseware is the docx format type, the python-docx library is used to open the docx file and extract the text content therein. For some descriptive charts in docx, OCR combined with OPENCV is used to extract the text in the image and merge it with the plain text content in the document for analysis.

[0080] When the format type of the training courseware is pptx, use the python-pptx library to read the text content, slide layout and other information in the ppt. For elements such as charts and pictures, use the plotly tool to convert charts into data and merge them with the text description of the slide for analysis. For audio and video elements in ppt, parse metadata, such as the duration of the audio, the key frames of the video, etc., and associate them with the text of the slide. For example, explain a concept in the slide based on the content of the audio, and determine the focus of the text description related to it based on the video content.

[0081] When the training courseware is in PDF format, pypdf2 is used to parse the text content of PDF, and tesseract OCR is used to process the image text in PDF. Interactive elements (hyperlinks, forms, etc.) in PDF are parsed and associated with the text content. The structure of PDF is parsed, including page layout, chapter division, title recognition, subtitle recognition, etc. These structural information are associated with audio and video for analysis to obtain more comprehensive document information.

[0082] When the format type of the training courseware is xlsx, use the openpyxl library to read the data in the xlsx file. If the spreadsheet records multimedia-related data (such as the number of video views, the relationship between the duration of audio and a certain classification, etc.), associate the data in the table with the actual audio and video files for analysis. For example, classify or filter audio and video based on the data in the spreadsheet. Analyze the relationship between data in the xlsx file, such as by finding duplicate data, calculating correlations, etc. If the table data is associated with other file formats (docx or ppt), use multimodal technology to combine them. For example, generate charts in ppt based on the data in xlsx, and maintain the consistency of data relationships during the parsing process.

[0083] like Figure 4 As shown, in a more specific embodiment, executing step S120 further specifically includes executing steps S121-S122:

[0084] S121, define index mapping;

[0085] Specifically, when creating an index, you need to define the index mapping, including field type, word breaker, lowercase conversion and other settings. These settings will determine how to index and query the documents stored in the index. By setting these parameters reasonably, you can optimize the index storage structure, improve search speed, reduce unnecessary resource waste, and provide users with a more accurate and efficient search experience.

[0086] Among them, field type refers to the data type of the field, including text, keywords, numbers, dates, and Boolean values. Different field types will affect the way Elasticsearch stores, indexes, and queries data; the tokenizer is the core component used to segment text in Elasticsearch. It splits a piece of text into multiple keywords for subsequent searching and matching; lowercase conversion refers to converting text to lowercase in Elasticsearch to achieve case-insensitive search.

[0087] S122. Convert the courseware content into vector representations.

[0088] Specifically, this embodiment is based on vector-based queries. The courseware content shards need to be converted into vector representations. This step involves using some text vectorization techniques (such as TF-IDF, Word2Vec, BERT, etc.) to convert text information into high-dimensional vectors.

[0089] As Figure 5 shown, in a more specific embodiment, performing the conversion of the courseware content shards into vector representations in step S122 further specifically includes performing steps S1221 - S1223:

[0090] S1221. Load a pre-trained word embedding model and convert each word in the text into a corresponding vector representation;

[0091] S1222. Store the vector representation in ES using a vector field;

[0092] S1223. Define the type, dimension, and distance metric of the vector field.

[0093] Specifically, first, select a suitable pre-trained word embedding model. These models include Word2Vec, GloVe, BERT, etc. When selecting a model, the training data, dimension, and performance of the model should be considered. Use appropriate libraries (such as gensim, transformers, etc.) to load the selected pre-trained model to ensure that the weights and parameters of the model are correctly loaded for subsequent word vector conversion. Perform word segmentation on the text of the courseware content. Word segmentation is the process of splitting text into independent lexical units. Use the loaded pre-trained model to convert each segmented word into a corresponding vector representation. These vectors represent the positions of the vocabulary in the semantic space and can be used for subsequent similarity calculation and retrieval. Create an index in Elasticsearch to store the vector representation of the courseware content, use the vector to represent the courseware content shards, and configure the vector field in the query to determine parameters such as the dimension of the vector and the distance metric.

[0094] As Figure 6 shown, in a more specific embodiment, performing step S130 further specifically includes performing steps S131 - S134.

[0095] S131. Perform keyword preprocessing by means of removing stop words, stemming, intent judgment, extracting keywords, and keyword weighting;

[0096] Specifically, stop words: refer to words that appear frequently in the text but contribute little to the meaning of the text, such as "de" (的), "le" (了), etc. In the keyword preprocessing stage, these stop words need to be removed first to reduce the complexity of subsequent processing and improve the purity of the keywords.

[0097] Stemming: It is the process of reducing words to their basic forms. For example, reducing "running" and "run" to the same stem "run". This helps to consider words with different forms but the same meaning as the same keyword, thus improving the keyword coverage rate.

[0098] Intention judgment: By performing semantic analysis on the text, the query intention of the user is judged, which helps to more accurately extract keywords related to the query intention subsequently.

[0099] Keyword extraction: Based on removing stop words, performing stemming, and intention judgment, natural language processing technology is used to extract keywords from the text. These keywords should be able to accurately reflect the theme and content of the text.

[0100] Keyword weighting: Different weights are assigned to keywords according to their importance levels in the text. It is achieved by statistically analyzing factors such as the frequency of keyword occurrence and position in the text. The weighted keywords will have higher priorities in subsequent processing.

[0101] S132. Query the processed keywords using an inverted index;

[0102] Specifically, an inverted index is a data structure that records in which documents each keyword appears and the positions of these keywords in the documents. Use the preprocessed keywords to search in the inverted index, which usually involves searching for items that exactly match the query keywords in the index. For each matching keyword, the inverted index will provide a list of documents containing that keyword (and possibly position information). If the query contains multiple keywords, the query results for each keyword need to be merged. This usually involves using Boolean logic (such as AND, OR, NOT) to combine the result sets. For example, if the query is "machine learning AND deep learning", the system needs to find documents that contain both the keywords "machine learning" and "deep learning". Secondly, sort the merged results according to certain sorting criteria (such as relevance score, document quality, timestamp, etc.). Return the sorted result set to the user, usually including information such as document title, abstract, URL, etc.

[0103] S133. Calculate the similarity between the query problem vector and the slice vector of the courseware content;

[0104] Specifically, after converting the query problem into a vector representation, calculate its similarity with the slice vector of the courseware content, which can be achieved by using an embedding model (such as BERT, GPT, etc.) to convert the text into a vector and applying a metric method such as cosine similarity.

[0105] S134. Recall the slice of the courseware content related to the problem according to the similarity.

[0106] Specifically, based on the calculated similarity, the courseware content fragments with higher similarity are selected as the first recall results. These fragments have higher semantic relevance to the query question and are therefore more likely to provide useful information to users.

[0107] like Figure 7 As shown, in a more specific embodiment, executing step S140 to expand the content according to the preset rule further specifically includes executing steps S141-S142.

[0108] S141, setting forward expansion of N pages and backward expansion of M pages;

[0109] S142. Adjust the values ​​of N and M according to actual business needs and the characteristics of the courseware content.

[0110] Specifically, by setting the rules of expanding forward N pages and expanding backward M pages, the system can provide users with the previous and next page content associated with the current query content. This helps users better understand the context of the current page and improve the coherence and depth of learning. Secondly, the values ​​of N and M can be adjusted according to actual business needs and the characteristics of the courseware content. This means that the system is flexible and can be customized according to the needs of different scenarios. For example, the need to cross documents and expand pages.

[0111] In a more specific embodiment, the multiple dimensions in executing step S150 include: intent recognition, word aggregation density, keyword weight and fuzzy matching degree.

[0112] Specifically, in search engines, the initial sorting is usually based on factors such as the matching degree between documents and queries, field values, custom scripts, etc. However, in order to provide more accurate search results that better meet user needs, secondary sorting is often required to further optimize the display order of search results. In this embodiment, the ES results are secondary sorted using multiple dimensions such as intent recognition, word aggregation density, TF-IDF weights of keywords in certain fields, and fuzzy matching.

[0113] Among them, intent recognition includes using intent classification systems (such as Figure 3 The intent analysis of user queries is done by using the search engine's search engine's search engine classification or Do-Know-Go system. Different weights or priorities are assigned to search results based on the intent analysis results. For example, for navigational searches, the target website's official website can be ranked first; for informational searches, the ranking can be adjusted based on the specificity of the query (direct, indirect, etc.).

[0114] Word cluster density involves calculating the word cluster density of each document in the search results, that is, the frequency and distribution of specific words or phrases in the document. Search results are sorted according to the word cluster density. Generally, documents with higher word cluster density are more likely to contain the information that users need.

[0115] The weight of a keyword refers to the TF-IDF weight of a keyword in a specific field, which includes extracting keywords in a specific field and calculating the TF-IDF weight of these keywords in the search results. Search results are sorted according to the TF-IDF weight. The TF-IDF weight reflects the importance and uniqueness of a keyword in a document, so documents with higher weights are more likely to contain valuable information related to the user's query.

[0116] Fuzzy matching involves calculating the fuzzy matching between search results and queries using Elasticsearch's fuzzy matching function (such as match query). Search results are sorted according to the degree of fuzzy matching. Documents with higher fuzzy matching are more likely to meet the user's query needs, even if they do not completely match the exact terms of the query.

[0117] In summary, the ranking results of the above multiple dimensions are integrated to obtain a comprehensive score. The search results are finally sorted according to the comprehensive score.

[0118] Compared with the existing technology, this solution optimizes the problems in the pre-processing and recall stages of the current RAG processing flow in the industry, and has achieved the following beneficial effects:

[0119] 1) Currently, most of the industry supports the search of pure text information. This solution adopts a multimodal approach to accurately extract data text information in various formats, thereby reducing the loss of text information and making the recalled content more accurate and complete;

[0120] 2) In view of the fact that the content recalled by es is not accurate enough and the information is easily missing, a second-order recall strategy is adopted. The first step is to comprehensively recall relevant documents through keywords and vectors, and the second step is to use the course ID and the page number of the first hit. According to the actual business, such as the need to cross documents and the problem of page expansion, the dimensions of the page number expansion before and after are determined, and the second recall is performed, which improves the recall accuracy and reduces the information missing;

[0121] 3) Through multi-faceted judgment and the use of large models, the order of recalled documents is reorganized to ensure the relevance and accuracy of the recalled content.

[0122] Figure 8 is a schematic block diagram of a multi-modal RAG device based on a large model provided in an embodiment of the present application. Figure 8As shown, corresponding to the above multimodal RAG based on a large model, the present application also provides a multimodal RAG device based on a large model 200. The multimodal RAG device based on a large model 200 includes a unit for executing the above multimodal RAG based on a large model, and the device can be configured in a terminal such as a desktop computer, a tablet computer, a laptop computer, etc.

[0123] Specifically, see Figure 8 , the large model-based multimodal RAG device 200 includes:

[0124] The preprocessing unit 210 is used to preprocess the text information in the large-scale corpus using multimodal technology.

[0125] The index building unit 220 is used to build courseware content segments according to the pre-processed text information and store them in the created index.

[0126] The index building unit 220 is also specifically used to: define the mapping of the index; convert the courseware content fragments into vector representations. The process of converting the courseware content fragments into vector representations includes: loading a pre-trained word embedding model to convert each word in the text into a corresponding vector representation; using a vector field to store the vector representation in the index; and defining the type, dimension, and distance measurement method of the vector field.

[0127] The first recall unit 230 is used to recall the courseware content segments related to the question based on the pre-processed keywords in the question sent by the user in combination with the index to generate the first recall result.

[0128] The secondary recall unit 240 is used to expand the content according to the preset rules based on the result of the first recall and to perform a secondary recall. The expansion of the previous and next page numbers according to the preset rules includes: setting the expansion of N pages forward and M pages backward; and adjusting the values ​​of N and M according to the actual business needs and the characteristics of the courseware content.

[0129] The sorting unit 250 is used to sort the results of the secondary recall in multiple dimensions, wherein the multiple dimensions include: intention recognition, word aggregation density, keyword weight and fuzzy matching degree.

[0130] The text generation unit 260 is used to integrate the sorted results with the user questions, make relevance judgments through the large model, and further filter the search results.

[0131] In addition, more specifically, the preprocessing unit 210 is specifically used to: use training courseware from different platforms as a corpus, the training courseware includes documents and audio and video in different formats; effectively extract text from the corpus, and the extraction method includes using a variety of different libraries and tools for extraction.

[0132] The first recall unit 230 is specifically used for: preprocessing keywords by removing stop words, stemming, judging intent, extracting keywords, and weighting keywords; querying processed keywords by using an inverted index; calculating the similarity between the query question vector and the courseware content segment vector; and recalling the courseware content segments related to the question based on the similarity.

[0133] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned multimodal RAG device based on a large model and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.

[0134] The above-mentioned large model-based multimodal RAG device 200 can be implemented in the form of a computer program. The computer program can be used in Fig. 9 Runs on the computer device shown.

[0135] See also Fig. 9 , Fig. 9 300 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 300 may be a terminal or a server, wherein the terminal may be an electronic device with communication functions such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The server may be an independent server or a server cluster composed of multiple servers.

[0136] See also Fig. 9 The computer device 300 includes a processor 302 , a memory and a network interface 305 connected via a system bus 301 , wherein the memory may include a non-volatile storage medium 303 and an internal memory 304 .

[0137] The non-volatile storage medium 303 may store an operating system 3031 and a computer program 3032. The computer program 3032 includes program instructions, which, when executed, may enable the processor 302 to execute a large model-based multimodal RAG.

[0138] The processor 302 is used to provide computing and control capabilities to support the operation of the entire computer device 300 .

[0139] The internal memory 304 provides an environment for the operation of the computer program 3032 in the non-volatile storage medium 303. When the computer program 3032 is executed by the processor 302, the processor 302 can execute a multi-modal RAG based on a large model.

[0140] The network interface 305 is used to communicate with other devices over the network. Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 300 to which the solution of the present application is applied. The specific computer device 300 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] It should be understood that in the embodiment of the present application, the processor 302 may be a central processing unit (CPU), and the processor 302 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0142] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0143] Therefore, the present application also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the following steps:

[0144] S110, using multimodal technology to pre-process text information in a large-scale corpus;

[0145] S120, constructing courseware content fragments according to the preprocessed text information and storing them in the created index;

[0146] S130, based on the pre-processed keywords in the question sent by the user, recall the courseware content segments related to the question in combination with the index to generate the first recall result;

[0147] S140, based on the result of the first recall, expand the content according to the preset rules and conduct a second recall;

[0148] S150, sorting the results of the secondary recall in multiple dimensions;

[0149] S160, integrating the sorted results with the user questions, making relevance judgments through a large model, and further filtering the search results.

[0150] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0151] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0152] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0154] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application.

[0156] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A multimodal RAG based on a large model, characterized in that: The following steps are involved: Use multimodal technology to preprocess text information in large-scale corpora; Construct courseware content segments based on preprocessed text information and store them in the created index; Based on the pre-processed keywords in the questions sent by users, the courseware content segments related to the questions are recalled in combination with the index to generate the first recall results; Based on the results of the first recall, the content is expanded according to the preset rules and a second recall is conducted; Sorting the results of the secondary recall in multiple dimensions; Integrate the sorted results with user questions, make relevance judgments through a large model, and further filter the search results.

2. A large model-based multimodal RAG according to claim 1, characterized in that: The step of preprocessing text information in a large-scale corpus using multimodal technology includes: Using training courseware from different platforms as a corpus, the training courseware includes documents in different formats and audio and video; Effectively extracting text from a corpus, the extraction method includes using a variety of different libraries and tools to extract.

3. The multimodal RAG based on a large model according to claim 1, characterized in that: The step of constructing courseware content segments according to the preprocessed text information and storing them in the created index comprises: Define the mapping of the index; Convert courseware content slices into vector representations.

4. The multimodal RAG based on a large model according to claim 3, characterized in that: The step of converting the courseware content segments into vector representations comprises: Load a pre-trained word embedding model to convert each word in the text into its corresponding vector representation; storing the vector representation in an index using a vector field; Defines the type, dimension, and distance measure of a vector field.

5. The multimodal RAG based on a large model according to claim 1, characterized in that: The step of generating the first recall result by combining the index to recall the courseware content segments related to the question based on the pre-processed keywords in the question sent by the user includes: Preprocess keywords by removing stop words, stemming, judging intent, extracting keywords, and weighting keywords; Use the inverted index to query the processed keywords; Calculate the similarity between the query question vector and the courseware content segment vector; Recall the courseware content segments related to the question based on similarity.

6. The multimodal RAG based on a large model according to claim 1, characterized in that: The content expansion according to the preset rules includes: Set to expand forward by N pages and backward by M pages; Adjust the values ​​of N and M according to actual business needs and the characteristics of the courseware content.

7. The multimodal RAG based on a large model according to claim 1, characterized in that: The results of the secondary recall are sorted in multiple dimensions, wherein the multiple dimensions include: intent recognition, word aggregation density, keyword weight and fuzzy matching degree.

8. A multi-modal RAG device based on a large model, characterized in that: The device comprises: A preprocessing unit, used to preprocess text information in a large-scale corpus using multimodal technology; An index building unit, used to build courseware content segments based on preprocessed text information and store them in the created index; The first recall unit is used to recall the courseware content segments related to the question based on the pre-processed keywords in the question sent by the user and in combination with the index to generate the first recall result; A secondary recall unit is used to expand the content and conduct a secondary recall according to the result of the first recall and the preset rules; A sorting unit, used to sort the results of the secondary recall in multiple dimensions; The text generation unit is used to integrate the sorted results with the user questions, make relevance judgments through the large model, and further filter the search results.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 can be implemented.

Images