Multi-modal data retrieval enhancement generation method and system based on RAG technology

Abstract

The invention discloses a multi-modal data retrieval enhancement generation method and system based on an RAG technology. The method comprises the following steps: performing character recognition on collected picture data by using a self-adaptive OCR (Optical Character Recognition) strategy; carrying out data loading on the collected document data and processing the collected document data into text data; performing text slicing and vectorization on the text data; using a vector database to store vectors and text blocks, and constructing an index; vectorizing the input text of the user, carrying out vector matching by using a matching algorithm, and querying a corresponding text; and generating and displaying a query result. According to the method, layout analysis is carried out on the picture, so that different OCR recognition strategies are adopted, and the accuracy of text recognition is improved; the multi-modal data is formed through the text identified by the picture and the text of the document, so that the retrieval and generation accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and natural language processing technology, and in particular to a multimodal data retrieval enhancement generation method and system combining image optical character recognition and text retrieval generation technology. Background Art

[0002] With the development of technology, enterprises need to process a large amount of image and text data, and the demand for information retrieval is increasing. Traditional retrieval methods are usually limited to text data and cannot effectively process image data, which limits the application scope and accuracy of the retrieval system. The retrieval systems on the market currently mainly rely on text retrieval technology and ignore the value of image data. In the existing technology, there is a lack of a multimodal retrieval system that can process image and text data simultaneously.

[0003] With the development of OCR technology, text information in image data can be recognized and converted into text, but how to efficiently combine image and text data for retrieval remains a difficulty. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal data retrieval enhancement generation method and system based on RAG technology to solve the problems raised in the background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: A multimodal data retrieval enhancement generation method based on RAG technology, the method comprising: Use adaptive OCR recognition strategy to perform text recognition on the collected image data; Load the collected document data and process it into text data; Perform text segmentation and vectorization on text data; Use vector database to store vectors and text blocks and build indexes; Vectorize the user's input text, use a matching algorithm to perform vector matching, and query the corresponding text; Generate query results and display them.

[0006] Preferably, the image is preprocessed before the text recognition.

[0007] Preferably, the preprocessing includes image resizing, cropping and grayscale processing.

[0008] Preferably, the adaptive OCR recognition strategy includes: When the image contains only text, PaddleOCR is used to recognize the text in the image; when the image contains a table, PP_Structure is used to recognize the text in the image.

[0009] Preferably, the data loading and processing includes: Load documents containing various formats; Determine the document type. If it is a WORD type document, determine whether the document contains a table. If a table exists, traverse the text in the table and store it together with other text in the document. Otherwise, store the text directly. If it is a PDF, Markdown or text document, directly use the corresponding type of document loader provided in Langchain to read the text and store it.

[0010] Preferably, the text slicing and vectorizing comprises: Use recursive character text segmentation method to segment text into strings, and concatenate the strings to form text blocks of fixed length; Use the text embedding model to convert text blocks into vectors and normalize the vectors.

[0011] Preferably, the text embedding module uses a BERT, RoBERTa or GPT model.

[0012] Preferably, the matching algorithm uses a cosine similarity algorithm or a similarity algorithm based on Euclidean distance.

[0013] A multimodal data retrieval enhancement generation system based on RAG technology, the system comprising: Image upload module: used to upload images in various formats; Document upload module: used to upload documents in various formats; OCR recognition module: uses adaptive OCR recognition strategy to recognize text in uploaded images and displays the recognized text; Document loading module: loads uploaded documents, extracts text from the documents and displays them; The text segmentation and vectorization module is used to obtain text segments based on text and vectorize them; Vector database storage module: used to store text blocks and corresponding vectors; Text retrieval module: used to match text blocks in the vector database module for the text entered by the user through a similarity algorithm; The retrieval enhancement generation module is used to generate and display query results based on matching text blocks using RAG technology.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention performs layout analysis on images and adopts different OCR recognition strategies to improve the accuracy of text recognition; multimodal data is formed through text recognized by images and text of documents to improve the accuracy of retrieval and generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is an architectural diagram of a multimodal data retrieval enhancement generation method based on RAG technology of the present invention.

[0016] Figure 2 The present invention is a flowchart of a multimodal data retrieval enhancement generation method based on RAG technology. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0018] Reference Figure 1-2 As shown, a multimodal data retrieval enhancement generation method based on RAG technology includes the following steps: S1, uses adaptive OCR recognition strategy to perform text recognition on the collected image data.

[0019] In the present invention, the image data may be a screenshot or data in other formats, including images in jpg, png, jpeg and other formats.

[0020] In S1, before text recognition, the image needs to be preprocessed, including resizing, cropping, grayscale processing, etc., to improve the accuracy of the OCR model.

[0021] OCR technology is the abbreviation of Optical Character Recognition. It converts the text of various bills, newspapers, books, manuscripts and other printed materials into image information through optical input methods such as scanning, and then uses text recognition technology to convert the image information into usable computer input technology.

[0022] The function of the OCR model is to recognize images and convert the text in the image into editable or searchable text format. PaddleOCR and PP_Structure are selected here to implement the adaptive OCR recognition strategy. Specifically: determine whether the image contains table information. If there is no table, use PaddleOCR to recognize the text information in the corresponding image. If the image contains table data, use PP_Structure to recognize the text content in the table data.

[0023] S2, loading the collected document data and processing it into text data.

[0024] In the present invention, the document types include PDF, WORD, Markdown and other format documents. In S2, it is necessary to identify the type of uploaded document, use the corresponding loader according to different types to load the provided documents of different types, and convert them into text data. Specifically, if the uploaded document is a WORD type document, it is necessary to determine whether the document contains a table. If there is a table, it is necessary to traverse the text data in the table and store it together with other plain text data in the document; if it is a PDF, Markdown or text type file, directly use the corresponding type of document loader provided in Langchain to read the data.

[0025] S3, text data is segmented and vectorized, specifically: S3.1, using a recursive character text segmentation method to segment the text into character strings, and concatenating the character strings to form text blocks of fixed length; S3.2, based on the existing text embedding model, each segmented text block is converted into a vector, and the converted vector is normalized.

[0026] The Recursive Character Text Segmentation method also known as Recursive Character Splitter is a tool for segmenting text incrementally. It segments text by a provided list of characters until the chunks reach a specified size.

[0027] Here, the size of the text block is adjusted according to the length of the text to be queried, and at the same time, each segmented text block must contain the information of the adjacent text blocks to ensure the coherence between blocks. Based on the preliminary segmentation, the boundaries of the blocks are adjusted to segment them as much as possible at the boundaries of sentences or paragraphs. This ensures the semantic coherence of the sentences and retains the integrity of the sentences.

[0028] By normalizing the vectors, the influence of the vector scale can be eliminated, making the comparison between vectors more accurate.

[0029] The text embedding model can use BERT, RoBERTa or GPT, etc., and technical personnel in this field can set it according to actual conditions.

[0030] S4, stores the text block data and vectorized data together in the vector database, and builds corresponding indexes for easy search.

[0031] S5, uses the text embedding model to convert the user input content into a vector, and uses the matching algorithm to match the vector and query the corresponding text.

[0032] In the present invention, the text embedding model used by S3.2 is consistent with that used by S5, ensuring the uniformity of the converted vectors.

[0033] Convert the user input into an input vector, perform similarity matching with the vectors stored in the database, and use the vector matching results to obtain the corresponding text block. Use similarity matching algorithms (such as cosine similarity or Euclidean distance) or efficient approximate nearest neighbor search algorithms to find a group of text blocks that are most similar to the input vector to improve the accuracy of retrieval.

[0034] Take the similarity matching algorithm as an example: Cosine similarity is widely used to calculate the similarity of document data. Its principle is the cosine value of the angle between two points in multidimensional space and the set point.

[0035] vector With vector The cosine similarity is:

[0036] Euclidean distance, also known as Euclidean distance, is the easiest way to calculate distance. It is derived from the distance formula between two points in Euclidean space.

[0037] In plane space and The Euclidean distance between: .

[0038] S6, based on the queried text block, uses RAG technology to generate and display query results.

[0039] Retrieval-Augmented Generation (RAG) technology cleverly combines the power of information retrieval and neural network generation models. By introducing relevant external information in the generation process, it achieves accurate, diverse, and contextual text generation based on a large-scale knowledge base. In S6, after obtaining the matching text block through S5, the RAG technology is used to generate and display relevant answers.

[0040] Another aspect of the present invention further provides a multimodal data retrieval enhancement generation system based on RAG technology, characterized in that the system comprises: Image upload module: used to upload images in various formats; Document upload module: used to upload documents in various formats; OCR recognition module: uses adaptive OCR recognition strategy to recognize text in uploaded images and displays the recognized text; Document loading module: loads the uploaded document, extracts the text in the document and displays it; The text segmentation and vectorization module is used to obtain text segments based on text and vectorize them; Vector database storage module: used to store text blocks and corresponding vectors; Text retrieval module: used to match text blocks in the vector database module for the text entered by the user through a similarity algorithm; The retrieval enhancement generation module is used to generate and display query results based on matching text blocks using RAG technology.

[0041] The system of the present invention can utilize the above modules to comprehensively process image and text data to construct a vector database, so as to complete the text retrieval enhancement generation function for the search content and improve the accuracy and practicality of the retrieval. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal data retrieval enhancement generation method based on RAG technology, characterized in that: The method comprises: Use adaptive OCR recognition strategy to perform text recognition on the collected image data; Load the collected document data and process it into text data; Perform text segmentation and vectorization on text data; Use vector database to store vectors and text blocks and build indexes; Vectorize the user's input text, use a matching algorithm to perform vector matching, and query the corresponding text; Generate and display query results using RAG technology.

2. The multimodal data retrieval enhancement generation method based on RAG technology as claimed in claim 1, characterized in that: Before the text recognition, the picture is preprocessed.

3. The multimodal data retrieval enhancement generation method based on RAG technology as claimed in claim 2, characterized in that: The preprocessing includes image resizing, cropping and grayscale processing.

4. The multimodal data retrieval enhancement generation method based on RAG technology as claimed in claim 1, characterized in that: The adaptive OCR recognition strategy includes: When the image contains only text, PaddleOCR is used to recognize the text in the image; when the image contains a table, PP_Structure is used to recognize the text in the image.

5. The multimodal data retrieval enhancement generation method based on RAG technology as claimed in claim 1, characterized in that: The data loading and processing includes: Load documents containing various formats; Determine the document type. If it is a WORD type document, determine whether the document contains a table. If a table exists, traverse the text in the table and store it together with other text in the document. Otherwise, store the text directly. If it is a PDF, Markdown or text document, directly use the corresponding type of document loader provided in Langchain to read the text and store it.

6. The multimodal data retrieval enhancement generation method based on RAG technology as claimed in claim 1, characterized in that: The text segmentation and vectorization includes: Use recursive character text segmentation method to segment text into strings, and concatenate the strings to form text blocks of fixed length; Use the text embedding model to convert text blocks into vectors and normalize the vectors.

7. The multimodal data retrieval enhancement generation method based on RAG technology as claimed in claim 6, characterized in that: The text embedding module uses the BERT, RoBERTa or GPT model.

8. The multimodal data retrieval enhancement generation method based on RAG technology as claimed in claim 1, characterized in that: The matching algorithm uses a cosine similarity algorithm or a similarity algorithm based on Euclidean distance.

9. A multimodal data retrieval enhancement generation system based on RAG technology, characterized in that: The system comprises: Image upload module: used to upload images in various formats; Document upload module: used to upload documents in various formats; OCR recognition module: uses adaptive OCR recognition strategy to recognize text in uploaded images and displays the recognized text; Document loading module: loads uploaded documents, extracts text from the documents and displays them; The text segmentation and vectorization module is used to obtain text segments based on text and vectorize them; Vector database storage module: used to store text blocks and corresponding vectors; Text retrieval module: used to match text blocks in the vector database module for the text entered by the user through a similarity algorithm; The retrieval enhancement generation module is used to generate and display query results based on matching text blocks using RAG technology.

Images