RAG-based multi-format data rapid query method

Abstract

The invention relates to the technical field of data management and query, and discloses a multi-format data quick query method based on RAG, which comprises the following steps: step a, a user uploads a target file through a front-end interface, and the file is any one of PDF, Word or Excel formats; b, carrying out format verification on the uploaded file; c, selecting a corresponding analysis tool according to a file format, and performing text and structured data extraction on the file by using a PyMuPDF library, a python-docx library or an openpyxl library; and step d, respectively storing the analyzed data into a PostgreSQL (Structured Query Language) database and a Faiiss database. By combining structured data storage and non-structured data vectorization storage, efficient vector similarity retrieval is carried out by utilizing Fiss, and the response time of data query is greatly shortened. Compared with a traditional query method based on full-text search, the method has the advantages that the result most related to user query can be quickly and accurately returned from a large amount of data, so that the data query efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management and query, and in particular to a multi-format data fast query method based on RAG. Background Art

[0002] With the continuous development of information technology, the amount of data generated by enterprises and individual users has exploded, and most of this data is unstructured data, such as text, pictures, audio, video, etc. In addition, structured data still occupies a core position in business processes, especially in enterprise management and information query. Traditional data management systems often rely on structured data storage, while the processing and query of unstructured data face significant challenges. How to efficiently manage, store and retrieve these two different types of data has become a problem that needs to be solved in modern data management and query technology.

[0003] Existing multi-format data query technologies usually use independent data storage and query methods. For example, for structured data, traditional relational databases (such as PostgreSQL, MySQL, etc.) usually use SQL query language for storage and retrieval. For unstructured data, such as files in PDF, Word, Excel and other formats, traditional methods often rely on full-text search engines or extract information through simple text retrieval technology. Although these methods have achieved certain success in their respective application fields, they have obvious limitations in cross-format query, multimodal data retrieval and efficiency.

[0004] A major problem with the prior art is that it is impossible to effectively integrate structured data with unstructured data for unified query. Existing systems usually use different processing and query mechanisms for different data formats, which results in users having to switch between multiple systems in actual applications, increasing the complexity of operations and reducing query efficiency. Especially when faced with complex query requirements across formats, existing systems often cannot provide efficient and accurate retrieval results. Therefore, how to efficiently integrate structured and unstructured data and provide fast and accurate query processing capabilities has become a problem that needs to be solved urgently. To address this problem, the present invention proposes a multi-format data fast query method based on RAG, which aims to solve the efficiency and accuracy problems existing in multi-format data queries through a unified query framework and efficient retrieval technology. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a RAG-based multi-format data fast query method, which solves the problem that structured data and unstructured data cannot be efficiently integrated for query in multi-format data query.

[0006] To achieve the above objectives, the present invention is implemented by the following technical scheme: a multi-format data fast query method based on RAG, comprising the following steps: Step (a) The user uploads the target file through the front-end interface, and the file is in any of the formats of PDF, Word or Excel; Step (b) verifying the format of the uploaded file and parsing the data after passing the verification; Step (c) selecting a corresponding parsing tool according to the file format, and using PyMuPDF, python-docx or openpyxl library to extract text and structured data from the file; Step (d) storing the parsed data in a PostgreSQL database and a Faiss database respectively, wherein PostgreSQL is used to store structured data, and Faiss is used to store high-dimensional vectors of unstructured text data; Step (e) generates a query instruction according to the user's query request and executes the query, and the query result returns relevant documents or data through vector similarity retrieval.

[0007] Preferably, in the step (d), the Faiss database stores high-dimensional text vectors generated by a natural language processing model, and the text vectors are generated by a BERT model.

[0008] Preferably, the query instructions include structured query instructions and unstructured query instructions, the structured query instructions are executed in a PostgreSQL database through SQL statements, and the unstructured query instructions perform similarity retrieval through a Faiss database.

[0009] Preferably, the similarity search is based on the vector index in the Faiss database, and the vector similarity calculation formula is as follows: ; Specifically, A and B are two vectors, where A represents the query vector, B Represents a data vector in the Faiss database; is a vector A and B The dot product of , calculate the inner product relationship between them; For vector A and B The mold length.

[0010] Preferably, in the step (b), the data analysis further comprises processing the table data in the uploaded file, converting the table data into a structured data format and storing it in a PostgreSQL database.

[0011] Preferably, the query instruction is automatically generated by a natural language processing module according to a user's query request, the query request is transmitted to the system via text or voice input, and the voice input is converted into text input via a voice recognition module.

[0012] Preferably, in step (d), the data stored in the Faiss database includes vectorized representation of document content, and is combined with structured data in an external knowledge base to perform multi-format data retrieval.

[0013] Preferably, the accuracy of the query results is further improved by a query optimization module, which performs weighted sorting on the search results based on historical query data and query type.

[0014] The present invention provides a multi-format data fast query method based on RAG, which has the following beneficial effects: 1. The present invention combines structured data storage with unstructured data vector storage, and uses Faiss to perform efficient vector similarity retrieval, which greatly shortens the response time of data queries. Compared with the traditional query method based on full-text search, this method can quickly and accurately return the most relevant results to the user's query from a large amount of data, thereby improving the efficiency of data queries.

[0015] 2. The present invention can process data in multiple formats such as PDF, Word, Excel, etc., and can use special parsing tools to process data in different formats. This function of supporting multi-format data uploading, parsing and storage breaks the limitation that traditional systems can only process data in a single format, allowing the system to more flexibly deal with data from different sources, and improving the versatility and adaptability of the system.

[0016] 3. The present invention stores structured data in the PostgreSQL database, and generates high-dimensional vectors for unstructured data through models such as BERT and stores them in the Faiss database, thus achieving effective integration of structured and unstructured data. This innovative solution enables users to obtain information through both SQL query and similarity retrieval, greatly enriching the query methods and improving the accuracy of query results. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Please see attached Figure 1 , an embodiment of the present invention provides a multi-format data fast query method based on RAG, including.

[0020] Step 1: Upload files and verify format; In the RAG-based multi-format data rapid query method of the present invention, file uploading and format verification are the first steps to ensure the smooth progress of the data processing flow. This process involves the system verifying and preliminarily processing the format of the file after the user uploads the file to ensure that the uploaded file meets the format required by the system and lays the foundation for subsequent data parsing and storage. Through this step, the system can efficiently identify the file type and perform reasonable preprocessing on it to avoid data errors or processing failures in subsequent steps caused by files that do not meet the standards. The specific implementation method includes checking and verifying the file format and necessary preprocessing.

[0021] In this embodiment, the steps of file uploading and format verification are described in detail as follows: File upload: Users upload target files through the front-end interface. Supported file formats include common office file formats such as PDF (.pdf), Word (.docx) and Excel (.xlsx). The front-end interface implements file selection through a file selection box, and users can select the files to be uploaded from their local devices. After uploading the file, the system will immediately perform format verification to ensure that the uploaded file meets the format requirements.

[0022] Format Verification: During the initial processing of file upload, the system first checks the format of the uploaded file. At this time, the system preliminarily determines the file type by the file suffix, and further verifies the authenticity of the file content by analyzing the internal structure of the file.

[0023] For PDF files, the system confirms whether the file is a valid PDF file by parsing the identifier in the file header (such as %PDF). If the file content matches the PDF format specification, the subsequent processing steps will continue; otherwise, the user will be prompted to upload a valid PDF file.

[0024] For Word files, the system will verify the file's docx suffix and XML structure to ensure that the uploaded file is a valid Word document. If the verification fails, the user will be prompted to re-upload the file in the correct format.

[0025] For Excel files, the system will also perform a preliminary check based on the file's .xlsx suffix and confirm that it is a valid Excel workbook by parsing the file's xl directory. For non-standard Excel format or damaged files, the system will refuse to upload and prompt the user to reselect the file.

[0026] File preprocessing and compression: After the file format verification is passed, the system enters the file preprocessing stage. The main purpose of this stage is to process the uploaded files to improve the efficiency of subsequent operations.

[0027] For large PDF files, the system will use compression algorithms to compress the files to reduce the storage and processing burden of the files. After compression, the file content will not be lost and can be parsed more efficiently.

[0028] For Word and Excel files, the system performs a preliminary analysis of the text content of the file, such as removing unnecessary formatting information and useless whitespace, to ensure more accurate processing during subsequent parsing.

[0029] Exception handling and error prompts: If the file format verification fails, the system will promptly feedback the error information to the user, prompting the user to upload a file that meets the requirements. At this time, the user can choose to re-upload the file to ensure that the file format meets the standards. The system can also provide solutions to common errors, such as file damage and format non-compliance, to help users quickly solve problems.

[0030] User Notification and Confirmation: After the file is uploaded and verified, the system will notify the user that the file has been successfully uploaded and is ready for the next step of processing. At this time, the system will automatically record the file's metadata (such as file name, upload time, file type, etc.) to provide support for subsequent data storage and query.

[0031] Through the above steps, the method of the present invention ensures the compliance and validity of files in terms of file uploading and format verification, and avoids the impact of incorrect or damaged files on subsequent data processing. This process not only improves the accuracy of file processing, but also provides the system with flexible expansion capabilities, which can support the uploading and parsing of multiple common file formats.

[0032] Step 2: Data analysis and extraction; In the RAG-based multi-format data rapid query method of the present invention, data parsing and extraction is one of the core links of file processing. Its purpose is to extract the content from the uploaded multi-format files and perform standardized processing to facilitate subsequent storage, indexing and query. Due to the diverse formats of files, data parsing and extraction not only need to support the extraction of text information, but also need to consider the characteristics of files in different formats, and adopt different parsing methods for structured data and unstructured data. The system uses a series of specialized parsing tools and algorithms to effectively extract file content to ensure the accuracy and integrity of the data.

[0033] In this embodiment, the process of data analysis and extraction is described in detail as follows: File content extraction and identification: After the format of the file uploaded by the user is verified, it enters the data parsing stage. At this time, the system needs to use different parsing methods to extract content according to the different formats of the file.

[0034] PDF Documents: For PDF files, the system uses the PyMuPDF library in Python to parse the files. PyMuPDF can read the text content, images, and other embedded data in PDF files. The system first extracts the text part of the PDF and removes non-text content such as image tags and table frames through regular expressions. Subsequently, the extracted text data is converted into a unified structured format and stored in the storage area.

[0035] Word Document: For Word files (.docx format), the system extracts the file content through the python-docx library in Python. python-docx can read paragraphs, tables, text styles and other contents in Word files. First, the system identifies the paragraphs in the document and extracts the text content. For the part containing the table, the system extracts the table data separately and stores the data in the table in rows and columns to form structured data.

[0036] Excel file: For Excel files (.xlsx format), the system uses the openpyxl library for data parsing. The system reads the Excel worksheet, extracts the content of each cell, and organizes and stores the data according to the row and column structure of the table. For larger Excel files, the system parses each page separately to avoid memory overflow and ensure the stability of data processing.

[0037] Data standardization and cleaning: During the data extraction process, the system performs standardization on the extracted data. The specific operations include: Remove irrelevant characters and spaces: The system will remove extraneous spaces, line breaks, and invalid characters to ensure data purity.

[0038] Standardized formats: For date, time, currency and other fields in different formats, the system will convert them into a standard format to avoid format inconsistencies in subsequent queries.

[0039] Field naming standardization: For example, for column names in Excel or titles in Word files, the system will automatically identify and perform unified naming based on the field content, making subsequent data management and query easier.

[0040] Structured data extraction and storage: For table data contained in Excel and Word files, the system will store the data in the PostgreSQL database through structured parsing. Each row in the table represents a data record, and each column represents a data field. The system will automatically identify the field names in the table and convert them into column names in the database table. For tables in Word files, the system will also determine whether there are merged cells and perform appropriate processing.

[0041] For example, when a table containing "name", "student number", and "grade" is extracted from a Word document, the system will store these fields as "name", "student number", and "grade" columns in PostgreSQL to ensure structured storage of data and facilitate subsequent queries.

[0042] Unstructured data extraction and vectorization: For unstructured text data (such as the text content in PDF and Word documents), the system will use natural language processing (NLP) technology for semantic analysis. Specifically, the system will use pre-trained NLP models (such as BERT or Word2Vec) to convert text data into fixed-length vectors. A vector will be generated for each paragraph or sentence in the file, which will be stored in the Faiss database for fast semantic similarity retrieval.

[0043] Vectorization steps: The system extracts text data from PDF or Word files.

[0044] Use BERT or Word2Vec model to convert each paragraph or sentence into a vector.

[0045] The generated vectors are stored in the Faiss database to form a vector representation of the document.

[0046] The vectorized text data not only retains the semantic information of the original text, but also helps the system find the content most relevant to the user's query by calculating the similarity between vectors.

[0047] File content is associated with metadata: During the data extraction process, the system will also extract and store relevant metadata for each file. These metadata include information such as the file name, upload time, file type, source, etc. The system associates these metadata with the structured and unstructured data in the file to facilitate precise positioning during subsequent queries.

[0048] Metadata storage: The file's metadata (such as file name, upload time, file size, format type, etc.) will be stored in the database together with the file's parsed data. For each file's content, the system will also record its storage path and index information in the database for quick retrieval and positioning.

[0049] Step 3: Data storage and indexing; In the RAG-based multi-format data fast query method of the present invention, data storage and indexing is one of the core steps. This step aims to efficiently store the parsed and processed data and provide fast indexing for subsequent query operations. Since the present invention supports multi-format data (such as PDF, Word, Excel, etc.), data storage and indexing not only need to process structured data, but also need to vectorize and store unstructured text data. By using two different types of databases (PostgreSQL and Faiss), the system can take into account the storage and query requirements of structured data and unstructured data, and use indexing technology to improve query speed and accuracy.

[0050] In this embodiment, the data storage and indexing process is described in detail as follows: Structured data storage (PostgreSQL): The system stores the table data extracted by parsing Word and Excel files in a PostgreSQL database. PostgreSQL is a relational database management system suitable for storing structured data such as tables, user information, knowledge items, etc. In specific implementation, the system will automatically generate a table structure based on the file content, and the table data in each file will be converted into row and column data in PostgreSQL.

[0051] Table data storage: Each table in an Excel file or table data in a Word document will be converted into a table in the PostgreSQL database. Each row in the table represents a record, and each column represents a field. For example, when storing an Excel file, the system will use the title of each column as the field name in the database table, and each row as a record in the database table.

[0052] Metadata storage: In addition to storing table data, PostgreSQL is also used to store file metadata (such as file name, upload time, file type, etc.). These metadata will help the system manage files and query quickly.

[0053] Example: For an Excel file containing student grades, the data in the table (such as student name, student ID, grade, etc.) will be stored in the "grade table" of PostgreSQL, and each field (such as "name", "student ID", "grade") corresponds to a column in the database table.

[0054] Unstructured Data Storage (Faiss): For unstructured text data in Word, PDF and other files, the system uses Faiss (Facebook AI Similarity Search) for vectorized storage. Faiss is an efficient vector retrieval library that can vectorize text data and provide fast similarity search. The main goal of this step is to convert text data into vectors for fast similarity retrieval and matching.

[0055] The vector similarity calculation formula is as follows: ; Specifically, A and B are two vectors, where A represents the query vector, B Represents a data vector in the Faiss database; is a vector A and B The dot product of , calculate the inner product relationship between them; For vector A and B The mold length.

[0056] Cosine similarity: It measures the directional similarity between two vectors. The larger the value, the more similar the vectors are. This formula can be used to determine the similarity between the query vector and each data vector, thereby finding the most relevant documents or data in the Faiss database.

[0057] Text vectorization: The system uses a natural language processing (NLP) model (such as BERT or Word2Vec) to convert the text content in each file into a high-dimensional vector. The text content of each document, paragraph, or sentence will generate a fixed-length vector through the NLP model to represent the semantic features of the text.

[0058] Vector storage: The generated vectors will be stored in the Faiss database. Faiss can efficiently store a large number of vectors and achieve fast similarity retrieval by building indexes (such as HNSW or IVF indexes). This enables the system to quickly find content similar to the user's query text and improve query efficiency.

[0059] Example: For the content in a PDF file, the system converts the text of each paragraph into a vector and stores these vectors in the Faiss database. When a user queries for information related to a certain paragraph of text, Faiss will quickly find the most relevant paragraph by calculating the similarity between the query text vector and the vectors of each paragraph in the database.

[0060] Index technology and optimization: To improve query efficiency, Faiss and PostgreSQL both use efficient index technology.

[0061] PostgreSQL index: PostgreSQL supports multiple indexing methods, such as B-tree index, hash index, etc. For structured data stored in PostgreSQL, the system will create appropriate indexes based on query requirements. For example, if users often query a column (such as "student name"), the system can create an index for the column to speed up the query.

[0062] Faiss index: Faiss accelerates vector queries by building efficient vector indexes such as HNSW and IVF. The HNSW (Hierarchical Navigable Small World) index provides fast approximate nearest neighbor search, while the IVF (Inverted File) index optimizes query speed by dividing vectors into multiple clusters.

[0063] Data update and management: The system supports the management of data in the database, including adding, deleting, and modifying data stored in PostgreSQL and Faiss.

[0064] Data upload and update: When a user uploads a new file, the system automatically parses the file and inserts the new data entry into the PostgreSQL database. At the same time, the system vectorizes the newly added text data and stores the new vectors in the Faiss database.

[0065] Data deletion and cleaning: Users can manage uploaded files, such as deleting unnecessary files or updating the content of existing files. The system will update the database accordingly based on the user's operations to ensure data consistency and accuracy.

[0066] Through the above data parsing and extraction process, the present invention can extract structured and unstructured data from files of different formats, and perform standardization, cleaning, vectorization and other processing on them. Structured data is stored in PostgreSQL, and unstructured data is stored in the Faiss database through vectorization, thereby providing efficient data support for subsequent fast queries. These steps ensure the accuracy and completeness of the data and provide a solid data foundation for the RAG-based query method.

[0067] Step 4: Query processing and execution; In the RAG-based multi-format data fast query method of the present invention, query processing and execution are key links in the entire query process, aiming to convert the user's query intention into efficient retrieval and accurate response to multi-format data. Query processing and execution include extracting key information from the query content input by the user, constructing query instructions, performing query retrieval in an external knowledge base, obtaining query results, and finally returning the results to the user through natural language generation technology. This process requires not only efficient extraction of relevant information from multi-format data, but also improving query speed and accuracy through optimized query execution strategies.

[0068] In this embodiment, query processing and execution is a multi-step process, covering the entire process from query intent understanding to final result presentation. By adopting RAG (retrieval enhanced generation) technology, the present invention can combine structured and unstructured data for efficient and accurate query processing. Users can initiate queries through text or voice commands, and the system generates appropriate query commands based on the input content, and uses an external knowledge base for rapid retrieval of multi-format data.

[0069] In this embodiment, the query processing and execution process is described in detail as follows: User query input: Users initiate query requests to the system through text input boxes or voice input. Whether through text or voice input, the system can understand and convert the user's query intent into an internally processable instruction format.

[0070] Text Input: The user directly enters the query content in the dialogue window, and the system parses the user's query through natural language processing (NLP) technology and extracts key information. For example, the user may enter "find Excel files related to sales reports", and the system will extract the two keywords "sales report" and "Excel file".

[0071] Voice Input: If the user enters a query via voice input, the system first converts the voice content into text using speech recognition technology (such as Google Speech-to-Text), then performs text parsing and extracts keywords for subsequent processing.

[0072] Query command generation: The system automatically generates query instructions based on the parsed query content and user needs. These query instructions can be SQL queries or vector similarity search requests for external knowledge bases (such as PostgreSQL databases or Faiss vector databases).

[0073] Structured data query: For queries involving structured data (such as Excel tables or table data in Word), the system will initiate a query request to the PostgreSQL database by generating SQL statements. For example, when a user queries "find all sales data in sales report", the system will generate an SQL query instruction based on the user input to retrieve records in the database that contain fields related to "sales report".

[0074] Unstructured data query: For unstructured data (such as the text content in PDF or Word documents), the system will generate vector query instructions based on the user's query content and use the Faiss vector database for similarity search. The user's query content will be converted into a vector and then compared with the text vectors stored in the database to find the most relevant files or paragraphs.

[0075] Query execution and data retrieval: After the query instruction is generated, the system will execute the query operation and initiate a query to the PostgreSQL database or the Faiss vector database according to different data types and storage methods.

[0076] Structured data retrieval: For structured data, the system initiates an SQL query request to the PostgreSQL database. The database retrieves data records that meet the conditions according to the query instructions and returns the results to the system. The system then performs necessary post-processing on the query results, such as sorting, filtering, or paging, to ensure that the most relevant data is returned to the user.

[0077] Unstructured Data Retrieval: For unstructured data, the system converts the query content entered by the user into a vector and initiates a similarity search request to the Faiss database. Faiss retrieves the most relevant documents or paragraphs based on the vector similarity and returns the corresponding search results. The system further screens the returned results to improve the accuracy of the query.

[0078] Results presentation and feedback: After the query results are returned, the system will format the data according to the query type and user needs, and present the query results to the user through natural language generation (NLG) technology. The system returns the query results to the user in an easy-to-understand form through a human-computer dialogue window.

[0079] Structured data results presentation: For structured data, the system will display the query results in a table or list format, providing relevant values, fields, or records. For example, when querying data in a sales report, the system will display all relevant sales records in a table format, including fields such as sales amount and sales date.

[0080] Unstructured data results are presented as follows: For unstructured data, the system presents query results to users in the form of text paragraphs. For example, if a user queries for information in a report, the system will return paragraphs related to the query and highlight the key content.

[0081] Optimization and caching mechanism: To improve the efficiency of query processing, the system also implements a query result cache mechanism. For common queries, the system caches the query results in memory to avoid repeated queries of the same data. Whenever a user initiates the same or similar query, the system directly obtains the results from the cache without having to perform heavy database query operations again. This effectively reduces query latency and improves the system's response speed.

[0082] User feedback and query optimization: The system will also optimize the query process based on user feedback on the query results. Users can evaluate the query results, and the system will adjust the query strategy and optimize the query model based on user feedback to continuously improve the accuracy of the query and user experience.

[0083] Through the above query processing and execution process, the present invention can efficiently and accurately retrieve relevant information from structured and unstructured data according to the user's query intention through a variety of technical means (such as SQL query, vector similarity search, natural language generation, etc.), ensuring the accuracy and rapid response of query results. This process not only improves the efficiency of query, but also enhances the convenience and intelligence level of user interaction.

[0084] Step 5: Query optimization and sorting; In the RAG-based multi-format data fast query method of the present invention, query optimization and sorting are key steps to improve query accuracy and response speed. After the query processing is completed, the optimization and sorting technology ensures that the final presented results are not only relevant, but also have the best ranking order so that users can quickly find the most matching answers. The goal of this process is to intelligently sort the query results according to the user's query intent, the relevance of the data, and other optimization strategies, thereby improving the efficiency and accuracy of the query.

[0085] In this embodiment, query optimization and sorting combines traditional sorting algorithms with modern AI intelligent algorithms, and uses the characteristics of structured data and unstructured data to dynamically optimize query results. Through refined sorting strategies, it can ensure that the results of user queries meet expectations at multiple levels and improve the overall performance of the system in processing queries.

[0086] In this embodiment, the query optimization and sorting process is specifically described as follows: Initial screening of query results: During the query execution phase, the system retrieves eligible data from the external knowledge base based on the generated query instructions. At this point, the retrieved results are raw, unsorted data sets. The first step in query optimization and sorting is to screen these initial screening results, remove redundant or irrelevant results, and retain the most relevant items. The screening process is based on a variety of criteria, such as the relevance of the results, timestamps, and data update frequency.

[0087] In this embodiment, by setting predefined screening conditions, the system first filters structured data and unstructured data separately. For example, for structured data (such as records in Excel tables), the system will perform preliminary screening based on field values ​​to exclude records that do not meet the query conditions; for unstructured data (such as text in PDF or Word documents), the system will screen the content through a text similarity algorithm to remove irrelevant or inaccurate paragraphs.

[0088] Relevance ranking: After the initial screening, the system will rank the remaining query results by relevance. This ranking is based on the match between the query intent and the data content, taking into account the user's query keywords, contextual information, and historical query records. By leveraging natural language processing (NLP) technology, the system is able to evaluate the relevance between the query content and the information stored in the database.

[0089] In this embodiment, the relevance ranking algorithm is based on traditional information retrieval models such as TF-IDF (term frequency-inverse document frequency) and BM25 (BestMatching 25), combined with modern deep learning algorithms for optimization. The system first calculates the similarity score between each query result and the user's query, and then sorts the query results according to the score to ensure that the most relevant content is ranked first.

[0090] For the sorting of structured data, the system also combines the database index structure and field weights, using different sorting strategies for different data types. For example, for sales data queries, the system can sort the query results in descending order based on the weight of the sales field.

[0091] Personalized sorting based on user behavior: To further improve the relevance of query results, the system will also perform personalized sorting based on the user's historical behavior and preferences. After the user makes multiple queries, the system will record their query history and analyze the results selected by the user during each query. By analyzing the content selected by the user, the system can gradually learn the user's interests and needs, and make personalized recommendations in future queries.

[0092] In this embodiment, personalized sorting is achieved through user preference models and recommendation algorithms. The system will train the model based on the user's click behavior on the query results, viewing time and other data, and gradually optimize the query sorting rules. In this process, the deep learning model will predict the most likely results of interest based on the user's interests and present them to the user first.

[0093] Multi-dimensional sorting strategy: In the query optimization and sorting process, in addition to relevance and personalized sorting, it is also necessary to comprehensively consider multi-dimensional factors of the query results, including but not limited to the timestamp, authority, update frequency, source reliability, etc. of the results.

[0094] In this embodiment, the system uses a multi-dimensional sorting strategy to comprehensively rank the query results. For structured data, the system weights the results according to the data update frequency and timestamp; for unstructured data, the system considers the credibility of the content and the authority of the source, and gives priority to displaying the latest data or documents from trusted sources.

[0095] User feedback and dynamic optimization: To further optimize the ranking of query results, the system will also adjust the ranking algorithm in real time based on user feedback. After the user views or selects a query result, the system will adjust the model weight or recalculate the ranking score based on the user's selection record. This mechanism ensures that query ranking can be dynamically optimized over time and as user needs change.

[0096] In this embodiment, user feedback is collected through query logs, click data and evaluation mechanisms. Whenever a user gives feedback on a query result, the system automatically adjusts the weights of relevance ranking and personalized ranking, thereby improving the accuracy of subsequent queries and user satisfaction.

[0097] Caching and performance optimization: During the query optimization process, the system will also use caching strategies to speed up the presentation of query results. For frequently queried content or highly relevant data, the system will cache it in memory to avoid repeated calculations and query operations. This effectively reduces the burden of database access and improves query response speed.

[0098] In this embodiment, the cache mechanism is implemented through an in-memory database (such as Redis). When the system recognizes that a query result is frequently requested in a short period of time, it will store the result in the cache. When the user initiates the same or similar query again, the system will directly return the result in the cache without having to re-query and sort.

[0099] Through the above-mentioned multi-dimensional query optimization and sorting strategy, the present invention can significantly improve the relevance and accuracy of query results, thereby enhancing the user's query experience and ensuring that query processing efficiency and accuracy can be effectively guaranteed in a multi-format data environment.

[0100] Step 6: Knowledge base management and expansion; In the RAG-based multi-format data fast query method of the present invention, knowledge base management and expansion are key parts to ensure that the system can continue to grow, maintain efficient operation and adapt to changes in user needs. Knowledge base management not only involves the organization and storage of data, but also includes how to efficiently expand new knowledge items and how to maintain and update existing knowledge according to actual needs. Through the effective management and expansion of the knowledge base, the accuracy and comprehensiveness of the data in the query process can be ensured, thereby improving the effect and precision of the query.

[0101] In this embodiment, knowledge base management and expansion supports dynamic update and expansion of multi-format data through flexible knowledge input, storage and maintenance mechanisms. The system introduces efficient storage structure, automated parsing and management processes, so that the knowledge base can add, update and delete data at any time according to user needs, and can support the expansion of new data types and file formats.

[0102] In this embodiment, the specific implementation method of knowledge base management and expansion is as follows: Data structure design of the knowledge base: In this embodiment, the system adopts an architecture that combines the PostgreSQL database and the Faiss vector database to support the storage and query of structured and unstructured data. PostgreSQL is used to store structured knowledge items, user information, query history, etc., while Faiss is used to store vectorized representations of unstructured data to support query operations based on similarity retrieval.

[0103] In this embodiment, the storage structure of the knowledge base is as follows: User table: stores user information (such as username, password hash, email, permissions, etc.).

[0104] Knowledge table: stores structured knowledge entries of individuals or teams, including metadata, keywords, topics, and other information of various documents.

[0105] Vector table: stores vector representation of documents or data, mainly used for similarity retrieval of unstructured data (such as PDF, Word, etc.).

[0106] Automated parsing and storage of knowledge items: In this embodiment, the expansion of the knowledge base is achieved through an automated parsing tool, which supports uploading and parsing of multiple file formats (such as PDF, Word, Excel, etc.), and stores the parsing results in a structured or vectorized form in the database.

[0107] The specific steps are as follows: When users upload files through the system's front-end interface, the system first verifies the file format to ensure that the file meets the upload requirements (such as PDF, Word, etc.).

[0108] Then, the system calls the backend parsing tool, using the PyMuPDF library to process PDF files and the python-docx library to process Word files to extract the text content in the file.

[0109] For unstructured text data, the system uses natural language processing (NLP) technology for preprocessing, extracts keywords, abstracts, topics and other information, and generates vector representations based on the content. The vector representations are then stored in the Faiss database.

[0110] For structured data (such as Excel tables), the system directly stores the table content and its metadata (such as field name, field type, etc.) in the PostgreSQL database.

[0111] Management and maintenance of knowledge base by users: In this embodiment, management and maintenance of knowledge base is realized through user authority mechanism. The system provides different management authorities for different users to ensure that users can upload, modify, delete and other operations of knowledge according to their needs.

[0112] Upload operation: Users can select files to upload through the front-end interface. The system will automatically parse the file content and add it to the knowledge base as a new knowledge entry.

[0113] Modification operation: Users can edit or modify existing knowledge items. For example, users can update the content of a knowledge item or change its metadata.

[0114] Delete operation: Users can delete knowledge items that are no longer needed or invalid data. The delete operation will also update the corresponding records in the database.

[0115] In this embodiment, all operations are subject to permission control. Only users with management permissions can perform modification and deletion operations, and ordinary users can only upload or view knowledge items.

[0116] Knowledge base expansion and update: As the amount of data increases, the knowledge base needs to be expanded and updated regularly. In order to ensure the flexibility and scalability of the system, the system supports docking with other knowledge bases or data sources through API interfaces to achieve cross-system data import and expansion of knowledge items.

[0117] Automatic expansion: When external knowledge sources provide new data, the system can automatically import new knowledge entries through predefined rules or API interfaces. For example, by integrating open data sources, the system can automatically crawl the latest industry knowledge or technical articles from the Internet and add them to the knowledge base.

[0118] Manual update: Users can also manually update the information in the knowledge base regularly. For example, users can regularly upload new technical documents or product manuals, and the system will automatically parse and integrate them into the knowledge base.

[0119] Query and optimization of knowledge base: In this embodiment, query of knowledge base not only relies on keyword matching, but also combines vector retrieval and intelligent algorithms to improve query effect. The system provides the most relevant query results by matching documents and entries in the knowledge base with user queries.

[0120] Vector retrieval: When a user initiates a query, the system first extracts the text content of the user's query and generates a corresponding vector representation. Then, the system performs a similarity search in the Faiss vector database, finds the knowledge items most similar to the user's query, and returns them to the user.

[0121] Hybrid search: For queries involving structured data, the system will also execute SQL queries in the PostgreSQL database and perform comprehensive sorting in combination with vector search results. This hybrid search method can effectively combine structured and unstructured data to improve query accuracy.

[0122] Structured query instruction SQL formula:

[0123] in: SQL Query express SQL Query command; table Is the name of the data table; condition It is the query condition.

[0124] Unstructured query vector calculation formula:

[0125] is the query vector, is the i-th vector in the database; is the cosine similarity between the query vector and the database vector; Return and query vectors The vector with the highest similarity.

[0126] This formula describes how to perform similarity retrieval of unstructured data based on Faiss.

[0127] Permission control and audit of knowledge base: In this embodiment, management operations of the knowledge base (such as uploading, deleting, and modifying) are subject to strict permission control. The system will assign different permissions based on user roles to ensure that only appropriate users can perform specific operations. In order to prevent data leakage or misoperation, the system will also audit all key operations and generate log records.

[0128] Role management: The system provides multiple user roles, such as administrator, common user, auditor, etc. Each role has different permissions. Administrators can fully manage the knowledge base, while common users can only view or upload knowledge items.

[0129] Operation log: Every time a user operates the knowledge base, the system will generate a detailed operation log, including the time of the operation, the content of the operation, the user of the operation, etc. The administrator can use the log to audit the use of the system and discover abnormal operations in a timely manner.

[0130] Through the above knowledge base management and expansion mechanism, this embodiment ensures that the knowledge base can be flexibly expanded, maintains the latest and accuracy of the content, supports efficient query and management operations, and at the same time ensures the security and integrity of the data.

[0131] Step 7: Security and authority management; In the multi-format data fast query method based on RAG, security and permission management are the core links to ensure system data protection, user privacy security, and compliance of various operations with authorization regulations. With the increase in data volume and the complexity of system functions, how to effectively manage user operations, ensure that only legally authorized users can access, modify or delete data, and prevent unauthorized access, has become a key element of system design. Effective permission management can not only improve the security of the system, but also guarantee the personalized needs of users, and improve the overall user experience through fine-grained permission control.

[0132] In this embodiment, security and permission management ensures the security and compliance of the system by combining user authentication, role authorization, operation auditing and other mechanisms. At the same time, the system can flexibly configure permissions according to different business needs to ensure that different users can perform different levels of operations, such as data query, upload, modification, deletion, etc.

[0133] In this embodiment, the specific implementation of security and authority management is as follows: User authentication and identity verification: In this embodiment, the system uses a username and password combined with a secondary verification (such as an email address, a mobile phone number verification code, or two-factor authentication) to verify the user's identity. When a user operates the system, he or she must first pass the login verification. The system will perform identity verification based on the username and password to ensure the authenticity of the user's identity.

[0134] Username and password: When a user logs in to the system, he or she enters the username and password through the front-end interface, and the system compares the password with the password stored in the database. To ensure security, the password is encrypted and stored using a hash algorithm.

[0135] Secondary verification: In operations that require higher security (such as modifying or deleting important data), the system supports secondary verification, and users need to confirm through a verification code received in their email or mobile phone after logging in. In addition, two-factor authentication (such as generating a one-time password through an authentication app) can also be used as an optional security measure to further enhance the security of the system.

[0136] Role and permission management: In this embodiment, the system adopts a role-based permission management (RBAC) model. Users are assigned to different roles, and the system defines the operation permissions of different users through roles. Each role has a different set of permissions, and the system authorizes users to operate according to the role.

[0137] Role definition: The system defines several standard roles, such as ordinary users, administrators, super administrators, etc. Each role has different permissions. Ordinary users can only perform data query operations, while administrators and super administrators can perform operations such as uploading, modifying, and deleting data.

[0138] Permission management: Through the mapping of roles and permissions, the system can accurately control what operations each user can perform. Specifically, administrators can manage user permissions, upload new data entries, and perform complex queries; while ordinary users can only perform data queries and browse existing knowledge base content, and cannot perform operations such as modification or deletion.

[0139] Fine-grained permission control: In this embodiment, in order to further ensure the security and flexibility of data, the system supports fine-grained permission control. By combining database field-level permission control and operation audit logs, the system can ensure that users' access to data is limited to their permission scope.

[0140] Field-level permission control: For tables containing sensitive data (such as user tables, financial data, etc.), the system can set field-level permissions to ensure that different roles can only access specific fields. For example, ordinary users can query product information, but cannot access the details in financial statements.

[0141] Operation audit: The system will record detailed operation logs for all operations involving data changes (such as data upload, modification, and deletion). The log content includes information such as the operating user, operation time, and operation type. These operation logs can be used by system administrators for subsequent review and audit to ensure that the system operation complies with safety regulations.

[0142] Data encryption and protection: In this embodiment, all stored sensitive data (such as user passwords, emails, etc.) are protected by encryption technology. When the data is stored, the sensitive data is encrypted using a standard encryption algorithm (such as AES-256) to ensure that even if the database is leaked, the sensitive information cannot be read.

[0143] Data transmission encryption: During the communication process between the user and the system, all data transmission uses the TLS encryption protocol to prevent data from being eavesdropped or tampered with during transmission.

[0144] File upload encryption: For uploaded file data, the system will encrypt it when uploading to ensure the security of the file during storage. In addition, the system can configure file encryption options to ensure that only authorized users can view or download encrypted files.

[0145] Dynamic management of permissions: In this embodiment, the system supports dynamic adjustment of user permissions and roles. When the user's needs or position change, the system administrator can adjust the user's role and permissions in real time. This dynamic management mechanism can not only improve the flexibility of the system, but also ensure that the user's operations always meet the latest security requirements in different business scenarios.

[0146] Role upgrade and downgrade: For example, when an ordinary user is promoted to an administrator, the system administrator can upgrade his role permissions to administrator through the background interface and grant him more operation permissions.

[0147] Revocation and suspension of permissions: When a user violates system regulations or leaves the company, the administrator can immediately revoke the user's permissions to ensure that the security of sensitive data is not threatened. The system will update the database in real time when user permissions change to avoid permission leaks.

[0148] Multi-factor authentication and advanced security strategies: In this embodiment, the system supports more advanced security strategies, such as security verification based on IP address, device fingerprint, etc. For high-risk operations (such as large fund transfers, data leaks, etc.), the system can trigger multi-factor authentication and require users to provide additional verification information.

[0149] IP address restriction: The system can restrict the user's permissions based on the IP address at the time of login. For example, only users can log in to the system from the company's intranet to prevent external IP addresses from accessing sensitive data.

[0150] Device fingerprint verification: When a user logs in, the system can record the fingerprint information of the user's commonly used device (such as device ID, operating system type, etc.) and use this information to determine whether it is a legitimate device. If the user logs in from an unfamiliar device, the system will require the user to perform a more stringent verification.

[0151] Permission and security report: In this embodiment, the system supports permission and security report functions, and the administrator can regularly check the security status of the system and analyze whether the permission configuration is reasonable and whether the operation complies with security regulations.

[0152] Regular audit reports: Administrators can regularly generate system permission configuration reports to review whether the allocation of user roles is reasonable and whether there are any excessive or insufficient permissions.

[0153] Security incident report: When the system detects abnormal operations (such as unauthorized operations, abnormal logins, etc.), the system will generate a security incident report and remind the administrator to handle it in a timely manner. Embodiment 1:

[0154] A company needs to process a large amount of documents and data, including PDF, Word and Excel files. Employees often need to quickly find relevant information at work, such as policy documents, technical manuals, customer records, etc. Existing search methods cannot effectively process multi-format data, and the query speed is slow and the accuracy is low.

[0155] Implementation process: File upload and format verification: An employee uploads a PDF document containing a technical manual through the company's internal document management system.

[0156] The system first performs format verification on the uploaded PDF file to ensure that the file complies with the PDF format specification. If the verification is successful, it enters the data parsing step.

[0157] Data parsing and extraction: The system uses the PyMuPDF library to parse PDF documents and extract text information and table data.

[0158] Through natural language processing technology, the system extracts technical terms, operating steps, and equipment maintenance information from the document and stores it as structured data.

[0159] Data storage and indexing: The parsed text content is stored in the Faiss database and converted into high-dimensional vectors for storage, which facilitates fast similarity retrieval.

[0160] The table data in the document is stored in a PostgreSQL database for structured storage to facilitate SQL-based queries.

[0161] Query request and execution: The employee uses voice input to query on the query interface: "Please query the latest equipment maintenance manual."

[0162] After the speech recognition module converts speech into text, a query instruction is generated. The system automatically compares the query request with the document content in the database and finds relevant documents through similarity retrieval of the Faiss database.

[0163] The query results return relevant technical manuals, operating instructions and maintenance records.

[0164] Query optimization and sorting: The system performs weighted sorting of query results based on historical query data and employee access behavior to improve the accuracy and relevance of query results.

[0165] Employees receive the most relevant technical documentation and can quickly perform follow-up actions.

[0166] result: Through this method, employees can quickly find relevant documents, improve work efficiency, and also improve the accuracy of data query. Compared with traditional search methods, the system's query speed and accuracy have been significantly improved. Embodiment 2:

[0167] A medical research institution has a large amount of medical literature, case data, and clinical trial reports in multiple formats such as PDF, Word, and Excel. Researchers need to quickly search and analyze relevant literature to support their research work. Traditional query methods do not support multi-format data retrieval and are inefficient. Implementation process: File upload and format verification: The researchers uploaded an Excel spreadsheet containing clinical trial data and a Word document containing a medical research report.

[0168] The system first performs format verification on the two files to ensure that they meet the Excel and Word format requirements. If the verification passes, the data parsing phase begins.

[0169] Data parsing and extraction: For Excel table data, the system uses the openpyxl library to extract clinical trial data in the table, including trial time, participant information, treatment methods, etc., and converts it into structured data and stores it in the PostgreSQL database.

[0170] For Word documents, the system uses the python-docx library to extract text information, parse the medical terms, treatment methods, experimental results, etc. in the text, and convert the text content into high-dimensional vectors and store them in the Faiss database.

[0171] Data storage and indexing: The parsed Word document content is converted into a high-dimensional vector, and the text vector is generated through the BERT model and stored in the Faiss database.

[0172] The structured data in Excel is stored in the PostgreSQL database to ensure data integrity and facilitate subsequent SQL queries.

[0173] Query request and execution: The researcher entered the query via text: "Find all clinical trial data related to cancer treatment."

[0174] The system automatically generates query instructions and, based on the query request entered by the user, first performs a text similarity search through Faiss to find relevant medical literature and research reports.

[0175] At the same time, the system queries relevant clinical trial data in PostgreSQL according to SQL instructions and returns the most relevant trial data.

[0176] Query optimization and sorting: The system weights and sorts query results based on query history and user behavior to ensure that the most relevant research literature and clinical trial data are displayed to researchers first.

[0177] The query results include not only relevant literature, but also relevant experimental data extracted from Excel tables, providing researchers with comprehensive reference materials.

[0178] This method helps researchers quickly find relevant medical literature and clinical data to support their scientific research. Compared with traditional query methods, the system can process multi-format data at the same time, improve query efficiency, and provide accurate query results, significantly improving researchers' work efficiency and scientific research results.

[0179] These two examples demonstrate the application of the RAG-based multi-format data fast query method in different fields, namely enterprise document management and medical data query. Through this method, users can efficiently query in multiple file formats, quickly extract relevant data, and ensure the accuracy and efficiency of the results through similarity retrieval, query optimization and other technologies. These examples fully demonstrate the versatility and practical application value of the technical solution.

[0180] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-format data fast query method based on RAG, characterized in that: The following steps are involved: Step a: The user uploads the target file through the front-end interface. The file is in any of the formats of PDF, Word or Excel. Step b, verifying the format of the uploaded file, and parsing the data after passing the verification; Step c: Select a corresponding parsing tool according to the file format, and use PyMuPDF, python-docx or openpyxl library to extract text and structured data from the file; Step d: Store the parsed data in a PostgreSQL database and a Faiss database respectively, where PostgreSQL is used to store structured data and Faiss is used to store high-dimensional vectors of unstructured text data; Step e: Generate a query instruction according to the user's query request and execute the query. The query result returns relevant documents or data through vector similarity retrieval.

2. The RAG-based multi-format data fast query method according to claim 1, characterized in that: The Faiss database stores high-dimensional text vectors generated by a natural language processing model, and the text vectors are generated by the BERT model.

3. The RAG-based multi-format data fast query method according to claim 1, characterized in that: The query instructions include structured query instructions and unstructured query instructions. The structured query instructions are executed in a PostgreSQL database through SQL statements, and the unstructured query instructions perform similarity retrieval through a Faiss database.

4. The RAG-based multi-format data fast query method according to claim 1, characterized in that: The similarity retrieval is based on the vector index in the Faiss database, and the vector similarity calculation formula is as follows: ； Specifically, A and B are two vectors, where A represents the query vector, B Represents a data vector in the Faiss database; is a vector A and B , and calculate the inner product relationship between them; For vector A and B The mold length.

5. The RAG-based multi-format data fast query method according to claim 1, characterized in that: The data analysis also includes processing the table data in the uploaded file, converting the table data into a structured data format and storing it in a PostgreSQL database.

6. The RAG-based multi-format data fast query method according to claim 1, characterized in that: The query instruction is automatically generated by the natural language processing module according to the user's query request, the query request is transmitted to the system through text or voice input, and the voice input is converted into text input by the voice recognition module.

7. The RAG-based multi-format data fast query method according to claim 1, characterized in that: The data stored in the Faiss database includes vectorized representation of document content, and is combined with structured data in an external knowledge base to perform multi-format data retrieval.

8. The RAG-based multi-format data fast query method according to claim 1, characterized in that: The accuracy of the query results is further improved by a query optimization module, which performs weighted sorting of the search results based on historical query data and query type.

Images