A lightweight document storage and vector retrieval system based on RustFS

Abstract

The application provides a lightweight document storage and vector retrieval system based on RustFS, and relates to the technical field of distributed storage and vector retrieval. The system adopts an integrated architecture without external vector database, external relational database and large model dependence, and realizes document storage, vectorization processing and semantic retrieval functions. The system includes eight modules such as a storage layer and an access layer, the storage layer is divided into three types of storage areas for persisting corresponding data, and the access layer receives various requests; the modules work cooperatively, the lightweight object vectorization module generates three types of vectors in parallel, the vector database engine retrieves through an independent index channel, and the results are fused and sorted through inverse ranking and output. The application solves the problems of traditional system component redundancy and complex deployment, is lightweight and easy to deploy, has low resource occupation, is suitable for small and medium-sized scenarios, can be used as an AI knowledge base and retrieval enhancement generation system bottom support, and has high practicability.

Description

Technical Field

[0001] This invention relates to the field of distributed storage and vector retrieval technology, specifically to a lightweight document storage and vector retrieval system based on RustFS. Background Technology

[0002] With the accelerated pace of enterprise digital transformation and the application of AI technology, the volume of unstructured documents (such as PDFs, Word documents, and scanned copies) has exploded. Small and medium-sized enterprises (SMEs) and small teams face a significant need for storing, managing, and retrieving semantic knowledge from a large amount of documents. Compared to large enterprises, SMEs generally have limited hardware resources, a shortage of professional technical personnel, and lower budgets, placing extremely high demands on document management systems that are lightweight, easy to deploy, low-cost, and easy to maintain.

[0003] Traditional intelligent document retrieval systems typically employ a separate deployment model of "independent object storage + external vector database + external relational database" to achieve semantic retrieval functionality. This architecture has the following problems:

[0004] 1. Redundant and complex components: Multiple independent components such as object storage, vector databases, and relational databases need to be deployed, resulting in cumbersome deployment processes and high operation and maintenance costs. 2. Excessive resource consumption: Each component independently occupies CPU and memory resources, coupled with middleware communication overhead, making it unsuitable for low-configuration environments. 3. Inefficient data interaction: Communication between different components via network interfaces results in high data transmission latency, affecting retrieval and storage efficiency. 4. High deployment and operation and maintenance threshold: Component integration and configuration require professional personnel, and subsequent operation and maintenance are complex, making it difficult for SMEs and edge computing scenarios to implement quickly. 5. Insufficient ecosystem adaptation: Existing lightweight storage systems often lack standardized APIs and the ability to interface with large model ecosystems, making them unsuitable as the underlying support for AI knowledge bases and Retrieval Augmentation (RAG) systems, requiring the development of additional adaptation layers.

[0005] In existing technologies, some solutions attempt to achieve component integration through intermediate layer adaptation, but there are still problems such as high intermediate layer overhead and poor compatibility, and the need for lightweight integration with the AI ​​ecosystem has not been resolved. Summary of the Invention

[0006] The purpose of this invention is to provide a lightweight document storage and vector retrieval system based on RustFS, which can solve the technical problems of existing document storage and retrieval systems that rely on external vector databases, have high resource consumption, complex deployment, and are not suitable for the use scenarios of small and medium-sized users. It realizes the integrated and lightweight deployment of document storage and vector intelligent retrieval, reducing the usage cost and technical threshold for small and medium-sized users. At the same time, it overcomes the defects of existing technologies, realizes native integration of storage and retrieval, and has a simple structure that is easy to deploy to meet the needs of small and medium-sized users.

[0007] This invention provides a lightweight document storage and vector retrieval system based on RustFS. It is built on RustFS distributed object storage developed in pure Rust language and adopts an integrated architecture without external vector databases, external relational databases, and large model dependencies to realize document storage, vectorization processing and semantic retrieval functions.

[0008] The system includes: a storage layer, an access layer, a business control module, a document management module, a vector database engine, a lightweight object vectorization module, a document preprocessing module, and a vectorized task scheduling module;

[0009] The storage layer is built on the RustFS underlying layer and is centrally divided into a raw document storage area, a vector data storage area, and an index file storage area, which are used to persistently store raw documents, vector data, and various index files, respectively.

[0010] The access layer serves as the unified external interaction entry point for the system, and is used to uniformly receive document upload requests or semantic search requests submitted by external systems. The document upload request carries the tenant ID, set ID, and the document to be uploaded, while the semantic search request carries the tenant ID, set ID, and search text.

[0011] The business control module is used to perform identity and permission verification on the document upload request, generate a unique document ID and a three-level storage path of tenant-collection-shard for the document to be uploaded; or to perform identity and permission verification on the semantic retrieval request and route the semantic retrieval request.

[0012] The document management module is used to perform format verification, compression and encryption on the document to be uploaded, and then write it in segments to the original document storage area of ​​the storage layer, and simultaneously record the document metadata.

[0013] The document preprocessing module is used to read the target document from the original document storage area of ​​the storage layer, and sequentially perform format parsing, text cleaning and adaptive block processing to generate standard text blocks;

[0014] The lightweight object vectorization module integrates three lightweight vectorization algorithms in parallel: Word2Vec, TF-IDF, and feature hashing. It is used to simultaneously perform three-way vectorization processing on the standard text blocks generated by the document preprocessing module and the retrieved text, and independently generate three types of vectors: semantic vector, statistical feature vector, and hash feature vector.

[0015] The vector database engine has a built-in ANN retrieval layer based on the HNSW algorithm, which is used to independently build indexes for the three types of vectors and store the indexes in the index file storage area of ​​the storage layer. During retrieval, the three feature vectors are retrieved in parallel through their respective independent index channels, and multiple candidate results are recalled. Finally, after comprehensive scoring and sorting by the inverse ranking fusion sorting strategy, the results are associated with the corresponding document metadata and encapsulated for external return.

[0016] Compared to existing technologies, the lightweight document storage and vector retrieval system based on RustFS provided by this invention is built on the RustFS distributed object storage system developed in pure Rust language. It adopts an integrated architecture without external vector databases, external relational databases, or large model dependencies, realizing document storage, vectorization processing, and semantic retrieval functions. The system includes eight modules, such as a storage layer and an access layer. The storage layer is divided into three types of storage areas for persistent data; the access layer receives various requests; and the modules work collaboratively. The lightweight object vectorization module generates three types of vectors in parallel, and the vector database engine retrieves them through independent index channels and outputs the results after inverse ranking fusion sorting. This invention solves the problems of component redundancy and complex deployment in traditional systems. It is lightweight, easy to deploy, and has low resource consumption, making it suitable for small to medium-scale scenarios. It can serve as the underlying support for AI knowledge bases and retrieval enhancement generation systems, demonstrating strong practicality. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0018] Figure 1 The diagram illustrates the structure of a lightweight document storage and vector retrieval system based on RustFS provided in an embodiment of the present invention.

[0019] Figure 2 A document storage flowchart provided by an embodiment of the present invention is shown;

[0020] Figure 3 A flowchart illustrating the vectorization processing provided in an embodiment of the present invention is shown;

[0021] Figure 4 A flowchart illustrating the semantic retrieval process provided in an embodiment of the present invention is shown. Detailed Implementation

[0022] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0023] It should be noted that, unless otherwise stated, the technical or scientific terms used in this invention should have the ordinary meaning as understood by one of ordinary skill in the art.

[0024] Furthermore, the terms "first" and "second," etc., are used to distinguish different objects, not to describe a specific order. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to those processes, methods, products, or apparatuses.

[0025] Please refer to Figure 1 , Figure 1 This is a schematic diagram of a lightweight document storage and vector retrieval system based on RustFS, provided by an embodiment of the present invention. The system is built upon the RustFS distributed object storage system developed in pure Rust language. It adopts an integrated architecture without external vector databases, external relational databases, or large model dependencies, realizing document storage, vectorization processing, and semantic retrieval functions. The system includes a storage layer 10, an access layer 20, a business control module 30, a document management module 40, a vector database engine 50, a lightweight object vectorization module 60, a document preprocessing module 70, and a vectorization task scheduling module 80. These modules are connected in series via a software logic bus and integrated into the same server operating environment. The specific structure is as follows:

[0026] (1) Storage layer 10

[0027] Built on the RustFS underlying layer, it communicates directly with the document management module 40 and the vector database engine 50. It centrally divides the storage area into the original document storage area, the vector data storage area, and the index file storage area, which are used to persistently store the original documents, vector data, and various index files, respectively. It centrally supports the data storage needs of the entire system and includes four independent storage areas:

[0028] Original document storage area: Various original documents are stored using software encryption mechanisms. The number of data copies can be configured from 2 to 4, ensuring the security and reliability of the original data.

[0029] Vector data storage area: Semantic vectors, statistical feature vectors, and hash feature vectors are stored in Parquet columnar storage format. The storage path is planned according to tenant-set-time partitioning to achieve independent and isolated storage of the three types of vectors, which facilitates management and retrieval.

[0030] Index file storage area: This area is dedicated to storing HNSW index files and quantized data files. It is managed with three-level path isolation based on tenant, collection, and index type, and corresponds to independent indexes for the three types of vectors. It supports fast loading and updating of the indexes.

[0031] Metadata snapshot storage area: Generates full metadata snapshots periodically. The snapshot generation cycle is configurable and is used for rapid recovery in case of metadata failure, ensuring system data consistency.

[0032] (2) Access Layer 20

[0033] As the unified external interaction entry point for the system, it is used to uniformly receive document upload requests or semantic search requests submitted by external systems. The document upload request carries the tenant ID, set ID, and the document to be uploaded, while the semantic search request carries the tenant ID, set ID, and the search text. This layer has an S3 protocol interface submodule and a vector search API interface submodule, and implements request routing through data forwarding logic.

[0034] S3 Protocol Interface Submodule: Compatible with S3 v4 signature specification, supports access from general S3 clients, and enables document upload, download, and deletion operations;

[0035] Vector retrieval API interface submodule: Provides a RESTful style retrieval interface, supports submission of retrieval text and return of results, and can be directly connected to large model services;

[0036] Data forwarding logic: used to verify the legitimacy of requests, forward document upload requests to the document management module, and forward semantic retrieval requests to the modified vector database engine.

[0037] The vector retrieval API of this layer is used to connect with the AI ​​knowledge base and the retrieval enhancement generation system. The output format of the retrieval results is adapted to the input specifications of the large model context. The system can run independently as a storage retrieval engine, or be embedded in enterprise office systems or local large model applications.

[0038] (3) Business control module 30

[0039] As the core scheduling hub of the system, it is used to perform identity and permission verification on document upload requests, generate a unique document ID and a three-level storage path of tenant-collection-shard for the document to be uploaded; or to perform identity and permission verification on semantic search requests and route the semantic search requests; specifically, it includes three sub-modules:

[0040] Tenant permission management submodule: Based on the RBAC (role-based access control) model, it implements tenant identity authentication and permission verification, and records operation logs. The log retention period is no less than 90 days to ensure that operations are traceable.

[0041] Request distribution submodule: Based on request type and tenant information, it routes requests to the corresponding functional modules to ensure the orderly flow of business processes;

[0042] Storage path planning submodule: Generates a unique storage path based on a three-level structure of tenant ID-set ID-shard ID, achieving physical isolation of multi-tenant data and ensuring data security.

[0043] (4) Document Management Module 40

[0044] This module is used to perform format verification, compression, and encryption on the document to be uploaded, then writes it in chunks to the original document storage area of ​​the storage layer, and simultaneously records document metadata; it implements document storage and metadata management based on RustFS, specifically including four sub-modules:

[0045] Metadata Index Submodule: Uses a local B+ tree index to store document metadata, which includes core information such as document ID, name, size, and storage path, enabling fast querying and management of metadata;

[0046] Data sharding submodule: shards documents into 64MB default sizes (preset sizes) and supports redundant shard storage to improve the reliability of data storage;

[0047] Vectorization task generation submodule: used to verify documents in preset formats such as PDF, Word, TXT, EXCEL, BMP, etc. It automatically generates standardized document vectorization tasks for documents that conform to the preset format, and does not generate vectorization tasks for document types that do not currently support vectorization.

[0048] Backup storage submodule: Implements scheduled backups of documents and metadata, supports full and incremental backup modes, can be flexibly configured according to actual needs, and ensures data recoverability.

[0049] (5) Document preprocessing module 70

[0050] This module is used to read target documents from the raw document storage area of ​​the storage layer, and sequentially perform format parsing, text cleaning, and adaptive chunking to generate standard text blocks, providing well-organized text data for subsequent vectorization processing. This module works in conjunction with the document management module and the vectorization task scheduling module, and specifically includes four sub-modules:

[0051] The format parsing submodule is used to extract the text content of structured or semi-structured documents such as PDF, Word, TXT, and EXCEL, and output plain text data.

[0052] OCR recognition submodule: Supports multilingual text recognition, used to recognize text in unstructured documents such as scanned documents and BMP images, and to extract text from unstructured documents;

[0053] Text cleaning submodule: Used to remove whitespace characters, special symbols and common stop words from text, while retaining the core semantic information of the text and improving the accuracy of subsequent vectorization processing;

[0054] The text segmentation submodule adaptively segments the text according to a configurable character window (adjustable range 128-1024 characters) and overlap rate (adjustable range 0-30%), and generates a unique identifier (text block ID) for each text block to facilitate subsequent vector association and retrieval.

[0055] (6) Lightweight object vectorization module 60

[0056] This module integrates three lightweight vectorization algorithms—Word2Vec, TF-IDF, and feature hashing—in parallel. These algorithms are used to simultaneously perform three-way vectorization processing on standard text blocks generated by the document preprocessing module and the retrieved text, generating three independent types of vectors: semantic vectors, statistical feature vectors, and hash feature vectors. This module contains three parallel and independently operating algorithm processing sub-modules, which work in conjunction with the document preprocessing module 70 and the vectorization task scheduling module 80. The three algorithms perform parallel vectorization on the same text, generating vectors of corresponding dimensions and independently building their own dedicated indexes. During the retrieval phase, after the results are retrieved synchronously from multiple indexes, they are sent to the Reciprocal Ranking (RRF) module to output a comprehensive retrieval result.

[0057] Word2Vec processing submodule: Based on a lightweight pre-trained model (model size not exceeding 50MB), it completes semantic representation, generates 128-256 dimensional semantic vectors, and maintains a separate semantic class index library;

[0058] TF-IDF processing submodule: Extracts text statistical features by dynamically constructing a word frequency matrix, generates 256-512 dimensional statistical vectors, and maintains a separate statistical feature index library;

[0059] Feature hashing processing submodule: Uses MurmurHash3 hashing algorithm to perform text feature mapping, generates 128-256 dimensional hash vectors, and maintains a separate hash feature index library.

[0060] (7) Vector Database Engine 50

[0061] The built-in HNSW algorithm-based ANN retrieval layer is used to independently build indexes for the three types of vectors, and stores each type of index separately in the index file storage area of ​​the storage layer. During retrieval, the three feature vectors are retrieved in parallel through their respective independent index channels, and multiple candidate results are recalled. Finally, after comprehensive scoring and ranking by the inverse ranking fusion (RRF) sorting strategy, the results are associated with the corresponding document metadata and encapsulated for external return. This engine is based on a deeply customized and modified Qdrant, and is built into the system runtime environment without external deployment. It specifically includes three sub-modules:

[0062] Storage Transformation Submodule (Qdrant Gridstore Transformation Submodule): Strips away the native file I / O dependency of the native Qdrant engine, directly calls the RustFS storage interface through standard APIs, realizes distributed read and write of index files and vector data, and achieves native adaptation with RustFS;

[0063] The HNSW index storage submodule has a built-in HNSW index construction logic with fixed parameters (M=12, efConstruction=150), which is adapted to the lightweight vector dimension of this system. The ANN retrieval layer has three independent index channels, which correspond to the independent storage and retrieval of semantic vectors, statistical feature vectors, and hash feature vectors, respectively. During the retrieval stage, the query text is encoded and retrieved in parallel through the three index channels, and the multiple candidate results are sorted by reciprocal ranking fusion (RRF) and the final comprehensive result is output.

[0064] Hot and cold data scheduling submodule: Configure a memory cache pool with a capacity of no more than 1GB for resident hot shard data, and load cold shard data on demand through the Least Recently Used (LRU) cache eviction policy, effectively reducing system resource consumption and improving retrieval efficiency.

[0065] (8) Vectorized task scheduling module 80

[0066] As a core module connecting document preprocessing and lightweight object vectorization, it is responsible for the unified scheduling, state management, automatic retry of failures, and dead-letter queue management of vectorization tasks. It has a built-in state machine-driven task scheduling mechanism, which automatically flows vectorization tasks according to the state machine. It supports automatic retry for processing failures and automatically puts over-limit tasks into the dead-letter queue for management. After the document is uploaded, it asynchronously performs compression, encryption, fragmented storage, and local index synchronization, and adapts to text chunking. Upload requests can be responded to instantly without waiting for the full processing to be completed.

[0067] The core workflow of the above system includes document storage, vectorization processing, and semantic retrieval. These processes work together to achieve integrated document storage, preprocessing, vectorization, and retrieval, as detailed below:

[0068] like Figure 2 As shown, the document storage process is as follows:

[0069] S101. External systems submit document upload requests through the S3 protocol interface or vector retrieval API of access layer 20, carrying the tenant ID, collection ID and the document to be uploaded;

[0070] S102, the access layer 20 verifies the legitimacy of the request through data forwarding logic and forwards the document upload request to the business control module 30;

[0071] S103, the business control module 30 completes identity and permission verification through the tenant permission management submodule, generates a unique document ID and a three-level storage path of tenant-collection-shard by the storage path planning submodule, and then routes the request to the document management module 40 through the request distribution submodule;

[0072] S104, the document management module 40 performs format verification, lightweight compression and encryption on the document, and writes it to the original document storage area of ​​storage layer 10 after being fragmented by the data fragmentation submodule according to the default size; the default size can be set to 64MB;

[0073] S105, the vectorization task generation submodule of the document management module 40 automatically generates standardized vectorization tasks for legally supported document formats such as PDF, WORD, EXCEL, TXT, and BMP, stores them in storage layer 10 in standardized JSON format, and hands them over to the vectorization task scheduling module 80 for unified scheduling and circulation.

[0074] S106, the metadata indexing submodule of the document management module 40 synchronously records document metadata, and the backup storage submodule automatically completes data backup, thus completing the complete document storage process.

[0075] like Figure 3 As shown, the vectorization process is as follows:

[0076] S201, Vectorization task scheduling module 80 schedules vectorization tasks and distributes them to document preprocessing module 70;

[0077] S202, the document preprocessing module 70 reads the target document from the original document storage area of ​​the storage layer 10, and sequentially passes through the format parsing submodule (or OCR recognition submodule, for image documents), text cleaning submodule, and block processing submodule to complete format parsing, OCR recognition, text cleaning, adaptive block processing, and generate standard text blocks.

[0078] S203. The processed standard text block is transmitted in parallel to the lightweight object vectorization module 60 through the built-in message queue.

[0079] S204, the lightweight object vectorization module 60 simultaneously sends the same text into the three sub-modules Word2Vec, TF-IDF and feature hashing to complete vectorization in parallel, generating semantic vectors, statistical feature vectors and hash feature vectors respectively;

[0080] S205. The three types of vector data are written to the vector data storage area of ​​storage layer 10 for isolated storage and are simultaneously pushed to the vector database engine.

[0081] S206, the HNSW index storage submodule of the vector database engine 50 constructs independent HNSW indexes for the three types of vectors, and the index files of each type are stored in the index file storage area of ​​storage layer 10.

[0082] like Figure 4 As shown, the semantic retrieval process is as follows:

[0083] S301, external systems or AI large model services submit semantic retrieval requests through the vector retrieval API of access layer 20, carrying tenant ID, set ID and retrieval text;

[0084] S302, the access layer 20 forwards the request to the business control module 30. After the business control module 30 completes the identity and permission verification, it routes the request to the vector database engine 50.

[0085] S303, Lightweight Object Vectorization Module 60 performs the same preprocessing logic (text cleaning, block segmentation) as the imported documents on the retrieved text, and generates three feature vectors in parallel.

[0086] S304 and the vector database engine 50 perform parallel retrieval through three independent HNSW index channels. Combined with the scheduling strategy of the hot and cold data scheduling submodule, they load hot shard (resident in memory cache pool) and cold shard (LRU cache loaded on demand) indexes to recall multiple candidate results respectively.

[0087] S305. The multi-way search results are comprehensively scored and sorted using the Reciprocal Ranking Fusion (RRF) ranking strategy, and relevant text fragments are extracted by associating and matching document metadata.

[0088] S306. During the retrieval process, only lightweight metadata associated with the loaded vectors is traversed and loaded. Similarity calculation is completed by loading vector objects on demand in a streaming manner. The top K (Top-K) optimal retrieval results are maintained through a min-heap structure. Finally, the original documents are read and the retrieval results are encapsulated and returned externally. It can adapt to the input format of large model contexts. The retrieval delay of hot data and cold data is hierarchically controllable.

[0089] For ease of understanding, this application describes the above system through two specific embodiments.

[0090] Example 1: Multi-tenant Deployment for Small and Medium Enterprises

[0091] This embodiment is suitable for the document management needs of departments with fewer than 50 people. It is deployed on a low-configuration server, requires no external database, and integrates document storage with lightweight semantic retrieval, fully meeting the technical features of claims 1-10.

[0092] (1) System deployment

[0093] The deployed device is configured with a CPU i5-12400 (4 cores), 16GB of memory, and a 1TB hard drive (Linux CentOS 7+). The system deployment is completed by executing a one-click installation script. The system automatically builds a storage layer based on RustFS. After initialization, the memory usage is ≤1.5GB, and there are no external vector databases or relational database dependencies. Three tenants (R&D Department, Finance Department, and Administration Department) are set up through configuration files, RBAC access control is enabled, and the storage path is planned according to tenant ID-collection ID-shard ID to achieve multi-tenant data isolation.

[0094] (2) Document processing flow

[0095] 1. Department employees upload 50 documents (including scanned copies of PDF, Word, Excel, TXT, and BMP files) via a web page or MinIO client (S3 protocol compatible), carrying the tenant ID and collection ID;

[0096] 2. The document management module performs format verification, compresses (reduces size by 30%) and encrypts valid documents, writes them to the original document storage area of ​​the storage layer in 64MB default size segments, and generates standardized vectorization tasks for documents with supported formats.

[0097] 3. The vectorized task scheduling module distributes tasks to the document preprocessing module, which sequentially performs format parsing / OCR recognition, text cleaning (removing whitespace and stop words), and block processing (configuring a 512-character window, 10% overlap rate, and generating a unique text block ID).

[0098] 4. The text block is transmitted in parallel to the lightweight object vectorization module through the built-in message queue. At the same time, it is vectorized in parallel by three sub-modules: Word2Vec, TF-IDF, and feature hashing, generating a 256-dimensional semantic vector, a 512-dimensional statistical vector, and a 256-dimensional hash vector, respectively.

[0099] 5. The three types of vectors are independently written to the vector data storage area of ​​the storage layer and synchronized to the vector database engine. The engine builds HNSW indexes for the three types of vectors respectively (parameters M=12, efConstruction=150). The index files are stored in the index file storage area according to the path of tenant ID-set ID-index type.

[0100] (3) Search test results

[0101] 1. R&D personnel submit a search request via the web: "Product R&D Technical Specifications for Q2 2025" (including the R&D department tenant ID);

[0102] 2. The access layer forwards requests, and the business control module verifies permissions before routing them to the vector database engine;

[0103] 3. The lightweight object vectorization module generates three vectors in parallel for the retrieved text. The vector database engine retrieves data in parallel through three independent index channels. Hot data shards are loaded through a memory cache pool, while cold data shards are loaded on demand.

[0104] 4. The multiple candidate results are sorted by inverse ranking fusion (RRF), and three matching documents are returned within 180ms. The core text fragments are extracted, and the retrieval accuracy is ≥95%. There is no external data interaction during the entire retrieval process, the memory usage is stable within 4GB, there is no risk of memory overflow, and it conforms to the retrieval logic of claim 9.

[0105] (4) Operation and maintenance management characteristics

[0106] The system automatically performs a full backup and index update every day at midnight, and clears the cache; after an administrative staff member deletes an expired document, the system synchronously marks the vector data and index with tombstone markers, and the background compaction mechanism merges them asynchronously; the operating status of each module can be monitored and operation logs can be queried (retained for 90 days) through the web management terminal, without the need for professional personnel to maintain it, which meets the automated operation and maintenance characteristics of claim 8.

[0107] Example 2: Deployment of the underlying search engine of the RAG system

[0108] This embodiment adapts to the requirements of the local RAG system, using the system as the retrieval base for the 7B parameter large model, providing low-latency contextual retrieval capabilities, and adapting to the ecological integration requirements of claim 10.

[0109] (1) Deployment and Integration

[0110] The deployed device is equipped with a CPU i5-12400 (4 cores), 16GB of memory, and a 2TB hard drive (Linux CentOS 7+). It connects with the local large model service through the system vector retrieval API, enables the API standardized output mode (adapting to the large model context input format), and has an integration cycle of ≤1 day, achieving rapid docking with the RAG system.

[0111] (2) Knowledge base construction and retrieval results

[0112] 1. Upload 50,000 technical documents (including API manuals, product descriptions, and troubleshooting guides). The system will automatically complete preprocessing, parallel vectorization, and three-index construction. The full processing time is ≤25 minutes.

[0113] 2. A developer asked through the large model's interactive interface: "How do I call the system's vector retrieval API to get the Top 5 matching results?"

[0114] 3. The large model transforms the question into search text, calls the system's API to initiate a search, the system generates three vectors in parallel, and searches in parallel through three index channels. The hot data search latency is ≤50ms, and three relevant document fragments are returned.

[0115] 4. The large model generates structured answers based on the retrieved context, with an accuracy of ≥92%; in high-concurrency scenarios (100 people querying at the same time), the system memory usage is ≤1.5GB, the retrieval latency is ≤80ms, and there is no performance fluctuation, fully demonstrating the advantages of this system in terms of lightweight design and high concurrency adaptability.

[0116] The beneficial effects of the lightweight document storage and vector retrieval system based on RustFS provided by this invention are as follows:

[0117] 1. Extremely lightweight, suitable for small to medium-sized scenarios.

[0118] Developed based on the RustFS lightweight file system, it eliminates all redundant functions, resulting in a small system size and extremely low resource consumption. It can run stably on ordinary office computers and low-configuration servers without requiring dedicated hardware, perfectly suited to the limited hardware resources of small and medium-sized users. Lightweight and easy to deploy, it requires no third-party dependencies or professional maintenance, reducing costs and precisely adapting to small and medium-sized data scenarios. The streaming loading mode ensures stable memory usage, and combined with memory management, stable retrieval latency meets performance targets.

[0119] 2. No external database required, zero barriers to deployment and maintenance.

[0120] By integrating the vector retrieval module into the RustFS system, a local unified vector index is built, eliminating the need to deploy external vector databases or relational databases. This enables one-click installation and deployment, and eliminates the need for professional maintenance personnel in the later stages, significantly reducing deployment costs and technical barriers. It addresses the pain points of small and medium-sized users with insufficient budgets and a lack of professional technical personnel, greatly reducing the digital office costs for small and medium-sized users.

[0121] 3. Integrated storage and retrieval for high retrieval efficiency.

[0122] Breaking away from the traditional architecture of separating storage and retrieval, this system integrates document storage, chunking, vector generation, and intelligent retrieval into a unified process. Data does not need to be transferred across databases, resulting in fast retrieval response and high semantic retrieval accuracy, far superior to traditional keyword retrieval systems. This meets the needs of small and medium-sized users for quickly finding documents. Hybrid retrieval combined with Reciprocal Ranking (RRF) sorting and streaming loading reduce IO overhead, while tagging hit information solves the problem of insufficient retrieval recall. Leveraging Rust's memory safety features, separate storage avoids performance degradation, deduplication and compression improve efficiency, and backup functionality ensures data security.

[0123] 4. High data security

[0124] All document and index data are stored locally without external data interaction, avoiding the risk of data leakage. It also features built-in document encryption and RBAC-based access control, making it suitable for the document security management needs of small to medium-sized teams. Vectors and original objects are bound to each other throughout their lifecycles, and multiple consistency mechanisms ensure data security and compliance. It is highly observable and easy to maintain. It can be flexibly expanded in terms of functionality and capacity, with unified module interfaces, support for partitioned retrieval, and convenient maintenance.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A lightweight document storage and vector retrieval system based on RustFS, characterized in that, RustFS, a distributed object storage system developed using the pure Rust language, adopts an integrated architecture without external vector databases, external relational databases, or large model dependencies, enabling document storage, vectorization processing, and semantic retrieval functions. The system includes: a storage layer, an access layer, a business control module, a document management module, a vector database engine, a lightweight object vectorization module, a document preprocessing module, and a vectorized task scheduling module; The storage layer is built on the RustFS underlying layer and is centrally divided into a raw document storage area, a vector data storage area, and an index file storage area, which are used to persistently store raw documents, vector data, and various index files, respectively. The access layer serves as the unified external interaction entry point for the system, and is used to uniformly receive document upload requests or semantic search requests submitted by external systems. The document upload request carries the tenant ID, set ID, and the document to be uploaded, while the semantic search request carries the tenant ID, set ID, and search text. The business control module is used to perform identity and permission verification on the document upload request, generate a unique document ID and a three-level storage path of tenant-collection-shard for the document to be uploaded; or to perform identity and permission verification on the semantic retrieval request and route the semantic retrieval request. The document management module is used to perform format verification, compression and encryption on the document to be uploaded, and then write it in segments to the original document storage area of ​​the storage layer, and simultaneously record the document metadata. The document preprocessing module is used to read the target document from the original document storage area of ​​the storage layer, and sequentially perform format parsing, text cleaning and adaptive block processing to generate standard text blocks; The lightweight object vectorization module integrates three lightweight vectorization algorithms in parallel: Word2Vec, TF-IDF, and feature hashing. It is used to simultaneously perform three-way vectorization processing on the standard text blocks generated by the document preprocessing module and the retrieved text, and independently generate three types of vectors: semantic vector, statistical feature vector, and hash feature vector. The vector database engine has a built-in ANN retrieval layer based on the HNSW algorithm, which is used to independently build indexes for the three types of vectors and store the various indexes separately in the index file storage area of ​​the storage layer. During retrieval, the three feature vectors are retrieved in parallel through their respective independent index channels, and multiple candidate results are recalled. Finally, after comprehensive scoring and sorting by the inverse ranking fusion sorting strategy, the results are associated with the corresponding document metadata and encapsulated for external return.

2. The system according to claim 1, characterized in that, The integrated access interface module is equipped with an S3 protocol interface submodule and a vector retrieval API interface submodule, and implements request routing through data forwarding logic; The S3 protocol interface submodule is compatible with the S3 v4 signature specification, supports access from general S3 clients, and enables document upload, download, and deletion operations. The vector retrieval API interface submodule provides a RESTful style retrieval interface, supports text submission and result return, and can be directly connected to large model services; the data forwarding logic is used to verify the legitimacy of the request, forward the document upload request to the RustFS file management module, and forward the semantic retrieval request to the modified vector database engine.

3. The system according to claim 1, characterized in that, The business control module includes: The tenant access control submodule implements tenant authentication and access control based on the RBAC model, and records operation logs. The request distribution submodule routes requests to the corresponding functional modules based on request type and tenant information; The storage path planning submodule generates a unique storage path based on a three-level structure of tenant-collection-shard, thereby achieving multi-tenant data isolation.

4. The system according to claim 1, characterized in that, The document management module includes: The metadata indexing submodule uses a local B+ tree index to store document metadata, which includes document ID, name, size, and storage path. The data sharding submodule shards documents according to a preset size and supports redundant storage of shards; The vectorization task generation submodule is used to verify the document format and generate document vectorization tasks for documents that conform to the preset format. The backup storage submodule enables scheduled backups of documents and metadata, supporting both full and incremental backup modes.

5. The system according to claim 1, characterized in that, The vector database engine is based on a customized version of Qdrant and is integrated into the system runtime environment, including: The storage modification submodule enables reading and writing of index files and vector data by calling the RustFS storage interface; The HNSW index storage submodule has built-in HNSW index construction logic with fixed parameters; the ANN retrieval layer has three independent index channels, which correspond to the independent storage and retrieval of semantic vectors, statistical feature vectors, and hash feature vectors, respectively; during the retrieval stage, the query text is encoded and retrieved in parallel through the three index channels, and the final comprehensive result is output after the multiple candidate results are merged and sorted by inverse ranking. The hot and cold data scheduling submodule is configured to keep hot data in a memory cache pool and load cold data on demand using an LRU cache eviction policy.

6. The system according to claim 1, characterized in that, The lightweight object vectorization module comprises three parallel and independently operating algorithm processing sub-modules, which work in conjunction with the document preprocessing module and the vectorization task scheduling module; wherein: The Word2Vec processing submodule completes semantic representation based on a pre-trained model, generates semantic vectors, and maintains a semantic class index library. The TF-IDF processing submodule extracts text statistical features by constructing a word frequency matrix, generates statistical vectors, and maintains a statistical feature index library. The feature hashing processing submodule uses a hash algorithm to map text features, generate hash vectors, and maintain a hash feature index library.

7. The system according to claim 1, characterized in that, The document preprocessing module works in conjunction with the document management module and the vectorized task scheduling module. The document preprocessing module includes: The format parsing submodule is used to extract the text content of structured or semi-structured documents and output plain text data. The OCR recognition submodule is used to recognize text in scanned documents and unstructured documents such as images; The text cleaning submodule is used to remove whitespace characters, special symbols, and common stop words from text; The text segmentation submodule adaptively segments the text into blocks according to a configurable character window and overlap rate, and generates a unique identifier for each text block.

8. The system according to claim 1, characterized in that, The storage layer is built on the RustFS architecture and communicates directly with the document management module and the vector database engine. The storage layer includes: The original document storage area uses software encryption to store original documents, and the number of data copies is configurable. The vector data storage area stores semantic vectors, statistical feature vectors, and hash feature vectors in columnar storage format. The storage path is planned according to tenant-set-time partitioning to achieve independent and isolated storage of the three types of vectors. The index file storage area stores HNSW index files and quantized data files, and is managed in a three-level path isolation system based on tenant, collection, and index type. Metadata snapshot storage area, which generates full metadata snapshots on a regular basis. The snapshot generation cycle is configurable and is used for metadata recovery.

9. The system according to claim 1, characterized in that, During semantic retrieval, the system performs preprocessing on semantic retrieval requests in the same way as the documents in the database. It uses three algorithms—Word2Vec, TF-IDF, and feature hashing—to generate corresponding vectors. It then uses near nearest neighbor retrieval and locality-sensitive hashing for matching. After multi-path synchronous recall, the results are output after inverse ranking fusion sorting. The retrieval process loads vector-related metadata, uses streaming on-demand loading of vector objects to complete similarity calculation, maintains the top K best retrieval results through a min-heap structure, reads the original document in association, and encapsulates the results for feedback.

10. The system according to claim 1, characterized in that, The vector retrieval API of the access layer is used to connect with the AI ​​knowledge base and the retrieval enhancement generation system. The output format of the retrieval results is adapted to the input specifications of the large model context. The system runs independently as a storage retrieval engine, or is embedded in enterprise office systems or local large model applications.

Images