Heterogeneous knowledge base unified management and asynchronous learning method based on RAG architecture

Abstract

The invention discloses a heterogeneous knowledge base unified management and asynchronous learning method based on an RAG architecture, and the method comprises the steps: receiving a knowledge document processing request, executing an asynchronous learning processing flow, and enabling the flow to achieve the transparent call of a heterogeneous RAG service provider through a unified adaptation layer; document processing is decoupled into an uploading storage stage, an asynchronous analysis stage and a vectorization indexing stage, and the last two stages are asynchronously connected with an event driver through a message queue. And meanwhile, performing association management on the original documents, the intermediate knowledge documents and the mapping identifiers in the RAG service under multiple tenants to generate traceable complete knowledge processing link metadata. According to the invention, architecture decoupling is realized, the processing stability and observability are improved, and the management and traceability of knowledge assets are enhanced.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and cloud computing technology, and in particular to a method for unified management and asynchronous learning of heterogeneous knowledge bases based on the RAG architecture. Background Technology

[0002] Currently, integrating RAG (Retrieval Enhancement Generation) capabilities into SaaS (Software as a Service) applications has become a common requirement. Existing technical solutions typically involve business systems directly calling the application programming interfaces (APIs) of specific RAG service providers (such as Dify and LangChain), synchronously uploading files and waiting for them to complete parsing and vectorization indexing. This model has significant drawbacks: First, the business logic is deeply and strongly coupled with the specific RAG provider's interface. When switching or supporting multiple RAG services simultaneously is required, extensive refactoring of the business code is necessary, resulting in poor scalability. Second, when processing large documents, such as policy documents hundreds of pages long, the synchronous upload and parsing process is prone to HTTP connection timeouts due to network latency or processing time. Furthermore, the lack of persistence of intermediate states makes it difficult to pinpoint the problem after a failure, leading to a poor user experience. Finally, native RAG services typically only focus on the vector index itself, lacking unified management of the relationships between the original files, parsing intermediate products (such as Markdown), tenant information, and the underlying vector index in a multi-tenant SaaS scenario. This makes knowledge tracing, auditing, and subsequent asset reuse difficult. Summary of the Invention

[0003] To address the issues of strong coupling between business systems and RAG services, instability in large file synchronization processing, and lack of unified and traceable knowledge asset management in existing technologies, this invention proposes a unified management and asynchronous learning method for heterogeneous knowledge bases based on the RAG architecture.

[0004] The specific technical solution is as follows: A method for unified management and asynchronous learning of heterogeneous knowledge bases based on RAG architecture, the steps of which include:

[0005] Receive knowledge document processing requests from business systems and perform asynchronous learning processing on the knowledge documents;

[0006] The asynchronous learning process includes:

[0007] Through a unified adaptation layer, the processing request is adapted and invoked to the specified heterogeneous underlying RAG service provider;

[0008] Document processing is decoupled into three stages: sequential upload and storage, asynchronous parsing, and vectorized indexing. Asynchronous parsing and vectorized indexing are asynchronously connected through message queues and event-driven mechanisms.

[0009] During the execution of this process, the original documents under multi-tenant environments, the intermediate knowledge documents generated through parsing, and their mapping identifiers in the underlying RAG service are associated and managed to generate traceable knowledge processing link metadata. This architecturally decouples the strong binding between the business system and specific RAG service providers, improving the system's scalability and flexibility. The use of asynchronous pipeline processing for large documents avoids timeout issues with synchronous interfaces, significantly improving the system's processing capacity and user experience. Furthermore, a metadata management system spanning the entire process is established, enabling full-link traceability and auditability of knowledge from upload to retrieval.

[0010] Furthermore, the step of adapting the processing request to the specified heterogeneous underlying RAG service provider through the unified adaptation layer includes:

[0011] Through the adapter factory, based on the request context or system configuration strategy, the unified knowledge base operation interface call of the business system is routed to the corresponding provider adapter, which encapsulates the specific RAG service provider's interaction protocol. This factory pattern enables unified management and dynamic scheduling of different RAG service provider adapters. The business system does not need to concern itself with the underlying implementation; it only needs to call the unified interface, greatly simplifying integration complexity and supporting the rapid integration of new RAG services without modifying core business logic.

[0012] Furthermore, the decoupling of document processing into three stages—sequential upload and storage, asynchronous parsing, and vectorized indexing—specifically includes:

[0013] During the upload and storage phase, the original knowledge document is received and stored, and the corresponding asynchronous parsing task is created and initialized. Then, a task receiving response is returned to the business system.

[0014] During the asynchronous parsing phase, the asynchronous parsing task is consumed, and the parsing process managed by the state machine drives the task to sequentially pass through the pending, parsing, and post-processing states, ultimately generating a standardized intermediate knowledge document.

[0015] During the vectorized indexing phase, the system listens for the successful completion event of the asynchronous parsing phase, retrieves the corresponding intermediate knowledge document based on the identifier information carried in the event, and submits it to the designated underlying RAG service through the unified adaptation layer to complete the vectorized indexing. The upload-and-return strategy ensures immediate responsiveness to user operations; the fine-grained state transitions of the parsing task are precisely controlled through a state machine, improving the controllability and observability of asynchronous tasks; and the event-driven mechanism achieves loose coupling between processing phases, making the responsibilities of each module clear and easy to maintain and extend.

[0016] Furthermore, during the asynchronous parsing phase, when the state machine is in the parsing state, a polling query is performed to check the progress of the third-party parsing service task; the polling query process includes a timeout threshold and failure judgment logic. Through proactive polling and timeout mechanisms, long periods of unresponsiveness or abnormal situations of third-party services can be detected and handled in a timely manner, preventing tasks from being suspended indefinitely due to external service failures, thus enhancing the robustness and reliability of the entire asynchronous processing flow.

[0017] Furthermore, the intermediate knowledge document is in Markdown format; the method also includes: storing and managing the Markdown-formatted intermediate knowledge document as a reusable knowledge asset independent of the vector index in a versioned manner, and supporting content proofreading, format optimization, or migration to other knowledge base engines based on this intermediate knowledge document. Managing structured Markdown documents as core knowledge assets not only provides high-quality input for vectorization, but also enables knowledge content to be parsed once and reused in multiple places, facilitating manual verification, continuous optimization, and cross-platform migration, transforming unstructured data into digital assets that can be operated long-term.

[0018] Furthermore, the traceable knowledge processing link metadata includes at least the following associated records: tenant identifier, knowledge base identifier, original document storage path, intermediate knowledge document storage path, RAG service provider type used, and the knowledge base identifier and document identifier of the document in the corresponding RAG service. By constructing a complete metadata association network, it is possible to quickly locate the original file, processing middleware, and its storage location in the underlying service corresponding to any search result, providing a solid data foundation for knowledge accuracy verification, problem investigation, cost accounting, and compliance auditing.

[0019] Furthermore, the polling process incorporates a configurable retry strategy. When polling times out or a temporary failure response is received, a limited number of automatic retries are performed according to the retry strategy. If the task still fails after reaching the maximum number of retries, the task status is marked as final failure, detailed error context information is recorded, and an error handling mechanism can be triggered. This intelligent retry mechanism effectively addresses network jitter or momentary service failures, increasing the probability of eventual task success. Clear failure marking, detailed logging, and the accompanying error handling mechanism provide clear fault location and recovery methods for system operation and maintenance, improving system maintainability.

[0020] Furthermore, in the post-processing state of the asynchronous parsing phase, post-processing and enhancement of the parsing results are performed, including: format standardization, content structure extraction, irrelevant information filtering, and metadata supplementation of the raw result data obtained from the third-party parsing service, to generate the intermediate knowledge document that meets preset quality standards. This ensures the uniformity of the intermediate knowledge document format, the purity of the content, and the richness of the information, thereby providing a reliable guarantee for downstream vectorized indexing and high-quality retrieval.

[0021] Furthermore, the method is applied to a multi-tenant SaaS platform, providing logically isolated message queues or consumer groups for asynchronous learning tasks of different tenants, as well as differentiated task priority scheduling and processing resource quota management. This achieves resource and data isolation among multiple tenants, ensuring business security and performance independence. Simultaneously, differentiated scheduling strategies based on tenant level or system load optimize overall resource utilization and ensure that high-priority or important tenant tasks are processed promptly, improving the service quality and customer satisfaction of the SaaS platform.

[0022] Furthermore, during the vectorized indexing stage, the documents submitted to the underlying RAG service are the standardized intermediate knowledge documents. By moving the complex and time-consuming document parsing work forward and having it performed by a dedicated service, the underlying RAG service only needs to process cleaned, structured, high-quality text. This significantly reduces the processing burden and error probability of the RAG service, improving the overall efficiency and success rate of index construction.

[0023] The above technical solution has the following advantages or technical effects:

[0024] 1. This invention achieves effective decoupling between business architecture and underlying technology. By introducing a unified adaptation layer and adapter factory, it shields the differences between heterogeneous RAG services, enabling upper-layer businesses to flexibly and seamlessly switch or mix and use multiple RAG services, greatly enhancing the system's scalability and ability to cope with technological changes.

[0025] 2. This invention significantly improves the stability and user experience of large-scale knowledge document processing. By adopting a phased asynchronous pipeline design of "upload-storage-parse-index", and combining message queues, state machines and event-driven mechanisms, it effectively solves the timeout and stability problems in large file synchronous processing, and realizes fast response to requests and reliable execution of background tasks.

[0026] 3. This invention constructs a complete knowledge asset management and traceability system. By persisting the parsed standardized intermediate documents (such as Markdown) as core assets and establishing a metadata link that runs through the entire process and connects multiple tenants and services, it not only realizes the reusability and optimization of knowledge content, but also ensures that the entire process from retrieval results to the original source is traceable and auditable.

[0027] 4. This invention enhances the fine-grained support for multi-tenant SaaS scenarios. By providing resource isolation, differentiated scheduling, and quota management for different tenants, it ensures the fairness and independence of service performance in a multi-tenant environment, and meets the requirements of enterprise applications for security, reliability, and service level agreements (SLAs). Attached Figure Description

[0028] Figure 1 This is a flowchart of the method of the present invention;

[0029] Figure 2 This is a schematic diagram of the overall system architecture of the present invention;

[0030] Figure 3 This is a diagram showing the state evolution of the document during the asynchronous processing lifecycle of this invention. Detailed Implementation

[0031] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] like Figure 1 As shown, a knowledge base management system deployed in a multi-tenant SaaS platform is based on a unified management and asynchronous learning method for heterogeneous knowledge bases using a RAG architecture. The steps include:

[0033] Receive knowledge document processing requests from business systems and execute asynchronous learning processing flows on the knowledge documents;

[0034] The asynchronous learning process includes:

[0035] Through a unified adaptation layer, the processing request is adapted and invoked to the specified heterogeneous underlying RAG service provider;

[0036] Document processing is decoupled into three stages: sequential upload and storage, asynchronous parsing, and vectorized indexing. Asynchronous parsing and vectorized indexing are asynchronously connected through message queues and event-driven mechanisms.

[0037] During the process execution, the original documents under multi-tenant environments, the intermediate knowledge documents generated by parsing, and their mapping identifiers in the underlying RAG service are associated and managed to generate traceable knowledge processing link metadata.

[0038] In practical applications, from the perspective of enterprise users (tenants), the experience of using this system for knowledge entry is smooth and asynchronous. Users select and upload files within the enterprise's internal SaaS application. The system quickly responds with "Document uploaded successfully, system is processing," along with a task ID. Users can then close the page, and the system will automatically complete all the complex processes from text parsing to intelligent index building in the background. Users can view the real-time status in the task center, or, after processing, directly ask questions about the document content in the enterprise intelligent assistant and receive accurate answers.

[0039] like Figure 2 As shown, a loosely coupled, highly available backend system architecture is adopted, which mainly includes:

[0040] Unified access and adaptation layer: Serves as the sole interface to the business, shielding it from the differences in the underlying RAG services;

[0041] Asynchronous learning pipeline: As the core processor, it robustly handles document parsing and indexing tasks using a "production-consumption" model;

[0042] Unified metadata and service layer: performs task scheduling, status management, and end-to-end data association.

[0043] The implementation of the unified access and adaptation layer specifically includes: building a unified adaptation layer to decouple business systems from specific RAG service providers. At the code level, this is reflected in a set of clearly defined interfaces and classes.

[0044] The system defines a core facade service, RagService, which provides business methods such as addDocument and retrieve. RagService itself does not implement any RAG logic. Instead, when it needs to call the underlying RAG service, it relies on a ProviderRagServiceFactory, which is an adapter factory. The core responsibility of this factory is to dynamically select and return a specific service adapter instance based on a strategy (e.g., tenant configuration read from the database).

[0045] The following pseudocode illustrates the core routing logic of the factory:

[0046] @Component

[0047] public class ProviderRagServiceFactory {

[0048] private final Map<String, ProviderRagService> ragServiceMap; / / Stores all adapters

[0049] public ProviderRagService getRagService(RagProviderType provider){

[0050] RagProviderType use = (provider == null) ?RagProviderType.getDefault() : provider;

[0051] String beanName = use.beanName(); / / For example, to get the bean name corresponding to "DIFY"

[0052] ProviderRagService service = ragServiceMap.get(beanName);

[0053] if (service == null) {

[0054] throw new IllegalArgumentException("Unsupported RAGprovider: " + use.name());

[0055] }

[0056] return service; / / Returns the found adapter, such as DifyRagService

[0057] }

[0058] }

[0059] All underlying adapters (such as DifyRagService and LangChainRagService) implement the unified ProviderRagService interface. Taking DifyRagService as an example, it encapsulates all the details of interacting with the Dify platform API. Once RagService obtains a DifyRagService instance through the factory, it can call its createDocument and other methods, while DifyRagService is responsible for converting general parameters into Dify-specific HTTP requests. This design allows adding a new RAG service provider (such as Azure AISearch) to simply implement a new adapter and register it in the factory, without any changes to the business code, perfectly achieving architectural decoupling and high scalability.

[0060] In the three-stage decoupled processing flow of the asynchronous learning pipeline, the three stages are asynchronously connected through queues and event-driven mechanisms. Specifically:

[0061] Phase 1: Upload, Storage, and Task Initialization

[0062] When a user uploads a file, RagService first persists the file to Object Storage Service (OSS), then creates a document record in the business database with a status of WAITED. At this point, the system immediately returns a success response to the user, and simultaneously packages the subsequent parsing task into a ParseTask entity and sends it to a highly reliable message queue (i.e., the "learning queue"). This stage separates processing that could take minutes or even longer from the user request, completely avoiding HTTP timeouts and greatly improving user experience and interface availability.

[0063] Phase Two: Asynchronous Resolution and State Machine-Driven Approach

[0064] One or more background parsing coordination service instances listen to and consume tasks in the queue. At its core is a state machine engine that precisely controls the lifecycle of each parsing task, such as... Figure 3 As shown.

[0065] The state machine is managed by DocResolveService. After acquiring a ParseTask, this service selects a matching handler (e.g., StartResolveState) from a set of DocResolveState handlers based on its current state (e.g., PENDING) to execute specific operations. The handler then calls the corresponding Resolver to interact with third-party OCR / document parsing services.

[0066] The following pseudocode snippet illustrates the core of the state machine-driven parsing process:

[0067] @Service

[0068] public class DocResolveService {

[0069] private final List <docresolvestate>states; / / All state handlers

[0070] public void resolve(ParseTask parseTask) {

[0071] / / Build execution context

[0072] DocResolveContext context = DocResolveContext.builder().parseTask(parseTask).status(parseTask.getStatus()).build();

[0073] / / State machine cycle driven

[0074] while(!context.isEnd()) {

[0075] / / 1. Retrieve the corresponding handler (e.g., QueryResultState) based on the current state.

[0076] DocResolveState state = getState(context.getStatus());

[0077] / / 2. Execute the processing logic in this state (such as polling third-party services).

[0078] state.handle(context);

[0079] / / 3. Update task status and persistence

[0080] updateParseTask(context);

[0081] }

[0082] }

[0083] private DocResolveState getState(ParseStatus status) {

[0084] / / Iterate through the state processors to find those that support the current state

[0085] for (DocResolveState state : states) {

[0086] if (state.support(status)) {

[0087] return state;

[0088] }

[0089] }

[0090] return null;

[0091] }

[0092] }

[0093] PENDING state: Handled by StartResolveState, which calls Resolver.start() to submit the file to a third-party parsing service. Upon successful submission, the state changes to PROVIDER_PARSING.

[0094] PROVIDER_PARSING state: Handled by QueryResultState. This state handler periodically calls Resolver.query() to poll the progress of the third-party service by task ID. The system is configured with a polling interval and a total timeout. If the polling is successful and parsing is complete, the state transitions to AFTER_PARSING; if a timeout occurs or the process fails, it transitions to the failure state and logs the changes.

[0095] AFTER_PARSING state: Handled by BuildMarkdownState. This state handler calls Resolver.buildMarkdown() to obtain the raw parsed data (such as JSON), performs post-processing such as format standardization, content cleaning, and metadata extraction, and finally generates a well-structured and clean Markdown file, which is then saved to OSS.

[0096] When the parsing is finally successful, the state machine transitions to SUCCESS, and DocResolveService publishes a DocumentResolved event, announcing that the intermediate knowledge asset (Markdown) of the document is ready.

[0097] Phase 3: Event-Driven Vectorized Indexing

[0098] This stage is implemented by DocumentEventHandler. It listens for the DocumentResolved event and performs the following operations:

[0099] Update the status of the corresponding document in the business database to LEARNING;

[0100] Read the Markdown file generated in Phase 2 from OSS;

[0101] Through a unified adaptation layer, the corresponding RAG service adapter (such as DifyRagService) is called to submit the Markdown content (instead of the original PDF) to the underlying RAG platform for vectorized indexing.

[0102] After successful indexing, the document status is updated to COMPLETED, and the document ID returned by the RAG platform is recorded.

[0103] By moving the most computationally intensive and error-prone document parsing work forward and having it done by a professional service, the RAG engine receives "clean" text, which greatly reduces the processing burden and complexity of the RAG service and significantly improves the success rate and efficiency of the final index construction.

[0104] This embodiment achieves precise management of the entire knowledge processing chain by designing a relational data model. The main entities include:

[0105] KnowledgeBase: Associated tenant (ent_code), local knowledge base identifier, type of RAG provider used, and actual knowledge base ID under that provider.

[0106] Document: Associates the original file key, the parsed Markdown file key, the status, and the final document ID (rag_document_id) in the RAG service.

[0107] ParseTask: Records detailed information about every step of asynchronous parsing, including state change times, third-party task IDs, and error messages.

[0108] By querying these entities in relation to each other, the system can clearly trace the answer fragments of any intelligent question and answer session.

[0109] In addition to the logical isolation achieved by the ent_code field in the data model mentioned above, at the resource level:

[0110] Queue isolation: Independent physical or logical message queues can be configured for important tenants to ensure that their task processing is not affected by other tenants; Scheduling and quotas: The system service can set different priorities for parsing tasks according to the tenant level and implement concurrent task quota management.

[0111] This embodiment achieves decoupling of business and technology through a unified adaptation layer, ensures the stability and efficiency of large file processing through an asynchronous learning pipeline (including state machine management), and realizes traceability and asset management of knowledge through full-link metadata association.

[0112] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.< / docresolvestate>

Claims

1. A method for unified management and asynchronous learning of heterogeneous knowledge bases based on RAG architecture, characterized by the following steps: include: Receive knowledge document processing requests from business systems and perform asynchronous learning processing on the knowledge documents; The asynchronous learning process includes: Through a unified adaptation layer, the processing request is adapted and invoked to the specified heterogeneous underlying RAG service provider; Document processing is decoupled into three stages: sequential upload and storage, asynchronous parsing, and vectorized indexing. Asynchronous parsing and vectorized indexing are asynchronously connected through message queues and event-driven mechanisms. During the execution of the process, the original documents under the multi-tenant system, the parsed intermediate knowledge documents, and their mapping identifiers in the underlying RAG service are associated and managed to generate traceable knowledge processing link metadata.

2. The method for unified management and asynchronous learning of heterogeneous knowledge bases based on RAG architecture according to claim 1, characterized in that, The process of adapting and invoking the processing request to the specified heterogeneous underlying RAG service provider through the unified adaptation layer includes: Through the adapter factory, based on the request context or system configuration strategy, the unified knowledge base operation interface call of the business system is routed to the corresponding provider adapter that encapsulates the specific RAG service provider interaction protocol for execution.

3. The method for unified management and asynchronous learning of heterogeneous knowledge bases based on RAG architecture according to claim 1, characterized in that, The decoupling of document processing into three stages—sequential upload and storage, asynchronous parsing, and vectorized indexing—specifically includes: During the upload and storage phase, the original knowledge document is received and stored, and the corresponding asynchronous parsing task is created and initialized. Then, a task receiving response is returned to the business system. During the asynchronous parsing phase, the asynchronous parsing task is consumed, and the parsing process managed by the state machine drives the task to sequentially pass through the pending, parsing, and post-processing states, ultimately generating a standardized intermediate knowledge document. During the vectorized indexing phase, the system listens for the successful completion of the asynchronous parsing phase, retrieves the corresponding intermediate knowledge document based on the identification information carried in the event, and submits it to the designated underlying RAG service through the unified adaptation layer to complete the vectorized indexing.

4. The method for unified management and asynchronous learning of heterogeneous knowledge bases based on RAG architecture according to claim 3, characterized in that, During the asynchronous parsing phase, when the state machine is in the parsing state, a polling query is performed to check the progress of the third-party parsing service task; the polling query process is equipped with a timeout threshold and failure judgment logic.

5. A method for unified management and asynchronous learning of heterogeneous knowledge bases based on RAG architecture according to claim 3 or 4, characterized in that, The intermediate knowledge document is in Markdown format; the method further includes: storing and managing the Markdown format intermediate knowledge document as a reusable knowledge asset independent of the vector index in a versioned manner, and supporting content proofreading, format optimization or migration to other knowledge base engines based on this intermediate knowledge document.

6. A method for unified management and asynchronous learning of heterogeneous knowledge bases based on RAG architecture according to claim 1 or 3, characterized in that, The traceable knowledge processing link metadata includes at least the following associated records: tenant identifier, knowledge base identifier, original document storage path, intermediate knowledge document storage path, type of RAG service provider used, and the knowledge base identifier and document identifier of the document in the corresponding RAG service.

7. The method for unified management and asynchronous learning of heterogeneous knowledge bases based on RAG architecture according to claim 4, characterized in that, The polling process incorporates a configurable retry strategy. When polling times out or a temporary failure response is received, a limited number of automatic retries are performed according to the retry strategy. If the maximum number of retries is reached and the task still fails, the task status is marked as final failure, detailed error context information is recorded, and an error handling mechanism can be triggered.

8. The method for unified management and asynchronous learning of heterogeneous knowledge bases based on RAG architecture according to claim 3, characterized in that, In the post-processing state of the asynchronous parsing phase, post-processing and enhancement of the parsing results are performed, including: format standardization, content structure extraction, irrelevant information filtering, and metadata supplementation of the raw result data obtained from the third-party parsing service, so as to generate the intermediate knowledge document that meets the preset quality standards.

9. The method for unified management and asynchronous learning of heterogeneous knowledge bases based on RAG architecture according to claim 1, characterized in that, The method is applied to a multi-tenant SaaS platform and provides logically isolated message queues or consumer groups for asynchronous learning tasks of different tenants, as well as differentiated task priority scheduling and processing resource quota management.

10. A method for unified management and asynchronous learning of heterogeneous knowledge bases based on RAG architecture according to claim 3, characterized in that, During the vectorized indexing stage, the documents submitted to the underlying RAG service are the standardized intermediate knowledge documents.

Images