Retrieval recall method and system integrating document fragmentation and authority control

Abstract

The invention provides a retrieval recall method and system fusing document fragmentation and authority control, and relates to the technical field of artificial intelligence and information retrieve.The method comprises the steps that a role of a user is obtained according to the identity of the user, and a corresponding authority strategy is determined in combination with a preset multi-level authority mapping relation; the permission strategy comprises a label allowed to be accessed, a data set and a document set; the query request is converted into a semantic vector, and initial candidate document fragments are retrieved from a vector database in combination with an associated data set or tag identifier; according to the method, the candidate results are subjected to access control filtering on the basis of the authority strategy, and the final retrieval recall result is generated, so that role-based fine-grained access control is realized while high retrieval precision is ensured, the problems of rough authority management and weak context processing capability in traditional retrieval are solved, and the security and efficiency of retrieval are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and information retrieval technology, and more specifically, to a retrieval and recall method and system that integrates document fragmentation and access control. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale models are increasingly being used in fields such as intelligent question answering, knowledge services, and natural language understanding. In existing technologies, knowledge service systems built on large-scale pre-trained language models typically utilize general knowledge bases to train the models, enabling them to possess basic language understanding and generation capabilities, and to generate corresponding answers based on user input. In practical applications, when a user initiates a query request, the system first determines whether the user has permission to access specific data: if the user has the appropriate permissions, the system retrieves relevant information from a controlled data source and, combined with the large-scale model's generation capabilities, provides the user with an accurate answer; if the user does not have access permissions, the system does not directly return restricted data, but instead relies on the large-scale model to generate an answer based on the general knowledge learned during its training, thereby preventing the unauthorized leakage of sensitive information.

[0003] Such mechanisms have been initially applied in various scenarios, including enterprise knowledge management, government service platforms, and healthcare consultation. However, due to differences in users' roles, responsibilities, and required information scope, simple permission determination alone is insufficient to meet the security and personalization needs of complex business environments. Furthermore, existing systems often lack a deep understanding and continuous tracking of contextual semantics when processing user queries, leading to an inability to accurately grasp the user's actual intent in multi-turn interactions or complex scenarios, thus affecting the relevance and security of the responses. Therefore, existing technologies, due to their inability to achieve granular permission management and their difficulty in adapting to complex contextual environments, limit the effective deployment and application of large models in scenarios with high security requirements. Summary of the Invention

[0004] The purpose of this invention is to provide a retrieval and recall method and system that integrates document fragmentation and access control, in order to solve the technical problems of low precision of access control and low ability to process context in the prior art.

[0005] In a first aspect, embodiments of the present invention provide a retrieval and recall method that integrates document fragmentation and access control. The method includes: receiving a query request input by a user, obtaining a corresponding user role based on the user's identity information, and determining an access control policy corresponding to the user role based on a pre-configured multi-level access control mapping relationship; the access control policy includes a set of tags, a set of datasets, and a set of documents that are allowed to be accessed; the query request is associated with at least one dataset identifier and / or a tag identifier. The query request is converted into a semantic vector, and based on the semantic vector and the dataset identifier and / or tag identifier associated with the query request, document fragments are retrieved from the vector database to obtain an initial set of candidate document fragments; wherein the document fragments are labeled, and the vector database stores the mapping relationship between document fragments and tags, datasets, and documents; Based on the above permission policy, the permission of the initial candidate document fragment set is verified, and the document fragments that the user does not have permission to access are filtered out to obtain the document fragment set that passes the permission verification, so as to generate the retrieval results.

[0006] In some optional implementations, the above method further includes: a preprocessing step before receiving a query request: splitting the documents in the knowledge base to obtain multiple document splits; attaching at least one tag to each of the document splits and establishing a mapping relationship between the document splits and tags, datasets and documents; configuring user roles for the splits and establishing a multi-level permission mapping relationship between the user roles and tags, datasets and documents based on the mapping relationship.

[0007] In some optional implementations, the above tags can be generated in the following ways: automatic generation from large models, keyword extraction, manual annotation, rule engine matching, or entity mapping based on knowledge graphs.

[0008] In some optional implementations, the aforementioned vector database is generated based on feature vector calculation and vectorization of data within the knowledge base; based on the aforementioned semantic vector retrieval of document fragments, a dense vector retrieval model is adopted to calculate and sort the similarity between the aforementioned semantic vector and the document fragment vector, and select the N document fragments with the highest similarity to form the aforementioned initial candidate document fragment set.

[0009] In some optional implementations, the above method further includes: reordering and merging the document fragment set that has passed the permission verification, wherein the reordering is based on the semantic similarity between the document fragment and the query request, and the tag signal corresponding to the document fragment; and generating retrieval recall results based on the reordered and merged document fragment set that has passed the permission verification.

[0010] In some optional implementations, the above method also includes: if the set of document fragments that have passed the permission verification is empty, a degradation mechanism is triggered to generate an answer based on general knowledge and prompt the user.

[0011] Secondly, embodiments of the present invention provide a retrieval and recall system that integrates document fragmentation and access control. The system includes: an access control module, configured to receive a query request input by a user, obtain the corresponding user role based on the user's identity information, and determine the access control policy corresponding to the user role based on a pre-configured multi-level access control mapping relationship; the access control policy includes a set of tags, a set of datasets, and a set of documents that are allowed to be accessed; the query request is associated with at least one dataset identifier and / or tag identifier. The retrieval module is used to convert the above query request into a semantic vector, and based on the above semantic vector and the dataset identifier and / or tag identifier associated with the above query request, retrieve document fragments from the vector database to obtain an initial candidate document fragment set; wherein the above document fragments are labeled, and the above vector database stores the mapping relationship between document fragments and tags, datasets, and documents; The verification module is used to perform permission verification on the initial candidate document fragment set based on the above permission policy, filter out the document fragments that the user does not have permission to access, and obtain the document fragment set that has passed the permission verification in order to generate the retrieval results.

[0012] In some optional implementations, the system further includes: a result generation module, used to rearrange and merge the document fragment set that has passed the permission verification, wherein the rearrangement is based on the semantic similarity between the document fragment and the query request, and the tag signal corresponding to the document fragment; the result generation module is also used to generate retrieval recall results based on the rearranged and merged document fragment set that has passed the permission verification.

[0013] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0014] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method described in any of the first aspects above.

[0015] This invention provides a retrieval and recall method and system that integrates document fragmentation and access control. Upon receiving a user query request, the method determines the user's role based on their identity and obtains accessible tags, datasets, and document sets based on that role. It then performs preliminary screening of document fragments in a vector database to construct a candidate set for retrieval. The query is converted into a semantic vector, and a vectorized retrieval is performed on the candidate set to obtain relevant document fragments. Subsequently, fine-grained access control is performed on each relevant fragment, eliminating content that is not authorized for access. Only the fragments that pass the verification are fed into a large model to generate the final recall results. This invention achieves high retrieval accuracy while implementing role-based fine-grained access control, solving the problems of coarse access management and weak context processing capabilities in traditional retrieval methods, and significantly improving retrieval security and efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a retrieval and recall method integrating document fragmentation and access control, provided in an embodiment of the present invention; Figure 2 This is an application diagram of a retrieval and recall method that integrates document fragmentation and access control, provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of a retrieval and recall system that integrates document fragmentation and access control, provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] In existing technologies, a typical intelligent question-answering and knowledge retrieval system usually includes the following processing flow: First, in the model building phase, a large model is trained using general knowledge to construct an intelligent question-answering or knowledge service capability with basic semantic understanding and generation capabilities. Then, a backend knowledge base is established to store pre-processed knowledge data, and a mapping relationship between user roles and data access permissions is set to achieve role-based data access control, with different roles assigned different access scopes. Next, the data in the knowledge base is vectorized, its semantic features are extracted and transformed into high-dimensional vector representations, and a vector database is constructed. Simultaneously, data connectivity between the vector database and the large model is achieved, enabling the large model to access results retrieved based on semantic similarity. When a user initiates a question, the system extracts features from the input question, generates a vector representation corresponding to the question, and performs similarity matching in the vector database to retrieve the most relevant knowledge fragments. Based on this, it determines whether the user has permission to access the search results. If permission is granted, the final answer is generated by combining the retrieved content with the large model; if permission is denied, restricted content is not returned, and the large model generates an answer solely based on its internal general knowledge, thus preventing information leakage due to unauthorized access. After each question and answer session, the system resets the background knowledge base to its initial state, preparing to respond to the next independent query. While the above-described intelligent question answering and knowledge retrieval process achieves basic search enhancement and permission isolation functions, it still has significant shortcomings in terms of granularity of permission control, contextual coherence, and support for complex interactions.

[0020] In other words, existing technologies for access control have a coarse granularity, resulting in insufficient precision in system access control; at the same time, their limited ability to process context restricts the dynamic adaptability and flexibility of access control policies. Based on this, this invention provides a retrieval and recall method and system that integrates document fragmentation and access control, enabling fine-grained access control and possessing a powerful context-aware and processing-capable access management mechanism.

[0021] To facilitate understanding of this embodiment, a retrieval and recall method integrating document fragmentation and access control, as disclosed in this embodiment of the invention, will first be described in detail. (See [link to relevant documentation]). Figure 1 The diagram illustrates a retrieval and recall method that integrates document fragmentation and access control. This method can be executed by an electronic device and mainly includes the following steps S102 to S108: Step S102: Receive the query request input by the user, obtain the corresponding user role based on the user's identity information, and determine the permission policy corresponding to the user role based on the pre-configured multi-level permission mapping relationship.

[0022] The query request entered by the user can be a question or retrieval instruction in natural language, and the query request is associated with at least one dataset identifier and / or tag identifier.

[0023] For example, a user can enter the following query requests: "Please provide all technical documents related to project A", "Query all fault records in dataset B", etc. These query requests can be submitted through the user terminal's input interface, supporting text input, speech-to-text input, and other formats, and serve as the basic input data for subsequent semantic parsing and vectorization processing.

[0024] The permission policies corresponding to user roles can include the set of tags, datasets, and documents that are allowed to be accessed.

[0025] In one embodiment, the above method may further include a preprocessing step before receiving a query request: (S01) performing document segmentation on the documents in the knowledge base to obtain multiple document segments; (S02) attaching at least one tag to each document segment and establishing a mapping relationship between the document segment and the tag, dataset and document; (S03) configuring user roles for the segments and establishing a multi-level permission mapping relationship between the user roles and the tag, dataset and document based on the mapping relationship.

[0026] The document chunking method can be implemented using a sliding window method, a semantic boundary segmentation method, or a chapter division method to ensure that each chunk is semantically complete and of appropriate length (e.g., within 512 tokens).

[0027] In some implementations, a tag can be metadata such as keywords, category tags, sensitivity levels (e.g., "confidential"), subject categories (e.g., "reimbursement process"), timestamps, or geographic locations.

[0028] It should be noted that the above-mentioned tag generation methods are not limited to manual setting, and may also include the following alternative methods: automatic generation by large models, keyword extraction, manual annotation, rule engine matching, entity mapping based on knowledge graphs, etc.

[0029] As a concrete example, this can be achieved through various methods: automatic generation of large models (e.g., using LLM to analyze document fragments and automatically summarize semantic tags); keyword extraction (e.g., extracting high-frequency or key terms as tags based on algorithms such as TF-IDF and TextRank); manual annotation (usually done by experts to ensure accuracy); rule engine matching (automatically labeling based on regular expressions or preset rules, such as labeling pre-tax salary as "salary"); and entity mapping based on knowledge graphs (linking named entities in fragments to knowledge graph nodes, inheriting their category tags). These multiple tag generation methods can be used in combination depending on the application scenario to balance efficiency and accuracy.

[0030] Step S104: The query request is converted into a semantic vector, and based on the semantic vector and the dataset identifier and / or tag identifier associated with the query request, document fragments are retrieved from the vector database to obtain an initial candidate document fragment set.

[0031] Document segments can be accompanied by corresponding tags, and the vector database stores the mapping relationship between document segments and tags, datasets, and documents.

[0032] In one embodiment, the aforementioned vector database can be generated based on feature vector calculation and vectorization processing of data within the knowledge base.

[0033] Document retrieval based on semantic vectors can employ a dense vector retrieval model. This model calculates and sorts the similarity between the semantic vector and the document shard vector, then selects the N document shards with the highest similarity to form an initial candidate document shard set.

[0034] Preferably, advanced dense retrieval models (such as bge-small-zh, text2vec-base-chinese, etc.) can be used for vectorization to support Chinese semantic understanding. The retrieval process can be executed efficiently in a GPU-accelerated environment.

[0035] In another implementation, a hybrid search approach can be used, combining keyword matching algorithms such as BM25 with vector similarity scores to further improve recall quality. Alternatively, an inverted index search or a knowledge graph-based entity search can also be employed.

[0036] Step S106: Based on the permission policy, perform permission verification on the initial candidate document fragment set, filter out document fragments that the user does not have permission to access, and obtain the document fragment set that has passed the permission verification to generate the retrieval result.

[0037] In this embodiment, the purpose of the permission verification process is to determine whether a user has the legal permission to access a certain document fragment based on a preset permission policy model and the user's identity attribute information, and to filter out unauthorized fragments accordingly, thereby generating a final set of secure candidate fragments that can be passed.

[0038] In one embodiment, the construction of the preset permission policy model may include: pre-configuring a structured permission policy system, which includes, but is not limited to, one or more of the following policy types: (1) Role-based access control (RBAC): Users are divided into different roles (such as "administrator", "department employee", "external collaborator"), and each role corresponds to a set of document tags or categories that are allowed to be accessed. (2) Attribute-based access control (ABAC): Define dynamic policy rules consisting of user attributes (such as department, job level, project group), resource attributes (such as document security level, business line, creation time), and environment attributes (such as access time, IP address). (3) Access control list (ACL): Maintain an access whitelist for each document fragment, explicitly listing the user IDs or user groups that can access it.

[0039] For example, if a document fragment in a financial knowledge base is labeled with "Confidentiality Level = Confidential" and "Department = Risk Control Department", then access will only be allowed if the user's attributes meet the condition `role='Risk Control Personnel' AND clearance>= 'Confidential'`.

[0040] In one embodiment, the method for obtaining user identity attribute information may include: upon receiving a user's query request, obtaining the current user's identity context information through an authentication module (such as OAuth 2.0, LDAP, SSO), such as: the organization path to which the user's unique identifier (User ID) belongs, the role list, the security clearance level, and project affiliation. This information can be used for subsequent matching and evaluation with permission policies.

[0041] In the steps described above, each document shard can be tagged with metadata during the index building phase, containing the attribute information required for its access control. This metadata is stored along with the document shard in a vector database or hybrid search engine to support rapid permission determination.

[0042] Next, permission verification is performed. In one embodiment, the execution logic for permission verification is as follows: For each fragment C_i in the initial candidate document fragment set, the following verification process is executed: 1. Extract the permission-related metadata M_i of C_i; 2. Obtain the identity attribute set U of the current user; 3. Calculate the expression \text{isAllowed}(U,M_i,P) according to the permission policy rule set P loaded by the system; 4. If it returns `true`, retain the fragment; otherwise, remove it.

[0043] This process can be implemented through a dedicated permission decision module (PDP, PolicyDecisionPoint), such as integrating policy engines like OpenPolicyAgent (OPA).

[0044] After the above verification, a new set of document fragments is output, in which all fragments are content that the user has the right to access, for subsequent sorting, aggregation and presentation.

[0045] By using the above step S106, it can be ensured that the system has removed content that the user does not have access to before returning the search results, thereby achieving fine-grained access control and data security compliance.

[0046] In one embodiment, the above method may further include: rearranging and merging the set of document fragments that have passed permission verification, wherein the rearranging is based on the semantic similarity between the document fragments and the query request, and the tag signals corresponding to the document fragments; and then generating retrieval results based on the rearranged and merged set of document fragments that have passed permission verification.

[0047] In this embodiment, the purpose of the reordering and fusion process is to reorder and fuse the set of document fragments that have passed the permission verification based on the semantic similarity between the document fragments and the query request, as well as the tag signals corresponding to the document fragments, in order to generate higher quality retrieval results.

[0048] The methods for calculating semantic similarity can include: First, calculate the semantic similarity between each document shard that passes permission verification and the query request. As a concrete example, this step can be implemented using the following methods: vector embedding, using a pre-trained language model (such as BERT, RoBERTa) to convert the query request and document shards into high-dimensional vector representations; and cosine similarity, calculating the cosine similarity between the query vector and each document shard vector to obtain a score between 0 and 1, representing the semantic relevance between the two.

[0049] In addition to semantic similarity, document fragment tag signals can be used to further optimize ranking. Tag signals can include, but are not limited to: document type (reports, meeting minutes, technical documents, etc.); importance level (high, medium, low); creation time (recently created documents may be more timely); author information (documents by certain authors may be more authoritative or relevant). The weights of these tag signals can be predefined.

[0050] As a concrete example, the specific steps for reordering a set of document fragments that have passed permission verification, combining semantic similarity and tag signal weights, are as follows: 1. Comprehensive score calculation: Multiply the semantic similarity and the tag signal weight to obtain the comprehensive score of each document segment.

[0051] 2. Sorting: Sort the document segments in descending order based on the overall score.

[0052] 3. Merging Processing: For document fragments with highly related content, they can be merged or summarized to reduce redundancy and improve readability.

[0053] In one embodiment, the above method may further include: if the set of document fragments that have passed the permission verification is empty, triggering a degradation mechanism, generating an answer based on general knowledge and prompting the user.

[0054] In this embodiment, the degradation mechanism is used to instruct the large model to generate answers based on general knowledge and to prompt the user. For example, the user prompt message after the degradation mechanism is triggered could be: "You do not have permission to view the relevant information" or "The relevant content is protected by permissions." This mechanism avoids the embarrassing situation of "zero results" caused by permission filtering, while reminding users to pay attention to access restrictions, thus enhancing user experience and security transparency.

[0055] This invention provides a retrieval and recall method that integrates document fragmentation and access control. The method obtains the user's role based on their identity and determines the corresponding access control policy by combining a preset multi-level access control mapping relationship. The access control policy includes allowed tags, datasets, and document sets. The method converts the query request into a semantic vector and, combined with the associated dataset or tag identifier, retrieves initial candidate document fragments from the vector database. Then, based on the access control policy, the candidate results are filtered for access control to generate the final retrieval and recall results. This method achieves high retrieval accuracy while implementing fine-grained access control based on roles, solving the problems of coarse access control and weak context processing capabilities in traditional retrieval, and significantly improving the security and efficiency of retrieval.

[0056] For ease of understanding, embodiments of the present invention also provide a retrieval and recall system, which can be used to execute a retrieval and recall method that integrates document fragmentation and access control. See [link to relevant documentation]. Figure 2 The diagram illustrates an application of a retrieval and recall method that integrates document fragmentation and access control.

[0057] In one embodiment, the retrieval and recall system described above may include: a knowledge management layer (KMS), an agent invocation layer (Dify), a retrieval enhancement layer (ragflow), and a unified access control layer (UIC).

[0058] The Knowledge Management System (KMS) is configured to: divide documents into chunks and store them in a knowledge base; each knowledge base belongs to a space / department; and attach tags at the chunk level to form a multi-layered annotation structure of "document → chunk → tag".

[0059] In the agent invocation layer (Dify), agents (agent1, agent2, ...) bind dataset keys and tag keys during configuration. Agents actually determine access permissions (i.e., the scope of knowledge they can access and retrieve) based on the tag / dataset keys.

[0060] In another example, the agent's retrieval scope is not limited to the dataset key / tag key, but can also be achieved through context templates, prompt injection, or calling the route configuration.

[0061] The retrieval enhancement layer (ragflow) is responsible for organizing the "document + data block" in the document dataset into searchable units. When a user asks a question, the retrieval enhancement layer (ragflow) can perform a precise search based on tags and dataset configuration.

[0062] The Unified Access Control Layer (UIC) can be used to define three types of access control mappings: User-Role-Tag, User-Role-DataSet, and User-Role-Doc. Access control is not limited to the data / document level but can be granular down to the tag level. It integrates with KMS / Dify to ensure that only users with the appropriate permissions can retrieve the corresponding shards.

[0063] In summary, the embodiments of the present invention introduce a tag system at the document chunk level and establish a multi-level mapping relationship between user roles and tags, datasets, and documents, thereby achieving more granular access control than existing technologies. This avoids the security deficiencies caused by traditional solutions that only support knowledge base or document-level authorization.

[0064] As a concrete example, the workflow of the above-mentioned retrieval and recall system when executing the retrieval and recall method that integrates document fragmentation and access control may include: (1) Management mode: used for permission and resource configuration, and index creation. Preferably, this step can be implemented through the following process: (11) Administrator / Operations: Configure user roles and permission mapping in UIC (configuration content: User-Role-Tag / User-Role-DataSet / User-Role-Doc). (12) UIC: Issue permission policies to the permission management module; (13) Resource Management: (131) Accessing / updating knowledge base resources; (132) The document is divided into chunks and tags are written (segment tagging); (14) Indexing service: Create / update indexes (creating an index includes: Chunk → Tag / DataSet / Doc mapping); (15) Knowledge base: Synchronize dataset and index metadata; (16) Workflow: Bind dataset key / tag key to each Agent; The aforementioned management state can be the complete process of system initialization and configuration performed by an AI application platform (such as Dify) in the management backend. Its goal is to prepare the necessary data foundation and permission rules for the secure and accurate operation of the system.

[0065] (2) Runtime: Used for precise retrieval and access control. Preferably, this step can be implemented through the following process: (21) User / Agent: Ask a question in Dify (Natural Language Query); (22) Dify: Encapsulates the retrieval request with the dataset key / tag key; (23) Indexing services: (231) Query parsing and vectorization; (232) Retrieving candidate chunk sets by dataset / tag / doc constraints; (24) Access Control: (241) Perform permission verification and filtering on the candidate set based on user identity and policy (User-Role-Tag / DataSet / Doc); (242) Determine whether the candidate is empty.

[0066] (243) If so, generate a permission denied / downgrade prompt; (244) If not, output the candidate set of permissions granted; (25) RagFlow: (251) Rearrange / merge candidate chunks (similarity + label signal + context constraint); (252) Obtain the sorted search results / evidence fragments; (26) Dify: (261) The Agent organizes the final answer based on the evidence fragments (optional: template / format constraints); (262) Returns the response to the user / Agent.

[0067] The above process describes the operation of a RAG system implemented on the Dify platform, integrating precise retrieval and fine-grained access control. Its core purpose is to ensure that user (or agent) queries not only retrieve relevant information but also strictly adhere to preset data security policies.

[0068] In summary, this embodiment of the invention divides the system into two modes: management mode and runtime mode. In the management mode, permission configuration, resource access and agent binding are performed, while in the runtime mode, retrieval and permission filtering are executed. This makes the system's operation and execution logic clearer and improves its scalability and feasibility.

[0069] As another specific example, in addition to the above-mentioned method of clearly distinguishing between management mode and runtime mode, the above embodiments can also adopt a unified service process to dynamically update permissions and indexes through API or batch scripts at runtime.

[0070] Furthermore, this embodiment of the invention limits the search scope by dataset key / tag key before retrieval, and filters candidate results again by combining user role permissions after retrieval, realizing two-layer permission control before and after retrieval, effectively improving the security and execution efficiency of retrieval. This embodiment of the invention combines tags, vector similarity, and contextual constraints, and rearranges and merges candidate fragments in the RagFlow module, so that the recall results not only meet permission constraints, but also have higher relevance and accuracy, significantly improving the problems of bias and redundancy in traditional vector retrieval results.

[0071] Based on the same inventive concept, this invention also provides a retrieval and recall system that integrates document fragmentation and access control. See [link to relevant documentation]. Figure 3 As shown, the system mainly includes the following parts: The permission management module 310 is used to receive a query request input by a user, obtain the corresponding user role based on the user's identity information, and determine the permission policy corresponding to the user role based on the pre-configured multi-level permission mapping relationship; the permission policy includes the allowed set of tags, set of datasets, and set of documents; the query request is associated with at least one dataset identifier and / or tag identifier; The retrieval module 320 is used to convert the query request into a semantic vector, and retrieve document fragments from the vector database based on the semantic vector and the dataset identifier and / or tag identifier associated with the query request to obtain an initial candidate document fragment set; wherein, the document fragments are labeled, and the vector database stores the mapping relationship between document fragments and tags, datasets, and documents; The verification module 330 is used to perform permission verification on the initial candidate document fragment set based on the permission policy, filter out document fragments that users do not have permission to access, and obtain a set of document fragments that have passed the permission verification in order to generate retrieval results.

[0072] In one embodiment, the system may further include: The result generation module is used to rearrange and merge the document fragment set that has passed the permission verification. The rearrangement is based on the semantic similarity between the document fragment and the query request, as well as the tag signal corresponding to the document fragment. The result generation module is also used to generate retrieval results based on the set of document fragments that have passed permission verification after rearrangement and fusion processing.

[0073] The system provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0074] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, such as... Figure 4 As shown, an electronic device 400 provided in this embodiment of the invention includes: a processor 410, a memory 420 and a bus. The memory 420 stores machine-readable instructions that can be executed by the processor 410. When the electronic device is running, the processor 410 communicates with the memory 420 through the bus, and the processor 410 executes the machine-readable instructions to perform the steps of the method described above.

[0075] Specifically, the memory 420 and processor 410 can be general-purpose memory and processor, without any specific limitations. When the processor 410 runs the computer program stored in the memory 420, it can execute the above method.

[0076] Processor 410 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 410 or by instructions in software form. The processor 410 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 420, and processor 410 reads the information from memory 420 and, in conjunction with its hardware, completes the steps of the above method.

[0077] Corresponding to the above method, this embodiment of the invention also provides a computer-readable storage medium storing machine-executable instructions. When the computer-executable instructions are called and run by a processor, the computer-executable instructions cause the processor to perform the steps of the above method.

[0078] The apparatus provided in this embodiment of the invention can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the apparatus provided in this embodiment of the invention are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the apparatus embodiments can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, apparatuses, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0079] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0080] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] In addition, the functional units in the embodiments provided by the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0083] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0085] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0086] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0087] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the scope of protection of the present invention.

Claims

1. A retrieval and recall method integrating document fragmentation and access control, characterized in that, The method includes: The system receives a query request input by a user, obtains the corresponding user role based on the user's identity information, and determines the permission policy corresponding to the user role based on a pre-configured multi-level permission mapping relationship; the permission policy includes a set of tags, a set of datasets, and a set of documents that are allowed to be accessed; the query request is associated with at least one dataset identifier and / or tag identifier. The query request is converted into a semantic vector, and based on the semantic vector and the dataset identifier and / or tag identifier associated with the query request, document fragments are retrieved from the vector database to obtain an initial candidate document fragment set; wherein, the document fragments are labeled, and the vector database stores the mapping relationship between document fragments and tags, datasets, and documents; Based on the permission policy, the initial candidate document fragment set is subjected to permission verification, and document fragments that the user does not have permission to access are filtered out to obtain a set of document fragments that have passed the permission verification, so as to generate retrieval results.

2. The retrieval and recall method integrating document fragmentation and access control according to claim 1, characterized in that, The method also includes a preprocessing step before receiving a query request: The documents in the knowledge base are segmented to obtain multiple document segments; At least one tag is attached to each document fragment, and a mapping relationship is established between the document fragments and tags, datasets, and documents; User roles are configured in segments, and based on the mapping relationship, a multi-level permission mapping relationship is established between the user roles and the tags, the datasets, and the documents.

3. The retrieval and recall method integrating document fragmentation and access control according to claim 2, characterized in that, The tags are generated in the following ways: automatic generation from large models, keyword extraction, manual annotation, rule engine matching, or entity mapping based on knowledge graphs.

4. The retrieval and recall method integrating document fragmentation and access control according to claim 1, characterized in that, The vector database is generated based on feature vector calculation and vectorization processing of data within the knowledge base; Based on the semantic vector retrieval of document fragments, a dense vector retrieval model is adopted. The similarity between the semantic vector and the document fragment vector is calculated and sorted. The N document fragments with the highest similarity are selected to form the initial candidate document fragment set.

5. The retrieval and recall method integrating document fragmentation and access control according to claim 1, characterized in that, The method further includes: The document fragment set that has passed the permission verification is rearranged and merged, wherein the rearrangement is based on the semantic similarity between the document fragment and the query request, and the tag signal corresponding to the document fragment; The retrieval and recall results are generated based on the document fragment set that has passed the permission verification after rearrangement and fusion processing.

6. The retrieval and recall method integrating document fragmentation and access control according to claim 1, characterized in that, The method further includes: If the set of document fragments that have passed the permission verification is empty, a degradation mechanism is triggered, and an answer is generated based on general knowledge and a prompt is given to the user.

7. A retrieval and recall system integrating document fragmentation and access control, characterized in that, The system includes: The permission management module is used to receive a query request input by a user, obtain the corresponding user role based on the user's identity information, and determine the permission policy corresponding to the user role based on a pre-configured multi-level permission mapping relationship; the permission policy includes a set of tags, a set of datasets, and a set of documents that are allowed to be accessed; the query request is associated with at least one dataset identifier and / or tag identifier; The retrieval module is used to convert the query request into a semantic vector, and retrieve document fragments from the vector database based on the semantic vector and the dataset identifier and / or tag identifier associated with the query request to obtain an initial candidate document fragment set; wherein, the document fragments are labeled, and the vector database stores the mapping relationship between document fragments and tags, datasets, and documents; The verification module is used to perform permission verification on the initial candidate document fragment set based on the permission policy, filter out document fragments that the user does not have permission to access, and obtain a set of document fragments that have passed the permission verification in order to generate retrieval results.

8. The retrieval and recall system integrating document fragmentation and access control according to claim 7, characterized in that, The system also includes: The result generation module is used to perform rearrangement and fusion processing on the document fragment set that has passed the permission verification, wherein the rearrangement is based on the semantic similarity between the document fragment and the query request, and the tag signal corresponding to the document fragment; The result generation module is also used to generate retrieval results based on the set of document fragments that have passed permission verification after rearrangement and fusion processing.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images