Large model corpus isolating and sorting method based on user permission

Abstract

The invention relates to a large model corpus isolating and sorting method based on user permission. The method is composed of five core parts including an isolation dimension management center, a knowledge base, a corpus, a corpus isolation method and a corpus sorting method, and aims to output corpora to a large model based on user permission. The isolation dimension management center is connected with the knowledge base through a dimension configuration channel and transmits an isolation dimension strategy based on user permission; the knowledge base is connected with the corpus through a knowledge conversion channel, and the knowledge subjected to authority isolation is converted into vectorized corpora; according to the corpus isolation method, a first isolation control channel is connected with a corpus, and access isolation control is performed on the corpus according to an isolation dimension strategy; according to the corpus sorting method, a second sorting control channel is connected with a corpus, and the corpus is screened and output according to an isolation dimension strategy. According to the method, authority supply and dynamic security isolation of corpora are realized, data security and access accuracy are effectively improved, and compliance and controllability of large model application are guaranteed.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and data security technology. Specifically, it relates to a method for isolating and sorting large model corpora based on user permissions, which is particularly suitable for situations where fine-grained access control and security management are required for corpora used in training or inference of large-scale language models. Background Technology

[0002] With the rapid development and widespread application of large-scale language model technology, the corpora relied upon for training and inference are becoming increasingly massive and complex in origin. In many enterprise-level and specialized application scenarios, such as finance, healthcare, law, and government, corpus data often contains a large amount of sensitive information, professional knowledge, or internally controlled data. Directly feeding all corpora into large models can lead to serious data security and compliance risks, such as the leakage of sensitive information, intellectual property infringement, and model output exceeding the scope of user authorization. Traditional corpus processing methods typically involve uniformly processing the collected data and using it for model training or fine-tuning, lacking fine-grained access control and isolation mechanisms based on user identity and permissions. This extensive management approach cannot meet the high standards of data security required in application scenarios, constituting one of the main obstacles to the secure implementation of large-scale models.

[0003] To address the issues of data security and access control for large-scale models, existing technologies have proposed several solutions. Some solutions involve building multiple independent, dedicated models to handle tasks with different security levels or domains, essentially training completely independent models for data with different permission levels or business lines. While this approach achieves physical isolation, it leads to high model maintenance costs, low resource utilization, and an inability to share common underlying language capabilities. Other solutions focus on intervention during the model's inference output stage, such as using post-processing filters or content security policies to restrict the model's responses and prevent the output of unauthorized information. However, these solutions are post-hoc remedial measures and cannot fundamentally prevent the model from accessing corpora it should not have permission to view during training or inference, resulting in fragile security boundaries and potentially affecting the coherence and usefulness of the model's output. Still other solutions attempt coarse-grained classification during data preprocessing, but these are mostly based on content themes or data types rather than directly linked to specific user permission systems, making it difficult to achieve dynamic and precise control over corpus supply.

[0004] In summary, existing technologies have significant shortcomings in achieving refined large-scale model corpora and permission-driven dynamic isolation and sorting. They either sacrifice efficiency and flexibility or fail to build robust security boundaries at the source. Therefore, there is an urgent need for a method that can tightly couple with the user permission system to achieve dynamic isolation and intelligent sorting at the corpus level, thereby ensuring that large models only learn and use corpora within authorized limits, and building a solid foundation for the secure and controllable operation of models from the data source. This invention aims to solve the above problems by designing a user permission-based corpus isolation and sorting mechanism to achieve on-demand supply and secure isolation of corpora. This has significant practical value and social benefits for promoting the secure and compliant application of large models in sensitive areas, protecting data assets, and meeting privacy regulations. Summary of the Invention

[0005] To address these issues, this invention provides a method for isolating and sorting large model corpora based on user permissions.

[0006] This invention is achieved through the following technical solutions: 1. A method for isolating and sorting a large model corpus based on user permissions, consisting of five core parts: an isolation dimension management center (1), a knowledge base (2), a corpus (3), a corpus isolation method (4), and a corpus sorting method (5), which realizes the output of corpus to a large model based on user permissions. The method is characterized in that: the isolation dimension management center (1) is connected to the knowledge base (2) through a dimension configuration channel (6) to transmit the isolation dimension strategy based on user permissions; the knowledge base (2) is connected to the corpus (3) through a knowledge transformation channel (7) to transform the knowledge isolated by permissions into vectorized corpus; the corpus isolation method (4) is connected to the corpus (3) through an isolation control channel (8) to perform access isolation control on the corpus according to the isolation dimension strategy; the corpus sorting method (5) is connected to the corpus (3) through a sorting control channel (9) to filter and output the corpus according to the isolation dimension strategy. The isolation dimension management center (1) as described in claim 1 is responsible for managing user permission dimensions and policies, characterized in that: it includes a permission definition unit (10) and a policy configuration unit (11), wherein the permission definition unit (10) and the policy configuration unit (11) are directly connected; the policy configuration unit (11) outputs the isolation dimension policy to the knowledge base (2), the corpus isolation method (4) and the corpus sorting method (5) through the dimension configuration channel (6).

[0007] The knowledge base (2) as described in claim 1 is responsible for storing original knowledge and implementing dimension-based knowledge isolation. It is characterized in that it includes a knowledge storage unit (12) and a permission filtering unit (13). The knowledge storage unit (12) is directly connected to the permission filtering unit (13). The permission filtering unit (13) outputs the permission-filtered knowledge content to the corpus (3) through the knowledge transformation channel (7).

[0008] The corpus (3) as described in claim 1 is responsible for storing vectorized corpus and providing an access interface, characterized in that: it includes a vector storage unit (14) and a corpus interface unit (15), wherein the vector storage unit (14) is directly connected to the corpus interface unit (15); the corpus interface unit (15) is connected to the corpus isolation method (4) and the corpus sorting method (5) respectively through the isolation control channel (8) and the sorting control channel (9).

[0009] The corpus isolation method (4) as described in claim 1 is responsible for implementing access control for corpus access, characterized in that: it includes an access control unit (16) and an access verification unit (17), wherein the access control unit (16) and the access verification unit (17) are directly connected; the access control unit (16) is connected to the corpus (3) through an isolation control channel (8) to control the user's access permissions to the corpus.

[0010] The corpus sorting method (5) as described in claim 1 is responsible for sorting and outputting corresponding corpora according to user permissions. It is characterized in that: it includes a corpus filtering unit (18) and an output control unit (19). The corpus filtering unit (18) is directly connected to the output control unit (19). The corpus filtering unit (18) is connected to the corpus (3) through the sorting control channel (9) to filter corpus content that meets the user permissions.

[0011] The permission definition unit (10) as described in claim 2 is responsible for defining the user permission level and scope, characterized in that: user permission definition information is output to the policy configuration unit (11) through the dimension configuration channel (6).

[0012] The permission filtering unit (13) as described in claim 3 is responsible for filtering knowledge content according to the permission policy. Its feature is that it receives permission filtering instructions through the knowledge transformation channel (7) and outputs permission-filtered knowledge vectors to the corpus (3).

[0013] The access control unit (16) as described in claim 5 is responsible for controlling user access to the corpus, characterized in that: it receives user access requests through the isolation control channel (8) and controls access permissions according to the verification result of the permission verification unit (17).

[0014] The corpus filtering unit (18) as described in claim 6 is responsible for filtering corpora that meet user permissions, characterized in that: it receives user permission information through the sorting control channel (9) and filters out corpus content with the corresponding permission level from the corpus (3).

[0015] Compared with traditional methods of managing and using large model corpora, the user-permission-based method for isolating and sorting large model corpora proposed in this invention has the following advantages: 1. Achieve refined management of corpus access, solidifying the foundation of large-scale model data security. Traditional large-scale model training and inference typically use hybrid corpora, lacking access control based on user identity and permissions, posing risks of sensitive information leakage and unauthorized access. This invention standardizes user permissions and organizational structure modeling through an isolation dimension management center, and uses this as the core to drive dual dynamic isolation of the corpus at the knowledge base and corpus levels. This is equivalent to building a dedicated knowledge base for each user or user group, fundamentally ensuring that large-scale models only access and use the corpus within authorized limits, constructing a robust security barrier from the data source, and meeting the stringent requirements for data compliance and privacy protection in highly sensitive fields such as finance, government affairs, and healthcare.

[0016] 2. Improve the accuracy and efficiency of corpus supply and optimize the output quality of large models. Existing methods struggle to quickly and accurately filter the most relevant and authorized information from massive corpora based on specific tasks and user context. The corpus sorting method proposed in this invention, based on a unified isolation dimension, can intelligently and in real-time sort the optimal corpus set matching the current permissions and context from isolated corpus subsets when a user initiates a query or the model performs inference. This not only avoids interference from irrelevant corpora, making the output of large models more accurate, reliable, and professional, but also improves the overall response efficiency of the system by reducing unnecessary corpus retrieval scope.

[0017] 3. Enhance system flexibility and scalability to adapt to complex and ever-changing business scenarios. An enterprise's permission system and organizational structure dynamically adjust as business develops. This invention uses isolation dimensions as the core management unit, supporting dynamic addition, deletion, modification, and querying of dimensions. When a new business line is added, department permissions are adjusted, or a new security category is introduced, configuration in the isolation dimension management center is all that's needed to automatically map it to the isolation logic of the knowledge base and the sorting strategy of the corpus, without requiring a reconstruction of the underlying data or model architecture. This design allows the system to easily adapt to the continuous evolution of organizational structure and permission models, possessing high flexibility and scalability.

[0018] 4. Ensuring high compatibility and ease of integration, lowering the deployment and implementation threshold. This invention does not rely on a specific large-scale model architecture. The knowledge base can be various structured or unstructured knowledge management systems, and the corpus is compatible with mainstream vector databases. The core isolation and sorting methods are provided as independent services and can be integrated with systems through standard APIs. This means that enterprises can upgrade based on their existing technology stack without replacing the core large-scale model or storage facilities, significantly reducing the complexity of technology integration and deployment costs, which is conducive to the rapid implementation and promotion of the solution.

[0019] 5. Supports auditing and source tracing, meeting compliance and regulatory requirements. Traditional mixed corpus models struggle to trace which training data a specific output originates from. This invention, through permission-based corpus isolation and sorting, naturally establishes a traceable link between the user, the permission dimension, and the subset of corpus used. The system can fully record the isolation dimension associated with each model call, providing clear and reliable log evidence for compliance review of model output, data leakage tracing, and algorithm auditing, effectively supporting the compliance requirements of various industries and regulations. Attached Figure Description

[0020] Figure 1 A detailed description of the implementation process for the knowledge base module. Figure 2 Detailed implementation process of the corpus module Figure 3 A detailed explanation of the implementation process for corpus isolation methods. Figure 4 A detailed description of the specific implementation process for corpus sorting. Detailed Implementation

[0021] The specific embodiments of the present invention will now be described clearly and in detail with reference to the accompanying drawings. The described embodiments are merely a part of the embodiments of the present invention.

[0022] The technical solution of this invention to solve the above-mentioned technical problems is: a method for isolating and sorting a large model corpus based on user permissions. This invention includes: an isolation dimension management center, a knowledge base, a corpus, a corpus isolation method, and a corpus sorting method.

[0023] I) Isolation Dimension Management Center The isolation dimension management center is the control core of the entire method, responsible for defining, managing, and mapping user permissions to specific corpus isolation and sorting dimensions. Specific implementation steps include: 1. Dimension Definition and Management: Based on the organizational structure and security policy, create and maintain a set of standardized isolated dimensions, supporting dynamic addition, deletion, modification, and query operations for dimensions; 2. Permission-Dimension Mapping: Establishes a mapping relationship between users or user groups and one or more isolation dimensions. When a user initiates a request, the center can quickly resolve all valid isolation dimension sets associated with that user; 3. Strategy Distribution: The parsed user isolation dimension set is used as the core control parameter and distributed in real time to the corpus isolation module and the corpus sorting module as the input basis for their execution logic.

[0024] II) Knowledge Base A knowledge base is a unit for storing and managing raw structured or unstructured knowledge, and it is pre-isolated logically or physically based on isolation dimensions. The specific implementation is as follows: 1. Knowledge entry and dimension tagging: When raw knowledge is entered into the database, it is tagged with one or more isolated dimension tags; 2. Knowledge Logic Isolation: At the storage or indexing level of the knowledge base, establish indexes or partitions based on isolation dimension tags to form clear logical boundaries; 3. Knowledge maintenance and synchronization: When the definition of isolated dimensions or user permissions change, the knowledge-dimension mapping relationship in the knowledge base is adjusted accordingly.

[0025] (iii) Corpus A corpus is a storage format for knowledge base content after vectorization. It is specifically designed to support semantic retrieval and understanding of large models, and its structure inherits the isolation logic of a knowledge base. The specific workflow is as follows: 1. Vectorization: Using a pre-trained embedding model, the knowledge content in the knowledge base that has been dimensionally labeled is transformed into a high-dimensional vector representation; 2. Vector storage and dimension association: The generated vectors are stored in a vector database, and each vector data is strongly associated with the isolated dimension label of the source knowledge in the knowledge base; 3. Vector library structure organization: Based on the isolation dimension, establish corresponding partitions or sets in the vector library to realize the physical or logical grouping of vector data.

[0026] (iv) Corpus Isolation Methods This method is a key step in implementing access control, ensuring that when supplying large models, access is limited to vectorized corpora within the authorized scope. The specific steps are as follows: 1. Receiving Permission Context: When the large model service receives a user request, it retrieves the user's current "set of valid isolation dimensions" from the "Isolation Dimension Management Center"; 2. Generate query boundaries: Based on the effective isolation dimension set, dynamically generate a query range in the corpus to limit the query to those vector partitions or sets in the vector metadata whose dimension labels meet the permission requirements; 3. Implement isolated queries: When performing retrieval in the vector database, the generated query boundary is used as a mandatory filtering condition to ensure that unauthorized vector data will not be retrieved and returned, thereby achieving isolation at the source of corpus retrieval.

[0027] (v) Corpus sorting methods This method, based on isolation, further intelligently filters the subset of corpus most relevant to the current request from the authorized corpus, thereby optimizing input quality. The specific implementation is as follows: 1. Initial screening of semantic relevance: Within the defined authorized corpus, using the user's query content as input, the most semantically relevant vectorized corpus fragments are initially recalled through vector similarity calculation; 2. Multi-dimensional weighted ranking: The initially recalled corpus fragments are comprehensively weighted and ranked by combining their semantic similarity scores and the matching degree between their associated isolation dimensions and the current user's permission context. 3. Generate the final corpus: Based on the weighted ranking results, select the top-ranked corpus and use it as the final corpus after permission filtering and relevance optimization. Output it to the large model for further processing.

Claims

1. A method for isolating and sorting a large model corpus based on user permissions, comprising five core parts: an isolation dimension management center (1), a knowledge base (2), a corpus (3), a corpus isolation method (4), and a corpus sorting method (5), which realizes the output of corpus to a large model based on user permissions, characterized in that: The isolation dimension management center (1) is connected to the knowledge base (2) through the dimension configuration channel (6) to transmit the isolation dimension strategy based on user permissions; The knowledge base (2) is connected to the corpus (3) through the knowledge transformation channel (7) to transform the knowledge that has been isolated by permissions into vectorized corpus; The corpus isolation method (4) is connected to the corpus (3) through the first isolation control channel (8) and performs access isolation control on the corpus according to the isolation dimension strategy; The corpus sorting method (5) is connected to the corpus (3) through the second sorting control channel (9) and filters and outputs the corpus according to the isolation dimension strategy.

2. The isolation dimension management center (1) as described in claim 1, responsible for managing user permission dimensions and policies, characterized in that: It includes a permission definition unit (10) and a policy configuration unit (11), wherein the permission definition unit (10) and the policy configuration unit (11) are directly connected; The strategy configuration unit (11) outputs the isolation dimension strategy to the knowledge base (2), the corpus isolation method (4) and the corpus sorting method (5) through the dimension configuration channel (6).

3. The knowledge base (2) as described in claim 1, responsible for storing original knowledge and implementing dimension-based knowledge isolation, characterized in that: It includes a knowledge storage unit (12) and a permission filtering unit (13), wherein the knowledge storage unit (12) and the permission filtering unit (13) are directly connected; The permission filtering unit (13) outputs the permission-filtered knowledge content to the corpus (3) through the knowledge conversion channel (7).

4. The corpus (3) as described in claim 1, responsible for storing vectorized corpora and providing access interfaces, characterized in that: It includes a vector storage unit (14) and a corpus interface unit (15), wherein the vector storage unit (14) is directly connected to the corpus interface unit (15); The corpus interface unit (15) is connected to the corpus isolation method (4) and the corpus sorting method (5) through the first isolation control channel (8) and the second sorting control channel (9), respectively.

5. The corpus isolation method (4) as described in claim 1, responsible for implementing access control for corpus access, characterized in that: It includes an access control unit (16) and a permission verification unit (17), wherein the access control unit (16) and the permission verification unit (17) are directly connected; The access control unit (16) is connected to the corpus (3) through the first isolation control channel (8) to control the user's access permissions to the corpus.

6. The corpus sorting method (5) as described in claim 1, responsible for sorting and outputting corresponding corpora according to user permissions, characterized in that: It includes a corpus filtering unit (18) and an output control unit (19), wherein the corpus filtering unit (18) and the output control unit (19) are directly connected; The corpus filtering unit (18) is connected to the corpus (3) through the second sorting control channel (9) to filter corpus content that meets the user's permissions.

7. The permission definition unit (10) as described in claim 2, responsible for defining user permission levels and scope, characterized in that: User permission definition information is output to the policy configuration unit (11) through the dimension configuration channel (6).

8. The permission filtering unit (13) as described in claim 3, responsible for filtering knowledge content according to the permission policy, characterized in that: The knowledge transformation channel (7) receives permission filtering instructions and outputs permission-filtered knowledge vectors to the corpus (3).

9. The access control unit (16) as described in claim 5, responsible for controlling user access to the corpus, characterized in that: The first isolation control channel (8) receives user access requests and controls access permissions based on the verification result of the permission verification unit (17).

10. The corpus filtering unit (18) as described in claim 6, responsible for filtering corpora that conform to user permissions, characterized in that: The second sorting control channel (9) receives user permission information and filters out the corresponding permission level of corpus content from the corpus (3).

Images