Document large-scale model construction method

Abstract

The invention relates to an official document large model construction method, and belongs to the field of artificial intelligence. According to the method, accurate format adaptability and content depth extraction are achieved for official documents of different types, and all analyzed data can be converted into a unified format to be output for official document large model fine adjustment; the client adopts a low-rank adaptive LoRA technology, selects Qwen3-8B as a base model, and performs efficient fine tuning by using local official document data on the basis of the base model; the server introduces RBAC (Role Based Access Control) of a role, constructs an encryption mechanism of an adaptive hierarchy, and ensures that the permission is matched with the data visibility; the server adopts a differential privacy enhancement-adaptive weight aggregation architecture, and integrates local fine tuning parameters encrypted and transmitted by all departments into global model parameters through dynamic noise injection and multi-dimensional weight calibration. According to the method, the contradiction between government affair data security and cooperative training is cracked, the suitability of the document large model field is improved, and the government affair scene landing threshold is reduced.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a method for constructing a large model of official documents. Background Technology

[0002] Official documents, as the core carrier of government information transmission and decision-making implementation, are characterized by strong format standardization, dense domain terminology, and high data sensitivity. Currently, the construction of large-scale official document models generally faces two major bottlenecks: first, the problem of data silos, where official document data from various functional departments cannot be aggregated across domains due to confidentiality requirements, resulting in insufficient multi-source data support for model training; second, privacy and security risks, as traditional centralized training requires the transmission of original official document texts, which can easily lead to the leakage of classified information.

[0003] Federated learning employs a decentralized training paradigm, allowing multiple participants (clients) to collaboratively train a global model without sharing their local raw data. Each client trains its model locally using its own data and then sends the calculated model updates (such as gradients or weight changes) to a central server. The central server aggregates these updates to improve the global model and distributes the updated model back to the clients. This process is repeated until the model converges. In this way, federated learning not only protects data privacy but also enables the use of broader and more diverse datasets, potentially improving the performance, generalization ability, and fairness of large models.

[0004] Federated learning, with its characteristics of "data remaining local and parameters flowing across domains," offers a solution to the challenges of building large-scale models of official documents. However, existing research still has limitations: some government federated learning methods focus on multimodal feature fusion or recommendation tasks, failing to optimize the fine-tuning process for the structured characteristics of official document texts; while some studies combine parameter-efficient fine-tuning (PEFT) with federated learning, they haven't addressed the heterogeneity of fine-tuning parameters caused by differences in document data formats; most importantly, official documents have multi-level confidentiality requirements, categorized according to their level of secrecy, with varying degrees of accessibility depending on departmental hierarchy. Existing solutions often employ a single encryption method for parameter transmission, making it difficult to balance security and robustness. Therefore, a method for building large-scale models that adapts to the characteristics of official documents and balances security and performance is urgently needed. Summary of the Invention

[0005] (a) Technical problems to be solved The technical problem this invention aims to solve is how to provide a method for constructing a large-scale official document model, so as to resolve the core contradiction between "data silos" and strict privacy protection requirements when constructing a large-scale official document model.

[0006] (II) Technical Solution To address the aforementioned technical problems, this invention proposes a method for constructing a large-scale official document model, which includes the following steps: S1. For official documents of different document types, it achieves precise format adaptation and in-depth content extraction. All parsed data will be converted into a unified JSON-LD format for output. S2. The client adopts low-rank adaptive LoRA technology and selects Qwen3-8B as the base model. Based on the Qwen3-8B model, it uses local document data for efficient fine-tuning. The LoRA adapter parameters trained by each client on proprietary document data are transmitted to the central server for aggregation and iterative updates after differential privacy protection and encryption. S3. The server introduces role-based access control (RBAC) and builds an encryption mechanism that adapts to different levels to ensure that permissions match data visibility. S4. The server adopts a differential privacy enhancement-adaptive weight aggregation architecture, which integrates the locally fine-tuned parameters transmitted by each department into global model parameters through dynamic noise injection and multi-dimensional weight calibration.

[0007] (III) Beneficial Effects This invention proposes a method for constructing a large-scale official document model. Addressing the core pain points in existing large-scale official document model construction methods, this invention achieves innovative breakthroughs by combining federated learning and efficient parameter fine-tuning techniques, with the following beneficial effects: 1. Resolving the conflict between government data security and collaborative training: This addresses the privacy risks caused by cross-domain data transfer in traditional centralized training, as well as the problem of data silos between departments hindering collaboration. Through a federated architecture that ensures "data stays within its domain and parameters flow securely," coupled with hierarchical encryption and differential privacy protection, multi-department collaboration is achieved without disclosing original official document data, making it more suitable for the confidentiality requirements of government data than existing technologies. 2. Enhance the domain adaptability of the official document model: Overcome the shortcomings of general-purpose models in learning the characteristics of official document types, format specifications, and specific terminology. Through targeted designs such as document type embedding, structured parsing, and terminology-enhanced loss functions, combined with an adaptive weight aggregation strategy, the model accurately captures the domain characteristics of official documents, making it more suitable for government business scenarios than existing general fine-tuning methods; 3. Lowering the barriers to implementation in government scenarios: It solves the problems of high computational consumption during training of existing large-scale models, high parameter transmission costs, and difficulty in adapting to the resource differences among government departments. Leveraging LoRA's lightweight fine-tuning significantly reduces communication overhead and computational burden in the federated learning process. Its adaptive aggregation strategy effectively balances the differences in data contributions and quality among different departments, improving training efficiency and model convergence speed under non-independent and identically distributed data. This enhances the system's scalability and practical deployment feasibility, providing an efficient and practical solution for large-scale cross-departmental collaboration. Attached Figure Description

[0008] Figure 1 This is a diagram of the federated fine-tuning architecture of the large-scale document model of this invention; Figure 2 This is a flowchart illustrating the construction process of the large document model of the present invention. Detailed Implementation

[0009] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0010] This patent aims to resolve the core contradiction between "data silos" and strict privacy protection requirements when building large-scale models of official documents in the government sector. Under the current system and regulations, official document data from various departments cannot be centrally shared due to the presence of sensitive information, making it difficult for general-purpose large-scale models to obtain high-quality, domain-specific training, resulting in problems such as non-standard formats and inaccurate terminology. While traditional federated learning can achieve "data not leaving the local machine," its general architecture has limitations in the context of official documents: on the one hand, directly transmitting model parameters still carries the risk of privacy leakage (e.g., back-calculating raw data through gradients); on the other hand, standard aggregation strategies are difficult to effectively integrate the differentiated knowledge of various departments in terms of document types, formats, and professional terminology, leading to insufficient understanding and generation quality of official document writing norms in the jointly trained model. This patent introduces differential privacy-enhanced LoRA (Low-Rank Adaptation) fine-tuning, adaptive weight aggregation based on document characteristics, and hierarchical encryption strategies. While ensuring absolute data locality and providing provable privacy protection, it significantly improves the accuracy of federated large-scale models in adapting to document formats, terminology, and department-specific needs, ultimately achieving the construction of secure, adaptable, efficient, and compliant large-scale models of official documents.

[0011] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0012] 1. Data Preprocessing The local data preprocessing module is the foundation and starting point for building the federated document model. Its core task is to address the heterogeneity of document data from various departments and transform it into high-quality, structured training samples. This module deeply integrates professional knowledge in the field of official documents, and through a multi-level, automated processing workflow, ensures that the raw data meets the standardization and consistency requirements for fine-tuning the large model while preserving complete semantic and format elements, thus laying a solid data foundation for subsequent federated fine-tuning.

[0013] 1.1 Document Structure Parsing Mechanism This mechanism employs a hybrid parsing strategy of "rule-guided and model-assisted" parsing, achieving precise format adaptation and in-depth content extraction for different document types such as requests, reports, and notices. Its core technology lies in deconstructing unstructured raw text into standardized data objects that machines can deeply understand and manipulate. The specific process is as follows: (1) Document type adaptive classification: The input official documents are identified by a government document type classifier to provide a basis for subsequent parsing template calls. The classifier is built based on a fine-tuned BERT-base model. The training data covers 15 types of standard documents (requests, reports, notices, letters, minutes, etc.) and 3 types of high-frequency non-standard documents (work plans, situation descriptions, briefings). The classification logic is optimized by introducing document type-specific features. For example, request documents often contain the ending "Please approve or disapprove" and notice documents often contain the guiding phrase "The following is hereby notified".

[0014] (2) Title and document number: By combining named entity recognition and high-precision rule matching based on regular expressions, various common variants are effectively identified and normalized. By analyzing their relative positional relationship with the document number (such as “XX Regulation

[2025] No. X”), the lossless extraction of key identification information is achieved, ensuring high recall and high accuracy.

[0015] (3) Classification and Confidentiality Period: Based on the location characteristics of the classification markings on government documents (usually located in the upper left corner of the title or the upper right corner of the first page), candidate areas are first filtered by coordinate positioning, and then matched with the classification keyword library and the term description. This mechanism also introduces a multi-level verification process to check whether the classification descriptions mentioned in the text are consistent with the declaration in the document header, effectively avoiding misjudging the classification information quoted in the text but not actually in the document as the document attribute, thereby significantly improving the reliability of the extraction results.

[0016] (4) Issuing and Receiving Authorities: Utilizing dictionary-based entity recognition and grammatical rule analysis. By incorporating the full names, standardized abbreviations, and common variant names of various levels of institutions and departments, joint determination can be made in common locations in official documents. For example, issuing authorities are often located at the end of the document, while receiving authorities are often located at the beginning of the main text as the primary recipient. This accurately identifies and standardizes institutional entities, providing support for subsequent analysis of document circulation relationships.

[0017] (5) Main text structure and content: Considering that the length of general official documents is usually within the effective context window of the mainstream large model (the federated official document large model of this method is based on the Qwen3-8B model, which supports a context length of 32K tokens, approximately 25,000 words, which is sufficient to handle the vast majority of official documents), the main text content is usually retained in its entirety after basic cleaning (such as removing irrelevant formatting symbols). At the same time, the parsing mechanism will identify and record the internal logical structure of the official document, such as "summary-based" and "level-by-level headings". This structural information can be used as metadata to embed prompt word templates, guiding the federated official document large model to better understand the document logic.

[0018] 1.2 Data Quality Control and Standardized Output To ensure consistent data quality across nodes in federated learning, the preprocessing module integrates a rigorous quality control process. All parsed data is converted to a unified JSON-LD (JSON for Linked Data) format for output. This format not only includes all extracted structured fields but also retains a copy of the original text, along with metadata such as language and parsing confidence level. This design facilitates automated processing in subsequent workflows and provides convenience for data traceability and quality auditing.

[0019] 2. LoRA-based parameter fine-tuning module The parameter fine-tuning module is the core technology component of this patent. It employs low-rank adaptive LoRA technology, selecting Qwen3-8B as the base model, and efficiently fine-tunes it using local document data. Compared to full fine-tuning, LoRA requires training only a very small number of parameters, significantly reducing computational, storage, and communication overhead, making it particularly suitable for federated learning environments. Each client uses LoRA technology on its own proprietary document dataset to fine-tune the shared large model base, specifically including: 2.1 Language Embedding Mechanism Unit Adding document type embeddings to the beginning of the input text sequence enhances the model's ability to recognize different document types. Specifically, special document type markers are added to the beginning of the input text sequence in the format [TYPE:document type name], such as [TYPE:request], [TYPE:report], etc. The concatenated new sequence is then input into the large model. Since the added special document type markers are all from the original token dictionary, they can be directly mapped to word embedding vectors without additional training. Subsequently, during training, they are fused with the text content through an attention mechanism. 2.2 LoRA Adapter Initialization Unit This method loads a common base model shared by all clients locally. Qwen3-8B is chosen as the base model. During fine-tuning based on proprietary document data on each client, the parameters of the bottom 33 layers are frozen, and only the top 3 layers are fine-tuned. Specifically, LoRA adaptation matrices A and B are initialized. Matrix A uses Kaiming initialization (an initialization method conforming to a random Gaussian distribution), and matrix B is initialized to zero. Initializing matrix B to zero ensures that the LoRA adapter output is zero at the start of training, consistent with the behavior of the original model. The non-zero nature of matrix A ensures that the partial derivatives of matrix B are non-zero during backpropagation, allowing matrix B to obtain effective gradients and begin updating. From the second iteration onwards, matrix B is non-zero, resulting in non-zero partial derivatives of matrix A during backpropagation, allowing matrix A to obtain effective gradients and begin updating. 2.3 Terminology Enhancement Weighted Loss Function This method employs a loss function that incorporates terminology enhancement weights to strengthen the learning of high-frequency official document terms specific to each department. The weighted loss function is implemented as follows:

[0020] in, The standard cross-entropy loss, which is the original loss function of the large model, is obtained by calculating the cross-entropy loss between the one-hot label of the real token and the probability of predicting the next token. To balance hyperparameters, Term-weighted loss:

[0021] in Given input Post-generation The probability, Given the length of the input text, output the length of the first line in the text. The terminology for each position corresponds to a token. Terminology weight Calculate using the following steps: Perform word frequency statistics on local official document data and extract words with frequencies exceeding a threshold. For candidate terms, calculate term importance scores:

[0022] in, For the term The frequency of a term appearing in local departmental document data reflects its common usage in current departmental documents. For example, assuming a department has 10 documents with a total of 5000 words, and the term "budget arrangement" appears 50 times, then: TF("budget arrangement") = 50 / 5000 = 0.01; For the term Inverse document frequency, which is used to measure the term The general importance of the term, whether it appears in most documents. For example, suppose a department has 99 documents, and the term "budget arrangement" appears in 49 of them, then: IDF("budget arrangement") = log((99+1) / (49+1)) + 1 = log(2) + 1 ≈ 1.3010299; For the term The degree of concentration within a specific department reflects the departmental specificity of the term. This value should be calculated by having each client collect its high-frequency words and send them to the central server before model training begins. The aggregated words are then distributed to each client for word frequency statistics. For example, if the term "budget arrangement" has a frequency of 0.01 in the current department and an average frequency of 0.001 in other departments, then: DS("budget arrangement") ≈ 0.01 / 0.001 = 10. Substituting these values ​​into the formula, the importance score for the term "budget arrangement" can be obtained as follows: 1.3010299 * 10 = 1.3010299.

[0023] The formula for calculating term weight is:

[0024] in This is the scaling factor. For all terms importance The minimum value, It is its maximum value. If the th in the sequence Each position is a term Then the weight of the corresponding position For the calculation Otherwise, the weight is 1. When encountering high-frequency terms with multiple characters, the weight of each character is set to 1. .

[0025] 2.4 Local Training Cycles and Early Stopping For each round of local training, a mini-batch of samples is sampled from the preprocessed document data, and the forward propagation loss and backpropagation gradient are calculated. The loss function is: The LoRA adapter parameters are updated using the optimizer (Adam), and the performance of the current LoRA adapter is evaluated on the local validation set. If the performance no longer improves, an early stopping mechanism is triggered to avoid overfitting. The LoRA adapter parameters trained by each client on proprietary document data are encrypted and differentially privacy protected before being transmitted to the central server for aggregation and iterative updates.

[0026] 3. Hierarchical encryption mechanism: Encrypting parameter transmission In the cross-domain parameter transfer process of the federal document big data model, a hierarchical encryption transmission mechanism is the core technical support for ensuring "data does not leave the domain and parameters are transferred securely." In government scenarios, different types of fine-tuning parameters contain varying densities of sensitive information, and therefore, differentiated encryption strategies should be adopted to ensure parameter privacy while also considering the efficiency of federal aggregation and the compliance of government authority management.

[0027] 3.1 Adapter Parameter Encryption Unit The LoRA adapter parameters (low-rank matrices A and B) are personalized parameters trained by each department for specific document scenarios. They are strongly bound to departmental business operations and require strict restrictions on cross-permission access. This unit introduces Role-Based Access Control (RBAC) to build an encryption mechanism adapted to the government hierarchy, ensuring that "permissions and data visibility match."

[0028] The server stores aggregated global adapter parameters and domain adapter parameters from various departments, each with access permission tags. When a user (department) requests to use a model, the system dynamically loads the corresponding adapter parameters from storage based on the user's authorized data domain scope, merges them with the base model parameters, and generates a custom model tailored for that user.

[0029] This enables strict data access control at the model level, meaning users can only access the model capabilities corresponding to the data domains they are authorized to access, and cannot access data knowledge in other unauthorized domains. In cross-departmental collaboration scenarios, departments with higher privileges can unlock more adapter parameters, promoting knowledge sharing while maintaining data sovereignty boundaries.

[0030] 3.2 Training Metadata Encryption Unit While training metadata (including local training epochs, loss values, and data volume statistics) does not directly contain official document content, it may be used to infer local data distribution, such as using changes in loss values ​​to estimate data size; this is considered low-sensitivity information. This unit employs the AES-256 symmetric encryption scheme, prioritizing transmission efficiency while ensuring basic privacy. Specific content includes: (1) Dynamic key generation and distribution: The central server generates a unique AES session key (256 bits) for each round of federated training and distributes it through a "one-time pad" mechanism: the key is encrypted with the public key of each department node and transmitted separately through a secure channel to avoid the risk of key reuse. The AES session key automatically expires after the parameter aggregation of this round and is regenerated in the next round of training to prevent the risk of batch information leakage caused by long-term key leakage.

[0031] (2) Lightweight encryption of metadata: For the structured characteristics of metadata, which are mostly integers or floating-point numbers, the “field-level encryption + overall verification” mode is adopted: each field (such as “loss value = 0.023”) is encrypted separately, and an overall message verification code (HMAC-SHA256) is generated at the same time, which not only ensures the privacy of the fields, but also quickly verifies the integrity of the transmission.

[0032] 4. Security aggregation submodule based on differential privacy The secure aggregation mechanism is a core component in building the federated document big data model, undertaking the crucial task of fusing encrypted local fine-tuning parameters (including LoRA adapter parameters and gradient information) transmitted by various departments into global model parameters. In government scenarios, this mechanism must simultaneously meet two core requirements: first, strictly protect the data privacy of each department, avoiding the reverse inference of local document content through aggregation parameters; second, ensure that the aggregated global model can fully absorb the high-quality features of multiple departments and adapt to the needs of cross-domain document processing. To this end, this mechanism designs a two-level architecture of "differential privacy enhancement - adaptive weight aggregation," achieving a precise balance between privacy security and model performance through dynamic noise injection and multi-dimensional weight calibration.

[0033] 4.1 Differential Privacy Enhancement Unit: Dynamic Noise Injection Strategy Based on Data Characteristics The differential privacy enhancement unit injects calibrated noise into the client's local parameters to ensure that the presence or absence of a single document sample does not significantly affect the global aggregation result, fundamentally resisting "membership inference attacks," which infer whether a specific document participated in training based on aggregation parameters. The core design of this unit revolves around "adaptive noise scale calibration" and "full lifecycle privacy budget management," specifically implemented as follows: (1) Gaussian noise injection: LoRA adapter parameters (low-rank matrix A) of each department (client) i B i Before uploading, a Gaussian noise injection mechanism is used, and the noise distribution satisfies... ,in Let the noise standard deviation of department i be denoted by the following formula:

[0034] in, Based on the basic noise level and the preset privacy level of government data, ensure the noise protection strength of highly sensitive data; The average amount of data from each department participating in federal training ( K is the total number of departments, which serves as the benchmark for noise calibration; For the client (department) The amount of local official document data (the number of valid samples after preprocessing).

[0035] The core logic of this formula is: the larger the department's data volume ( The larger the value, the more stable its statistical properties, and the lower the influence weight of a single sample. Therefore, the noise level can be appropriately reduced. and Inversely proportional to the data volume (e.g., grassroots units), to avoid excessive noise causing loss of model accuracy; conversely, departments with smaller data volumes (e.g., grassroots units) (Smaller samples) have a more significant impact on parameters, requiring the injection of more noise to enhance privacy protection.

[0036] 4.2 Adaptive Weight Aggregation Unit: Multi-dimensional Weight Calibration Based on Data Quality and Quantity The goal of the adaptive weight aggregation unit is to enable global parameters to more fully absorb the features of high-quality data through scientific weight allocation, avoiding the negative impact of "data quantity-driven but low-quality" factors on model performance. This unit constructs a two-dimensional weight system of "quantity and quality," achieving accurate aggregation through dynamic calibration, as detailed below: The server calculates aggregation weights based on the data quality and quantity from each department, achieving accurate aggregation of model parameters. Specifically, for the client (department)... The server calculates dual weights: Quantity weight:

[0037] in This represents the number of valid samples for department i (the number of duplicate and incomplete data points after preprocessing), reflecting the contribution of department i's data volume to the global model. This weight ensures that departments with larger data volumes occupy a fundamental dominant position in aggregation, conforming to the principle that "data scale determines the coverage of basic features." ② Quality weight:

[0038] in To determine the format compliance rate, the preprocessed data of department i is compared with the degree of conformity with the official document format standard (GB / T 9704-2012). Specifically, it includes 12 indicators such as field completeness (e.g., whether the title, document number, and issuing authority are missing) and layout standardization (e.g. whether the font, line spacing, and page number position are standard). Each indicator is scored from 0 to 1 for compliance and the average value is taken. The terminology accuracy is calculated by fine-tuning the model in department i on a cross-department terminology validation set (containing 1000 standard terms and variants), which is the matching rate between the terms generated by the model and the standard terms in the validation set. In this method, the matching rule is exact matching + semantic approximation matching, and semantic approximation is determined by cosine similarity ≥ 0.8.

[0039] The final weighting considers both quantity and quality, and the calculation formula is as follows:

[0040] The coefficients are set based on the following criteria: quantity is the foundation of model learning (accounting for 70%), but quality determines learning efficiency (accounting for 30%). This avoids the situation where "garbage data dominates the model" due to overemphasizing quantity, or where overemphasizing quality ignores the unique needs of small departments.

[0041] Global parameter aggregation and validation are based on the aforementioned weights, global LoRA parameters. The aggregation formula is:

[0042] in, For the client (department) The result is obtained by adding carefully calibrated Gaussian noise to the finely tuned LoRA adaptation parameters.

[0043] After aggregation, the central server distributes the updated parameters to each client for the next round of LoRA fine-tuning, and obtains the retrained LoRA adaptation parameters from each client for further updates. This process is repeated until the model converges. The model aggregation process uses two checks to ensure the validity of the results: ① Weight normalization verification: Verification To avoid parameter offsets caused by incorrect weight calculations; ② Parameter stability verification: Calculate the cosine similarity between the current global parameter and the previous round (requirement ≥0.95). If it is lower than the threshold, trigger anomaly detection (check whether there is malicious department injecting abnormal parameters), and use 70% of the previous round parameter and 30% of the current result as the correction value.

[0044] This invention addresses the core pain points in the construction of existing large-scale official document models, achieving innovative breakthroughs by combining federated learning and efficient parameter fine-tuning techniques, with the following beneficial effects: 1. Resolving the conflict between government data security and collaborative training: This addresses the privacy risks caused by cross-domain data transfer in traditional centralized training, as well as the problem of data silos between departments hindering collaboration. Through a federated architecture that ensures "data stays within its domain and parameters flow securely," coupled with hierarchical encryption and differential privacy protection, multi-department collaboration is achieved without disclosing original official document data, making it more suitable for the confidentiality requirements of government data than existing technologies. 2. Enhance the domain adaptability of the official document model: Overcome the shortcomings of general-purpose models in learning the characteristics of official document types, format specifications, and specific terminology. Through targeted designs such as document type embedding, structured parsing, and terminology-enhanced loss functions, combined with an adaptive weight aggregation strategy, the model accurately captures the domain characteristics of official documents, making it more suitable for government business scenarios than existing general fine-tuning methods; 3. Lowering the barriers to implementation in government scenarios: It solves the problems of high computational consumption during training of existing large-scale models, high parameter transmission costs, and difficulty in adapting to the resource differences among government departments. Leveraging LoRA's lightweight fine-tuning significantly reduces communication overhead and computational burden in the federated learning process. Its adaptive aggregation strategy effectively balances the differences in data contributions and quality among different departments, improving training efficiency and model convergence speed under non-independent and identically distributed data. This enhances the system's scalability and practical deployment feasibility, providing an efficient and practical solution for large-scale cross-departmental collaboration.

[0045] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for constructing a large-scale official document model, characterized in that, The method includes the following steps: S1. For official documents of different document types, it achieves precise format adaptation and in-depth content extraction. All parsed data will be converted into a unified JSON-LD format for output. S2. The client adopts low-rank adaptive LoRA technology and selects Qwen3-8B as the base model. Based on the Qwen3-8B model, it uses local document data for efficient fine-tuning. The LoRA adapter parameters trained by each client on proprietary document data are encrypted and transmitted to the central server for aggregation and iterative updates after differential privacy protection. S3. The server introduces role-based access control (RBAC) and builds an encryption mechanism that adapts to different levels to ensure that permissions match data visibility. S4. The server adopts a differential privacy enhancement-adaptive weight aggregation architecture, which integrates the locally fine-tuned parameters transmitted by each department into global model parameters through dynamic noise injection and multi-dimensional weight calibration.

2. The document large-scale model construction method as described in claim 1, characterized in that, The S1 method, which enables precise format adaptation and in-depth content extraction for official documents of different document types, includes: (1) Adaptive classification of document types: The input official documents are identified by a government document type classifier to provide a basis for subsequent parsing template calls; the classifier is built based on a fine-tuned BERT-base model, and the training data covers 15 standard document types and 3 high-frequency non-standard document types, and document type-specific features are introduced to optimize the classification logic; (2) Title and document number: By combining named entity recognition and high-precision rule matching based on regular expressions, various common variants are effectively identified and normalized. By analyzing their relative positional relationship with the document number, the key identification information is extracted without loss. (3) Classification and confidentiality period: Based on the location characteristics of the classification of government documents, the candidate area is first screened by coordinate positioning, and then matched with the classification keyword library and the term description; a multi-level verification process is introduced to check whether the classification description mentioned in the text is consistent with the declaration in the document header, which effectively avoids misjudging the classification information quoted in the text but not the actual classification information in the text as the attributes of the document. (4) Issuing and Receiving Authorities: By using dictionary-based entity recognition and grammatical rule analysis, the full names, standardized abbreviations, and common variant names of various levels of institutions and departments are included in the joint determination of common positions in official documents, so as to accurately identify and standardize institutional entities; (5) Main text structure and content: After the basic cleaning is completed, the main text content is retained in the whole form; at the same time, the parsing mechanism will identify and record the internal logical structure of the document. This structural information is embedded as metadata into the prompt word template to guide the federated document big model to better understand the document logic.

3. The document large-scale model construction method as described in claim 1, characterized in that, S2 specifically includes: S21, Document Type Embedding Mechanism Unit: Adds document type embedding to the beginning of the input text sequence to enhance the model's ability to recognize different document types; S22. Load the common base model of each client from the local client. During the fine-tuning process of each client based on the proprietary document data, freeze the parameters of the bottom 33 layers and only make fine-tuning on the top 3 layers. S23. Enhance the weighted loss function by designing terminology to strengthen the learning of high-frequency official document terminology specific to each department; S24. For each round of local training, sample a small batch of samples from the preprocessed document data and calculate the forward propagation loss and backpropagation gradient.

4. The document large-scale model construction method as described in claim 3, characterized in that, S22 includes: initializing LoRA adaptation matrices A and B. Matrix A is initialized using Kaiming, and matrix B is initially set to zero. Initializing matrix B to zero ensures that the LoRA adapter output is zero at the start of training, consistent with the behavior of the original model. The non-zero nature of matrix A ensures that the partial derivatives of matrix B are non-zero during backpropagation, thus allowing matrix B to obtain effective gradients and begin updating. Starting from the second iteration, matrix B is non-zero, thus the partial derivatives of matrix A during backpropagation are non-zero, allowing matrix A to obtain effective gradients and begin updating.

5. The document large-scale model construction method as described in claim 4, characterized in that, In S23, the weighted loss function is specifically implemented as follows: in, The standard cross-entropy loss, which is the original loss function of the large model, is obtained by calculating the cross-entropy loss between the one-hot label of the real token and the probability of predicting the next token. To balance hyperparameters, Term-weighted loss: in Given input Post-generation The probability, Given the length of the input text, output the length of the first line in the text. The terminology for each position corresponds to a token. Terminology weight Calculate using the following steps: Perform word frequency statistics on local official document data and extract words with frequencies exceeding a threshold. For candidate terms, calculate term importance scores: in, For the term The frequency of the term in local departmental document data reflects its common usage in current departmental documents; For the term Inverse document frequency, which is used to measure the term Its universal importance, and whether it appears in most documents; For the term The degree of concentration in a specific department reflects the exclusivity of the term to the current department. This value should be calculated by having each client collect its high-frequency words to the central server before the model training begins, and then distribute the collected words to each client for word frequency statistics. The formula for calculating term weight is: in This is the scaling factor. For all terms importance The minimum value, It is its maximum value; if the th in the sequence Each position is a term Then the weight of the corresponding position For the calculation Otherwise, the weight is 1; when encountering high-frequency terms with multiple characters, the weight of each corresponding character is set to 1. .

6. The document large-scale model construction method as described in claim 5, characterized in that, S24 includes: a loss function of... The optimizer updates the LoRA parameters and evaluates the performance of the current LoRA adapter on the local validation set. If the performance no longer improves, an early stopping mechanism is triggered to avoid overfitting. The LoRA parameters trained by each client on proprietary document data are transmitted to the central server for aggregation and iterative updates after differential privacy protection and encryption.

7. The document large-scale model construction method as described in claim 1, characterized in that, In S3, the server stores aggregated global adapter parameters and domain adapter parameters from various departments, each with access permission tags. When a user requests to use a model, the corresponding adapter parameters are dynamically loaded from storage based on the user's authorized data domain scope. These parameters are then merged with the base model parameters to generate a customized model for that user. Users can only access the model capabilities corresponding to the data domains they have permission to access, and cannot access data knowledge from other unauthorized domains. In cross-departmental collaboration scenarios, departments with higher permissions can unlock more adapter parameters, promoting knowledge sharing while maintaining data sovereignty boundaries.

8. The method for constructing a large document model as described in claim 1, characterized in that, In S3, the server employs the AES-256 symmetric encryption scheme, specifically including: (1) Dynamic key generation and distribution: The central server generates a unique AES session key for each round of federated training and distributes it through the "one-time pad" mechanism: the key is encrypted by the public key of each department node and transmitted separately through a secure channel to avoid the risk of key reuse; the AES session key automatically expires after the parameter aggregation of this round and is regenerated in the next round of training. (2) Lightweight encryption of metadata: For metadata, the "field-level encryption + overall verification" mode is adopted: each field is encrypted separately, and an overall message verification code is generated at the same time.

9. The method for constructing a large document model as described in claim 1, characterized in that, In S4, the server's differential privacy enhancement unit injects calibrated noise into the client's local parameters to ensure that the presence or absence of a single document sample does not significantly affect the global aggregation result, fundamentally resisting "membership inference attacks," which infer whether a specific document participated in training through aggregation parameters; the specific implementation is as follows: (1) Gaussian noise injection: Before uploading, the LoRA adapter parameters of each client are injected with Gaussian noise, and the noise distribution satisfies ,in Let the noise standard deviation of department i be denoted by the following formula: in, Based on the basic noise level and the preset privacy level of government data, ensure the noise protection strength of highly sensitive data; This represents the average amount of data from each department participating in the federal training. K represents the total number of departments, which serves as the benchmark for noise calibration. For the client The amount of local official document data.

10. The document large-scale model construction method as described in claim 9, characterized in that, In S4, the server's adaptive weight aggregation unit constructs a two-dimensional weight system of "quantity and quality," achieving precise aggregation through dynamic calibration, as detailed below: The server calculates aggregation weights based on the quality and quantity of data from each department, achieving accurate aggregation of model parameters; specifically, for the client... The server calculates dual weights: Quantity weight: in The effective sample size of department i reflects the contribution of department i's data volume to the global model. Quality weight: in The format compliance rate is calculated by comparing the preprocessed data of department i with the degree of conformity with the official document format standards. Specifically, it includes multiple indicators, and each indicator is scored from 0 to 1 for compliance and the average value is taken. The terminology accuracy is calculated by fine-tuning the model in department i on the cross-department terminology validation set, which is the matching rate between the terms generated by the model and the standard terms in the validation set. The matching rule is exact matching + semantic approximation matching, and semantic approximation is determined by cosine similarity ≥ 0.

8. The final weighting considers both quantity and quality, and the calculation formula is as follows: Global parameter aggregation and validation are based on the aforementioned weights, global LoRA parameters. The aggregation formula is: in, For the client The result is obtained by adding carefully calibrated Gaussian noise to the finely tuned LoRA adaptation parameters; After aggregation, the central server distributes the updated parameters to each client for the next round of LoRA fine-tuning, and obtains the retrained LoRA adaptation parameters from each client for updates. This process is repeated until the model converges. The aggregation process uses two checks to ensure the validity of the results: ① Weight normalization verification: Verification To avoid parameter offsets caused by incorrect weight calculations; ② Parameter stability verification: Calculate the cosine similarity between the current global parameter and the previous round. If it is lower than the threshold, anomaly detection is triggered, and 70% of the previous round parameter and 30% of the current result are fused as the correction value.

Images