A government affair text auxiliary writing system and method based on a multi-modal large model

Abstract

The application discloses a government affair text auxiliary writing system and method based on a multi-modal large model, and relates to the technical field of artificial intelligence and informationization; comprising: step 1: converting heterogeneous government affair data into unified semantic representation, realizing cross-modal deep alignment through fine-grained contrast learning; encoding text, image and voice features by using an encoder respectively, and constructing a training set containing 100,000 pairs of government affair image-text samples; weighting and fusing the text, image and voice features, dynamically adjusting the weight through an attention mechanism, training a large model, step 2: performing lightweight deployment of the large model, and step 3: using the large model to understand and identify user intention, and outputting government affair text.

Description

Technical Field

[0001] This invention discloses a government document writing assistance system and method based on a multimodal large model, which relates to the interdisciplinary fields of artificial intelligence and information technology. Background Technology

[0002] The field of government document processing currently faces some technical limitations of traditional models and intelligent technologies, such as: I. Limitations of Multimodal Processing: Single Input and Semantic Separation. Existing government writing systems only support text-based input. When processing multimodal data such as images and voice, manual conversion is required, which can easily lead to information loss.

[0003] II. Challenges in Deploying Large-Scale Models: High Resource Consumption and Insufficient Adaptability at the Grassroots Level. Existing government writing tools based on large-scale models adopt a "pre-training + full fine-tuning" model, with model parameters generally exceeding 10 bytes and memory usage of 24-32GB, requiring support from high-end GPUs such as NVIDIA A100. However, grassroots government terminals are mostly ordinary PCs with Intel i5 processors and 16GB of RAM, resulting in low hardware compatibility.

[0004] III. Limitations of Small Sample Learning: Data scarcity and the risk of overfitting. Government document writing is highly scenario-dependent, with significant differences in writing styles and terminology across different fields. Existing technologies use general large models for direct transfer, but in small sample scenarios (<2000 labeled data points), the accuracy of intent recognition drops sharply.

[0005] Fourth, timeliness lag: The contradiction between static knowledge base and dynamic updates. The policy knowledge base of the existing system is generally updated more than 24 hours a day. The "full crawling-batch update" model leads to the risk of policy statements being outdated. Summary of the Invention

[0006] This invention addresses the problems of existing technologies by providing a government document writing assistance system and method based on a multimodal large model, aiming to improve the efficiency and standardization of government document editing through intelligent means.

[0007] The specific solution proposed in this invention is as follows: This invention provides a method for assisting in the writing of government documents based on a multimodal large model, comprising: Step 1: Transform heterogeneous government data into a unified semantic representation, and achieve cross-modal deep alignment through fine-grained contrastive learning: Encode text, image, and speech features separately using encoders, and construct a training set containing 100,000 pairs of government text and image samples. Weighted fusion of text, image, and speech features is performed, with weights dynamically adjusted through an attention mechanism, to train a large model. Step 2: Deploy the large model in a lightweight manner: Constructing a calibration set: Text samples covering 15 government sectors were selected and included contextual terminology. Perform dynamic range analysis: Traverse the weights of each layer of the model, record the 25th / 50th / 75th / 99th quantiles of the weight distribution, identify outliers and perform clipping. Asymmetric quantization is performed: asymmetric quantization is applied to the weights, and dynamic quantization is used for the activation values. End-to-end quantization is implemented using the PyTorchQuantization API. Precision restoration: Weight calibration is performed by minimizing KL divergence, and mixed-precision quantization is used for the quantization-sensitive layer. Deployment optimization: Utilizing a dedicated government inference engine, heterogeneous computing is performed using CPU / GPU. Step 3: Use a large model to understand and identify user intent, and output government documents.

[0008] Furthermore, in step 1 of the method for assisting government text writing based on a multimodal large model, the BERT-base model is pre-trained on the government corpus and then the vocabulary is expanded for text encoding; the SwinTransformer-B model is used to process images, and the local features of seals and handwritten annotations are captured through a sliding window attention mechanism; the Whisper-large-v3 model is used for speech-to-text processing and speech features are extracted, including semantic features, prosodic features, and structural features. Semantic features are generated through the token sequence of the Whisper model to capture keywords; prosodic features are obtained by calculating the mean of MFCC and speech rate to identify urgent matters; and structural features are obtained by statistically analyzing the pause rate and sentence completeness to adapt to the templated expression of government affairs.

[0009] Furthermore, in step 1 of the method for assisting government document writing based on a multimodal large model, when constructing the training set, the InfoNCE loss function with a temperature parameter τ=0.07 is used to ensure that the cosine similarity of feature vectors of similar government data is ≥0.92, and the similarity of dissimilar data is ≤0.3. When dynamically adjusting the weights of text, image, and speech features through the attention mechanism, the weights of text features are in the range of [0.6-0.8], image features in the range of [0.15-0.3], and speech features in the range of [0.05-0.1].

[0010] Furthermore, in step 3 of the aforementioned method for assisting in writing government documents based on a multimodal large model, user intent recognition is performed in small sample scenarios. A lightweight LoRA fine-tuning process is introduced to improve scenario adaptability while maintaining general government knowledge. This includes: Low-rank matrix insertion: In the 12 attention layers of the BERT model, a low-rank matrix W=BA^T is inserted into the Query and Value matrices, where B∈R^d×r, A∈R^r×k, rank r=8, and scaling parameter α=16. A hierarchical fine-tuning strategy was created: the parameters of the first 9 layers of the model were frozen, and only the LoRA parameters and classification head of the last 3 layers were trained. A learning rate of 5e-4 and cosine annealing scheduling were used. The model converged after 28-32 epochs of training on 1200 labeled samples, and the loss change rate on the validation set remained stable at <0.3%. The 1200 labeled samples covered 15 core document types, with ≥80 samples per class. Implement catastrophic forgetting mitigation: Employ the Elastic Weight Consolidation (EWC) algorithm to apply regularization constraints to general government knowledge-related parameters.

[0011] This invention also provides a government document writing assistance system based on a multimodal large model, including a multimodal data processing module, a lightweight deployment module, and an intent understanding and output module. The multimodal data processing module transforms heterogeneous government data into a unified semantic representation and achieves cross-modal deep alignment through fine-grained contrastive learning: text, image, and speech features are encoded separately using encoders, and a training set containing 100,000 pairs of government text and image samples is constructed. Weighted fusion of text, image, and speech features is performed, with weights dynamically adjusted through an attention mechanism to train a large model. The lightweight deployment module enables lightweight deployment of large models: Constructing a calibration set: Text samples covering 15 government sectors were selected and included contextual terminology. Perform dynamic range analysis: Traverse the weights of each layer of the model, record the 25th / 50th / 75th / 99th quantiles of the weight distribution, identify outliers and perform clipping. Asymmetric quantization is performed: asymmetric quantization is applied to the weights, and dynamic quantization is used for the activation values. End-to-end quantization is implemented using the PyTorchQuantization API. Precision restoration: Weight calibration is performed by minimizing KL divergence, and mixed-precision quantization is used for the quantization-sensitive layer. Deployment optimization: Utilizing a dedicated government inference engine, heterogeneous computing is performed using CPU / GPU. The intent understanding and output module uses a large model to understand and identify user intent and output government documents.

[0012] Furthermore, the multimodal data processing module of the government text-assisted writing system based on a multimodal large model employs a BERT-base model pre-trained on government corpora and then expands the vocabulary for text encoding; it uses a Swin Transformer-B model to process images and captures local features of seals and handwritten annotations through a sliding window attention mechanism; it uses a Whisper-large-v3 model for speech-to-text processing and extracts speech features, including semantic features, prosodic features, and structural features. Semantic features are generated through token sequences from the Whisper model to capture keywords; prosodic features are obtained by calculating the mean of MFCC and speech rate to identify urgent matters; and structural features are obtained by statistically analyzing pause rates and sentence completeness to adapt to government template expressions.

[0013] Furthermore, when constructing the training set for the multimodal data processing module of the government document auxiliary writing system based on a multimodal large model, the InfoNCE loss function with a temperature parameter τ=0.07 is used to ensure that the cosine similarity of feature vectors of similar government data is ≥0.92, and the similarity of dissimilar data is ≤0.3. When dynamically adjusting the weights of text, image, and speech features through the attention mechanism, the weights of text features are in the range of [0.6-0.8], image features in the range of [0.15-0.3], and speech features in the range of [0.05-0.1].

[0014] Furthermore, the intent understanding and output module of the government document auxiliary writing system based on a multimodal large model is designed for user intent recognition in small sample scenarios. It introduces a LoRA lightweight fine-tuning process to improve scenario adaptability while maintaining general government knowledge, including: Low-rank matrix insertion: In the 12 attention layers of the BERT model, a low-rank matrix W=BA^T is inserted into the Query and Value matrices, where B∈R^d×r, A∈R^r×k, rank r=8, and scaling parameter α=16. A hierarchical fine-tuning strategy was created: the parameters of the first 9 layers of the model were frozen, and only the LoRA parameters and classification head of the last 3 layers were trained. A learning rate of 5e-4 and cosine annealing scheduling were used. The model converged after 28-32 epochs of training on 1200 labeled samples, and the loss change rate on the validation set remained stable at <0.3%. The 1200 labeled samples covered 15 core document types, with ≥80 samples per class. Implement catastrophic forgetting mitigation: Employ the Elastic Weight Consolidation (EWC) algorithm to apply regularization constraints to general government knowledge-related parameters.

[0015] The advantages of this invention are: Significantly improved multimodal processing accuracy: Deep semantic alignment of text, image, and voice data is achieved, and cross-modal retrieval accuracy is increased from 65% to 80% compared to existing technologies. Among these, the accuracy rate for recognizing government charts reaches 98.5%, and the accuracy rate for recognizing handwritten annotations reaches 92%, representing a 22 percentage point improvement over traditional OCR technology. Material collection time is reduced by 80%, solving the problem of "multimodal information silos."

[0016] Model deployment costs are significantly reduced: INT8 quantization technology reduces model memory usage from 24GB to 7GB while maintaining terminology recognition accuracy, resulting in a 60% reduction in hardware procurement costs. Real-time inference (single text ≤300ms) can be achieved on a regular Intel i5 processor, and the compatibility rate with grassroots government terminals has increased from 28% to 85%.

[0017] Excellent performance in small-sample intent recognition: LoRA fine-tuning technology improves user intent recognition accuracy from 62% to 92% in scenarios with only 1200 labeled data points, reduces the fine-tuning parameter size from 110M to 5.8M, shortens training time from 30 hours to 2 hours, and reduces energy consumption by 95%. The EWC algorithm effectively mitigates catastrophic forgetting; when switching between scenarios, the term recognition accuracy loss is only 4.5%, far lower than the 35% loss of traditional fine-tuning methods.

[0018] Improve timeliness: resolve the contradiction between static knowledge base and dynamic updates, shorten the policy knowledge base update cycle, and eliminate the risk of policy statements being outdated. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the application process of the method of the present invention. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention. Example

[0021] This invention provides a method for assisting in the writing of government documents based on a multimodal large model, comprising: Step 1: The FG-CLIP feature extractor can be used to convert heterogeneous government data into a unified semantic representation, and cross-modal deep alignment can be achieved through fine-grained contrastive learning. Encoders are used to encode text, image, and speech features separately. The BERT-base model is pre-trained on government corpora and then expanded with a vocabulary for text encoding. The Swin Transformer-B model is used to process images, and a sliding window attention mechanism is employed to capture local features of seals and handwritten annotations. The Whisper-large-v3 model is used for speech-to-text processing, extracting speech features including semantic, prosodic, and structural features. Specifically, a 128-dimensional vector is generated from the token sequence of the Whisper model to capture keywords such as 'request for instructions' and 'approval'. Prosodic features are obtained by calculating the mean MFCC and speech rate to identify urgent matters. Structural features are obtained by statistically analyzing pause rates and sentence completeness, adapting to templated government expressions.

[0022] A training set containing 100,000 pairs of government document images and text samples was constructed, covering 15 types of official documents such as requests and reports. The InfoNCE loss function with a temperature parameter τ=0.07 was used to ensure that the cosine similarity of feature vectors for similar government documents was ≥0.92, and the similarity for dissimilar documents was ≤0.3. Weighted fusion of text, image, and speech features is performed, with the weights dynamically adjusted via an attention mechanism. This dynamic adjustment ensures that the text feature weights are within the range of [0.6-0.8], the image feature weights are within the range of [0.15-0.3], and on a test set of 1000 recorded government meetings, ablation experiments determined that the speech feature weights are within the range of [0.05-0.1]. Train a large model using the fused feature data.

[0023] Step 2: Deploy the large model in a lightweight manner: Constructing a calibration set: Text samples covering 15 government sectors were selected and included contextual terminology. Perform dynamic range analysis: Traverse the weights of each layer of the model, record the 25th / 50th / 75th / 99th quantiles of the weight distribution, identify outliers and perform clipping. Asymmetric quantization is performed: asymmetric quantization is applied to the weights, and dynamic quantization is used for the activation values. End-to-end quantization is implemented using the PyTorchQuantization API. Precision restoration is achieved by minimizing the KL divergence for weight calibration, and by employing mixed-precision quantization for quantization-sensitive layers such as attention layers. Deployment optimization: Utilizing a government-specific inference engine and employing heterogeneous computing with CPU / GPU, the inference time for a single text is ≤300ms on an Intel i5-10400 processor, meeting the requirements for real-time writing.

[0024] Step 3: Use a large model to understand and identify user intent, and output government documents.

[0025] For user intent recognition in small sample scenarios, a lightweight fine-tuning process based on LoRA is introduced to improve scenario adaptability while maintaining general government knowledge, including: Low-rank matrix insertion is performed: In the 12 attention layers of the BERT model, a low-rank matrix W=BA^T is inserted into the Query and Value matrices, where B∈R^d×r, A∈R^r×k, rank r=8, and scaling parameter α=16. In small-sample government text tasks, such as 1,200 labeled samples, r=8 balances the model's expressive power and computational efficiency, while α=16 ensures that the scale of low-rank updates matches the pre-trained weights, avoiding overfitting. This configuration significantly reduces the number of parameters compared to general NLP tasks where r≥16, while maintaining the accuracy of government text classification.

[0026] A hierarchical fine-tuning strategy was created: the parameters of the first 9 layers of the model were frozen, and only the LoRA parameters and classification head of the last 3 layers were trained. A learning rate of 5e-4 and cosine annealing scheduling were used. The model converged after 28-32 training cycles on 1200 labeled samples, and the loss change rate on the validation set remained stable at <0.3%. The 1200 labeled samples covered 15 core document types, with ≥80 samples in each class. Combining LoRA low-rank adaptation and gradient pruning strategies, convergence was achieved after 28-32 iterations. The strong structured characteristics of government documents, such as fixed templates and standardized terminology databases, significantly accelerated convergence, shortening the training cycle by an average of 40% compared to general text tasks.

[0027] Implement catastrophic forgetting mitigation: Employ the Elastic Weight Consolidation (EWC) algorithm to apply regularization constraints to general government knowledge-related parameters.

[0028] This invention achieves high efficiency through the coordinated operation of three flows: feature flow, control flow, and knowledge flow. Feature Stream: Multimodal data is encoded into 512-dimensional features via FG-CLIP, quantized, and then input into the intent recognition model. During feature transfer, a checksum mechanism can be used to ensure data integrity. Control flow: Based on the confidence level of intent recognition, a dynamic decision is made on whether to trigger knowledge query. A state machine can be used to manage the state transitions of encoding-recognition-query-generation. Knowledge Flow: Updated government entities are written into the knowledge graph, and cross-departmental knowledge integration is achieved through parameter exchange, with a knowledge update delay of ≤30 seconds. Example

[0029] This invention also provides a government document writing assistance system based on a multimodal large model, including a multimodal data processing module, a lightweight deployment module, and an intent understanding and output module. The multimodal data processing module transforms heterogeneous government data into a unified semantic representation and achieves cross-modal deep alignment through fine-grained contrastive learning: text, image, and speech features are encoded separately using encoders, and a training set containing 100,000 pairs of government text and image samples is constructed. Weighted fusion of text, image, and speech features is performed, with weights dynamically adjusted through an attention mechanism to train a large model. The lightweight deployment module enables lightweight deployment of large models: Constructing a calibration set: Text samples covering 15 government sectors were selected and included contextual terminology. Perform dynamic range analysis: Traverse the weights of each layer of the model, record the 25th / 50th / 75th / 99th quantiles of the weight distribution, identify outliers and perform clipping. Asymmetric quantization is performed: asymmetric quantization is applied to the weights, and dynamic quantization is used for the activation values. End-to-end quantization is implemented using the PyTorchQuantization API. Precision restoration: Weight calibration is performed by minimizing KL divergence, and mixed-precision quantization is used for the quantization-sensitive layer. Deployment optimization: Utilizing a dedicated government inference engine, heterogeneous computing is performed using CPU / GPU. The intent understanding and output module uses a large model to understand and identify user intent and output government documents.

[0030] The information interaction and execution process between the modules in the above system are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description in the method embodiment of the present invention, and will not be repeated here.

[0031] Similarly, the advantages of the system of the present invention are: Significantly improved multimodal processing accuracy: Deep semantic alignment of text, image, and voice data is achieved, and cross-modal retrieval accuracy is increased from 65% to 80% compared to existing technologies. Among these, the accuracy rate for recognizing government charts reaches 98.5%, and the accuracy rate for recognizing handwritten annotations reaches 92%, representing a 22 percentage point improvement over traditional OCR technology. Material collection time is reduced by 80%, solving the problem of "multimodal information silos."

[0032] Model deployment costs are significantly reduced: INT8 quantization technology reduces model memory usage from 24GB to 7GB while maintaining terminology recognition accuracy, resulting in a 60% reduction in hardware procurement costs. Real-time inference (single text ≤300ms) can be achieved on a regular Intel i5 processor, and the compatibility rate with grassroots government terminals has increased from 28% to 85%.

[0033] Excellent performance in small-sample intent recognition: LoRA fine-tuning technology improves user intent recognition accuracy from 62% to 92% in scenarios with only 1200 labeled data points, reduces the fine-tuning parameter size from 110M to 5.8M, shortens training time from 30 hours to 2 hours, and reduces energy consumption by 95%. The EWC algorithm effectively mitigates catastrophic forgetting; when switching between scenarios, the term recognition accuracy loss is only 4.5%, far lower than the 35% loss of traditional fine-tuning methods.

[0034] It should be noted that not all steps and modules in the above processes and system structures are mandatory; some steps or modules can be omitted as needed. The execution order of each step is not fixed and can be adjusted as required. The system structures described in the above embodiments can be physical or logical structures. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0035] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method for assisting in the writing of government documents based on a multimodal large model, characterized by: include: Step 1: Transform heterogeneous government data into a unified semantic representation, and achieve cross-modal deep alignment through fine-grained contrastive learning: Encode text, image, and speech features separately using encoders, and construct a training set containing 100,000 pairs of government text and image samples. Weighted fusion of text, image, and speech features is performed, with weights dynamically adjusted through an attention mechanism, to train a large model. Step 2: Deploy the large model in a lightweight manner: Constructing a calibration set: Text samples covering 15 government sectors were selected and included contextual terminology. Perform dynamic range analysis: Traverse the weights of each layer of the model, record the 25th / 50th / 75th / 99th quantiles of the weight distribution, identify outliers and perform clipping. Asymmetric quantization is performed: asymmetric quantization is applied to the weights, and dynamic quantization is used for the activation values. End-to-end quantization is implemented using the PyTorchQuantization API. Precision restoration: Weight calibration is performed by minimizing KL divergence, and mixed-precision quantization is used for the quantization-sensitive layer. Deployment optimization: Utilizing a dedicated government inference engine, heterogeneous computing is performed using CPU / GPU. Step 3: Use a large model to understand and identify user intent, and output government documents.

2. The method for assisting in writing government documents based on a multimodal large model as described in claim 1, Its features include: in step 1, the BERT-base model is pre-trained on government corpora and the vocabulary is expanded to encode the text; the Swing Transformer-B model is used to process images and capture local features of seals and handwritten annotations through a sliding window attention mechanism; the Whisper-large-v3 model is used for speech-to-text processing and speech features are extracted, including semantic features, prosodic features and structural features, among which semantic features are generated through the token sequence of the Whisper model to capture keywords; The mean MFCC and speech rate are calculated to obtain prosodic features and identify urgent matters; the pause rate and sentence integrity are statistically analyzed to obtain structural features and adapt to government template expressions.

3. A method for assisting in writing government documents based on a multimodal large model, as described in claim 1 or 2, characterized in that, when constructing the training set in step 1, an InfoNCE loss function with a temperature parameter τ = 0.07 is used, ensuring that the cosine similarity of feature vectors of similar government data is ≥ 0.92, and the similarity of dissimilar data is ≤ 0.

3. When dynamically adjusting the weights of text, image, and speech features through the attention mechanism, the weights of text features are in the range of [0.6-0.8], image features in the range of [0.15-0.3], and speech features in the range of [0.05-0.1].

4. The method for assisting in writing government documents based on a multimodal large model as described in claim 1, characterized in that: Step 3 addresses user intent recognition in small-sample scenarios by introducing a lightweight LoRA fine-tuning process. This improves scenario adaptability while maintaining general government knowledge, including: Low-rank matrix insertion: In the 12 attention layers of the BERT model, a low-rank matrix W=BA^T is inserted into the Query and Value matrices, where B∈R^d×r, A∈R^r×k, rank r=8, and scaling parameter α=16. A hierarchical fine-tuning strategy was created: the parameters of the first 9 layers of the model were frozen, and only the LoRA parameters and classification head of the last 3 layers were trained. A learning rate of 5e-4 and cosine annealing scheduling were used. The model converged after 28-32 epochs of training on 1200 labeled samples, and the loss change rate on the validation set remained stable at <0.3%. The 1200 labeled samples covered 15 core document types, with ≥80 samples per class. Implement catastrophic forgetting mitigation: Employ the Elastic Weight Consolidation (EWC) algorithm to apply regularization constraints to general government knowledge-related parameters.

5. A government document writing assistance system based on a multimodal large model, characterized by: It includes a multimodal data processing module, a lightweight deployment module, and an intent understanding and output module. The multimodal data processing module transforms heterogeneous government data into a unified semantic representation and achieves cross-modal deep alignment through fine-grained contrastive learning: text, image, and speech features are encoded separately using encoders, and a training set containing 100,000 pairs of government text and image samples is constructed. Weighted fusion of text, image, and speech features is performed, with weights dynamically adjusted through an attention mechanism to train a large model. The lightweight deployment module enables lightweight deployment of large models: Constructing a calibration set: Text samples covering 15 government sectors were selected and included contextual terminology. Perform dynamic range analysis: Traverse the weights of each layer of the model, record the 25th / 50th / 75th / 99th quantiles of the weight distribution, identify outliers and perform clipping. Asymmetric quantization is performed: asymmetric quantization is applied to the weights, and dynamic quantization is used for the activation values. End-to-end quantization is implemented using the PyTorchQuantization API. Precision restoration: Weight calibration is performed by minimizing KL divergence, and mixed-precision quantization is used for the quantization-sensitive layer. Deployment optimization: Utilizing a dedicated government inference engine, heterogeneous computing is performed using CPU / GPU. The intent understanding and output module uses a large model to understand and identify user intent and output government documents.

6. The government document writing assistance system based on a multimodal large model according to claim 5, characterized in that it is multimodal. The data processing module uses the BERT-base model pre-trained on government corpora and then expands the vocabulary to encode the text; it uses the Swin Transformer-B model to process images and captures local features of seals and handwritten annotations through a sliding window attention mechanism; it uses the Whisper-large-v3 model for speech-to-text processing and extracts speech features, including semantic features, prosodic features, and structural features. Semantic features are generated through the token sequence of the Whisper model to capture keywords. The mean MFCC and speech rate are calculated to obtain prosodic features and identify urgent matters; the pause rate and sentence integrity are statistically analyzed to obtain structural features and adapt to government template expressions.

7. A government document writing assistance system based on a multimodal large model as described in claim 5 or 6, characterized in that it is multimodal. When constructing the training set, the data processing module uses the InfoNCE loss function with a temperature parameter τ=0.07 to ensure that the cosine similarity of feature vectors for similar government data is ≥0.92, and the similarity for dissimilar data is ≤0.

3. When dynamically adjusting the weights of text, image, and speech features through the attention mechanism, the weights of text features are in the range of [0.6-0.8], image features in the range of [0.15-0.3], and speech features in the range of [0.05-0.1].

8. A government document writing assistance system based on a multimodal large model according to claim 5, characterized in that: The intent understanding and output module is designed for user intent recognition in small-sample scenarios. It incorporates a lightweight LoRA fine-tuning process to improve scenario adaptability while maintaining general government knowledge, including: Low-rank matrix insertion: In the 12 attention layers of the BERT model, a low-rank matrix W=BA^T is inserted into the Query and Value matrices, where B∈R^d×r, A∈R^r×k, rank r=8, and scaling parameter α=16. A hierarchical fine-tuning strategy was created: the parameters of the first 9 layers of the model were frozen, and only the LoRA parameters and classification head of the last 3 layers were trained. A learning rate of 5e-4 and cosine annealing scheduling were used. The model converged after 28-32 epochs of training on 1200 labeled samples, and the loss change rate on the validation set remained stable at <0.3%. The 1200 labeled samples covered 15 core document types, with ≥80 samples per class. Implement catastrophic forgetting mitigation: Employ the Elastic Weight Consolidation (EWC) algorithm to apply regularization constraints to general government knowledge-related parameters.

Images