Large language model training method for official document writing, computer device and storage medium

Abstract

This invention discloses a method, computer device, and storage medium for training a large language model for official document writing. The method includes the following steps: adding a low-rank matrix in parallel to the weight matrix of a pre-trained model; performing a first training process on the pre-trained model and the low-rank matrix using an official document dataset to obtain a base model; performing a second training process on the base model using a reward dataset based on human feedback reinforcement learning to obtain a reward model; and performing proximal policy optimization on the reward model to obtain a large language model. This invention can train a large language model, allowing input of official document writing instructions and prompts, and the large language model generates official documents that conform to human writing and reading habits. Using the large language model to generate official documents can assist in official document writing, thereby improving writing efficiency and maintaining consistent writing quality. This invention has wide applications in the field of official document writing technology.

Description

Technical Field

[0001] This invention relates to the field of official document writing technology, and in particular to a method, computer device, and storage medium for training a large language model for official document writing. Background Technology

[0002] Official documents are crucial tools for information transmission and decision-making within and outside government agencies, public institutions, enterprises, and organizations, playing a vital role in ensuring the smooth operation of these institutions and facilitating informed decision-making. Traditionally, official document writing is handled personally by staff. However, due to the sheer volume and variety of documents, as well as the tedious and complex writing process, the writing skills and time available to staff often limit the quality of the documents, leading to inaccurate information and unclear expression. These problems result in low efficiency and inconsistent quality in official document writing. Summary of the Invention

[0003] In view of the technical problems of low efficiency and unstable quality in current official document writing, the purpose of this invention is to provide a large language model training method, computer device and storage medium for official document writing.

[0004] On one hand, embodiments of the present invention include a method for training a large language model for official document writing, the method comprising the following steps:

[0005] Obtain a document dataset and a reward dataset; the document dataset includes multiple document writing instruction prompts and document texts, and the reward dataset includes multiple document writing instruction prompts.

[0006] Obtain a pre-trained model;

[0007] A low-rank matrix is ​​added in parallel to the weight matrix of the pre-trained model. The document dataset is used to perform a first training process on the pre-trained model and the low-rank matrix to obtain a base model. During the first training process, the parameters of the weight matrix are fixed and the low-rank matrix is ​​updated. After the first training process is completed, the parameters of the low-rank matrix are merged into the weight matrix.

[0008] A second training process based on human feedback reinforcement learning is performed on the base model using the reward dataset to obtain a reward model;

[0009] The reward model is optimized by performing a proximal policy to obtain a large language model.

[0010] Furthermore, the acquisition of the official document dataset and the reward dataset includes:

[0011] The corpus of official documents was obtained from publicly available sources; the corpus of official documents contains official document content.

[0012] Obtain general corpus;

[0013] The official document corpus and the general corpus are preprocessed;

[0014] The official document corpus and the general corpus are combined in a certain proportion to obtain the texts of each official document;

[0015] Each of the official document texts is annotated with writing prompts to obtain the corresponding official document writing instruction prompts.

[0016] Furthermore, obtaining the pre-trained model includes:

[0017] Obtain the general large model Qwen-72B as the pre-trained model.

[0018] Further, the step of performing a first training process on the pre-trained model and the low-rank matrix using the document dataset to obtain the base model includes:

[0019] Set a dimension-reducing matrix and a dimension-increasing matrix as the low-rank matrix;

[0020] The reduced-dimensional matrix is ​​randomized using a Gaussian random distribution, and the increased-dimensional matrix is ​​initialized as a zero matrix.

[0021] The document writing instruction prompt is input into the pre-trained model after adding the low-rank matrix. The document writing instruction prompt is processed by the weight matrix and then by the dimension reduction matrix and the dimension increase matrix. The processing results of the weight matrix and the low-rank matrix are fused to obtain the first output result. The parameters of the weight matrix are fixed, and the parameters of the dimension reduction matrix and the dimension increase matrix are updated according to the first output result and the document text.

[0022] The result of multiplying the reduced-dimensional matrix and the increased-dimensional matrix after parameter updates is superimposed on the weight matrix, thereby updating the weight matrix;

[0023] The pre-trained model updated with the weight matrix parameters is used as the base model.

[0024] Further, the step of performing a first training process on the pre-trained model and the low-rank matrix using the document dataset to obtain the base model includes:

[0025] The pre-trained model is split into multiple sub-models, and each sub-model is distributed to a corresponding sub-process.

[0026] In each subprocess, a corresponding low-rank matrix is ​​added to the sub-model in parallel, and the first training process is performed on the sub-model and the low-rank matrix using the document dataset.

[0027] The sub-models trained by each sub-process are merged to obtain the base model.

[0028] Further, the second training process using the reward dataset to perform reinforcement learning based on human feedback on the base model to obtain the reward model includes:

[0029] Input the document writing instruction prompt into the base model;

[0030] Obtain multiple texts to be scored generated by the base model in response to the document writing instruction prompt;

[0031] Obtain the score value for each of the texts to be scored;

[0032] The value of the loss function is determined based on the highest and lowest scores among the given scores;

[0033] The parameters of the base model are updated based on the value of the loss function;

[0034] The base model with updated parameters is used as the reward model;

[0035] Wherein, the loss function is

[0036] L ranking =-log(σ(r) θ (x,y c )-r θ (x,y r )-m(r)))

[0037] L ranking Let r be the value of the loss function. θ (x, y) represents the score value corresponding to the document writing instruction prompt x and the text y to be scored generated by the base model in response to x when the weight matrix parameter of the base model is θ. c The highest score is y r The minimum score is given by σ, which is the sigmoid function, i.e., σ(z) = Sig(z) = (1 + e^(-1 / 2)) / ( ... -z ) -1 m(r) is y c The corresponding text and y r The corresponding text margin label correction, used to describe y c The corresponding text and y r The difference between the corresponding texts, if yc The corresponding text and y r The corresponding text has a "significant difference" (e.g., y). c The corresponding text and y r If the Manhattan distance between the corresponding texts is greater than a preset threshold, the value of m(r) can be adjusted to increase the gradient value and speed up the parameter update.

[0038] Further, the step of performing proximal policy optimization on the reward model to obtain a large language model includes:

[0039] Set optimization goals;

[0040] Based on the optimization objective, a near-end policy optimization algorithm is executed on the reward model to iteratively update the parameters of the reward model;

[0041] The updated reward model is used as the large language model.

[0042] Wherein, the optimization objective is

[0043] argmax π E p～D,g～π [R(g|p)]

[0044] p represents the document writing instruction prompt, D represents the dataset containing the document writing instruction prompt p, g represents the strategy used by the near-end policy optimization algorithm, π represents the dataset containing strategy g, E represents the expectation function, and R(g|p) represents the reward function for prompt p under strategy g. The entire formula represents optimizing a strategy set π such that the expected total reward under this strategy is maximized when prompt p conforms to the dataset distribution D.

[0045] Furthermore, the method for training a large language model for official document writing also includes:

[0046] The model interface for the large language model is built using vLLM and Fschat.

[0047] On the other hand, embodiments of the present invention also include a computer device, including a memory and a processor, the memory for storing at least one program, and the processor for loading at least one program to execute the large language model training method for official document writing in the embodiments.

[0048] On the other hand, embodiments of the present invention also include a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the large language model training method for official document writing in the embodiments.

[0049] The beneficial effects of this invention are as follows: The large language model training method for official document writing in the embodiments can train a large language model, which can input prompts for official document writing instructions. These prompts can be a summary or keywords of the document to be generated, or instructions for writing the document. The large language model generates official documents that conform to human writing and reading habits. By using the large language model to generate official documents, official document writing can be assisted, thereby improving writing efficiency and maintaining consistent writing quality. Attached Figure Description

[0050] Figure 1 This is a schematic diagram illustrating the steps of the large language model training method for official document writing in the embodiment.

[0051] Figure 2 This is a schematic diagram illustrating the principle of the large language model training method for official document writing in the embodiment.

[0052] Figure 3 This is a flowchart illustrating the first training process in the embodiment;

[0053] Figure 4 This is a schematic diagram illustrating the principle of the first training process in the embodiment;

[0054] Figure 5 This is a flowchart illustrating the first training process performed using Zero3 technology in this embodiment. Detailed Implementation

[0055] Some terms mentioned in this embodiment have the following meanings:

[0056] Knowledge graph: A technique that uses graph theory to abstractly model and analyze concepts, entities, events and their relationships in the objective world.

[0057] Large models: Large models are ultra-large-scale neural network models containing billions of parameters, used in fields such as natural language processing. They achieve more powerful and accurate model performance by leveraging the huge model size, powerful computing resources, and techniques such as knowledge distillation.

[0058] LoRA: LoRA (Low-Rank Adaptation) is an efficient method for fine-tuning large pre-trained models. Its core principle is to use low-rank matrix factorization to achieve effective updates of model parameters.

[0059] Fine-tuning: Fine-tuning involves further training and adjustments to a pre-trained large neural network model (such as BERT, GPT, etc.) for a specific task or dataset. The purpose of fine-tuning is to enable the model to better adapt to and cope with the needs of a specific domain or task, thereby improving its performance on those tasks.

[0060] Transfer learning: Its core idea is to transfer knowledge gained from one or more source tasks to related but different target tasks. In traditional machine learning, each new task usually requires training the model from scratch, while transfer learning allows the model to use an existing pre-trained model as a starting point, especially when the training data for the target task is limited.

[0061] Jaccard similarity: The Jaccard index, also known as the Jaccard similarity coefficient, is used to compare the similarity and differences between a finite set of samples. The larger the Jaccard coefficient value, the higher the sample similarity.

[0062] Named entity recognition (NAME) is a technique that automatically detects and extracts meaningful entities from unstructured text data and categorizes them into predefined classes. These entities typically include: person names, place names, organization names, etc.

[0063] Zero3: ZeRO (Zero Redundancy Optimizer) improves the scalability and efficiency of model training by optimizing memory usage in data-parallel training. ZeRO reduces the storage of redundant data, especially when distributing model states across multiple GPUs. It is divided into several different optimization levels (ZeRO-DP1, ZeRO-DP2, ZeRO-DP3), each providing different degrees of memory optimization and scalability.

[0064] vLLM: vLLM is an LLM (Large Language Model) inference and deployment service library that combines iterative-level scheduling (continuous batching) and the PagedAttention algorithm to improve service throughput. Its core is the PagedAttention algorithm, inspired by operating system virtual memory and paging concepts. This algorithm stores previously contiguous key-value caches in non-contiguous space to avoid wasting GPU memory associated with key-value caching.

[0065] Iterative-level Schedule (Continuous Batching): Iterative-level scheduling, or continuous batch processing, is a strategy that combines multiple small batches of data into larger batches for processing when handling continuous data streams.

[0066] PagedAttention is an optimization technique for accelerating inference in large-scale language models, especially when dealing with long input sequences. Self-attention is a core component of the Transformer architecture, used to capture long-distance dependencies in text sequences. However, as model size and input sequence length increase, the time and space complexity of self-attention computation also increases. PagedAttention works by decomposing traditional global attention into local attention computations, using pagination or partitioning to reduce computation. Its basic idea is similar to the pagination mechanism in operating system virtual memory management; instead of loading and calculating the entire sequence's attention matrix at once, it divides the long sequence into several "pages" or "windows," performing attention computation on only the elements within one window at a time, and utilizing cross-window information sharing to reduce computation and memory usage.

[0067] NLP: Natural Language Processing is an important field in computer science and artificial intelligence. It can be applied to machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, Chinese OCR, and other fields.

[0068] RLHF: Reinforcement Learning from Human Feedback. By incorporating human feedback into the training process, it provides machines with a natural and human-like interactive learning process.

[0069] PPO: Proximal Policy Optimization is a reinforcement learning algorithm used to optimize the policy of an agent. It attempts to maintain stability during policy updates and prevents the learning process from becoming unstable due to excessive policy updates.

[0070] To automate the writing of official documents, introducing artificial intelligence technology is a viable solution. Large language models, as powerful natural language processing tools, possess the ability to understand and generate natural language. By introducing a large amount of specialized official document data and conducting targeted domain adaptation and fine-tuning training, these models can acquire the ability to write official documents, thereby improving the quality and efficiency of automated document generation. This effectively supports the daily work of governments, enterprises, institutions, and other related organizations, achieving the goals of automating and standardizing official document writing.

[0071] For example, graph neural networks (GNNs) can be used to generate text, and therefore can be used for official document writing. However, methods for building GNNs based on knowledge graphs have several drawbacks: First, building and maintaining knowledge graphs requires significant human and material resources, as they typically require expert knowledge to ensure the accuracy and completeness of information; second, knowledge graphs have limited coverage and may not encompass knowledge from all domains or keep up with the pace of knowledge updates, which limits the generalization ability and timeliness of GNN models built upon them; furthermore, GNNs generally have high computational complexity, especially when processing large-scale graph data, which can lead to efficiency issues; finally, the interpretability of GNN models may be inferior to some rule-based methods because the decision-making process of GNNs can be complex and opaque; although knowledge graph-based GNN methods can leverage structured knowledge to improve the performance of specific tasks, they may not be as flexible and efficient as large-model fine-tuning methods in terms of the breadth, depth, update speed, computational efficiency, and interpretability of knowledge.

[0072] Models with fewer parameters often cannot fully capture the rich features of large-scale datasets. If a base model with more parameters is selected, it can capture more complex and detailed data patterns and features, and usually has higher accuracy and generalization ability when generating official documents. NLP technology can play a role in screening and cleaning official document data, but at present, NLP technology still faces challenges in understanding complex contexts, recognizing professional terms, and parsing multi-layered logical structures, which may lead to the omission or misjudgment of some key information. In addition, when using large models for official document writing, it is also necessary to consider the real-time response speed and user experience of the large models, as well as the differences in requirements such as specific document styles and industry standards.

[0073] Based on the above principles, this embodiment provides a method for training a large language model for official document writing. (Refer to...) Figure 1 The training method for a large language model used in official document writing includes the following steps:

[0074] S1. Obtain the official document dataset and the reward dataset;

[0075] S2. Obtain the pre-trained model;

[0076] S3. Add a low-rank matrix in parallel to the weight matrix of the pre-trained model, and perform the first training process on the pre-trained model and the low-rank matrix using the official document dataset to obtain the base model;

[0077] S4. Use the reward dataset to perform a second training process based on human feedback reinforcement learning on the base model to obtain the reward model;

[0078] S5. Perform proximal policy optimization on the reward model to obtain a large language model.

[0079] The principle of steps S1-S5 is as follows: Figure 2 As shown.

[0080] In step S1, the official document dataset includes multiple document writing instruction prompts and document texts, while the reward dataset includes multiple document writing instruction prompts. The document texts in the official document dataset can be texts containing official document content, including organizational announcements, reports, notices, meeting minutes, orders, requests, departmental letters, external letters, press releases, etc. The document writing instruction prompts in the official document dataset correspond to the document texts and can be a summary of the document text's content, keywords, or instructions requiring personnel or equipment to write the document text. For example, for a document text with the content "Announcement from XX Organization Regarding the Procurement of Computer Equipment…", the corresponding document writing instruction prompt could be "equipment, procurement, announcement" (a summary or keywords of the document text) or "Please write an announcement regarding the procurement of computer equipment" (an instruction requiring document writing). The document writing instruction prompts in the reward dataset can have the same content and number as those in the official document dataset.

[0081] In step S2, a general-purpose large-scale model, Qwen-72B, with 72 billion parameters and pre-trained on 3 trillion tokens of data, can be selected as the pre-trained model. Qwen-72B has a large parameter scale, which can capture more complex and detailed data patterns and features, and usually has higher accuracy and generalization ability when generating official documents.

[0082] In step S3, LoRA and Zero3 technologies can be combined to train the pre-trained model using a document dataset (i.e., the first training process). The pre-trained model trained through the first training process is referred to as the base model in this embodiment. By executing step S3, the pre-trained model can be fine-tuned using the document dataset, resulting in high training efficiency.

[0083] In step S4, the base model obtained in step S3 is trained using the reward dataset (i.e., the second training process), which is based on RLHF. The base model trained through the second training process is referred to as the reward model in this embodiment. Since the document output of the base model is random, by executing step S4, the base model can be trained using human feedback reinforcement learning. The resulting reward model's document output is more in line with human preferences, thus enabling the output of more personalized documents that conform to official document writing standards.

[0084] In step S5, proximal policy optimization is performed on the reward model obtained in step S4 to obtain a large language model. By performing step S5, the reward model can learn more deeply from human feedback, and the resulting large language model can easily generate official documents that are closer to human writing.

[0085] In this embodiment, by executing steps S1-S5, a large language model can be trained. A prompt for official document writing instructions can be input into the large language model. The content of the prompt for official document writing instructions can be a summary or keywords of the official document text to be generated, or an instruction to write the official document text. The large language model generates official documents that conform to human writing and reading habits. By using a large language model to generate official documents, official document writing can be assisted, thereby improving writing efficiency and maintaining stable writing quality. Specifically, during the training of the large language model, the pre-trained model is fine-tuned on a specific official document dataset, improving its performance in processing official documents and generating official documents. The fine-tuned pre-trained model (i.e., the base model) is then trained using reinforcement learning based on human feedback. The resulting reward model can more accurately understand and generate the unique format, terminology, and style of official documents. Finally, proximal policy optimization is performed on the reward model to obtain the large language model, allowing for further fine-tuning of the model's parameters. This makes the large language model more closely resemble human writing and reading styles, providing more efficient and professional text processing tools for government agencies and enterprises. Relevant departments can use this fine-tuned model to assist in official document writing, improve office efficiency, and ensure the quality and responsiveness of office services.

[0086] In this embodiment, when performing step S1, which is to obtain the official document dataset and the reward dataset, the following steps can be performed:

[0087] S101. Obtain official document corpora from publicly available sources;

[0088] S102. Obtain general corpus;

[0089] S103. Preprocess the official document corpus and general corpus;

[0090] S104. Combine official document corpora and general corpora in a certain proportion to obtain the texts of each official document;

[0091] S105. Mark each official document text with writing prompts to obtain the corresponding official document writing instructions.

[0092] In step S101, you can collect publicly released official documents in the form of announcements, reports, notices, meeting minutes, etc., with different themes, fields, and styles from the organization's official website, the organization's superior department, or regulatory department, to ensure the diversity and breadth of the official documents.

[0093] The official document corpus obtained in step S101 contains official document content. In step S102, general corpus that does not contain official document content can be obtained from sources such as news websites and literary works.

[0094] In step S103, the official document corpus and general corpus are preprocessed to ensure that they meet the specifications of the pre-trained model. Taking the preprocessing of the official document corpus as an example, the preprocessing includes the following data cleaning process:

[0095] ① Filtering Invalid Documents: First, algorithms such as Jaccard similarity are used to calculate the similarity of text content in the articles, and highly repetitive documents are removed to ensure the uniqueness and accuracy of the dataset. Second, based on criteria such as keywords and topics, the documents are relevance-checked to filter out content irrelevant to the research objectives, making the data more targeted. Finally, outdated and invalid documents are removed, retaining only valid documents within a specific time frame to ensure the timeliness and usability of the information.

[0096] ② Convert all official documents to a unified file format, such as converting PDF to JSON files, for easier subsequent processing. Simultaneously, ensure all text adheres to a unified encoding standard, such as UTF-8, to maintain format consistency. Furthermore, perform structured processing on the documents, extracting structured information such as title, date, issuing unit, and body text, and store them according to a unified template to facilitate information organization and retrieval.

[0097] The preprocessing steps for general corpora are the same as those for official document corpora. By preprocessing the collected official document corpora and general corpora, problems such as formatting errors and inconsistent layouts in the corpora can be removed.

[0098] In step S104, official document corpora and general corpora are combined in a certain ratio to obtain various official document texts. Among all the obtained official document texts, some belong to the official document corpora, and others belong to the general corpora. Specifically, the ratio of official document texts belonging to the official document corpora to those belonging to the general corpora can be 1:1. By combining official document corpora and general corpora into various official document texts, data balance can be achieved. When using such an official document dataset to fine-tune the pre-trained model, it helps to mitigate the catastrophic forgetting problem that may occur during training.

[0099] In step S105, writing prompts are annotated for each official document text to obtain the corresponding official document writing instruction prompts. Specifically, when executing step S105, to ensure data security and privacy, each official document text can first undergo desensitization processing. Desensitization involves the use of various technologies, including Named Entity Recognition (NER) and regularization methods. NER technology is particularly helpful in identifying key entities in the text, such as names of people, places, and organizations. Regular expressions identify formatted sensitive data, including phone numbers, email addresses, social security numbers, or credit card numbers. Once these specific data patterns are identified, the sensitive information is replaced with forged data or masked. In this embodiment, desensitization can be performed using methods such as masking, spoofing, generalization, and randomization. Masking replaces part or all of the sensitive data with asterisks, random characters, or other unrecognizable placeholders. Spoofing replaces sensitive data with fake but seemingly real data to protect the privacy of the original data. Generalization replaces the precise values ​​of sensitive data with a broader range or category; for example, replacing the specific age "31" with the age range "30-40". Mechanization methods modify the data by adding random noise to the sensitive data, making its original values ​​indeterminate.

[0100] In step S105, after anonymizing each official document text, annotation can be performed manually by annotators to add writing prompts, or a large model can be used to assist manual annotation. When annotators manually add writing prompts, they can read the content of each document and summarize its content as a summary or keywords, or summarize the instructions for writing the document, thus obtaining the corresponding writing instruction prompts. When using a large model to assist manual annotation, the document dataset can be loaded into the model, allowing it to generate diverse writing instruction prompts for each data point based on its learned patterns and context. The human annotator then reviews and corrects the writing instruction prompts provided by the large model.

[0101] In this embodiment, when performing step S3, which is to perform the first training process on the pre-trained model and the low-rank matrix using the official document dataset to obtain the basis model, the following steps can be specifically performed:

[0102] S301. Set the reduced-dimensional matrix and the increased-dimensional matrix as low-rank matrices;

[0103] S302. Randomize the dimension-reducing matrix using a Gaussian random distribution and initialize the dimension-increasing matrix as a zero matrix;

[0104] S303. Input the document writing instruction prompt into the pre-trained model after adding a low-rank matrix. The document writing instruction prompt is processed by the weight matrix and then by the dimension reduction matrix and the dimension increase matrix. The processing results of the weight matrix and the low-rank matrix are fused to obtain the first output result. The parameters of the weight matrix are fixed, and the parameters of the dimension reduction matrix and the dimension increase matrix are updated according to the first output result and the document text.

[0105] S304. Multiply the updated dimension-reduced matrix and the dimension-increasing matrix by the result, and then superimpose the result on the weight matrix to update the weight matrix.

[0106] S305. Use the pre-trained model updated with weight matrix parameters as the base model.

[0107] The process of steps S301-S305 is as follows: Figure 3 As shown. (Refer to...) Figure 3 LoRA (Low-Rank Adaptation) can be used to perform the first training process, effectively fine-tuning the pre-trained model with limited resources by updating only the parameters of the side-path matrix, rather than all parameters of the entire model. LoRA is an efficient parameter-efficient fine-tuning technique that updates the model by introducing a low-rank matrix into the model's weight matrix.

[0108] Specifically, in step S301, a low-rank matrix is ​​added as a bypass structure on the basis of the pre-trained model, i.e. the large model Qwen-72B. The low-rank matrix includes a dimension-reduced matrix A and a dimension-increasing matrix B.

[0109] In step S302, the dimension reduction matrix A is initialized using a random Gaussian distribution, while the dimension increase matrix B is initialized to a zero matrix. That is, after executing step S302 and before executing step S303, the values ​​of the elements in the dimension reduction matrix A are randomly determined according to a Gaussian distribution, and the values ​​of the elements in the dimension increase matrix B are all 0. Here, if we assume the weight matrix of the pre-trained model Qwen-72B is W... 0 ∈R d×k Then B∈R d×r ,A∈R r×k , r << min(d,k).

[0110] In steps S303-S305, refer to Figure 3 The parameters (including input / output dimensions, etc.) of the fixed pre-trained model, i.e. the large model Qwen-72B, remain unchanged. In the output stage of the pre-trained model, the increased dimension matrix B and the reduced dimension matrix A are multiplied and then merged with the parameters of the Qwen-72B model to achieve effective fusion of the original information and the newly added information.

[0111] Specifically, when performing the first training process on the pre-trained model, the fine-tuning update strategy is as follows:

[0112] In step S303, let the weight matrix of the pre-trained model Qwen-72B be W. 0 ∈R d×k The constraints for updating are represented by low-rank decomposition, and W is used during training. 0 Once fixed, gradient updates will no longer be performed; refer to Figure 4 A document writing instruction prompt x is input into the pre-trained model Qwen-72B. The document writing instruction prompt x is then processed by the weight matrix W of the pre-trained model Qwen-72B itself. 0 The process involves continuous processing using a dimensionality reduction matrix A and an increased dimensionality matrix B, culminating in a fusion weight matrix W. 0 The processing results of the low-rank matrix (dimension-reduced matrix A and dimension-increasing matrix B) and the processing results of the low-rank matrix are used to obtain the first output result h, and the weight matrix W is fixed. 0 The parameters remain unchanged. The loss function value is calculated based on the first output result h and the corresponding document text x (for example, the Manhattan distance between the first output result h and the corresponding document text x can be calculated as the loss function value). The parameters of the dimension reduction matrix A and the dimension increase matrix B are updated based on the loss function value.

[0113] Figure 4 The diagram shows the flow of one execution of step S303. After executing step S303 multiple times, the first training process ends, and step S304 is executed, where the result of multiplying the parameter-updated dimensionality reduction matrix A and the dimensionality increase matrix B is superimposed on the weight matrix W. 0 Thus, the weight matrix W 0 Update. Step S304 can be expressed by the formula W. 0 =W 0 +ΔW=W 0 +BA. For the input document writing instruction prompt x, the model's forward propagation process updates h = W. 0 x + ΔWx = W 0 x+BAx.

[0114] By executing steps S301-S304, the weight matrix parameters W in the pre-trained model can be adjusted. 0 In step S305, the parameters W of the weight matrix are updated. 0 The pre-trained model is then used as the base model.

[0115] By executing steps S301-S305, the parameters of the Qwen-72B model can be fixed during training, and only the parameters of the newly added A and B matrices can be updated. This ensures that the performance of the original model is not affected, allowing the focus to be on optimizing the bypass structure and improving the efficiency of the first training process.

[0116] In this embodiment, the first training process can be performed by combining LoRA and Zero3 technologies. Specifically, refer to... Figure 5 During steps S301-S305, the pre-trained model can be split into multiple sub-models, and each sub-model can be distributed to a corresponding sub-process. Specifically, multiple sub-processes can be created, and the parameters W of the weight matrix of the pre-trained model can be... 0 The model is copied to various sub-processes, each of which obtains its corresponding sub-model. Each sub-process executes the first training process (similar to step S303) in parallel, updating the parameters of its respective sub-model. Once the parameters of each sub-model in each sub-process are updated, the trained sub-models are merged to obtain the base model. Specifically, the average parameter value of each sub-model in each sub-process can be calculated and used as the parameters of the base model.

[0117] ZeRo is a novel memory optimization technique for large-scale distributed deep learning. Zero3 reduces the required GPU memory by sharding optimizer states, gradients, and model parameters across each data-parallel worker process. The Zero3 workflow specifically includes:

[0118] ① Model parameter segmentation: Model parameters are segmented and distributed to various processes before loading. ZeRO3 performs this segmentation and distribution during model initialization.

[0119] ② Model Parameter Collection and Initialization: To enable the model to effectively access and manage its parameters during forward and backward propagation, a crucial step in the ZeRO3 initialization process is creating hook functions for each model submodule. Through these hook functions, each submodule automatically acquires the necessary local parameters for computation during each iteration, and promptly releases these memory resources after computation, thereby achieving efficient collection and initialization allocation of model parameters in a multi-GPU environment.

[0120] ③ In the forward propagation, model parameters are obtained. Each submodule in each process only needs to collect parameters before calculation and release parameters after calculation, which greatly reduces the redundant space occupation.

[0121] ④ The model parameters are split and released during backpropagation, thereby reducing the space occupied by the model parameters.

[0122] ⑤ Adjust hyperparameters: AdamW optimizer, β1 = 0.9, β2 = 0.95, eps = 10 -5 The weight decay is 0.1, and the learning rate is 10. -5 .

[0123] ⑥ Model testing: The converged model parameters are merged with the pre-trained model parameters, and the fine-tuned merged model is evaluated on the test set to obtain the model's performance on unknown data.

[0124] In this embodiment, when performing step S4, which is the second training process of reinforcement learning based on human feedback on the base model using the reward dataset to obtain the reward model, the following steps can be specifically performed:

[0125] S401. Input the document writing instruction prompts into the base model;

[0126] S402. Obtain multiple texts to be scored generated by the base model in response to the document writing instruction prompt;

[0127] S403. Obtain the score value for each text to be scored;

[0128] S404. Determine the value of the loss function based on the highest and lowest scores among the scores;

[0129] S405. Update the parameters of the basis model based on the value of the loss function;

[0130] S406. Use the base model with updated parameters as the reward model.

[0131] After executing steps S1-S3, the trained base model possesses the performance to respond to official document writing instruction prompts and generate text containing official document content. In steps S401-S402, the official document writing instruction prompts from the reward dataset are input into the base model, thereby triggering the base model to respond to the official document writing instruction prompts and output multiple texts containing official document content, i.e., the texts to be scored.

[0132] In step S403, human experts can read each text to be scored output by the basis model and score each text based on human document writing and reading norms, as well as subjective feelings, thereby determining the score value for each text. For example, in response to the input document writing instruction prompt x, the basis model generates and outputs texts y1, y2, ... awaiting scoring, and the score values ​​obtained for these texts are respectively "score". y1 score y2 ... score y1 score y2 The maximum value in ..., i.e., the highest score, is denoted as y. c The minimum value, or the lowest score, is denoted as y. r .

[0133] In step S404, with the weight matrix parameter of the basis model being θ, the loss function L is used. ranking =-log(σ(r) θ (x,y c )-r θ (x,y r The calculation is performed to obtain the value L of the loss function. ranking .

[0134] In step S405, based on the value L of the loss function... ranking The size of the value is used to update the weight matrix parameters θ of the base model, thereby obtaining the reward model.

[0135] By performing steps S401-S405, a reward model can be trained to convert human preferences into a binary ranking label format and ensure that the score of the preferred text is higher than that of responses that do not conform to human preferences. This allows the reward model to be guided by human feedback to generate more personalized outputs that conform to official document writing standards.

[0136] In this embodiment, when performing step S5, which is to perform proximal policy optimization on the reward model to obtain a large language model, the following steps can be specifically performed:

[0137] S501. Set optimization goals;

[0138] S502. Based on the optimization objective, perform a proximal policy optimization algorithm on the reward model to iteratively update the parameters of the reward model;

[0139] S503. Use the updated reward model as the large language model.

[0140] In step S501, the optimization objective is...

[0141] argmaxπ E p～D,g～π [R(g|p)]

[0142] p represents the prompt for official document writing, D represents the dataset containing prompt p, g represents the policy used by the near-end policy optimization algorithm, π represents the dataset containing policy g, E represents the expectation function, and R(g|p) represents the reward function for prompt p under policy g. The entire formula represents optimizing a policy set π such that the expected total reward under this policy is maximized when prompt p conforms to the dataset distribution D.

[0143] The optimization objective in step S501 can be considered as the true reward function (human preference), and can serve as a strategy for optimizing the reward model. Step S502 involves sampling the document writing instruction prompt p from dataset D and sampling the policy g from the policy dataset π to iteratively improve the policy, using the PPO algorithm and loss function to achieve this objective. The PPO algorithm updates the parameters of the reward model iteratively; that is, the execution process of the PPO algorithm is iterative. Each iteration includes three steps: model execution, feedback collection, and policy adjustment. After completing these three steps in one iteration, the next iteration begins.

[0144] When performing steps S501-S503, the following training parameters can be used: using the AdamW optimizer, β1 = 0.9, β2 = 0.95, eps = 10 -5 The weight decay is 0.1, the gradient clipping is 1.0, and the learning rate is 10. -6 Batch size = 512, PPO trimming threshold 0.2.

[0145] By executing steps S501-S503, the reward model can learn more and more knowledge from human feedback. The accumulated knowledge is stored in the form of parameters of the reward model. After the parameters of the reward model are no longer updated, the reward model becomes a large language model. The official document text output by the large language model in response to the prompts of official document writing instructions can be close to human writing and reading habits.

[0146] In this embodiment, after executing steps S1-S5, a model interface can be built using vLLM and Fschat to accelerate model inference for large language models. Specifically, vLLM is an LLM (Large Language Model) inference and deployment service library that combines iterative-level scheduling (continuous batching) and the PagedAttention algorithm to improve service throughput. Its core is the PagedAttention algorithm, which allows storing contiguous keys and values ​​in non-contiguous memory spaces.

[0147] vLLM and Fschat can perform attention computation efficiently and support efficient memory sharing, further reducing the memory overhead of complex sampling algorithms and improving throughput, thereby accelerating model inference.

[0148] After establishing the model interface using vLLM and Fschat, functional, performance, and speed tests can be performed on the large language model. The tested model can then be deployed to a production environment to provide document writing services. When staff members need to write official documents, they can edit writing instructions based on the desired content, input these instructions into the large language model, and receive the generated document text. Staff members can then review and modify the generated text to obtain the desired document content.

[0149] The large language model trained using the large language model training method for official document writing in this embodiment is fine-tuned based on a general pre-trained model. Therefore, it can adapt to the specific style, structure, and content requirements of official documents, enhancing the model's adaptability and generalization ability across different writing tasks and styles, so as to better meet users' personalized writing needs. Fine-tuning the model through LoRA and human feedback reinforcement learning consumes fewer resources, enabling efficient model optimization with limited resources, and also avoiding overfitting of the large language model on specific tasks.

[0150] In some optional implementations of the large language model training method for official document writing, a specialized data balancing strategy is employed to ensure a 1:1 ratio of professional domain writing data to general language corpora during training, mitigating catastrophic forgetting. This strategy not only helps the model better understand the professionalism and standardization of official document writing but also enables it to maintain excellent performance in general language understanding. Targeted document preprocessing steps are used to classify document topics, convert formats, and anonymize data, thereby improving the model's recognition and processing capabilities. Combining LoRA and Zero3 techniques reduces the hardware resource requirements for model training; this method updates only the parameters of the bypass matrix, rather than all parameters of the entire model. By combining memory optimization techniques to reduce the hardware resource requirements during model training and inference, costs are saved and efficiency is improved. Reinforcement learning-based fine-tuning enables customized training for specific document writing standards and styles, specifically improving document quality, including key elements such as logical clarity, linguistic rigor, and legal validity. The PagedAttention algorithm, based on operating system virtual memory and paging concepts, reduces computational load, thus accelerating model inference. A model interface is built using vLLM and Fschat to achieve streaming output and accelerated model inference.

[0151] A computer program for training a large language model for official document writing as described in this embodiment can be written into a computer device or storage medium. When the computer program is read out and run, the large language model training method for official document writing as described in this embodiment is executed, thereby achieving the same technical effect as the large language model training method for official document writing in this embodiment.

[0152] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. Furthermore, the descriptions of "upper," "lower," "left," and "right" used in this disclosure are only relative to the relative positional relationships of the components of this disclosure in the accompanying drawings. The singular forms "a," "an," and "the" used in this disclosure are also intended to include the plural forms, unless the context clearly indicates otherwise. Moreover, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this embodiment specification is only for describing particular embodiments and is not intended to limit the invention. The term "and / or" as used in this embodiment includes any combination of one or more of the associated listed items.

[0153] It should be understood that although the terms first, second, third, etc., may be used to describe various elements in this disclosure, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, a first element may also be referred to as a second element without departing from the scope of this disclosure, and similarly, a second element may also be referred to as a first element. The use of any and all instances or exemplary language (“e.g.,” “such as,” etc.) provided in this embodiment is intended only to better illustrate embodiments of the invention and, unless otherwise required, does not impose a limitation on the scope of the invention.

[0154] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can be implemented using standard programming techniques—including a non-transitory computer-readable storage medium configured with a computer program, wherein such a storage medium causes the computer to operate in a specific and predefined manner—according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).

[0155] Furthermore, the procedures described in this embodiment can be performed in any suitable order unless otherwise indicated by this embodiment or clearly contradicted by the context. The procedures (or variations and / or combinations thereof) described in this embodiment can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. A computer program includes multiple instructions executable by one or more processors.

[0156] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention of this embodiment includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques of the invention, the invention also includes the computer itself.

[0157] A computer program can be applied to input data to perform the functions of this embodiment, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects generated on the display.

[0158] The above are merely preferred embodiments of the present invention. The present invention is not limited to the above-described embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention, as long as they achieve the technical effects of the present invention by the same means, should be included within the scope of protection of the present invention. Within the scope of protection of the present invention, the technical solutions and / or implementation methods can have various modifications and variations.

Claims

1. A method for training a large language model for official document writing, characterized in that, The method for training a large language model for official document writing includes: Obtain a document dataset and a reward dataset; the document dataset includes multiple document writing instruction prompts and document texts, and the reward dataset includes multiple document writing instruction prompts. Obtain the general-purpose large model Qwen-72B as a pre-trained model; A low-rank matrix is ​​added in parallel to the weight matrix of the pre-trained model. The document dataset is used to perform a first training process on the pre-trained model and the low-rank matrix to obtain a base model. During the first training process, the parameters of the weight matrix are fixed and the low-rank matrix is ​​updated. After the first training process is completed, the parameters of the low-rank matrix are merged into the weight matrix. Input the document writing instruction prompt into the base model; Obtain multiple texts to be scored generated by the base model in response to the document writing instruction prompt; Obtain the score value for each of the texts to be scored; The value of the loss function is determined based on the highest and lowest scores among the given scores; The parameters of the base model are updated based on the value of the loss function; The base model with updated parameters is used as the reward model; Wherein, the loss function is (s( (r))) The value of the loss function, The weight matrix parameters of the basis model are: At that time, according to the prompts in the official document writing instructions And the basis model responds to Generated text to be scored The corresponding score, The highest score is... The minimum score is given by σ, which is the sigmoid function. (r) is The corresponding text and Marginal label correction for the corresponding text; the marginal label correction is used to describe The corresponding text and The difference between the corresponding texts; Perform near-end policy optimization on the reward model to obtain a large language model; The step of performing a first training process on the pre-trained model and the low-rank matrix using the document dataset to obtain the base model includes: The pre-trained model is split into multiple sub-models, and each sub-model is distributed to a corresponding sub-process. In each subprocess, a corresponding low-rank matrix is ​​added to the sub-model in parallel, and the first training process is performed on the sub-model and the low-rank matrix using the document dataset. The sub-models trained by each sub-process are merged to obtain the base model.

2. The method for training a large language model for official document writing according to claim 1, characterized in that, The acquisition of the official document dataset and the reward dataset includes: The corpus of official documents was obtained from publicly available sources; the corpus of official documents contains official document content. Obtain general corpus; The official document corpus and the general corpus are preprocessed; The official document corpus and the general corpus are combined in a certain proportion to obtain the texts of each official document; Each of the official document texts is annotated with writing prompts to obtain the corresponding official document writing instruction prompts.

3. The method for training a large language model for official document writing according to claim 1, characterized in that, The step of performing a first training process on the pre-trained model and the low-rank matrix using the document dataset to obtain the base model includes: Set a dimension-reducing matrix and a dimension-increasing matrix as the low-rank matrix; The reduced-dimensional matrix is ​​randomized using a Gaussian random distribution, and the increased-dimensional matrix is ​​initialized as a zero matrix. The document writing instruction prompt is input into the pre-trained model after adding the low-rank matrix. The document writing instruction prompt is processed by the weight matrix and then by the dimension reduction matrix and the dimension increase matrix. The processing results of the weight matrix and the low-rank matrix are fused to obtain the first output result. The parameters of the weight matrix are fixed, and the parameters of the dimension reduction matrix and the dimension increase matrix are updated according to the first output result and the document text. The result of multiplying the reduced-dimensional matrix and the increased-dimensional matrix after parameter updates is superimposed on the weight matrix, thereby updating the weight matrix; The pre-trained model updated with the weight matrix parameters is used as the base model.

4. The method for training a large language model for official document writing according to claim 1, characterized in that, The step of performing proximal policy optimization on the reward model to obtain a large language model includes: Set optimization goals; Based on the optimization objective, a near-end policy optimization algorithm is executed on the reward model to iteratively update the parameters of the reward model; The updated reward model is used as the large language model. Wherein, the optimization objective is [R( g | p )] This is a prompt for official document writing instructions. Instructions for official document writing The dataset in which it is located The strategy used by the near-end policy optimization algorithm. For strategy The dataset in which it is located Let R(g|p) represent the expectation function, and R(g|p) represent the policy. Prompt for issuing official document writing instructions The reward function.

5. The method for training a large language model for official document writing according to any one of claims 1-4, characterized in that, The large language model training method for official document writing also includes: The model interface for the large language model is built using vLLM and Fschat.

6. A computer device, characterized in that, It includes a memory and a processor, the memory being used to store at least one program, and the processor being used to load at least one program to execute the large language model training method for official document writing as described in any one of claims 1-5.

7. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the large language model training method for official document writing as described in any one of claims 1-5.

Images