A method and system for dynamic word count control of large language model text output

Abstract

The present invention relates to the technical field of large models, and discloses a method and system for dynamic word count control of text output of a large language model. By collecting data, high-quality question-answer pair data and question data are automatically generated based on the large model; the large model is fine-tuned using the question-answer pair data set to obtain a professional model; a reward model is trained using the question data set to score the content generated by the large model according to the word count; the parameters of the word count control model are constructed and optimized based on the question-answer pair data set and the reward model based on a reinforcement learning algorithm; when a user inputs a question and a word count requirement, the word count requirement is assigned to each chapter, and the word count control model is iteratively called in units of chapters to generate each chapter, and finally the entire article is spliced ​​together. The present invention can accurately control the output word count of the large model, has the advantages of accurate and controllable generation, thereby meeting the needs of specific application scenarios, avoiding publishing problems or format errors caused by too many or too few words, and enhancing user experience.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to a method and system for dynamic word count control of large language model text output. Background Art

[0002] In today's rapidly developing field of artificial intelligence, large language models, leveraging their vast data sets and complex algorithmic structures, are now capable of performing a variety of language processing tasks, including question-answering, text generation, and writing assistance. Large language models not only provide real-time feedback but can also create stories, write reports, and even proofread and edit manuscripts.

[0003] However, despite the remarkable achievements of large language models in text generation, they still face challenges in controlling the number of words in their output. Sometimes, the text generated by the model may exceed the expected word count, which not only makes the content too lengthy but also dilutes key information. Conversely, if the number of words generated is insufficient, important information may be omitted, affecting the integrity and persuasiveness of the content. In many application scenarios, there are strict requirements on the number of words in articles or texts, such as news reports, academic papers, and social media posts. Precisely controlling the number of words in the output can avoid publishing problems or formatting errors caused by too many or too few words. Therefore, how to accurately control the number of words in the text output of large models to meet the needs of specific application scenarios and improve user experience has become an urgent problem to be solved. Summary of the Invention

[0004] The present invention provides a method and system for dynamic word count control of large language model text output, aiming to solve at least one of the above technical problems.

[0005] To achieve the above objectives, the present invention provides a method for dynamic word count control of text output in a large language model, comprising:

[0006] S1: Prepare reference materials, generate several question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair dataset and a question dataset; each question includes a word count requirement;

[0007] S2: Use the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model;

[0008] S3: Input the question dataset into the professional model, generate multiple answers, determine the reward value based on the number of words for each answer, and build and train the reward model;

[0009] S4: constructing an initial word count control model, using the question dataset and the reward model, optimizing the parameters of the initial word count control model based on a reinforcement learning algorithm, and generating a final word count control model;

[0010] S5: Obtaining a target question and a target word count requirement input by a user, generating a text outline including several chapters based on the target question, and determining a chapter word count allocation strategy for the target question based on the text outline and the target word count requirement;

[0011] S6: According to the chapter word count allocation strategy, the word count control model is used to generate text for each chapter in the text outline in turn to obtain a final generated text.

[0012] Optionally, S1: prepare reference materials, generate several question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair dataset and a question dataset, specifically including:

[0013] S11: Obtain relevant information; wherein the relevant information is configured as professional information related to the target problem collected through various channels;

[0014] S12: extracting text content from the relevant materials, dividing the relevant materials into several sections of reference materials according to each preset number of words as a reference material text;

[0015] S13: For each reference material, use the target large model to generate several question-answer pairs, construct a question-answer pair dataset, extract the question data from each question-answer pair, and construct a question dataset; wherein each question includes a word count requirement.

[0016] Optionally, S2: using the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model step, specifically including:

[0017] S21: A lora adapter is set between the q layer and the v layer of each transformer layer of the target large model to capture the knowledge features in the reference material;

[0018] S22: For each question-answer pair in the question-answer pair dataset, input the correct answer and multiple irrelevant answers as input data into the target large model;

[0019] S23: Calculate the loss function based on the difference between the answer output by the target large model and the correct answer, and update the parameters of the lora adapter through the back propagation algorithm to minimize the loss function;

[0020] S24: After iterative training, until the model performance reaches the predetermined standard or the training reaches the preset rounds, a professional model is generated based on the parameters of the original target large model and the fine-tuned lora adapter parameters.

[0021] Optionally, S3: inputting the question dataset into a professional model, generating multiple answers, determining a reward value based on the number of words in each answer, and building and training a reward model, specifically includes:

[0022] S31: Sampling questions from the question dataset to obtain a training question dataset, repeatedly inputting the questions in the training question dataset into the professional model to generate multiple answers;

[0023] S32: Calculate the word count of each answer, sort the answers according to how close the word count is to the requirement, and normalize them into a scalar value, using the scalar value as a reward value to represent the preference for word count;

[0024] S33: Input each question, answer, and corresponding reward value in the training question dataset into the opt-350M model to train and obtain a reward model.

[0025] Optionally, the reward model is configured to use the opt-350M model and change the last output layer to a feedforward network with a single output node, thereby outputting a scalar value representing the reward; during training, the reward model calculates a loss function for each of the multiple question-answer pairs, and then adds and averages the results. The loss function uses a binary cross-entropy loss function, so that the model can predict the reward value for a given number of words;

[0026] The expression of the loss function is specifically:

[0027] ;

[0028] in, is the total number of times to judge. If multiple questions and answers are calculated in pairs, then , is the total number of question-answer pairs; is the Sigmoid function, which maps real numbers to between (0,1); For the current parameter The following answers The reward value; For the current parameter The following answers The reward value.

[0029] Optionally, S4: constructing an initial word count control model, using the problem dataset and the reward model, optimizing the parameters of the initial word count control model based on a reinforcement learning algorithm, and generating a final word count control model, specifically includes:

[0030] S41: Sampling questions from the question dataset to obtain an optimized question dataset, constructing an initial word count control model having the same network structure and initial parameters as the professional model, inputting the questions in the optimized question dataset into the initial word count control model to obtain a first answer, and forming a first optimized question-answer pair based on the corresponding questions and the first answers;

[0031] S42: Input the first optimized question-answer pair into the reward model to obtain a reward value output by the reward model;

[0032] S43: Constructing a reference model with the same network structure and initial parameters as the professional model, inputting the question in the optimization problem dataset into the reference model to obtain a second answer, and forming a second optimized question-answer pair based on the corresponding question and the second answer;

[0033] S44: Calculating the data distribution gap between the first answer and the second answer by using KL divergence, using the reward value minus the data distribution gap as a reward function for reinforcement learning, optimizing the parameters of the initial word count control model based on the reinforcement learning algorithm, and generating a final word count control model;

[0034] The expression of the data distribution gap is specifically:

[0035] ;

[0036] in, is the KL divergence, which indicates the degree of deviation of the word count control model; represents the first answer to the question by the word count control model; represents the reference model's second answer to the question;

[0037] in, Represents a question-answer pair consisting of question x and the answer y obtained by the word count control model for question x; Represents the answer obtained by question x and the reference model of question x input The question and answer pairs constituted;

[0038] The expression of the reward function is specifically:

[0039] in, is the penalty coefficient. When the KL divergence is large, When the value is large, and the KL divergence is small, The value is smaller.

[0040] Optionally, the step of optimizing the parameters of the initial word count control model based on the reinforcement learning algorithm specifically includes: using the word count control model as an intelligent agent in the reinforcement learning algorithm, and using the optimization problem data set as a state in the reinforcement learning algorithm; when optimizing the parameters of the initial word count control model, inputting the problem in the optimization problem data set into the word count control model to obtain the model output, inputting the problem and the model output into the reward model to calculate the reward value, and calculating feedback through the reward function according to the reward value to determine the parameter update.

[0041] Optionally, S5: obtaining a target question and a target word count requirement input by a user, generating a text outline including several chapters based on the target question, and determining a chapter word count allocation strategy for the target question based on the text outline and the target word count requirement, specifically comprising:

[0042] S51: Obtaining a target question and a target word count requirement input by a user, and generating a text outline including several chapters and an importance level of each chapter based on the target question;

[0043] S52: Based on the importance level and target word count requirement of each chapter in the text outline, generate the text word count for each chapter according to the proportion corresponding to each importance level, and determine the chapter word count allocation strategy for the target problem.

[0044] Optionally, S6: generating text for each chapter in the text outline in sequence using the word count control model according to the chapter word count allocation strategy to obtain a final generated text step, specifically comprising:

[0045] S61: Generate word counts based on the text of each chapter, and iteratively call the word count control model to generate the text of each chapter in units of chapters;

[0046] S62: After generating the text of each chapter, determining the word count requirement for the remaining chapters based on the word count of the currently generated chapter and the target word count requirement, and re-determining the chapter word count allocation strategy for the remaining chapters in the text outline for which the text has not been generated based on the remaining chapter word count requirement;

[0047] S63: Execute text generation for the remaining chapters that have not yet been generated according to the re-determined chapter word count allocation strategy until text generation is completed for each chapter to obtain a final generated text.

[0048] In addition, to achieve the above-mentioned purpose, the present invention also provides a large language model text output dynamic word count control system, comprising:

[0049] A data construction module is used to prepare reference materials, generate multiple question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair dataset and a question dataset; wherein each question includes a word count requirement;

[0050] A professional model generation module is used to use the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model;

[0051] A reward value evaluation module, configured to input the question dataset into a professional model, generate multiple answers, determine a reward value based on the word count of each answer, and construct and train a reward model;

[0052] a model optimization module, configured to construct an initial word count control model, optimize the parameters of the initial word count control model based on a reinforcement learning algorithm using the problem dataset and the reward model, and generate a final word count control model;

[0053] A word count allocation module is configured to obtain a target question and a target word count requirement input by a user, generate a text outline comprising a plurality of chapters based on the target question, and determine a chapter word count allocation strategy for the target question based on the text outline and the target word count requirement;

[0054] The article generation module is used to generate text for each chapter in the text outline in turn according to the chapter word count allocation strategy and adopt the word count control model to obtain a final generated text.

[0055] The beneficial effects of the present invention are as follows: a method and system for dynamic word count control of text output of a large language model are proposed, which collects professional-related information through multiple channels; based on the collected information, high-quality question-answer pair data and question data are automatically generated based on the large model; the large model is fine-tuned using the question-answer pair data set to obtain a professional model, making it more suitable for specific professional tasks; the reward model is trained using the question data set, and the content generated by the large model is scored according to the word count, so that the model can accurately predict the preference for a given word count; the question-answer pair data set and the reward model are used to optimize the parameters of the word count control model based on the reinforcement learning algorithm; when the user inputs the question and word count requirement, the word count requirement is allocated to each chapter (or paragraph) using a word count allocation method, thereby strengthening the control of the word count; after completing the output word count allocation, a prompt is dynamically constructed, and the word count control model is iteratively called in units of chapters to generate each chapter, and finally the chapters are spliced ​​together to generate the entire article; for the generated article, a summary is performed based on the word count control model to further standardize the word count. The present invention can accurately control the number of output words of a large model, and has the advantages of accurate and controllable generation, thereby meeting the needs of specific application scenarios, avoiding publishing problems or format errors caused by too many or too few words, and enhancing user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of a method for dynamic word count control of a large language model text output according to an embodiment of the present invention;

[0057] Figure 2 Schematic diagram of the structure of a dynamic word count control system for large language model text output according to an embodiment of the present invention.

[0058] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] The embodiment of the present invention provides a method for controlling the number of dynamic words in a large language model text output. Figure 1 , Figure 1 This is a flow chart of a method for dynamically controlling the number of words in a large language model text output according to an embodiment of the present invention. In this embodiment, the method for dynamically controlling the number of words in a large language model text output includes the following steps:

[0061] S1: Prepare reference materials, generate several question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair dataset and a question dataset; each question includes a word count requirement;

[0062] S2: Use the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model;

[0063] S3: Input the question dataset into the professional model, generate multiple answers, determine the reward value based on the number of words for each answer, and build and train the reward model;

[0064] S4: constructing an initial word count control model, using the question dataset and the reward model, optimizing the parameters of the initial word count control model based on a reinforcement learning algorithm, and generating a final word count control model;

[0065] S5: Obtaining a target question and a target word count requirement input by a user, generating a text outline including several chapters based on the target question, and determining a chapter word count allocation strategy for the target question based on the text outline and the target word count requirement;

[0066] S6: According to the chapter word count allocation strategy, the word count control model is used to generate text for each chapter in the text outline in turn to obtain a final generated text.

[0067] It should be noted that although large language models have made significant achievements in text generation, they still face challenges in controlling the number of words in their output. Sometimes, the text generated by the model may exceed the expected word count, which not only makes the content too lengthy but also dilutes key information. Conversely, if the number of words generated in the text is insufficient, important information may be omitted, affecting the integrity and persuasiveness of the content. In many application scenarios, there are strict requirements on the number of words in articles or texts, such as news reports, academic papers, and social media posts. Precisely controlling the number of words in the output can avoid publishing problems or formatting errors caused by too many or too few words. Therefore, how to accurately control the number of words in the text output of large models to meet the needs of specific application scenarios and improve user experience has become an urgent problem to be solved.

[0068] To address the above-mentioned issues, this embodiment collects professional-related materials through multiple channels. Based on the collected materials, high-quality question-answer pair data and question data are automatically generated based on a large model. The large model is fine-tuned using the question-answer pair data set to obtain a professional model, making it more suitable for specific professional tasks. A reward model is trained using the question data set to score the content generated by the large model based on word count, enabling the model to accurately predict preferences for a given word count. The question-answer pair data set and the reward model are used to optimize the parameters of a word count control model based on a reinforcement learning algorithm. When a user enters a question and word count requirement, a word count allocation method is used to allocate the word count requirement to each chapter (or paragraph), thereby strengthening word count control. After the output word count allocation is completed, a prompt is dynamically constructed, and the word count control model is iteratively called on a chapter-by-chapter basis to generate each chapter, and finally the chapters are spliced ​​together to generate the entire article. The generated article is summarized based on the word count control model to further standardize the word count. The present invention can accurately control the output word count of the large model, has the advantages of accurate and controllable generation, thereby meeting the needs of specific application scenarios, avoiding publishing problems or format errors caused by excessive or insufficient word count, and enhancing the user experience.

[0069] In order to explain the present invention more clearly, a specific example application of the method for dynamic word count control of large language model text output in actual scenarios is provided below.

[0070] Step 1: Data preparation.

[0071] Collect professional-related information through various channels such as libraries, archives, and the Internet. Based on the collected information, automatically generate high-quality question-answer data based on the big model to build a data set ; Question-Answer Dataset Extract the questions and build a question dataset , used for subsequent model training and word count restrictions. The question data includes word count requirements.

[0072] The method for generating question-answer pairs involves segmenting preprocessed data into 200-word segments, which serve as reference material for generating question-answer pairs. For paragraphs shorter than 200 words, these are directly used as reference material. For each reference, a large model (such as Qwen or llama) is used to generate two to five question-answer pairs closely related to the reference material. The generated question-answer pairs are manually reviewed and revised to construct a high-quality dataset.

[0073] Furthermore, the format of the question-answer pair output is:

[0074] Output in the following json format [

[0076] {

[0077] "question": <question>,

[0078] "answer": <answer>

[0079] } ]

[0081] Furthermore, the generated question-answer pair examples are:

[0082] "question":Why is the sky blue? The answer should be 40 words.

[0083] "answer":The sky appears blue because molecules and other particles in the atmosphere scatter the blue light from sunlight. This phenomenon is called Rayleigh scattering.

[0084] Step 2: Professional model fine-tuning.

[0085] Select a basic large model and use the question-answer dataset Perform LoRa fine-tuning to obtain a professional model , making it more suitable for specific professional tasks and improving the accuracy and relevance of the model in answering professional questions.

[0086] The basic large models include qwen, llama, etc., from which basic models with good performance and suitable for fine-tuning are selected according to needs and preferences, ensuring that the selected models have sufficient capacity and generalization ability so that they can adapt to professional tasks through fine-tuning.

[0087] The LoRa fine-tuning method is to set up a LoRa adapter between the Q layer (query layer) and the V layer (value layer) of each transformer layer to capture and store specific professional knowledge. For each question, the correct answer and multiple irrelevant answers are used as input data, and the large model is asked to answer relevant questions based on the input data. The loss function is calculated based on the difference between the model's answer and the correct answer, and the parameters of the LoRa adapter are updated through the backpropagation algorithm to minimize the loss function. The training process will iterate multiple times until the model performance reaches the predetermined standard or the training reaches the preset rounds. After training is completed, the parameters of the original large model and the fine-tuned LoRa adapter parameters are merged to generate a professional large model.

[0088] Furthermore, the input examples for model fine-tuning are:

[0089] {

[0090] "instruction":"Use <data>< / data> Mark the content as your knowledge to answer the question. Question: <question>",

[0091] "input":" <data>

[0092] 1 Answer

[0093] ---

[0094] 2 Answers

[0095] ---

[0096] 3 Answers

[0097] ---

[0098] 4 Answers

[0099] < / data> ",

[0100] "output": <answer>,

[0101] "system":"You are an expert and need to answer questions about your professional knowledge."

[0102] }

[0103] Step 3: Reward model training.

[0104] Combining reinforcement learning with human feedback, by incorporating feedback on the number of generated words into the training process, the number of words is used as a reward signal to guide model training, providing a natural and humane learning process for the large model, thereby enhancing the large model's understanding and compliance with the word count requirements.

[0105] One of the key steps in controlling the number of words generated by the reinforcement learning-based control model is to train the reward model to score the content generated by the large model based on the number of words, so that the model can accurately predict the preference for a given number of words.

[0106] Specifically, the process of training the reward model is:

[0107] Using random sampling method, we can select Sampling Problems Building Problem Datasets . Combine the question and the word count requirement to form a new question.

[0108] The problem dataset Issues in duplicate entry of specialized models , generate multiple answers. Calculate the word count of each answer, sort the answers by how close they are to the requirement, and normalize them to a scalar value , as a reward value, which numerically represents the preference for the number of words.

[0109] Input the questions, answers and corresponding reward values ​​into the opt-350M model and train the reward model .

[0110] The random sampling method can ensure the randomness and representativeness of the sample, is suitable for situations where each part of the population has the same probability, and can better reflect the characteristics of the population.

[0111] The goal of the reward model is to accept a sequence of text and return a scalar reward representing the word count preference. Specifically, the opt-350M model is used, and the final output layer is converted to a feed-forward network with a single output node, outputting a scalar value representing the reward. During model training, the loss function is calculated for each of the multiple question-answer pairs and then summed and averaged. The loss function uses a binary cross-entropy loss function, enabling the model to accurately predict the reward value for a given word count.

[0112] ;

[0113] in, is the total number of times to judge. If multiple questions and answers are calculated in pairs, then , is the total number of question-answer pairs; is the Sigmoid function, which maps real numbers to between (0,1); For the current parameter The following answers The reward value; For the current parameter The following answers The reward value.

[0114] The OPT-350M model is a large-scale language model developed by Meta AI. It is a decoder-only Transformer model primarily used for text processing. The OPT-350M model has 24 layers, a hidden layer size of 1024, 16 attention heads, and a head size of 64.

[0115] Step 4: Word count control model training.

[0116] Using the question-answer pair dataset and reward model , optimize the word count control model based on reinforcement learning algorithm Parameters.

[0117] The word count control model The network structure and initial parameters and model same.

[0118] Specifically, the specific process of word count control model training is as follows:

[0119] Using random sampling method, we can select Sampling problems, building new problem datasets .

[0120] The problem dataset Problems in Input word count control model , get the answer , forming a question-answer pair .

[0121] Correctly answer the questions Input Reward Model , get reward value .

[0122] To prevent the model Too much emphasis is placed on the number of words generated and too little emphasis is placed on the content generated. A reference model needs to be added. , input the problem into the reference model , get the answer , forming a question-answer pair The reference model The network structure and parameters of the professional model same.

[0123] Answered by KL divergence calculation and The data distribution gap between them.

[0124]

[0125] in, is the KL divergence, which indicates the degree of deviation of the word count control model; Represents a large model On the issue Answer ; Represents a large model On the issue Answer .

[0126] Use the reward value minus the data distribution gap as the reward function of reinforcement learning, and optimize the model based on the reinforcement learning algorithm Parameters.

[0127] The reinforcement learning algorithm, also known as reinforcement learning, evaluation learning, or enhanced learning, is used to describe and solve the problem of how an agent learns strategies to maximize rewards or achieve specific goals during its interaction with the environment. A common model for reinforcement learning is the standard Markov Decision Process (MDP).

[0128] Reinforcement learning is an agent learning process using a trial-and-error approach. It obtains rewards and guides its behavior by interacting with the environment. The goal is to maximize the reward for the agent.

[0129] Elements of reinforcement learning include:

[0130] Agent (intelligent body, robot, agent): The main body of reinforcement learning training is the agent.

[0131] Environment: The overall background of the entire process is the environment.

[0132] State: The current state of the Agent.

[0133] Action: What actions can the agent take based on the current state?

[0134] Reward: After the agent takes a specific action in the current state, it will receive certain feedback from the environment, which may be a reward or a punishment.

[0135] A reinforcement learning algorithm is used to train a word count control model. The agent is the word count control model, the state is the question dataset, and the action is the output of the word count control model. The question is input into the word count control model to obtain the model output. The model question and output are then input into the reward model to calculate the reward value. The reward function then calculates feedback based on the reward value, thereby determining parameter updates.

[0136] Furthermore, the reward function is:

[0137] ;

[0138] in, is the penalty coefficient. When the KL divergence is large, When the value is large, and the KL divergence is small, A smaller value encourages the word count control model to not overemphasize the generated word count and ignore the generated content.

[0139] Step 5: Output word count allocation.

[0140] When users generate large amounts of text, such as writing a complete proposal or report, they face the challenge of effectively controlling word count. Models for long texts typically generate text in segments. The cumulative word count deviations within each segment can lead to significant deviations in the final word count. Therefore, a word count allocation method is used to distribute the word count requirements to each chapter (or paragraph), thereby strengthening word count control.

[0141] Specifically, the main process of output word count allocation is:

[0142] When the user enters the question and word count requirement, the total word count requirement is first obtained.

[0143] When the user wants to generate a large number of words, he or she should first generate a text outline to determine the number of chapters to be generated, the importance of each chapter, and its role in the overall text, so as to reasonably allocate the number of words.

[0144] Then divide the word count requirements by chapter and calculate the word count requirements for each chapter.

[0145] Among them, the main chapters are allocated more words, while the prologue and conclusion generally have fewer words and are less important, so they account for a smaller proportion of the total word requirement.

[0146] For example, if a proposal consists of five chapters with a total word count of 5,000 words, the word count can be allocated according to the importance of the chapters as follows:

[0147] Prologue: 500 words (10%)

[0148] Chapter 1: 1,000 words (20%)

[0149] Chapter 2: 1500 words (30%);

[0150] Chapter 3: 1,000 words (20%);

[0151] Conclusion: 500 words (10%).

[0152] Step 6: Dynamic prompt settings.

[0153] After the output word count is assigned, the word count control model is iteratively called on a chapter-by-chapter basis to generate each chapter, and finally the chapters are concatenated to generate the entire article. The generated articles are summarized based on the word count control model to further standardize the word count.

[0154] During chapter generation, a dynamic adjustment mechanism is implemented. If a chapter's word count exceeds expectations, the word count allocated to subsequent chapters can be appropriately reduced. Specifically, the word count for the generated chapters is first calculated, and then the word count for the remaining chapters is reallocated using the word count allocation method described in Step 5. Once the word count is allocated, a dynamic prompt is constructed based on the outline and the allocated word count, and the large model is fed into it to obtain a response.

[0155] Reference Figure 2 , Figure 2 Schematic diagram of the structure of a dynamic word count control system for large language model text output according to an embodiment of the present invention.

[0156] like Figure 2 As shown, the large language model text output dynamic word count control system proposed in the embodiment of the present invention includes:

[0157] The data construction module 10 is used to prepare reference materials, generate a number of question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair dataset and a question dataset; wherein each question includes a word count requirement;

[0158] A professional model generation module 20 is used to perform LoRa fine-tuning on the target large model using the question-answer pair dataset to generate a professional model;

[0159] a reward value evaluation module 30 for inputting the question dataset into a professional model, generating multiple answers, determining a reward value based on the word count of each answer, and constructing and training a reward model;

[0160] A model optimization module 40 is configured to construct an initial word count control model, optimize the parameters of the initial word count control model based on a reinforcement learning algorithm using the problem dataset and the reward model, and generate a final word count control model;

[0161] The word count allocation module 50 is configured to obtain a target question and a target word count requirement input by a user, generate a text outline comprising a plurality of chapters based on the target question, and determine a word count allocation strategy for each chapter of the target question based on the text outline and the target word count requirement;

[0162] The article generation module 60 is used to generate text for each chapter in the text outline in turn according to the chapter word count allocation strategy and adopt the word count control model to obtain a final generated text.

[0163] Other embodiments or specific implementations of the large language model text output dynamic word count control system of the present invention can refer to the above-mentioned method embodiments and will not be repeated here.

[0164] It should be understood that, throughout this specification, references to terms such as "one embodiment," "another embodiment," "other embodiments," or "first through Nth embodiments" mean that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0165] It should be noted that, in this document, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0166] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.< / answer> < / question> < / answer> < / question>

Claims

1. A method for dynamic word count control of large language model text output, characterized in that: include: S1: Prepare reference materials, generate several question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair dataset and a question dataset; each question includes a word count requirement; S2: Use the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model; S3: Input the question dataset into the professional model, generate multiple answers, determine the reward value based on the number of words for each answer, and build and train the reward model; S4: constructing an initial word count control model, using the question dataset and the reward model, optimizing the parameters of the initial word count control model based on a reinforcement learning algorithm, and generating a final word count control model; S5: Obtaining a target question and a target word count requirement input by a user, generating a text outline including several chapters based on the target question, and determining a chapter word count allocation strategy for the target question based on the text outline and the target word count requirement; S6: According to the chapter word count allocation strategy, the word count control model is used to generate text for each chapter in the text outline in turn to obtain a final generated text.

2. The method for dynamic word count control of large language model text output according to claim 1, characterized in that: S1: Prepare reference materials, generate several question-answer pairs of the reference materials based on the target large model, and construct question-answer pair datasets and question datasets, specifically including: S11: Obtain relevant information; wherein the relevant information is configured as professional information related to the target problem collected through various channels; S12: extracting text content from the relevant materials, dividing the relevant materials into several sections of reference materials according to each preset number of words as a reference material text; S13: For each reference material, use the target large model to generate several question-answer pairs, construct a question-answer pair dataset, extract the question data from each question-answer pair, and construct a question dataset; wherein each question includes a word count requirement.

3. The method for dynamic word count control of large language model text output according to claim 1, characterized in that: S2: Use the question-answer pair dataset to perform Lora fine-tuning on the target large model to generate a professional model, specifically including: S21: A lora adapter is set between the q layer and the v layer of each transformer layer of the target large model to capture the knowledge features in the reference material; S22: For each question-answer pair in the question-answer pair dataset, input the correct answer and multiple irrelevant answers as input data into the target large model; S23: Calculate the loss function based on the difference between the answer output by the target large model and the correct answer, and update the parameters of the lora adapter through the back propagation algorithm to minimize the loss function; S24: After iterative training, until the model performance reaches the predetermined standard or the training reaches the preset rounds, a professional model is generated based on the parameters of the original target large model and the fine-tuned lora adapter parameters.

4. The method for dynamic word count control of large language model text output according to claim 1, wherein: S3: Input the question dataset into the professional model, generate multiple answers, determine the reward value based on the word count of each answer, and build and train the reward model, specifically including: S31: Sampling questions from the question dataset to obtain a training question dataset, repeatedly inputting the questions in the training question dataset into the professional model to generate multiple answers; S32: Calculate the word count of each answer, sort the answers according to how close the word count is to the requirement, and normalize them into a scalar value, using the scalar value as a reward value to represent the preference for word count; S33: Input each question, answer, and corresponding reward value in the training question dataset into the opt-350M model to train and obtain a reward model.

5. The method for dynamic word count control of large language model text output according to claim 4, characterized in that: The reward model is configured to use the opt-350M model and convert the last output layer into a feedforward network with a single output node, thereby outputting a scalar value representing the reward. During training, the reward model calculates a loss function for each of the multiple question-answer pairs and then adds and averages the results. The loss function uses a binary cross-entropy loss function, enabling the model to predict the reward value for a given word count. The expression of the loss function is specifically: ； in, is the total number of times to judge. If multiple questions and answers are calculated in pairs, then , is the total number of question-answer pairs; is the Sigmoid function, which maps real numbers to between (0,1); For the current parameter The following answers The reward value; For the current parameter The following answers The reward value.

6. The method for dynamic word count control of large language model text output according to claim 1, characterized in that: S4: Constructing an initial word count control model, using the problem dataset and the reward model, optimizing the parameters of the initial word count control model based on a reinforcement learning algorithm, and generating a final word count control model, specifically including: S41: Sampling questions from the question dataset to obtain an optimized question dataset, constructing an initial word count control model having the same network structure and initial parameters as the professional model, inputting the questions in the optimized question dataset into the initial word count control model to obtain a first answer, and forming a first optimized question-answer pair based on the corresponding questions and the first answers; S42: Input the first optimized question-answer pair into the reward model to obtain a reward value output by the reward model; S43: Constructing a reference model with the same network structure and initial parameters as the professional model, inputting the question in the optimization problem dataset into the reference model to obtain a second answer, and forming a second optimized question-answer pair based on the corresponding question and the second answer; S44: Calculating the data distribution gap between the first answer and the second answer by using KL divergence, using the reward value minus the data distribution gap as a reward function for reinforcement learning, optimizing the parameters of the initial word count control model based on the reinforcement learning algorithm, and generating a final word count control model; The expression of the data distribution gap is specifically: ； Among them, is the KL divergence, which indicates the degree of deviation of the word count control model; represents the first answer to the question by the word count control model; represents the reference model's second answer to the question; in, Represents a question-answer pair consisting of question x and the answer y obtained by the word count control model for question x; Represents the answer obtained by question x and the reference model of question x input The question and answer pairs constituted; The expression of the reward function is specifically: ； in, is the penalty coefficient. When the KL divergence is large, When the value is large, and the KL divergence is small, The value is smaller.

7. The method for dynamic word count control of large language model text output according to claim 6, characterized in that: The step of optimizing the parameters of the initial word count control model based on the reinforcement learning algorithm specifically includes: using the word count control model as an intelligent agent in the reinforcement learning algorithm and using the optimization problem data set as a state in the reinforcement learning algorithm; when optimizing the parameters of the initial word count control model, inputting the problem in the optimization problem data set into the word count control model to obtain the model output, inputting the problem and the model output into the reward model to calculate the reward value, and calculating feedback through the reward function according to the reward value to determine the parameter update.

8. The method for dynamic word count control of large language model text output according to claim 1, wherein: S5: Obtaining a target question and a target word count requirement input by the user, generating a text outline containing several chapters based on the target question, and determining a chapter word count allocation strategy for the target question based on the text outline and the target word count requirement, specifically including: S51: Obtaining a target question and a target word count requirement input by a user, and generating a text outline including several chapters and an importance level of each chapter based on the target question; S52: Based on the importance level and target word count requirement of each chapter in the text outline, generate the text word count for each chapter according to the proportion corresponding to each importance level, and determine the chapter word count allocation strategy for the target problem.

9. The method for dynamic word count control of large language model text output according to claim 8, characterized in that: S6: According to the chapter word count allocation strategy, the word count control model is used to sequentially generate text for each chapter in the text outline to obtain the final generated text step, specifically including: S61: Generate word counts based on the text of each chapter, and iteratively call the word count control model to generate the text of each chapter in units of chapters; S62: After generating the text of each chapter, determining the word count requirement for the remaining chapters based on the word count of the currently generated chapter and the target word count requirement, and re-determining the chapter word count allocation strategy for the remaining chapters in the text outline for which the text has not been generated based on the remaining chapter word count requirement; S63: Execute text generation for the remaining chapters that have not yet been generated according to the re-determined chapter word count allocation strategy until text generation is completed for each chapter to obtain a final generated text.

10. A large language model text output dynamic word count control system, characterized in that: include: A data construction module is used to prepare reference materials, generate multiple question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair dataset and a question dataset; wherein each question includes a word count requirement; A professional model generation module is used to use the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model; A reward value evaluation module, configured to input the question dataset into a professional model, generate multiple answers, determine a reward value based on the word count of each answer, and construct and train a reward model; a model optimization module, configured to construct an initial word count control model, optimize the parameters of the initial word count control model based on a reinforcement learning algorithm using the problem dataset and the reward model, and generate a final word count control model; A word count allocation module is configured to obtain a target question and a target word count requirement input by a user, generate a text outline comprising a plurality of chapters based on the target question, and determine a chapter word count allocation strategy for the target question based on the text outline and the target word count requirement; The article generation module is used to generate text for each chapter in the text outline in turn according to the chapter word count allocation strategy and adopt the word count control model to obtain a final generated text.

Images