Large language model text output dynamic word number control method and system

Abstract

The invention relates to the technical field of large models, and discloses a large language model text output dynamic word number control method and system, and the method comprises the steps: collecting data, and automatically generating high-quality question and answer pair data and question data based on a large model; performing fine tuning on the large model by utilizing the question-answer pair data set to obtain a professional model; training a reward model by using the question data set, and scoring the content generated by the large model according to the number of words; constructing and optimizing parameters of the word number control model based on a reinforcement learning algorithm by utilizing the question and answer pair data set and the reward model; when a user inputs questions and word number requirements, the word number requirements are distributed to each chapter, the word number control model is iteratively called with the chapter as the unit to generate each chapter, and finally the whole article is generated through splicing. According to the method, the output word number of the large model can be accurately controlled, and the method has the advantages of accurate and controllable generation and the like, so that the requirements of specific application scenes are met, publishing problems or format errors caused by too many or too few word numbers are avoided, and the user experience is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to a large language model text output dynamic word count control method and system. Background Art

[0002] In today's rapidly developing field of artificial intelligence, large language models, with their massive data training and complex algorithmic structures, have been able to perform various language processing tasks, including knowledge question answering, text generation, and assisted writing. Large language models can not only provide real-time information feedback, but also create stories, write reports, and even proofread and edit manuscripts.

[0003] However, despite the remarkable achievements of large language models in text generation, they still face challenges in controlling the number of words in output. Sometimes, the text generated by the model may exceed the expected word count, which will not only make the content too lengthy, but also dilute key information. On the contrary, if the number of words generated by the text is insufficient, important information may be missed, affecting the integrity and persuasiveness of the content. In many application scenarios, there are strict requirements on the number of words in articles or texts, such as news reports, academic papers, social media posts, etc. Accurately controlling the number of output words can avoid publishing problems or format errors caused by too many or too few words. Therefore, how to accurately control the number of words in the text output of large models to meet the needs of specific application scenarios and improve user experience has become an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a large language model text output dynamic word count control method and system, aiming to solve at least one of the above technical problems.

[0005] To achieve the above object, the present invention provides a method for controlling the number of dynamic words in a large language model text output, comprising: S1: Prepare reference materials, generate several question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair dataset and a question dataset; wherein each question includes a word count requirement; S2: Use the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model; S3: inputting the question data set into a professional model, generating multiple answers, determining a reward value based on the number of words of each answer, and constructing and training a reward model; S4: constructing an initial word count control model, using the problem data set and the reward model, optimizing the parameters of the initial word count control model based on a reinforcement learning algorithm, and generating a final word count control model; S5: Obtaining the target problem and the target word count requirement input by the user, generating a text outline including several chapters based on the target problem, and determining a chapter word count allocation strategy for the target problem according to the text outline and the target word count requirement; S6: According to the chapter word count allocation strategy, the word count control model is used to generate text for each chapter in the text outline in turn to obtain a final generated text.

[0006] Optionally, S1: preparing reference materials, generating a plurality of question-answer pair data of the reference materials based on the target large model, and constructing a question-answer pair dataset and a question dataset step, specifically including: S11: Acquire relevant information; wherein the relevant information is configured as professional information related to the target problem collected through various channels; S12: extracting text content from the relevant material, dividing the relevant material into several sections of reference material according to each preset number of words as a reference material text; S13: For each piece of reference material, use the target large model to generate several question-answer pair data, construct a question-answer pair data set, extract the question data in each question-answer pair data, and construct a question data set; wherein each question includes a word count requirement.

[0007] Optionally, S2: using the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model step, specifically including: S21: A lora adapter is set between the q layer and the v layer of each transformer layer of the target large model to capture the knowledge features in the reference material; S22: for each question-answer pair in the question-answer pair data set, inputting a correct answer and a plurality of irrelevant answers together as input data into a target large model; S23: Calculate the loss function based on the difference between the answer result output by the target large model and the correct answer, and update the parameters of the lora adapter through the back propagation algorithm to minimize the loss function; S24: After iterative training, until the model performance reaches the predetermined standard or the training reaches the preset rounds, a professional model is generated based on the parameters of the original target large model and the fine-tuned lora adapter parameters.

[0008] Optionally, S3: inputting the question data set into a professional model, generating multiple answers, determining a reward value based on the number of words in each answer, and constructing and training a reward model step, specifically including: S31: sampling questions from the question data set to obtain a training question data set, and repeatedly inputting the questions in the training question data set into the professional model to generate multiple answers; S32: Calculate the number of words in each answer, sort the answers according to how close the number of words is to the requirement, and normalize them into a scalar value, and use the scalar value as a reward value to represent the preference for the number of words; S33: Input each question, answer and corresponding reward value in the training question data set into the opt-350M model, and train to obtain a reward model.

[0009] Optionally, the reward model is configured to use the opt-350M model and change the last output layer into a feed-forward network so that it has a single output node, thereby outputting a scalar value representing the reward; when training the reward model, the loss function is calculated for multiple question-answer pairs in pairs, and then the average is added, and the loss function uses a binary cross entropy loss function, so that the model can predict the reward value for a given number of words; The expression of the loss function is specifically: ; in, is the total number of judgments required. If multiple questions and answers are calculated in pairs, then , is the total number of question and answer pairs; is the Sigmoid function, which maps real numbers to (0,1); For the current parameter Next answer The reward value of For the current parameter Next answer The reward value.

[0010] Optionally, S4: constructing an initial word count control model, using the problem data set and the reward model, optimizing the parameters of the initial word count control model based on a reinforcement learning algorithm, and generating a final word count control model step specifically includes: S41: sampling questions from the question data set to obtain an optimized question data set, constructing an initial word count control model having the same network structure and initial parameters as the professional model, inputting the questions in the optimized question data set into the initial word count control model to obtain a first answer, and forming a first optimized question-answer pair based on the questions and the first answer having a corresponding relationship; S42: Input the first optimized question-answer pair into the reward model to obtain a reward value output by the reward model; S43: constructing a reference model with the same network structure and initial parameters as the professional model, inputting the question in the optimization problem data set into the reference model to obtain a second answer, and forming a second optimization question-answer pair based on the question and the second answer having a corresponding relationship; S44: calculating the data distribution gap between the first answer and the second answer by KL divergence, using the reward value minus the data distribution gap as a reward function for reinforcement learning, optimizing the parameters of the initial word count control model based on the reinforcement learning algorithm, and generating a final word count control model; The expression of the data distribution gap is specifically: ; in, is the KL divergence, which indicates the degree of deviation of the word count control model; represents the first answer to the question by the word count control model; represents the second answer to the question by the initial word count control model; The expression of the reward function is specifically: ; in, is the penalty coefficient. When the KL divergence is large, When the value is large, and the KL divergence is small, The value is smaller.

[0011] Optionally, the step of optimizing the parameters of the initial word count control model based on a reinforcement learning algorithm specifically includes: using the word count control model as an intelligent agent in the reinforcement learning algorithm, and using the optimization problem data set as a state in the reinforcement learning algorithm; when optimizing the parameters of the initial word count control model, inputting the problem in the optimization problem data set into the word count control model to obtain the model output, inputting the problem and the model output into the reward model to calculate the reward value, and calculating feedback through the reward function according to the reward value to determine the parameter update.

[0012] Optionally, S5: obtaining a target problem and a target word count requirement input by a user, generating a text outline containing several chapters based on the target problem, and determining a chapter word count allocation strategy step for the target problem according to the text outline and the target word count requirement, specifically comprising: S51: Obtaining a target question and a target word count requirement input by a user, and generating a text outline including several chapters and an importance level of each chapter based on the target question; S52: According to the importance level and target word count requirement of each chapter in the text outline, the text word count of each chapter is generated according to the proportion corresponding to each importance level, and the chapter word count allocation strategy of the target problem is determined.

[0013] Optionally, S6: according to the chapter word count allocation strategy, using the word count control model to sequentially generate text for each chapter in the text outline to obtain a final generated text step specifically includes: S61: Generate a word count according to the text of each chapter, and iteratively call a word count control model in units of chapters to generate the text of each chapter; S62: After generating the text of each chapter, determine the word count requirement of the remaining chapters according to the word count of the currently generated chapter and the target word count requirement, and re-determine the chapter word count allocation strategy for the remaining ungenerated text chapters in the text outline based on the remaining chapter word count requirement; S63: Execute text generation for the remaining chapters that have not yet been generated according to the re-determined chapter word count allocation strategy, until text generation for each chapter is completed to obtain a final generated text.

[0014] In addition, in order to achieve the above-mentioned purpose, the present invention also provides a large language model text output dynamic word count control system, comprising: A data construction module is used to prepare reference materials, generate a number of question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair dataset and a question dataset; wherein each question includes a word count requirement; A professional model generation module is used to use the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model; A reward value evaluation module, used to input the question data set into a professional model, generate multiple answers, determine a reward value based on the number of words in each answer, and build and train a reward model; A model optimization module, used to construct an initial word count control model, and optimize the parameters of the initial word count control model based on a reinforcement learning algorithm using the problem data set and the reward model to generate a final word count control model; A word count allocation module is used to obtain a target problem and a target word count requirement input by a user, generate a text outline containing several chapters based on the target problem, and determine a chapter word count allocation strategy for the target problem based on the text outline and the target word count requirement; The article generation module is used to generate text for each chapter in the text outline in turn according to the chapter word count allocation strategy and adopt the word count control model to obtain the final generated text.

[0015] The beneficial effects of the present invention are as follows: a method and system for dynamic word count control of text output of a large language model are proposed, and professional-related materials are collected through multiple channels; high-quality question-answer pair data and question data are automatically generated based on the large model for the collected materials; the large model is fine-tuned using the question-answer pair data set to obtain a professional model, making it more suitable for specific professional tasks; the reward model is trained using the question data set, and the content generated by the large model is scored according to the word count, so that the model can accurately predict the preference for a given word count; the question-answer pair data set and the reward model are used to optimize the parameters of the word count control model based on the reinforcement learning algorithm; when the user inputs the question and word count requirement, the word count requirement is allocated to each chapter (or paragraph) using the word count allocation method, thereby strengthening the control of the word count; after completing the output word count allocation, a prompt is dynamically constructed, and the word count control model is iteratively called in units of chapters to generate each chapter, and finally the chapters are spliced ​​to generate the entire article; for the generated article, a summary is performed based on the word count control model to further standardize the word count. The present invention can accurately control the number of output words of a large model, and has the advantages of accurate and controllable generation, thereby meeting the needs of specific application scenarios, avoiding publishing problems or format errors caused by too many or too few words, and enhancing user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a flow chart of a method for controlling the number of dynamic words in a large language model text output according to an embodiment of the present invention; Figure 2 It is a structural schematic diagram of a large language model text output dynamic word count control system according to an embodiment of the present invention.

[0017] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] The embodiment of the present invention provides a method for controlling the number of dynamic words in a large language model text output, referring to Figure 1 , Figure 1 The flowchart of the method for controlling the number of words in the text output of a large language model according to an embodiment of the present invention is shown in FIG. In this embodiment, the method for controlling the number of words in the text output of a large language model comprises the following steps: S1: Prepare reference materials, generate several question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair dataset and a question dataset; wherein each question includes a word count requirement; S2: Use the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model; S3: inputting the question data set into a professional model, generating multiple answers, determining a reward value based on the number of words of each answer, and constructing and training a reward model; S4: constructing an initial word count control model, using the problem data set and the reward model, optimizing the parameters of the initial word count control model based on a reinforcement learning algorithm, and generating a final word count control model; S5: Obtaining the target problem and the target word count requirement input by the user, generating a text outline including several chapters based on the target problem, and determining a chapter word count allocation strategy for the target problem according to the text outline and the target word count requirement; S6: According to the chapter word count allocation strategy, the word count control model is used to generate text for each chapter in the text outline in turn to obtain a final generated text.

[0020] It should be noted that although large language models have made remarkable achievements in text generation, they still face challenges in controlling the number of output words. Sometimes, the text generated by the model may exceed the expected word count, which will not only make the content too lengthy, but also dilute key information. On the contrary, if the number of words generated by the text is insufficient, important information may be missed, affecting the integrity and persuasiveness of the content. In many application scenarios, there are strict requirements on the number of words in articles or texts, such as news reports, academic papers, social media posts, etc. Accurately controlling the number of output words can avoid publishing problems or format errors caused by too many or too few words. Therefore, how to accurately control the number of words in the text output of large models to meet the needs of specific application scenarios and improve user experience has become an urgent problem to be solved.

[0021] In order to solve the above problems, this embodiment collects professional related information through multiple channels; for the collected information, high-quality question-answer pair data and question data are automatically generated based on the big model; the big model is fine-tuned using the question-answer pair data set to obtain a professional model, making it more suitable for specific professional tasks; the reward model is trained using the question data set, and the content generated by the big model is scored according to the number of words, so that the model can accurately predict the preference for a given number of words; the parameters of the word count control model are optimized based on the reinforcement learning algorithm using the question-answer pair data set and the reward model; when the user inputs the question and the word count requirement, the word count requirement is allocated to each chapter (or paragraph) using the word count allocation method, thereby strengthening the control of the word count; after completing the output word count allocation, a prompt is dynamically constructed, and the word count control model is iteratively called in units of chapters to generate each chapter, and finally the chapters are spliced ​​to generate the entire article; for the generated article, a summary is made based on the word count control model to further standardize the word count. The present invention can accurately control the output word count of the big model, has the advantages of accurate generation and controllability, thereby meeting the needs of specific application scenarios, avoiding publishing problems or format errors caused by too many or too few words, and enhancing user experience.

[0022] In order to explain the present invention more clearly, a specific example application of the large language model text output dynamic word count control method in an actual scenario is provided below.

[0023] Step 1: Data preparation.

[0024] Collect professional-related information through libraries, archives, the Internet and other channels, and automatically generate high-quality question-answer data based on the collected information and the big model to build a data set ; The question-answer pair dataset Extract the questions and build a question dataset , which is used for subsequent model training and word count limit. The question data includes the word count requirement.

[0025] The method for generating question-answer pairs is to segment the preprocessed data into 200-word segments as reference materials for generating question-answer pairs. For paragraphs with less than 200 words, the paragraph is directly used as a reference material. For each reference material, a large model (such as qwen, llama, etc.) is used to generate 2-5 question-answer pairs that are closely related to the content of the reference material. The generated question-answer pairs are manually checked and modified to construct a high-quality data set.

[0026] Furthermore, the format of the question-answer pair output is: Output in the following json format [ { "question": <question>, "answer": <answer> } ] Furthermore, the generated question-answer pair example is: "question":Why is the sky blue? The answer should be 40 words.

[0027] "answer":The sky appears blue because molecules and other particles in the atmosphere scatter the blue light in sunlight. This phenomenon is called Rayleigh scattering.

[0028] Step 2: Professional model fine-tuning.

[0029] Select a basic large model and use the question-answer pair dataset Perform LoRa fine-tuning to obtain a professional model , making it more suitable for specific professional tasks and improving the accuracy and relevance of the model when answering professional questions.

[0030] The basic large models include qwen, llama, etc., from which basic models with good performance and suitable for fine-tuning are selected according to needs and preferences, ensuring that the selected models have sufficient capacity and generalization ability so as to adapt to professional tasks through fine-tuning.

[0031] The LoRa fine-tuning method is to set a LoRa adapter between the Q layer (query layer) and the V layer (value layer) of each transformer layer to capture and store specific professional knowledge. For each question, the correct answer and multiple irrelevant answers are used as input data, and the large model is allowed to answer relevant questions based on the input data. The loss function is calculated based on the difference between the model's answer and the correct answer, and the parameters of the LoRa adapter are updated through the back propagation algorithm to minimize the loss function. The training process will iterate multiple times until the model performance reaches the predetermined standard or the training reaches the preset rounds. After the training is completed, the parameters of the original large model and the fine-tuned LoRa adapter parameters are merged to generate a professional large model.

[0032] Furthermore, an example input for model fine-tuning is: { "instruction":"Use <data>< / data> Mark the content in as your knowledge to answer the question. Question: <question>", "input":" <data> 1 Answer --- 2 Answers --- 3 Answers --- 4 Answers < / data> ", "output": <answer>, "system":"You are an expert and need to answer questions about professional knowledge." } Step 3: Reward model training.

[0033] Combining reinforcement learning with human feedback, by incorporating feedback on the number of generated words into the training process, the number of words is used as a reward signal to guide the training of the model, providing a natural and humane learning process for the large model, thereby enhancing the large model's understanding and compliance with the word count requirements.

[0034] One of the key steps in controlling the number of words generated by the model based on reinforcement learning is to train the reward model to score the content generated by the large model according to the number of words, so that the model can accurately predict the preference for a given number of words.

[0035] Specifically, the process of training the reward model is: Using random sampling method, from the problem data set Sampling Problems Building Problem Datasets . Combine the question and the word count requirement to form a new question.

[0036] The problem dataset Problems in duplicate input professional model , generate multiple answers. Calculate the number of words in each answer, sort the answers by how close the number of words is to the requirement, and normalize them to a scalar value , as a reward value, which numerically represents the preference for the number of words.

[0037] Input the questions, answers and corresponding reward values ​​into the opt-350M model and train the reward model .

[0038] The random sampling method can ensure the randomness and representativeness of the sample, is suitable for situations where each part of the population has the same probability, and can better reflect the characteristics of the population.

[0039] The goal of the reward model is to receive a series of texts and return a scalar reward representing the word count preference. Specifically, the opt-350M model is used and the last output layer is changed to a feedforward network with a single output node, which outputs a scalar value representing the reward. When training the model, the loss function is calculated for multiple question-answer pairs and then the average is added. The loss function uses a binary cross entropy loss function, which enables the model to accurately predict the reward value for a given word count.

[0040]

[0041] in, is the total number of judgments required. If multiple questions and answers are calculated in pairs, then , is the total number of question and answer pairs; is the Sigmoid function, which maps real numbers to (0,1); For the current parameter Next answer The reward value of For the current parameter Next answer The reward value.

[0042] Furthermore, the OPT-350M model is a large language model developed by Meta AI. It is a Transformer model consisting of only a decoder (decoder-only), mainly used for text processing. The OPT-350M model has 24 layers, a hidden layer size of 1024, 16 attention heads, and a head size of 64.

[0043] Step 4: Word count control model training.

[0044] Using the Question-Answer Pairs Dataset and reward model , optimize the word count control model based on reinforcement learning algorithm Parameters.

[0045] The word count control model The network structure and initial parameters of the model same.

[0046] Specifically, the specific process of word count control model training is: Using random sampling method, from the problem data set Sampling problems and building new problem datasets .

[0047] The problem dataset Problems in Input word count control model , get the answer , forming a question-answer pair .

[0048] Answer the questions correctly Input Reward Model , get reward value .

[0049] To prevent the model Too much emphasis is placed on the number of words generated and too little emphasis is placed on the content generated. A reference model needs to be added , input the problem into the reference model , get the answer , forming a question-answer pair The reference model The network structure and parameters of the professional model same.

[0050] Answered by calculating KL divergence and The data distribution gap between them.

[0051]

[0052] in, is the KL divergence, which indicates the degree of deviation of the word count control model; Represents a large model Questions Answer ; Represents a large model Questions Answer .

[0053] Use the reward value minus the data distribution gap as the reward function of reinforcement learning, and optimize the model based on the reinforcement learning algorithm Parameters.

[0054] The reinforcement learning algorithm is also called reinforcement learning, evaluation learning or enhanced learning, and is used to describe and solve the problem of an agent maximizing rewards or achieving specific goals through learning strategies during the interaction with the environment. A common model of reinforcement learning is the standard Markov Decision Process (MDP).

[0055] Reinforcement learning is a process in which an agent learns by trial and error, obtaining rewards to guide behavior through interaction with the environment, with the goal of enabling the agent to obtain the maximum reward.

[0056] Elements of reinforcement learning include: Agent (intelligent body, robot, agent): The subject of reinforcement learning training is the Agent.

[0057] Environment: The overall background of the whole process is the environment.

[0058] State: The current state of the Agent.

[0059] Action: What actions can the agent take based on the current state? Reward: After the agent takes a certain action in the current state, it will receive certain feedback from the environment, which may be a reward or a punishment.

[0060] The word count control model is trained using a reinforcement learning algorithm. The agent is the word count control model, the state is the problem data set, and the action is the output of the word count control model. The problem is input into the word count control model to obtain the model output. The model problem and output are input into the reward model to calculate the reward value. The reward function is used to calculate the feedback based on the reward value to determine the parameter update.

[0061] Furthermore, the reward function is:

[0062] in, is the penalty coefficient. When the KL divergence is large, When the value is large, and the KL divergence is small, A smaller value encourages the word count control model not to place too much emphasis on the generated word count and ignore the generated content.

[0063] Step 5: Output word count allocation.

[0064] When users generate a large amount of text, such as writing a complete plan or report, they will face the challenge of how to effectively control the number of words. For long text models, segmented generation is usually adopted. The accumulation of word count deviations in each segment will lead to a large deviation in the final word count. Therefore, the word count allocation method is adopted to allocate the word count requirements to each chapter (or paragraph) to strengthen the control of the word count.

[0065] Specifically, the main process of output word count allocation is: When the user enters the question and word count requirement, the total word count requirement is first obtained.

[0066] When the user wants to generate a large number of words, he or she should first generate a text outline to determine the number of chapters to be generated, the importance of each chapter, and its role in the overall text, so as to reasonably allocate the number of words.

[0067] Then divide the word count requirement by chapters and calculate the word count requirement for each chapter.

[0068] Among them, the main chapters are allocated more words, while the prologue and conclusion generally have fewer words and are less important, so they account for a smaller proportion of the total word requirement.

[0069] For example, if a proposal consists of five chapters with a total word count of 5,000 words, the word count can be allocated according to the importance of the chapters as follows: Prologue: 500 words (10%); Chapter 1: 1,000 words (20%); Chapter 2: 1,500 words (30%); Chapter 3: 1,000 words (20%); Conclusion: 500 words (10%).

[0070] Step 6: Dynamic prompt settings.

[0071] After the output word count is allocated, the word count control model is iteratively called to generate each chapter, and finally the chapters are spliced ​​to generate the entire article. For the generated articles, a summary is made based on the word count control model to further standardize the word count.

[0072] During the chapter generation process, a dynamic adjustment mechanism is set up. If the number of words in a chapter exceeds expectations, the number of words allocated to subsequent chapters can be appropriately reduced. Specifically, the number of words in the generated chapters is first calculated, and then the number of words in the remaining chapters is reallocated according to the output word allocation method described in step 5. After the word allocation is completed, a dynamic prompt is constructed based on the outline and the allocated word count, and the large model is input to obtain a response.

[0073] Reference Figure 2 , Figure 2 It is a structural schematic diagram of a large language model text output dynamic word count control system according to an embodiment of the present invention.

[0074] like Figure 2 As shown, the large language model text output dynamic word count control system proposed in the embodiment of the present invention includes: The data construction module 10 is used to prepare reference materials, generate a number of question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair data set and a question data set; wherein each question includes a word count requirement; A professional model generation module 20 is used to perform LoRa fine-tuning on the target large model using the question-answer pair dataset to generate a professional model; A reward value evaluation module 30, for inputting the question data set into a professional model, generating multiple answers, determining a reward value based on the number of words in each answer, and constructing and training a reward model; A model optimization module 40 is used to construct an initial word count control model, and optimize the parameters of the initial word count control model based on a reinforcement learning algorithm using the problem data set and the reward model to generate a final word count control model; The word count allocation module 50 is used to obtain the target problem and the target word count requirement input by the user, generate a text outline containing several chapters based on the target problem, and determine the chapter word count allocation strategy of the target problem according to the text outline and the target word count requirement; The article generation module 60 is used to generate text for each chapter in the text outline in turn according to the chapter word count allocation strategy and adopt the word count control model to obtain the final generated text.

[0075] Other embodiments or specific implementations of the large language model text output dynamic word count control system of the present invention can refer to the above-mentioned method embodiments and will not be repeated here.

[0076] It is understood that, in the description of this specification, the reference terms "one embodiment", "another embodiment", "other embodiments", or "first to Nth embodiments" etc. mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0077] It should be noted that, in this article, the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0078] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.< / answer> < / question> < / answer> < / question>

Claims

1. A method for controlling the number of dynamic words in text output of a large language model, characterized in that: include: S1: Prepare reference materials, generate several question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair dataset and a question dataset; wherein each question includes a word count requirement; S2: Use the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model; S3: inputting the question data set into a professional model, generating multiple answers, determining a reward value based on the number of words of each answer, and constructing and training a reward model; S4: constructing an initial word count control model, using the problem data set and the reward model, optimizing the parameters of the initial word count control model based on a reinforcement learning algorithm, and generating a final word count control model; S5: Obtaining the target problem and the target word count requirement input by the user, generating a text outline including several chapters based on the target problem, and determining a chapter word count allocation strategy for the target problem according to the text outline and the target word count requirement; S6: According to the chapter word count allocation strategy, the word count control model is used to generate text for each chapter in the text outline in turn to obtain a final generated text.

2. The large language model text output dynamic word count control method according to claim 1, characterized in that: S1: Prepare reference materials, generate several question-answer pairs of the reference materials based on the target large model, and construct question-answer pair datasets and question datasets, specifically including: S11: Acquire relevant information; wherein the relevant information is configured as professional information related to the target problem collected through various channels; S12: extracting text content from the relevant material, dividing the relevant material into several sections of reference material according to each preset number of words as a reference material text; S13: For each section of reference material, use the target large model to generate several question-answer pair data, construct a question-answer pair data set, extract the question data in each question-answer pair data, and construct a question data set; wherein each question includes a word count requirement.

3. The large language model text output dynamic word count control method according to claim 1, characterized in that: S2: Use the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model, specifically including: S21: A lora adapter is set between the q layer and the v layer of each transformer layer of the target large model to capture the knowledge features in the reference material; S22: for each question-answer pair in the question-answer pair data set, inputting a correct answer and a plurality of irrelevant answers together as input data into a target large model; S23: Calculate the loss function based on the difference between the answer result output by the target large model and the correct answer, and update the parameters of the lora adapter through the back propagation algorithm to minimize the loss function; S24: After iterative training, until the model performance reaches the predetermined standard or the training reaches the preset rounds, a professional model is generated based on the parameters of the original target large model and the fine-tuned lora adapter parameters.

4. The large language model text output dynamic word count control method according to claim 1, characterized in that: S3: Inputting the question data set into a professional model, generating multiple answers, determining a reward value based on the number of words in each answer, and constructing and training a reward model, specifically including: S31: sampling questions from the question data set to obtain a training question data set, and repeatedly inputting the questions in the training question data set into the professional model to generate multiple answers; S32: Calculate the number of words in each answer, sort the answers according to how close the number of words is to the requirement, and normalize them into a scalar value, and use the scalar value as a reward value to represent the preference for the number of words; S33: Input each question, answer and corresponding reward value in the training question data set into the opt-350M model, and train to obtain a reward model.

5. The large language model text output dynamic word count control method according to claim 4, characterized in that: The reward model is configured to use the opt-350M model and change the last output layer to a feedforward network so that it has a single output node, thereby outputting a scalar value representing the reward; when training the reward model, the loss function is calculated for multiple question-answer pairs in pairs, and then the average is added. The loss function uses a binary cross entropy loss function, so that the model can predict the reward value for a given number of words; The expression of the loss function is specifically: ； in, is the total number of times to judge. If multiple questions and answers are calculated in pairs, then , is the total number of question and answer pairs; is the Sigmoid function, which maps real numbers to (0,1); For the current parameter Next answer The reward value of For the current parameter Next answer The reward value.

6. The large language model text output dynamic word count control method according to claim 1, characterized in that: S4: constructing an initial word count control model, using the problem data set and the reward model, optimizing the parameters of the initial word count control model based on a reinforcement learning algorithm, and generating a final word count control model step, specifically including: S41: sampling questions from the question data set to obtain an optimized question data set, constructing an initial word count control model having the same network structure and initial parameters as the professional model, inputting the questions in the optimized question data set into the initial word count control model to obtain a first answer, and forming a first optimized question-answer pair based on the questions and the first answer having a corresponding relationship; S42: Input the first optimized question-answer pair into the reward model to obtain a reward value output by the reward model; S43: constructing a reference model with the same network structure and initial parameters as the professional model, inputting the question in the optimization problem data set into the reference model to obtain a second answer, and forming a second optimization question-answer pair based on the question and the second answer having a corresponding relationship; S44: calculating the data distribution gap between the first answer and the second answer by KL divergence, using the reward value minus the data distribution gap as a reward function for reinforcement learning, optimizing the parameters of the initial word count control model based on the reinforcement learning algorithm, and generating a final word count control model; The expression of the data distribution gap is specifically: ； in, is the KL divergence, which indicates the degree of deviation of the word count control model; represents the first answer to the question by the word count control model; represents the second answer to the question by the initial word count control model; The expression of the reward function is specifically: ； in, is the penalty coefficient. When the KL divergence is large, When the value is large, and the KL divergence is small, The value is smaller.

7. The method for controlling the dynamic word count of a large language model text output according to claim 6, wherein: The step of optimizing the parameters of the initial word count control model based on the reinforcement learning algorithm specifically includes: taking the word count control model as an intelligent agent in the reinforcement learning algorithm and taking the optimization problem data set as a state in the reinforcement learning algorithm; when optimizing the parameters of the initial word count control model, inputting the problem in the optimization problem data set into the word count control model to obtain the model output, inputting the problem and the model output into the reward model to calculate the reward value, and calculating feedback through the reward function according to the reward value to determine the parameter update.

8. The large language model text output dynamic word count control method according to claim 1, characterized in that: S5: Obtaining the target problem and the target word count requirement input by the user, generating a text outline containing several chapters based on the target problem, and determining the chapter word count allocation strategy steps for the target problem according to the text outline and the target word count requirement, specifically including: S51: Obtaining a target question and a target word count requirement input by a user, and generating a text outline including several chapters and an importance level of each chapter based on the target question; S52: According to the importance level and target word count requirement of each chapter in the text outline, the text word count of each chapter is generated according to the proportion corresponding to each importance level, and the chapter word count allocation strategy of the target problem is determined.

9. The method for controlling the number of dynamic words in large language model text output according to claim 8, characterized in that: S6: According to the chapter word count allocation strategy, the word count control model is used to sequentially generate text for each chapter in the text outline to obtain a final generated text step, specifically including: S61: Generate a word count according to the text of each chapter, and iteratively call a word count control model in units of chapters to generate the text of each chapter; S62: After generating the text of each chapter, determine the word count requirement of the remaining chapters according to the word count of the currently generated chapter and the target word count requirement, and re-determine the chapter word count allocation strategy for the remaining ungenerated text chapters in the text outline based on the remaining chapter word count requirement; S63: Execute text generation for the remaining chapters that have not yet been generated according to the re-determined chapter word count allocation strategy, until text generation for each chapter is completed to obtain a final generated text.

10. A large language model text output dynamic word count control system, characterized in that: include: A data construction module is used to prepare reference materials, generate a number of question-answer pairs of the reference materials based on the target large model, and construct a question-answer pair dataset and a question dataset; wherein each question includes a word count requirement; A professional model generation module is used to use the question-answer pair dataset to perform LoRa fine-tuning on the target large model to generate a professional model; A reward value evaluation module, used to input the question data set into a professional model, generate multiple answers, determine a reward value based on the number of words in each answer, and build and train a reward model; A model optimization module, used to construct an initial word count control model, and optimize the parameters of the initial word count control model based on a reinforcement learning algorithm using the problem data set and the reward model to generate a final word count control model; A word count allocation module is used to obtain a target problem and a target word count requirement input by a user, generate a text outline containing several chapters based on the target problem, and determine a chapter word count allocation strategy for the target problem based on the text outline and the target word count requirement; The article generation module is used to generate text for each chapter in the text outline in turn according to the chapter word count allocation strategy and adopt the word count control model to obtain the final generated text.

Images