BART-based reinforcement learning text generation method and system

Abstract

The invention provides a reinforcement learning text generation method and system based on BART, and belongs to the technical field of natural language processing. The method comprises the steps of obtaining a BART model for text generation; dialogue data of the intelligent outbound system is obtained, a text data set is formed, and a pre-trained BART model is obtained; constructing a question and answer pair data set based on dialogue data of the intelligent outbound system, and obtaining a fine tuning BART model based on the data set; further obtaining a trained BART model through reinforcement learning optimization based on the fine tuning BART model; and finally, performing text generation by utilizing the trained BART model to obtain a generated text. According to the method, language styles and subtle differences can be accurately captured, the stability of RLHF is improved, the training cost is reduced, the efficiency is improved, and a more stable and efficient solution is provided for wide application of text generation in commercial application.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing (NLP), and in particular relates to a reinforcement learning text generation method and system based on BART. Background Art

[0002] Natural language processing is an important branch of artificial intelligence and computational linguistics, which mainly studies how to make computers understand, generate and manipulate human natural language. With the rapid development of deep learning and big data technology, NLP has made remarkable progress in many application fields such as text analysis, information retrieval, machine translation, dialogue systems, etc. Among the many tasks of NLP, text generation is a core issue, which involves automatically generating natural language text that meets grammatical and semantic requirements from input data. Text generation technology has broad application prospects in open domain dialogue, machine translation, automatic summarization, content creation, etc. In order to improve the quality of text generation, researchers have developed a variety of methods based on statistics, rules and deep learning to enhance the understanding and generation capabilities of the model.

[0003] In the field of text generation, deep learning-based generation methods have gradually become mainstream, especially pre-trained language models such as GPT, BERT, BART, and T5 have made significant progress in natural language processing tasks. These models are pre-trained on large-scale corpora to capture complex semantic relationships in text and apply them to specific tasks after fine-tuning. These deep learning technology solutions each have unique advantages and have demonstrated excellent performance in multiple text generation applications. Although existing text generation technologies have made significant progress in many fields, there are still some shortcomings that cannot be ignored, especially when dealing with specific application scenarios with commercial value. These shortcomings become particularly prominent.

[0004] First, there is the problem of capturing language style and expression details. In many commercially valuable applications, such as customer service chatbots, digital human anchors, content creation, etc., the language style of text generation often needs to be highly customized. These application scenarios usually have unique language styles, common phrases, and popular terms, and these language habits may change frequently. Existing pre-trained models often have difficulty capturing these subtle differences when dealing with language generation in this specific style. For example, two very similar expressions may have significant differences in advantages and disadvantages in actual applications. These subtle differences are difficult for existing models to learn and recognize, which may result in generation results that do not meet user expectations.

[0005] Secondly, the effect of reinforcement learning with human feedback (RLHF) in text generation is unstable. Although RLHF is a promising technology that optimizes the quality of model generation by introducing human feedback, in practical applications, the effect of this method is still very unstable. Existing studies have shown that the training process of RLHF is easily affected by factors such as data quality and the initial state of the model, resulting in large volatility in training results. At present, RLHF has not been widely used in commercial applications, mainly due to its instability and high difficulty in implementation. This makes it difficult to ensure the quality and consistency of generated text in practical applications, especially in complex and changing business environments.

[0006] In addition, there are issues with the high cost and efficiency of model training. Existing large-scale pre-trained models such as BART and T5, although they have powerful generation capabilities, are very expensive to train in practical applications, especially when dealing with tasks in specific fields. These models usually require a lot of computing resources and time for fine-tuning, which is a significant bottleneck for many commercial applications. Therefore, how to improve training efficiency and reduce resource consumption while ensuring generation quality is a key issue that has not yet been solved by existing technologies. Summary of the invention

[0007] In order to solve the problems in the prior art, the present invention provides a reinforcement learning text generation method and system based on BART.

[0008] The technical solution adopted by the present invention is as follows:

[0009] In a first aspect, the present invention discloses a method for generating text using reinforcement learning based on BART, comprising the following steps:

[0010] 1) Obtaining a BART model for text generation; obtaining conversation data of an intelligent outbound call system and forming a text data set, performing random noise perturbation on the text in the text data set to generate a damaged input sequence, and pre-training the BART model based on the input sequence to obtain a pre-trained BART model;

[0011] 2) constructing a question-answer pair dataset based on the conversation data of the intelligent outbound call system, wherein the question-answer pair dataset includes multiple question-answer pairs, using the question text in the question-answer pair as the input of the pre-trained BART model, and performing supervised fine-tuning training on the pre-trained BART model based on the answer text in the question-answer pair to obtain a fine-tuned BART model;

[0012] 3) The question text in the question-answer pair is used as the input of the fine-tuned BART model, the answer text in the question-answer pair is used as the first candidate answer, and the output of the fine-tuned BART model when the question text is last input in step 2) is used as the second candidate answer. The fine-tuned BART model is subjected to reinforcement learning sorting training based on the two candidate answers to obtain a trained BART model;

[0013] 4) Use the trained BART model to generate text based on the input question to obtain generated text.

[0014] In a second aspect, the present invention discloses a reinforcement learning text generation system based on BART, comprising:

[0015] A pre-training module is used to obtain a BART model for text generation; obtain the conversation data of the intelligent outbound call system and form a text data set, perform random noise perturbation on the text in the text data set to generate a damaged input sequence, and pre-train the BART model based on the input sequence to obtain a pre-trained BART model;

[0016] A fine-tuning module is used to construct a question-answer pair dataset based on the conversation data of the intelligent outbound call system, wherein the question-answer pair dataset includes multiple question-answer pairs, and the question text in the question-answer pair is used as the input of the pre-trained BART model, and the pre-trained BART model is supervised and fine-tuned based on the answer text in the question-answer pair to obtain a fine-tuned BART model;

[0017] A reinforcement learning ranking training module is used to use the question text in the question-answer pair as the input of the fine-tuning BART model, the answer text in the question-answer pair as the first candidate answer, and the output of the fine-tuning BART model when the question text is last input into the fine-tuning module as the second candidate answer. The fine-tuning BART model is subjected to reinforcement learning ranking training based on the two candidate answers to obtain a trained BART model;

[0018] The text generation module is used to generate text based on input questions using the trained BART model to obtain generated text.

[0019] In a third aspect, the present invention discloses an electronic device including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the BART-based reinforcement learning text generation method.

[0020] In a fourth aspect, the present invention discloses a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, they are used to implement the BART-based reinforcement learning text generation method.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] (1) Reusing the same BART model for text generation and reordering: The present invention reuses the same BART model architecture for both text generation and candidate text reordering tasks. The integration of this architecture brings the following beneficial effects.

[0023] (1.1) Capturing unique expressions in business scenarios: Using the same model for generation and reordering allows the model to pay close attention to the subtle differences in language throughout the RL training process. This architecture allows the model to better learn and capture unique expressions in business scenarios, especially in specific application scenarios such as intelligent outbound robot customer service and digital human anchors. The generated text is more in line with industry standards and customer needs, significantly improving business adaptability and user experience.

[0024] (1.2) Simplifying the RLHF (Reinforcement Learning Based on Human Feedback) project architecture: By integrating the generation and reordering tasks into the same model, the present invention effectively simplifies the system architecture. The simplified architecture reduces the system complexity in the development process, reduces the interdependence between models, and makes the project easier to manage and maintain. In practical applications, RLHF usually faces problems such as unstable training and difficult to reproduce results. By using a unified BART model, the present invention shares the same model parameters and architecture in the generation and reordering tasks, significantly improving the stability of RLHF training and the reproducibility of the results. This improvement greatly reduces the difficulty of RLHF in commercial applications, making it more operational.

[0025] (1.3) Reduce development and commercial implementation costs: By reusing models, the present invention not only reduces the computing resources required during the development process, but also significantly reduces the resource consumption during model deployment and actual runtime. This design makes subsequent commercial implementation more economical and efficient, especially in large-scale applications, where the cost savings will be even more significant.

[0026] (2) Innovative joint rearrangement loss function loss = loss generation +β·loss rank

[0027] (2.1) The loss function proposed in this paper solves the problem of inconsistency and difficulty in capturing subtle language differences caused by independent training of the generation and re-ranking models in the current RLHF system. In traditional methods, the generation model and the re-ranking model are usually trained independently, resulting in a lack of coordination between the two in practical applications, making it difficult to learn the language characteristics in specific application scenarios.

[0028] (2.2) This paper introduces an innovative joint loss function that combines the generation loss and the reranking loss, thereby ensuring that the model pays better attention to the subtle differences in text expressions during the text generation process, significantly improving the adaptability and robustness of the model.

[0029] (3) Innovative model usage strategy: Generate candidate texts using models from previous training rounds. In traditional reinforcement learning and text generation tasks, models usually rely only on the strategy of the current round for optimization, while ignoring the experience accumulated during previous training. The present invention adopts an innovative strategy similar to the experience replay mechanism, which generates candidate texts for the current round by using models from previous training rounds. This strategy brings the following significant benefits:

[0030] (3.1) Combination of stability and exploration: Using the model of the previous round as the candidate generator of the current round helps to maintain the consistency of the strategy during the training process, while introducing moderate variability. This combination of stability and exploration enables the model to avoid falling into the local optimal solution during the training process, while ensuring the stability of policy optimization.

[0031] (3.2) Experience replay-like mechanism: This strategy of using previous models to generate candidates is similar to the experience replay mechanism in reinforcement learning. This method allows the model to learn from past decisions and gradually optimize its performance. Especially in text generation and re-ranking tasks, this strategy provides a wider learning space for the model, enabling it to better adapt to different application scenarios and improve the quality of the final generated results. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a schematic diagram of the DAE (denoising autoencoder) pre-training input data composition;

[0033] Figure 2 This is an example of how the BART model learns the correct text features based on the context during DAE pre-training.

[0034] Figure 3 Is to calculate the reordering loss loss rank Flowchart of the process. DETAILED DESCRIPTION

[0035] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.

[0036] The present invention is proposed to solve the following problems of the prior art in text generation. The problems of the prior art mainly include: difficulty in capturing language style details in specific scenarios, instability of reinforcement learning with human feedback (RLHF) in practical applications, and high cost and low efficiency of model training.

[0037] Before introducing the technical solution of the present invention, the three types of training data used in the present invention are explained first. The first type of data is unlabeled text, which is used to pre-train the basic BART model with denoising autoencoder (DAE). For example, in an AI outbound call scenario, this type of data can be a robot dialogue script written by a professional AI customer service outbound call script editor. The second type of data is used to train a BART-based generative model. This part of the data is usually in the form of question-answer pairs. In the AI ​​outbound call scenario, the question is a user question and the answer is the expected customer service response. The third type of data is used to train a BART-based sorting model. This part of the data includes a question and two answers with different scores, one is a high-scoring answer and the other is a low-scoring answer. The third type of training data can also be expanded by automatic model generation.

[0038] The present invention is achieved through the following technical solutions:

[0039] Step 1: Pre-training of denoising autoencoder (DAE). First, obtain the BART model for text generation; then obtain the conversation data of the intelligent outbound call system and form a text dataset. The conversation data of the intelligent outbound call system can be obtained through recording and other methods.

[0040] The pre-training stage mainly uses text data composed of conversation data based on the intelligent outbound call system. By performing random noise perturbations on the text to generate damaged input sequences, the BART model is trained to restore the original complete text and obtain a pre-trained BART model.

[0041] There are many ways to damage input data through text construction. This invention mainly uses random string masking. The detailed process can be referred to Figure 1 . During the DAE pre-training process, the corrupted input text (sequence) is used as the input of the model, and the model outputs a sequence that matches the original text. The model loss uses the standard cross-entropy loss function, which is defined as the difference between the generated output and the original text. The goal of pre-training is to train a basic model that can effectively generate high-quality text by maximizing the likelihood of the original text in the model output. In this stage, the main purpose is to enable the model to learn the language habits of a specific field, including common grammar, sentence structure, tone and intonation, so that it has the ability to generate expected content in a specific field. Appendix Figure 2It is an intuitive illustration of the features that the model is expected to learn during the pre-training process. In the illustrated example, the character "酒" in "酒吧" is masked, and the masked text is used as the input. During the encoding and decoding process of the BART model, BART has the opportunity to correctly reconstruct the masked character "酒" based on the subsequent characters "吧" and "酒", as well as other information in the context that may affect the selection of the current character.

[0042] Step 2, supervised fine-tuning (SFT) training of the generation model. The purpose of this fine-tuning training is to adjust the behavior pattern of the model. For example, in the scenario of artificial intelligence outbound calls, the question-and-answer pair dataset constructed based on the dialogue data of the intelligent outbound call system is used during the fine-tuning stage. The question-and-answer pair dataset includes multiple question-and-answer pairs. The questions in the question-and-answer pairs are used as the input to the pre-trained BART model, and the expected answers are used as the output of the model to obtain the fine-tuned BART model. In this stage, the standard cross-entropy loss function is also used to measure the matching degree between the generated answer and the actual answer (the answer in the question-and-answer pair), and the generated answer will be used for the training in the next stage. By adjusting the model parameters to minimize the loss function, the model gradually learns to provide high-quality responses in various customer service scenarios, ensuring that the generated text content is not only correct in grammar and style but also needs to be able to accurately understand the customer's question and give an accurate answer.

[0043] Step 3, reinforcement learning (RL) ranking model training. The training in this stage mainly relies on the third type of data, that is, it contains a question and two candidate answers - a winning answer and a losing answer; the answer text in the question-and-answer pair is used as the winning answer (the first candidate answer), and the output of the previous generation model when inputting the question text is used as the losing answer (the second candidate answer). During the training process, the questions in the question-and-answer pair dataset in the question-and-answer pair are used as the input to the fine-tuned BART model, and the model will calculate the scores of the winning answer and the losing answer respectively. The loss function (which will be described in detail later) optimizes the ranking ability of the model by encouraging the model to increase the gap between the score of the winning answer and the score of the losing answer. When using the previous generation model to generate training data in combination with the second type of data (question-and-answer pairs), the answer generated by the previous generation model will be used as the losing answer, and the answer in the question-and-answer pair will be used as the winning answer.

[0044] The loss function used in this step is described in detail below: loss = loss generation +α·loss rank

[0045] where: loss is the optimization objective in this stage, the total loss function. loss generation is the loss in the generation stage, which is the same as the loss in Step 2, and the standard cross-entropy loss is used to measure the gap between the generated text and the target text (the winning answer). loss rankIt is the core part of this stage and represents the ranking loss, which measures the difference between the winning answer and the losing answer. α is the weight coefficient used to balance the importance of generation loss and ranking loss.

[0046] Reordering loss rank The specific definitions are as follows (see Appendix Figure 3 Sorting loss diagram):

[0047]

[0048] Among them, J is the index set when the winning answer and the losing answer are at the same character position in the output sequence and the logit values ​​corresponding to the characters are different, Indicates the logits value corresponding to the character at position j in the output sequence in the winning answer. Indicates the logits value corresponding to the character at the same position j in the output sequence in the failed answer. σ is the Sigmoid function, which is used to convert the logits difference into probability. By calculating the difference between the winning answer and the failed answer, the model can learn how to better rank the candidate answers. α i is the delay attenuation factor of the i-th character in set J, α i+1 <α i <1.

[0049] When calculating the sorting loss, the present invention introduces a delay decay factor α, whose value is less than 1, to adjust the influence of different positions. The earlier the character, the greater the influence on the sorting; as the position is later, the weight α i Gradually decreases, thereby reducing the impact of this position on the final ranking loss. This design allows the model to pay more attention to the differences in early positions between candidate answers so as to better learn the importance of these differences for ranking.

[0050] This expression represents the difference between the text of the winning answer and the text of the losing answer. First, a question x = (x1, x2, x3...) is input to the BART model, and the BART model generates a sequence output y = (logits y1, logits y2, logits y3...). The length of this sequence usually corresponds to the length of the generated answer. Each element logits yj in the sequence is a list, and the length of the list is equal to the number of all characters in the dictionary. The numerical value in each list represents the probability distribution of the character at that position. Therefore, through the output of the BART model, the probability logits y of each character at each position j of the winning answer and the losing answer can be calculated. w j and logits y l j, take logits y wj and logits y l When j is not equal, the index j of the character position is included in the new set J; for convenience, the winning answer and the losing answer are represented as The constructed new J sequence is used for the calculation of the above loss.

[0051] The present invention is used The reason for reordering the different characters of the winning and losing answers from 1 is that we hope that when calculating the ranking loss based on the different characters of the winning and losing answers, the influence is only related to the relative position. For example, the first different character of the winning and losing answers may be the 10th character, the 100th character, or the 1000th character in the answer, but in set J they are all the first, so the weight is also α 1 , instead of α 10 ,α 100 , or α 1000 .

[0052] The reranking loss is calculated based on the difference in the likelihood of each character between the winning answer and the losing answer. Specifically, first calculate how much greater the likelihood of the character of the winning answer is than the likelihood of the character of the losing answer. The larger this difference is, the better the model performs in ranking. However, according to the definition of the loss function, this difference is added with a negative sign, which means that the smaller the value, the smaller the loss of the model, so this function is called the "loss function".

[0053] Through this ranking loss, the model can quantify the subtle differences between winning answers and losing answers, so that it can learn how to produce significant differences in business results through tiny text differences in specific application scenarios. During training, the re-ranking model of the present invention reuses the BART model of the generation phase. By strengthening the re-ranking learning, the model can better understand the subtle differences between different expressions, so that it can not only generate text that conforms to the expected language style (the effect of step one), but also accurately answer customer questions (the effect of step two), and further, it can also generate the most user-friendly and persuasive expression (the effect of step three).

[0054] In this embodiment, a reinforcement learning text generation system based on BART is also provided, including:

[0055] A pre-training module is used to obtain a BART model for text generation; obtain the conversation data of the intelligent outbound call system and form a text data set, perform random noise perturbation on the text in the text data set to generate a damaged input sequence, and pre-train the BART model based on the input sequence to obtain a pre-trained BART model;

[0056] A fine-tuning module is used to construct a question-answer pair dataset based on the conversation data of the intelligent outbound call system, wherein the question-answer pair dataset includes multiple question-answer pairs, and the question text in the question-answer pair is used as the input of the pre-trained BART model, and the pre-trained BART model is supervised and fine-tuned based on the answer text in the question-answer pair to obtain a fine-tuned BART model;

[0057] A reinforcement learning ranking training module is used to use the question text in the question-answer pair as the input of the fine-tuning BART model, the answer text in the question-answer pair as the first candidate answer, and the output of the fine-tuning BART model when the question text is last input into the fine-tuning module as the second candidate answer. The fine-tuning BART model is subjected to reinforcement learning ranking training based on the two candidate answers to obtain a trained BART model;

[0058] The text generation module is used to generate text based on input questions using the trained BART model to obtain generated text.

[0059] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0060] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0061] An embodiment of the present invention further provides an electronic device, including a memory and a processor;

[0062] The memory is used to store computer programs;

[0063] The processor is used to implement the above-mentioned BART-based reinforcement learning text generation method when executing the computer program.

[0064] An embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned BART-based reinforcement learning text generation method is implemented.

[0065] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0066] Obviously, the above-described embodiments and drawings are only some examples of the present application. For those of ordinary skill in the art, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that, although the work done in this development process may be complicated and lengthy, for those of ordinary skill in the art, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application. Without departing from the concept of the present application, several variations and improvements can also be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A reinforcement learning text generation method based on BART, characterized in that: The following steps are involved: 1) Obtaining a BART model for text generation; obtaining conversation data of an intelligent outbound call system and forming a text data set, performing random noise perturbation on the text in the text data set to generate a damaged input sequence, and pre-training the BART model based on the input sequence to obtain a pre-trained BART model; 2) constructing a question-answer pair dataset based on the conversation data of the intelligent outbound call system, wherein the question-answer pair dataset includes multiple question-answer pairs, using the question text in the question-answer pair as the input of the pre-trained BART model, and performing supervised fine-tuning training on the pre-trained BART model based on the answer text in the question-answer pair to obtain a fine-tuned BART model; 3) The question text in the question-answer pair is used as the input of the fine-tuned BART model, the answer text in the question-answer pair is used as the first candidate answer, and the output of the fine-tuned BART model when the question text is last input in step 2) is used as the second candidate answer. The fine-tuned BART model is subjected to reinforcement learning sorting training based on the two candidate answers to obtain a trained BART model; 4) Use the trained BART model to generate text based on the input question to obtain generated text.

2. The method for generating text using reinforcement learning based on BART according to claim 1, characterized in that: In the step 1), the BART model takes the input sequence as input, and the BART model outputs the restored complete text based on the context information of the damaged text. The parameters of the BART model are updated according to the first loss function between the restored complete text and the original text before damage to obtain a pre-trained BART model.

3. The method for generating text using reinforcement learning based on BART according to claim 1, characterized in that: In the step 2), the pre-trained BART model uses the question text in the question-answer pair as input to generate a predicted answer, and the parameters of the pre-trained BART model are updated according to a second loss function between the predicted answer and the true answer in the question-answer pair to obtain a fine-tuned BART model.

4. The method for generating text using reinforcement learning based on BART according to claim 1, characterized in that: In the step 3), the fine-tuned BART model takes the question text as input, generates an output sequence and its corresponding generated text, and the fine-tuned BART model calculates the scores of the first candidate answer and the second candidate answer respectively based on the output sequence; By constructing a third loss function and updating the parameters of the fine-tuned BART model based on the third loss function, a trained BART model is finally obtained.

5. The method for generating text using reinforcement learning based on BART according to claim 4, characterized in that: The third loss function is: loss=loss generation +β·loss rank Among them, loss is the loss function; loss generation is the cross entropy loss function between the first candidate answer and the generated one; loss rank is the re-ranking loss function; β is the weight coefficient; Where J is the index set when the first candidate answer and the second candidate answer are at the same character position in the output sequence and the logit values ​​corresponding to the characters are different, is the logits value corresponding to the character at position j in the output sequence in the first candidate answer, is the logits value corresponding to the character at the same position j in the output sequence in the second candidate answer; σ is the Sigmoid function, which is used to convert the logits difference into probability; is the logits value corresponding to the i-th character in the first candidate answer in set J; is the logits value corresponding to the i-th character in the second candidate answer in set J; α i is the delay attenuation factor of the i-th character in set J, α i+1 <α i <1.

6. A reinforcement learning text generation system based on BART, characterized in that: include: A pre-training module is used to obtain a BART model for text generation; obtain the conversation data of the intelligent outbound call system and form a text data set, perform random noise perturbation on the text in the text data set to generate a damaged input sequence, and pre-train the BART model based on the input sequence to obtain a pre-trained BART model; A fine-tuning module is used to construct a question-answer pair dataset based on the conversation data of the intelligent outbound call system, wherein the question-answer pair dataset includes multiple question-answer pairs, and the question text in the question-answer pair is used as the input of the pre-trained BART model, and the pre-trained BART model is supervised and fine-tuned based on the answer text in the question-answer pair to obtain a fine-tuned BART model; A reinforcement learning ranking training module is used to use the question text in the question-answer pair as the input of the fine-tuning BART model, the answer text in the question-answer pair as the first candidate answer, and the output of the fine-tuning BART model when the question text is last input into the fine-tuning module as the second candidate answer. The fine-tuning BART model is subjected to reinforcement learning ranking training based on the two candidate answers to obtain a trained BART model; The text generation module is used to generate text based on input questions using the trained BART model to obtain generated text.

7. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the BART-based reinforcement learning text generation method according to any one of claims 1 to 5.

8. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, which, when called and executed by a processor, are used to implement the BART-based reinforcement learning text generation method described in any one of claims 1 to 5.

Images