Model training method and device, computer equipment and storage medium

Through pre-training models and reinforcement learning optimization technology, the efficiency and accuracy problems of low-quality comment recognition and high-quality comment generation in the content platform are solved, the stability and diversity of the model are achieved, and the real-time update needs are met.

CN119004092AInactive Publication Date: 2024-11-22TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310561244.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-18
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When identifying and generating high-quality comments in content platforms, the existing technology is inefficient and accurate, and real-time updates cannot be achieved. It relies on manual annotation and model characteristics limitations, resulting in the generated content being single and cannot meet real-time iteration. Update requirements.

Method used

By obtaining the pre-trained generative model, copying and performing reinforcement learning, generating text with prompt information, and performing quality evaluation and reward mechanism on the generated text, optimizing the generative model until the preset stop conditions are met, and the model is self-optimized and diversified. generate.

Benefits of technology

It improves the stability and accuracy of model training, reduces costs, meets the requirements of real-time updates, and improves the accuracy and diversity of generated texts, which are suitable for different business scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004092A_ABST
    Figure CN119004092A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a model training method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a pre-trained generation model, copying the generation model, and obtaining a to-be-optimized generation model; performing text generation based on the prompt information through a generation model to obtain a first text and a first accuracy measurement parameter corresponding to the first text; performing text generation based on the prompt information through the to-be-optimized generation model to obtain a second text and a second accuracy measurement parameter corresponding to the second text; performing quality evaluation on the second text to obtain a quality score, and determining a reinforcement learning reward according to the quality score; and performing reinforcement learning on the to-be-optimized generation model according to the reward, the first accuracy measurement parameter and the second accuracy measurement parameter until a preset stop condition is met, and obtaining a trained generation model. And the efficiency and accuracy of model training are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, apparatus, computer equipment and storage medium. Background Art

[0002] In order to ensure the interactivity of the content platform, comments that meet the requirements (such as high-quality comments or interesting comments) can be generated for interaction. In order to maintain a good environment of the content platform, low-quality comments posted on the content platform can be identified to purify the content platform.

[0003] At present, manual review is usually used to identify low-quality comments posted on content platforms, or to publish comments that meet the requirements. This manual review method is inefficient and inaccurate, and cannot be updated in real time. Alternatively, a large number of comment cases for specific tasks are manually annotated as training samples to train the model, so that the trained model can be used to determine whether it is a low-quality comment or a generated comment. However, the cost of manual labeling is very high, the quality is unstable, and the labeling cycle is long. In addition, in actual business, the review rules are often adjusted, which makes it difficult to meet the requirements of model iteration and update, and it is impossible to update in real time, which reduces the efficiency and accuracy of model training. Moreover, due to the limitations of the characteristics of the model itself, the generated comments are repetitive and single, which reduces the accuracy of the model-generated comments. Summary of the invention

[0004] The embodiments of the present application provide a model training method, apparatus, computer equipment and storage medium, which can improve the efficiency and accuracy of model training.

[0005] To solve the above technical problems, the present application provides the following technical solutions:

[0006] The present application embodiment provides a model training method, including:

[0007] Obtain a pre-trained generative model, and copy the generative model to obtain a generative model to be optimized;

[0008] Generate text based on the prompt information by using the generation model to obtain a first text and a first accuracy measurement parameter corresponding to the first text;

[0009] Generate text based on the prompt information by using the generation model to be optimized to obtain a second text and a second accuracy measurement parameter corresponding to the second text;

[0010] Performing a quality assessment on the second text to obtain a quality score, and determining a reward for reinforcement learning according to the quality score;

[0011] According to the reward, the first accuracy measurement parameter and the second accuracy measurement parameter, reinforcement learning is performed on the generation model to be optimized until a preset stopping condition is met to obtain a trained generation model.

[0012] According to one aspect of the present application, a model training device is also provided, comprising:

[0013] An acquisition unit, used to acquire a pre-trained generative model and copy the generative model to obtain a generative model to be optimized;

[0014] A first generating unit, configured to generate text based on the prompt information by using the generating model, and obtain a first text and a first accuracy measurement parameter corresponding to the first text;

[0015] A second generating unit, configured to generate text based on the prompt information by using the generation model to be optimized, to obtain a second text and a second accuracy measurement parameter corresponding to the second text;

[0016] an evaluation unit, configured to perform a quality evaluation on the second text to obtain a quality score, and determine a reward for reinforcement learning according to the quality score;

[0017] A learning unit is used to perform reinforcement learning on the generation model to be optimized according to the reward, the first accuracy measurement parameter and the second accuracy measurement parameter until a preset stop condition is met to obtain a trained generation model.

[0018] According to one aspect of the present application, a computer device is also provided, including a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, any model training method provided in the embodiments of the present application is executed.

[0019] According to one aspect of the present application, a storage medium is also provided, which is used to store a computer program, and the computer program is loaded by a processor to execute any model training method provided in the embodiments of the present application.

[0020] According to one aspect of the present application, a computer program product is also provided, including a computer program, which is loaded by a processor to execute any model training method provided in the embodiments of the present application.

[0021] The embodiment of the present application can obtain a pre-trained generation model, and copy the generation model to obtain a generation model to be optimized, and then the generation model can be used to generate text based on prompt information to obtain a first text and a first accuracy measurement parameter corresponding to the first text, and the generation model to be optimized can be used to generate text based on prompt information to obtain a second text and a second accuracy measurement parameter corresponding to the second text. The second text generated by the generation model to be optimized is evaluated for quality, a quality score is obtained, and the reward for reinforcement learning is determined according to the quality score. At this time, the generation model to be optimized can be reinforced for learning according to the reward, the first accuracy measurement parameter and the second accuracy measurement parameter until the preset stop condition is met to obtain the generation model after training. On the one hand, the training of the generation model to be optimized is coordinated with the training of the generation model to be optimized to avoid the deviation of the generation model to be optimized during the training process, and the stability and accuracy of the model training are improved, so that the accuracy of the text generated by the training generation model can be improved. On the other hand, the pre-trained generation model and the generation model to be optimized are automatically generated to perform reinforcement learning with diversified texts, which do not rely on artificial marks and existing texts, reduce the cost of model training, meet the requirements of real-time update, and improve the efficiency and accuracy of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a schematic diagram of a scenario in which the model training method provided in an embodiment of the present application is applied;

[0024] Figure 2 It is a flowchart of the model training method provided in the embodiment of the present application;

[0025] Figure 3 It is a schematic diagram of the model pre-training provided in the embodiment of the present application;

[0026] Figure 4 is another schematic diagram of model pre-training provided in an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of quality assessment provided in an embodiment of the present application;

[0028] Figure 6 It is a schematic diagram of model reinforcement learning provided in an embodiment of the present application;

[0029] Figure 7is another flow chart of the model training method provided in the embodiment of the present application;

[0030] Figure 8 is a schematic diagram of a model training device provided in an embodiment of the present application;

[0031] Fig. 9 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0033] The embodiments of the present application provide a model training method, apparatus, computer equipment and storage medium.

[0034] See also Figure 1 , Figure 1 This is a schematic diagram of a scenario in which the model training method provided in an embodiment of the present application is applied. The model training method is applied to a model training system. The model training system may include a server 10 and a terminal 20, etc. The server 10 may be integrated with the model training device provided in the present application. The server 10 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, but is not limited to these.

[0035] The server 10 and the terminal 20 may be directly or indirectly connected via wired or wireless communication, which is not limited in the present application. The terminal 20 may be a mobile phone, a tablet computer, a laptop computer, a desktop computer, or a wearable device.

[0036] Among them, the server 10 can be used to obtain a pre-trained generation model, and copy the generation model to obtain the generation model to be optimized, and then the generation model can be used to generate text based on prompt information (such as a title or abstract, etc.) to obtain a first text and a first accuracy measurement parameter corresponding to the first text, and the generation model to be optimized can be used to generate text based on the prompt information to obtain a second text and a second accuracy measurement parameter corresponding to the second text. And the second text generated by the generation model to be optimized is evaluated for quality to obtain a quality score, and the quality score can be used as a reward for reinforcement learning. At this time, the generation model to be optimized can be reinforced for learning based on the reward, the first accuracy measurement parameter and the second accuracy measurement parameter until the preset stop condition is met to obtain the trained generation model, thereby improving the efficiency and accuracy of model training, wherein the preset stop condition can be the convergence of the loss function constructed based on the reward, the first accuracy measurement parameter and the second accuracy measurement parameter. After obtaining the trained generation model, the server 10 can accurately generate the target text through the trained generation model and send the target text to the terminal 20 for display. Alternatively, the server 10 can receive the text to be evaluated sent by the terminal 20, perform quality evaluation on the received text through the trained generation model, and send the evaluation result to the terminal 20; and so on.

[0037] It should be noted that Figure 1 The scenario diagram of the model training method application shown is only an example. The model training method application and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the application of the model training method and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0038] In this embodiment, artificial intelligence reinforcement learning is used to train a generative model with text generation capabilities, so as to generate text based on the trained generative model. It should be noted that artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0039] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which deep learning (DL) is a new research direction in machine learning. It is introduced into machine learning to make it closer to the original goal, namely artificial intelligence. At present, deep learning is mainly used in machine vision, speech processing technology, natural language processing and other fields.

[0040] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0041] In this embodiment, the description will be made from the perspective of a model training device, which can be specifically integrated into a computer device such as a server or a terminal.

[0042] See also Figure 2 , Figure 2 : is a flow chart of a model training method provided in an embodiment of the present application. The model training method may include:

[0043] S101. Obtain a pre-trained generative model, and copy the generative model to obtain a generative model to be optimized.

[0044] Among them, the pre-trained generative model can be a model obtained by pre-training the generative model to be trained based on the training sample to fine-tune the parameters of the generative model. The pre-stored pre-trained generative model can be directly obtained in the database for storing the model, or the training sample can be obtained first, and then the generative model to be trained is trained based on the training sample to obtain the pre-trained generative model. The generative model can be a generative pre-trained Transformer model (Generative Pre-Trained Transformer, GPT), a natural language model (Bloom), a large language model (Llama), a generalized linear model (Generalized Linear Model, GLM), a chat generative pre-trained transformer (ChatGenerative Pre-trained Transformer, ChatGPT), a bidirectional encoder representation model from a transformer (Bidirectional Encoder Representations from Transformers, BERT), a robustly optimized BERT model (A Robustly Optimized BERT, RoBERTa), etc. Of course, the generative model can also be other models, or include a combination of multiple models, which is not limited here. The generative model to be optimized may be a model that is consistent with the pre-trained generative model before reinforcement learning is performed. During the iterative process of reinforcement learning, the generative model to be optimized is continuously adjusted and optimized.

[0045] In one embodiment, obtaining a pre-trained generation model may include: obtaining training samples, the training samples including text samples, text sample prompt information corresponding to the text samples, and true labels corresponding to the text samples; generating predicted text and predicted labels corresponding to the predicted text based on the text sample prompt information by the generation model to be trained; calculating a loss value according to the true label and the predicted label; and adjusting the parameters of the generation model to be trained according to the loss value to obtain the pre-trained generation model.

[0046] In order to improve the reliability of the pre-trained generative model, the generative model to be trained can be pre-trained based on the training samples to preliminarily fine-tune the generative model to be trained to obtain a pre-trained generative model. The generative model to be trained can be an original generative model that has not been trained. Specifically, Figure 3As shown, the training samples can be obtained from a database for storing samples on a server, or the generated samples can be obtained from a terminal for generating text, to obtain the training samples; and so on. The training samples may include text samples, text sample prompt information corresponding to the text samples, and true labels corresponding to the text samples, etc. The training samples may also include other information, which is not limited here. The text samples may be texts such as comments or articles, the text sample prompt information may be relevant information such as the title or abstract corresponding to the text sample, and the true label may be the probability value of the corresponding word at each position in the text sample, or a parameter used to characterize the accuracy of the text sample generation, etc., which is not limited here.

[0047] Then, if Figure 3 As shown, the prediction text and the prediction label corresponding to the prediction text can be generated by the to-be-trained generation model based on the prompt information of the text sample. For example, the title and abstract can be input into the to-be-trained generation model, and the to-be-trained generation model can output the text based on the prompt information of the title and abstract. The text is the prediction text. At the same time, the to-be-trained generation model can also output the probability value of the corresponding word at each position in the prediction text, for example, the probability value of the word A appearing at the i-th position in the prediction text. The total probability value obtained by summing up the probabilities of the corresponding words at all positions in the prediction text or performing other operations can be used as the prediction label.

[0048] At this time, the loss value can be calculated based on the true label and the predicted label. For example, the true label and the predicted label can be compared, and the difference between the true label and the predicted label can be used as the loss value, or the loss value can be calculated based on the true label and the predicted label through a cross entropy function, etc.

[0049] In one embodiment, the true label includes a first probability value of a word corresponding to different positions in a text sample, and the predicted label includes a second probability value of a word corresponding to different positions in the predicted text. Calculating a loss value based on the true label and the predicted label may include: obtaining the text length of the text sample; and calculating the loss value based on the text length, the first probability value, and the second probability value.

[0050] In order to improve the accuracy of loss value calculation, the loss value can be calculated by the cross entropy function. For example, the text length of the text sample can be obtained, and the text length can be the number of characters contained in the text sample. Since the true label includes the first probability value of the word corresponding to different positions in the text sample, and the predicted label includes the second probability value of the word corresponding to different positions in the predicted text, the loss value can be calculated according to the text length, the first probability value and the second probability value. The calculation formula can be as follows:

[0051]

[0052] Among them, N represents the length of the text, yi represents the probability of the word corresponding to the ith position in the true distribution of the text sample (i.e., the first probability value), f(x i ) represents the probability of the word corresponding to the i-th position of the predicted text (i.e., the second probability value), log represents the logarithm, and loss represents the loss value calculated by the cross entropy function.

[0053] At this point, the parameters of the generative model to be trained can be adjusted according to the loss value, and the loss can be minimized through iterative training, so that a pre-trained generative model can be obtained. The generative model to be trained is fine-tuned by using prompt learning to obtain a pre-trained generative model, so that the pre-trained generative model has the ability to receive prompt information such as titles and summaries as input and generate corresponding text.

[0054] In one embodiment, the training sample also includes media sample prompt information corresponding to the text sample. Generating predicted text and predicted labels corresponding to the predicted text based on the text sample prompt information by the generation model to be trained may include: generating candidate text based on the text sample prompt information by the generation model to be trained, and extracting first feature information of the candidate text; generating predicted media based on the media sample prompt information by the generation model to be trained, and extracting second feature information of the predicted media; fusing the first feature information and the second feature information to obtain fused feature information; generating predicted text and predicted labels corresponding to the predicted text based on the fused feature information.

[0055] In addition to training the generative model based on unimodal text sample prompt information, the generative model can also be trained based on multimodal prompt information such as text sample prompt information and media sample prompt information, thereby improving the reliability and flexibility of model training. Specifically, Figure 4 As shown, candidate texts can be generated by the generative model to be trained based on text sample prompt information such as titles and abstracts. Candidate texts can be generated by the language model of the generative model to be trained. For example, the title and abstract can be input into the generative model to be trained, and the generative model to be trained outputs text based on the title and abstract. The text is used as the candidate text, and the feature vector of the candidate text is extracted to obtain the first feature information, which is the feature vector of the text contained in the candidate text.

[0056] Furthermore, the prediction media can be generated based on the media sample prompt information through the generative model to be trained, and the media sample prompt information can include prompt information related to images or videos. The generative model to be trained can include a model branch with the ability to generate text, and can also include a model branch with the ability to generate media such as images, such as a residual network (Residual Network, ResNet), etc. Figure 4As shown, the predicted media can be generated by the image model of the generative model to be trained. The type and structure of the image model can be flexibly set according to actual needs and are not limited here. For example, the image model can be a residual network or other models that can be used for image processing. The relevant media sample prompt information such as images or videos can be input into the generative model to be trained, and the generative model to be trained outputs media such as images based on the media sample prompt information. The media is used as the predicted media, and the second feature information of the predicted media is extracted. The second feature information is the feature vector of the media contained in the predicted media.

[0057] Then, if Figure 4 As shown, the first feature information and the second feature information can be fused to obtain fused feature information. For example, the first feature information and the second feature information can be spliced ​​to obtain fused feature information, or the first feature information and the second feature information can be weighted to obtain fused feature information. The specific fusion method can also be flexibly set according to actual needs and is not limited here.

[0058] At this time, the predicted text and the predicted label corresponding to the predicted text can be generated according to the fused feature information. For example, the fused feature information can be input into the generative model to be trained, and the predicted text and the predicted label corresponding to the predicted text can be output by the generative model to be trained based on the fused feature information. Thus, the loss value can be calculated according to the true label and the predicted label to adjust the parameters of the generative model to be trained until the loss reaches the minimum value, and the pre-trained generative model can be obtained. The generative model can be copied to obtain the generative model to be optimized.

[0059] S102: Generate text based on the prompt information using a generation model to obtain a first text and a first accuracy measurement parameter corresponding to the first text.

[0060] Among them, the prompt information may include information such as title and abstract, and the prompt information such as title and abstract can be extracted from the big data of the business according to actual needs, or the prompt information such as title and abstract generated by the editing instruction can be received, and the prompt information is input into the generation model, and the text is generated based on the prompt information by the generation model, so that the first text can be output, and the first accuracy measurement parameter corresponding to the first text can be output, and the first accuracy measurement parameter can be output. The first accuracy measurement parameter can be the probability value of the corresponding word at each position in the first text, for example, the probability of the word a appearing at the i-th position in the first text, and the probabilities of the corresponding words at all positions in the first text can be summed or weighted as the first accuracy measurement parameter. The first text and the first accuracy measurement parameter corresponding to the first text are generated based on the prompt information by the pre-trained generation model, without the need to collect and manually annotate the text, saving the annotation cost, ensuring the diversity of the ability to generate text, and improving the efficiency of model iteration and update.

[0061] S103 , generating text based on the prompt information by using the generation model to be optimized to obtain a second text and a second accuracy measurement parameter corresponding to the second text.

[0062] For example, prompt information such as title and abstract can be input into the generation model to be optimized, and text can be generated based on the prompt information by the generation model to be optimized, so that a second text can be output, and a second accuracy measurement parameter corresponding to the second text can be output. The second accuracy measurement parameter can be the probability value of the corresponding word at each position in the second text, for example, the probability of word a appearing at the i-th position in the second text. The probabilities of the corresponding words at all positions in the second text can be summed or weighted and used as the second accuracy measurement parameter.

[0063] In one embodiment, the prompt information includes text prompt information such as a title and a summary, and may also include related media prompt information such as an image or a video frame. The text model in the generation model to be optimized (such as a language model, etc.) generates text based on the prompt information to obtain a second text and a second accuracy measurement parameter corresponding to the second text. This may include: generating text based on the text prompt information by the generation model to be optimized to obtain a candidate text and a first accuracy corresponding to the candidate text; generating media based on the media prompt information by an image processing model in the generation model to be optimized (such as a residual network, etc.) to obtain a candidate media and a second accuracy corresponding to the candidate media, wherein the first accuracy is used to measure the accuracy of candidate text generation, and the second accuracy is used to measure the accuracy of candidate media generation. Then, a second text and a second accuracy measurement parameter corresponding to the second text can be generated based on the candidate text, the first accuracy, the candidate media and the second accuracy. For example, the candidate text and the candidate media can be fused (for example, by splicing or adjusting the candidate text based on the candidate media, etc.) to obtain the second text, and the first accuracy and the second accuracy can be fused and calculated (for example, by weighted operation) to obtain the second accuracy measurement parameter, thereby improving the flexibility and reliability of text generation, and by introducing more diverse prompt information input, the generation model to be optimized can output text that is more in line with the actual content.

[0064] It should be noted that the execution order of step S102 and step S103 can be simultaneous execution, or step S102 can be executed first and then step S103, or step S103 can be executed first and then step S102, etc., which is not specifically limited here.

[0065] S104: Perform a quality assessment on the second text to obtain a quality score, and determine a reward for reinforcement learning based on the quality score.

[0066] Among them, the quality score can be used to characterize whether the second text is a low-quality or high-quality text. The higher the quality score, the higher the quality of the second text. Alternatively, the lower the quality score, the higher the quality of the second text. The quality scoring standard can be set according to actual needs and is not limited here.

[0067] In one embodiment, the generation model to be optimized may include a pre-trained discriminant model, perform quality assessment on the second text to obtain a quality score, and determine a reward for reinforcement learning based on the quality score. This may include: performing quality assessment on the second text generated by the generation model to be optimized through a pre-trained discriminant model to obtain a quality score; and using the quality score as a reward for reinforcement learning.

[0068] In order to improve the convenience and accuracy of obtaining the quality score, the quality of the second text can be evaluated by a discriminant model to obtain the quality score. The discriminant model can be a trained discriminant model. The type and structure of the discriminant model can be flexibly set according to actual needs and are not limited here. Figure 5 As shown, the discriminant model included in the generative model to be optimized can be used to evaluate the quality of the second text generated by the generative model to be optimized to obtain a quality score, which can be used directly as a reward for reinforcement learning, or the quality score can be normalized or otherwise calculated to obtain an optimized score as a reward for reinforcement learning. This not only makes the generative model to be optimized more diverse and brings new capabilities to the discriminant model, but also takes advantage of adversarial learning to improve the capabilities of the generative model to be optimized and the discriminant model simultaneously.

[0069] It should be noted that the discriminant model can be dynamically adjusted according to actual needs. For example, if the discriminant model is a model for identifying low-quality text, when it is necessary to generate low-quality text, the discriminant model can be used to perform quality assessment on the second text generated by the optimization generation model to obtain a quality score. When it is necessary to generate high-quality text, the discriminant model can be used to perform quality assessment on the second text generated by the optimization generation model to obtain a quality score of -1, thereby converting the quality score to obtain the final quality score. For another example, if the discriminant model is a model for identifying high-quality text, when it is necessary to generate high-quality text, the discriminant model can be used to perform quality assessment on the second text generated by the optimization generation model to obtain a quality score. When it is necessary to generate low-quality text, the discriminant model can be used to perform quality assessment on the second text generated by the optimization generation model to obtain a quality score of -1, thereby converting the quality score to obtain the final quality score.

[0070] In the process of training the discriminant model, multiple labeled annotated texts can be obtained. The label can be the quality score of the annotated text, and the annotated text can be a high-quality text to ensure the performance of the discriminant model. After obtaining the annotated text, the discriminant model can be trained based on the annotated text to fine-tune the parameters of the discriminant model, and the probability value output by the discriminant model is used as the quality score of the annotated text. In order to make the result of the discriminant model meet the requirements of the "score", further processing can be performed on the output layer of the discriminant model. For example, a Sigmoid activation function can be added to the output layer of the discriminant model to normalize the quality score output by the discriminant model to a real number in the range of 0 to 1. The closer the quality score is to 0, the less likely it is that the annotated text is a low-quality text; conversely, the closer the quality score is to 1, the more likely it is that the annotated text is a low-quality review. Since the real quality assessment standard may be difficult to fully cover and accurately describe with rules and text, using a discriminant model for quality assessment allows the discriminant model to autonomously choose the direction of evolution to achieve the purpose of aligning with the quality assessment standard.

[0071] S105. Perform reinforcement learning on the generation model to be optimized according to the reward, the first accuracy measurement parameter and the second accuracy measurement parameter until a preset stop condition is met to obtain a trained generation model.

[0072] After obtaining the reward of reinforcement learning, the first accuracy measurement parameter corresponding to the first text generated by the generative model, and the second accuracy measurement parameter corresponding to the second text generated by the generative model to be optimized, reinforcement learning can be performed on the generative model to be optimized according to the reward, the first accuracy measurement parameter, and the second accuracy measurement parameter until a preset stop condition is met to obtain a trained generative model. The preset stop condition may be the convergence of a loss function constructed based on the reward, the first accuracy measurement parameter, and the second accuracy measurement parameter, or the reinforcement learning iteration reaches a preset number of times.

[0073] In one embodiment, reinforcement learning is performed on the generative model to be optimized according to the reward, the first accuracy measurement parameter, and the second accuracy measurement parameter until a preset stopping condition is met to obtain a trained generative model, including: determining a first loss function of the generative model to be optimized and a second loss function between the generative model to be optimized and the generative model according to the reward, the first accuracy measurement parameter, and the second accuracy measurement parameter; reinforcement learning is performed on the generative model based on the first loss function and the second loss function until a preset stopping condition that the first loss function and the second loss function converge is met to obtain a trained generative model.

[0074] To improve the reliability of reinforcement learning, the convergence of the loss function constructed by the reinforcement learning algorithm (Proximal Policy Optimization, PPO) can be used as the preset stopping condition of reinforcement learning, for example, Figure 6 As shown, the first loss function of the generative model to be optimized can be determined based on the reward, the first accuracy measurement parameter and the second accuracy measurement parameter. The first loss function can be the loss function of the generative model to be optimized. The reward of the current iteration can be calculated by the importance ratio and the discounted reward. The importance ratio can be the ratio of the probability of containing the same words between the text generated by the generative model to be optimized in the current iteration and the text generated by the generative model to be optimized in the previous iteration. And, the second loss function between the generative model to be optimized and the generative model can be determined based on the reward, the first accuracy measurement parameter and the second accuracy measurement parameter. The second loss function can be a loss function that characterizes the difference between the texts generated by the generative model to be optimized and the generative model in the reinforcement learning algorithm. At this time, as Figure 6 As shown, the generative model can be reinforced based on the first loss function and the second loss function until the first loss function and the second loss function converge to obtain a trained generative model. Alternatively, the first loss value calculated by the first loss function and the second loss value calculated by the second loss function can be fused to obtain a fusion loss until the fusion loss is minimized to obtain a trained generative model.

[0075] In one embodiment, determining a first loss function of the generative model to be optimized and a second loss function between the generative model to be optimized and the generative model based on a reward, a first accuracy measurement parameter, and a second accuracy measurement parameter may include: identifying differences between the first accuracy measurement parameter and the second accuracy measurement parameter to obtain difference information; determining the first loss function of the generative model to be optimized and a second loss function between the generative model to be optimized and the generative model based on the difference information and the reward.

[0076] In order to avoid the uncontrollable evolution direction caused by reinforcement learning, the texts generated by the generative model to be optimized and the generative model can be compared to calculate the penalty term of the difference. In order to improve the accuracy of the calculation, relative entropy can be introduced as a penalty term in the loss function of the training phase to ensure that the optimized generative model does not deviate too much from the generation direction of the original generative model. This relative entropy can be called KL divergence. Specifically, Figure 6 As shown, the difference between the first accuracy measurement parameter and the second accuracy measurement parameter can be identified to obtain difference information, for example, Figure 6As shown, the KL divergence calculation can be performed on the first accuracy measurement parameter and the second accuracy measurement parameter to obtain a divergence value, and the divergence value is used as the difference information between the first text generated by the generative model and the second text generated by the generative model to be optimized. Based on the difference information and the reward, the first loss function of the generative model to be optimized and the second loss function between the generative model to be optimized and the generative model can be determined.

[0077] In one embodiment, the generation model to be optimized includes the generation model to be optimized of the current iteration and the generation model to be optimized of the previous iteration. Determining the first loss function of the generation model to be optimized based on difference information and rewards may include: obtaining the ratio of the probability of containing the same words between the text generated by the generation model to be optimized of the current iteration and the text generated by the generation model to be optimized of the previous iteration; obtaining the first feature vector of the text generated by the generation model to be optimized of the current iteration and the second feature vector of the text generated by the generation model to be optimized of the previous iteration; determining the first loss function of the generation model to be optimized based on the ratio, the first feature vector, the second feature vector, the difference information and the reward.

[0078] Among them, the first feature vector may be a feature vector of the text generated by the generation model to be optimized in the current iteration, which is used to characterize the semantics and other information of the text generated by the generation model to be optimized in the current iteration, and the second feature vector may be a feature vector of the text generated by the generation model to be optimized in the previous iteration relative to the current iteration, which is used to characterize the semantics and other information of the text generated by the generation model to be optimized in the previous iteration. The formula of the first loss function may be as follows:

[0079]

[0080] in, It can represent the ratio of the probability of containing the same words between the text generated by the generative model to be optimized in the current iteration and the text generated by the generative model to be optimized in the previous iteration, and can be used to represent the importance ratio. P1 can represent the probability corresponding to the generative model to be optimized in the current iteration, and P2 can represent the probability corresponding to the generative model to be optimized in the previous iteration.

[0081] r can represent the sum of difference information (such as divergence value) and reward (i.e. reward value), γ can be a hyperparameter used to set the conversion coefficient, V1 can represent the first feature vector of the text generated by the generative model to be optimized in the current iteration, V2 can represent the second feature vector of the text generated by the generative model to be optimized in the previous iteration, and loss pg It can represent the first loss function and can be used to calculate the loss value.

[0082] In one embodiment, determining the second loss function between the generative model to be optimized and the generative model based on difference information and rewards can include: obtaining the third feature vector of the first text generated by the generative model and the fourth feature vector of the second text generated by the generative model to be optimized; determining the second loss function between the generative model to be optimized and the generative model based on the third feature vector, the fourth feature vector, the difference information and the reward.

[0083] Among them, the third feature vector can be a feature vector of the first text generated by the generative model, which is used to characterize the semantics and other information of the first text generated by the generative model, and the fourth feature vector can be a feature vector of the second text generated by the generative model to be optimized, which is used to represent the semantics and other information of the second text generated by the generative model to be optimized. The second loss function can be used to judge the difference between the first text generated by the generative model and the second text generated by the generative model to be optimized. The formula of the second loss function can be as follows:

[0084] loss value =||V pred -(r+V next )||

[0085] Among them, loss value It can represent the second loss function, which can be used to calculate the loss value, V pred The third feature vector V of the first text generated by the generative model can be represented next can represent the fourth eigenvector of the second text generated by the generative model to be optimized, and r can represent the sum of the difference information and the reward.

[0086] In one embodiment, after obtaining the trained generation model, the following may be included: obtaining target prompt information, and generating a target text based on the target prompt information through the trained generation model.

[0087] After obtaining the trained generative model, the target prompt information can be input into the trained generative model, and the trained generative model can accurately output the required target text based on the target prompt information. The target prompt information can be related information such as a title, summary, image or video frame. For example, the trained generative model can generate high-quality comments or great comments to achieve an active atmosphere. The trained generative model is obtained by performing reinforcement learning on the optimized generative model through the above-mentioned model training method, so that the trained generative model can generate text with good effect, high fluency, and good diversity, and can be flexibly applied to different business scenarios.

[0088] In one implementation, after obtaining the trained generated model, the following may be included: obtaining a text to be evaluated, and performing quality evaluation on the text using the trained generated model.

[0089] After obtaining the trained generative model, the text to be evaluated can be obtained. The text to be evaluated can be a comment or article to be published, or a comment or article that has been published. The text to be evaluated can be input into the trained generative model, and the quality of the text can be evaluated by the trained generative model. According to the quality evaluation result, it is determined whether the text to be evaluated is a low-quality text or a high-quality text. If the text to be evaluated is a low-quality text, the text to be evaluated can be prohibited from being published, or the text to be evaluated can be deleted, so that the content platform for publishing texts can be purified. If the text to be evaluated is a high-quality text, the text to be evaluated can be published. The trained generative model is obtained by performing reinforcement learning on the generative model to be optimized through the above-mentioned model training method, so that the trained generative model can accurately evaluate the quality of the text.

[0090] When it is necessary to perform quality assessment on multiple texts to be evaluated, the quality of each text can be evaluated separately through the trained model to obtain the quality score corresponding to each text. The multiple texts to be evaluated can be sorted and displayed in order from high to low or from low to high according to the scores, which improves the convenience and accuracy of text sorting.

[0091] The embodiment of the present application can obtain a pre-trained generation model, and copy the generation model to obtain a generation model to be optimized, and then the generation model can be used to generate text based on prompt information to obtain a first text and a first accuracy measurement parameter corresponding to the first text, and the generation model to be optimized can be used to generate text based on prompt information to obtain a second text and a second accuracy measurement parameter corresponding to the second text. The second text generated by the generation model to be optimized is evaluated for quality, a quality score is obtained, and the reward for reinforcement learning is determined according to the quality score. At this time, the generation model to be optimized can be reinforced for learning according to the reward, the first accuracy measurement parameter and the second accuracy measurement parameter until the preset stop condition is met to obtain the generation model after training. On the one hand, the training of the generation model to be optimized is coordinated with the training of the generation model to be optimized to avoid the deviation of the generation model to be optimized during the training process, and the stability and accuracy of the model training are improved, so that the accuracy of the text generated by the training generation model can be improved. On the other hand, the pre-trained generation model and the generation model to be optimized are automatically generated to perform reinforcement learning with diversified texts, which do not rely on artificial marks and existing texts, reduce the cost of model training, meet the requirements of real-time update, and improve the efficiency and accuracy of model training.

[0092] The method described in the above embodiment is further described in detail below with examples.

[0093] This embodiment takes the model training device integrated in the server as an example. Figure 7 , Figure 7A flow chart of a model training method provided in an embodiment of the present application. The method flow may include:

[0094] S201. Obtain training samples, and pre-train a generative model to be trained using the training samples to obtain a pre-trained generative model.

[0095] The server can obtain training samples from a database for storing samples, or receive training samples uploaded by a terminal, etc. The training samples may include text samples, text sample prompt information corresponding to the text samples, and true labels corresponding to the text samples, etc. The text samples may be texts such as comments or articles, and the text sample prompt information may be relevant information such as a title or summary corresponding to the text sample. The true label may be a parameter used to characterize the accuracy of text sample generation, etc.

[0096] The server can generate a predicted text and a predicted label corresponding to the predicted text based on the text sample prompt information through the model to be trained. The predicted label can be the probability value of word A appearing at the i-th position in the predicted text, or it can be the total probability value obtained by summing up or performing other operations on the probabilities of corresponding words at all positions in the predicted text as the predicted label.

[0097] Then, the server can compare the true label with the predicted label, and use the difference between the true label and the predicted label as the loss value, or can calculate the loss value based on the true label and the predicted label through a cross entropy function, etc. For example, the text length of the text sample can be obtained, the first probability value of the word corresponding to different positions in the text sample in the true label can be extracted, and the second probability value of the word corresponding to different positions in the predicted text in the predicted label can be extracted, and the loss value can be calculated according to the text length, the first probability value and the second probability value through a cross entropy function. At this time, the server can adjust the parameters of the generative model to be trained according to the loss value, and minimize the loss through iterative training, so as to obtain a pre-trained generative model. By using the prompt learning method to fine-tune the generative model to be trained, a pre-trained generative model is obtained, so that the pre-trained generative model has the ability to receive prompt information as input and generate corresponding text.

[0098] S202: Copy the generated model to obtain a generated model to be optimized.

[0099] After obtaining the pre-trained generative model, the server can copy the pre-trained generative model to obtain the generative model to be optimized. That is, the generative model to be optimized can be a model that is consistent with the pre-trained generative model before reinforcement learning. During the iterative process of reinforcement learning, the generative model to be optimized is continuously adjusted and optimized, while the generative model can maintain parameters unchanged, that is, not adjusted.

[0100] S203: Generate text based on the prompt information using a generation model to obtain a first text and a first accuracy measurement parameter corresponding to the first text.

[0101] The server can input prompt information such as the title and summary into the generation model, and generate text based on the prompt information through the generation model, so as to output the first text and the first accuracy measurement parameter corresponding to the first text, and the first accuracy measurement parameter is used to characterize the accuracy of the first text generated by the generation model. By automatically generating text and accuracy through the generation model, there is no need to collect and manually annotate the accuracy of the text, which saves the annotation cost and improves the efficiency of model iteration and update.

[0102] S204 , generating text based on the prompt information by using the generation model to be optimized to obtain a second text and a second accuracy measurement parameter corresponding to the second text.

[0103] The server can input prompt information such as the title and summary into the generation model to be optimized, and generate text based on the prompt information through the generation model to be optimized, so as to output the second text, and can output the second accuracy measurement parameter corresponding to the second text, and the second accuracy measurement parameter is used to characterize the accuracy of the second text generated by the generation model to be optimized. The efficiency of model training is improved by automatically generating text and accuracy through the generation model to be optimized.

[0104] It should be noted that the execution order of step S203 and step S204 can be simultaneous execution, or step S203 can be executed first and then step S204, or step S204 can be executed first and then step S203, etc., which is not specifically limited here.

[0105] S205. Perform quality assessment on the second text using the pre-trained discriminant model to obtain a quality score, and use the quality score as a reward for reinforcement learning.

[0106] Among them, the generative model to be optimized may include a pre-trained discriminative model. The server may evaluate the quality of the second text through the discriminative model to obtain a quality score. The quality score may be directly used as a reward for reinforcement learning, or the quality score may be normalized or otherwise calculated to obtain an optimized score as a reward for reinforcement learning. By leveraging the advantages of adversarial learning, the capabilities of the generative model to be optimized and the discriminative model can be improved simultaneously, improving the convenience of obtaining quality scores and the efficiency and reliability of model training.

[0107] S206: Perform divergence calculation on the first accuracy measurement parameter and the second accuracy measurement parameter to obtain a divergence value.

[0108] The server can perform KL divergence calculation on the first accuracy measurement parameter and the second accuracy measurement parameter to obtain a divergence value, and use the divergence value to characterize the difference between the first text generated by the generation model and the second text generated by the generation model to be optimized. The divergence value can be used as a penalty for the difference, thereby avoiding the uncontrollable evolution direction of the generation model to be optimized during the reinforcement learning process and improving the accuracy of reinforcement learning.

[0109] S207. Determine a first loss function of the generation model to be optimized and a second loss function between the generation model to be optimized and the generation model according to the divergence value and the reward.

[0110] The server can obtain the ratio of the probability of containing the same words between the text generated by the generative model to be optimized in the current iteration and the text generated by the generative model to be optimized in the previous iteration, obtain the first feature vector of the text generated by the generative model to be optimized in the current iteration, and the second feature vector of the text generated by the generative model to be optimized in the previous iteration; then, the first loss function of the generative model to be optimized can be determined based on the ratio, the first feature vector, the second feature vector, the divergence value and the reward.

[0111] The server can obtain the third feature vector of the first text generated by the generative model and the fourth feature vector of the second text generated by the generative model to be optimized, and then determine the second loss function between the generative model to be optimized and the generative model based on the third feature vector, the fourth feature vector, the divergence value and the reward. The second loss function can be used to judge the difference between the first text generated by the generative model and the second text generated by the generative model to be optimized.

[0112] S208, performing reinforcement learning on the generation model to be optimized based on the first loss function and the second loss function until the first loss function and the second loss function converge to obtain a trained generation model.

[0113] The server can use the convergence of the first loss function and the second loss function constructed by the reinforcement learning algorithm as the stopping condition of the reinforcement learning, that is, reinforcement learning is performed on the optimized generation model based on the first loss function and the second loss function until the first loss function and the second loss function converge to achieve the purpose of minimizing the loss, thereby obtaining the trained generation model and improving the reliability of the reinforcement learning.

[0114] S209, generating a target text based on the target prompt information through the trained generation model.

[0115] After obtaining the trained generation model, the server can input the target prompt information into the trained generation model, and the trained generation model can accurately output the required target text based on the target prompt information. For example, high-quality reviews can be generated through the trained generation model.

[0116] On the one hand, the embodiment of the present application cooperates with the training of the generative model to be optimized through the pre-trained generative model to avoid the deviation of the generative model to be optimized during the training process, thereby improving the stability and accuracy of the model training. In addition, the pre-trained generative model and the generative model to be optimized are used to automatically generate a variety of texts for reinforcement learning, which does not rely on artificial marks and existing texts, reduces the cost of model training, meets the requirements of real-time updates, and improves the efficiency and accuracy of model training. On the other hand, the trained generative model obtained by reinforcement learning has good effect in generating text, the generated text has high fluency and accuracy, and has good diversity, and can be flexibly applied to different business scenarios.

[0117] In order to facilitate better implementation of the model training method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above model training method. The meanings of the nouns are the same as those in the above model training method, and the specific implementation details can refer to the description in the method embodiment.

[0118] See also Figure 8 , Figure 8 A structural schematic diagram of a model training device provided in an embodiment of the present application, wherein the model training device 300 may include an acquisition unit 301, a first generation unit 302, a second generation unit 303, an evaluation unit 304, and a learning unit 305, etc.

[0119] Among them, the acquisition unit 301 is used to obtain the pre-trained generation model and copy the generation model to obtain the generation model to be optimized.

[0120] The first generating unit 302 is configured to generate text based on the prompt information by using a generation model to obtain a first text and a first accuracy measurement parameter corresponding to the first text.

[0121] The second generating unit 303 is used to generate text based on the prompt information by using the generation model to be optimized, and obtain a second text and a second accuracy measurement parameter corresponding to the second text.

[0122] The evaluation unit 304 is used to perform quality evaluation on the second text to obtain a quality score, and determine a reward for reinforcement learning according to the quality score.

[0123] The learning unit 305 is used to perform reinforcement learning on the generation model to be optimized according to the reward, the first accuracy measurement parameter and the second accuracy measurement parameter until a preset stop condition is met to obtain a trained generation model.

[0124] In one implementation, the learning unit 305 may include:

[0125] a determination module, configured to determine a first loss function of the generative model to be optimized and a second loss function between the generative model to be optimized and the generative model according to the reward, the first accuracy measurement parameter and the second accuracy measurement parameter;

[0126] The learning module is used to perform reinforcement learning on the generation model based on the first loss function and the second loss function until the preset stop condition of the convergence of the first loss function and the second loss function is met to obtain the trained generation model.

[0127] In one embodiment, the determination module can be specifically used to: identify the difference between the first accuracy measurement parameter and the second accuracy measurement parameter to obtain difference information; determine the first loss function of the generation model to be optimized and the second loss function between the generation model to be optimized and the generation model based on the difference information and the reward.

[0128] In one embodiment, the determination module can be specifically used to: obtain the ratio of the probability of containing the same words between the text generated by the generation model to be optimized in the current iteration and the text generated by the generation model to be optimized in the previous iteration; obtain the first feature vector of the text generated by the generation model to be optimized in the current iteration, and the second feature vector of the text generated by the generation model to be optimized in the previous iteration; determine the first loss function of the generation model to be optimized based on the ratio, the first feature vector, the second feature vector, the difference information and the reward.

[0129] In one embodiment, the determination module can be specifically used to: obtain a third feature vector of the first text generated by the generative model, and a fourth feature vector of the second text generated by the generative model to be optimized; determine a second loss function between the generative model to be optimized and the generative model based on the third feature vector, the fourth feature vector, the difference information and the reward.

[0130] In one implementation, the acquiring unit 301 may include:

[0131] An acquisition module is used to acquire training samples, where the training samples include text samples, text sample prompt information corresponding to the text samples, and true labels corresponding to the text samples;

[0132] A generation module, used to generate predicted text and predicted labels corresponding to the predicted text based on the text sample prompt information through the training generation model;

[0133] The calculation module is used to calculate the loss value based on the true label and the predicted label;

[0134] The adjustment module is used to adjust the parameters of the generative model to be trained according to the loss value to obtain a pre-trained generative model.

[0135] In one embodiment, the generation module can be specifically used to: generate candidate texts based on text sample prompt information through the generation model to be trained, and extract first feature information of the candidate texts; generate predicted media based on media sample prompt information through the generation model to be trained, and extract second feature information of the predicted media; fuse the first feature information and the second feature information to obtain fused feature information; generate predicted text and a prediction label corresponding to the predicted text based on the fused feature information.

[0136] In one embodiment, the true label includes a first probability value of a word corresponding to different positions in a text sample, and the predicted label includes a second probability value of a word corresponding to different positions in the predicted text. The calculation module can be specifically used to obtain the text length of the text sample; and calculate the loss value based on the text length, the first probability value, and the second probability value.

[0137] In one implementation, the evaluation unit 304 may be specifically configured to: perform quality evaluation on the second text generated by the generation model to be optimized using a pre-trained discriminant model to obtain a quality score; and use the quality score as a reward for reinforcement learning.

[0138] In one embodiment, the model training device 300 may further include: a third generation unit, configured to obtain target prompt information, generate a model after training, and generate a target text based on the target prompt information.

[0139] In one embodiment, the model training device 300 may further include: a processing unit, configured to obtain a text to be evaluated, and perform quality evaluation on the text by generating a model after training.

[0140] In the embodiment of the present application, the acquisition unit 301 can acquire the pre-trained generation model, and copy the generation model to obtain the generation model to be optimized, and then the first generation unit 302 can generate text based on the prompt information through the generation model to obtain the first text and the first accuracy measurement parameter corresponding to the first text, and the second generation unit 303 can generate text based on the prompt information through the generation model to be optimized to obtain the second text and the second accuracy measurement parameter corresponding to the second text. The evaluation unit 304 performs a quality evaluation on the second text generated by the generation model to be optimized to obtain a quality score, and determines the reward of reinforcement learning according to the quality score. At this time, the learning unit 305 can perform reinforcement learning on the generation model to be optimized according to the reward, the first accuracy measurement parameter and the second accuracy measurement parameter until the preset stop condition is met to obtain the trained generation model. On the one hand, by coordinating the training of the generation model to be optimized with the pre-trained generation model, the deviation of the generation model to be optimized during the training process is avoided, and the stability and accuracy of the model training are improved, so that the accuracy of the text generated by the training generation model can be improved. On the other hand, through the pre-trained generative model and the generative model to be optimized, diversified texts are automatically generated for reinforcement learning, which does not rely on manual labels and existing texts, reduces the cost of model training, meets the requirements of real-time updates, and improves the efficiency and accuracy of model training.

[0141] The present application also provides a computer device, which may be a terminal or a server, etc. Fig. 9 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0142] The computer device may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will appreciate that Fig. 9 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0143] The processor 401 is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the computer device and processes data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0144] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0145] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0146] The computer device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0147] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0148] Acquire a pre-trained generative model, and copy the generative model to obtain the generative model to be optimized; generate text based on the prompt information through the generative model to obtain a first text and a first accuracy measurement parameter corresponding to the first text; generate text based on the prompt information through the generative model to be optimized to obtain a second text and a second accuracy measurement parameter corresponding to the second text; perform quality assessment on the second text to obtain a quality score, and determine a reward for reinforcement learning based on the quality score; perform reinforcement learning on the generative model to be optimized based on the reward, the first accuracy measurement parameter, and the second accuracy measurement parameter until a preset stop condition is met to obtain a trained generative model.

[0149] In the above embodiments, the description of each embodiment has its own focus. For the parts that are not described in detail in a certain embodiment, please refer to the detailed description of the model training method above, which will not be repeated here.

[0150] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementations in the above-mentioned embodiments.

[0151] It will be understood by those skilled in the art that all or part of the steps in the various methods of the above embodiments can be completed by computer instructions, or by controlling related hardware through computer instructions, and the computer instructions can be stored in a computer-readable storage medium (i.e., storage medium) and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium in which a computer program is stored, and the computer program may include computer instructions, which can be loaded by a processor to execute any model training method provided in the embodiment of the present application.

[0152] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0153] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0154] Since the instructions stored in the storage medium can execute the steps in any model training method provided in the embodiments of the present application, the beneficial effects that can be achieved by any model training method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0155] The above is a detailed introduction to a model training method, device, computer equipment and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model training method, characterized in that: include: Obtain a pre-trained generative model, and copy the generative model to obtain a generative model to be optimized; Generate text based on the prompt information by using the generation model to obtain a first text and a first accuracy measurement parameter corresponding to the first text; Generate text based on the prompt information by using the generation model to be optimized to obtain a second text and a second accuracy measurement parameter corresponding to the second text; Performing a quality assessment on the second text to obtain a quality score, and determining a reward for reinforcement learning according to the quality score; According to the reward, the first accuracy measurement parameter and the second accuracy measurement parameter, reinforcement learning is performed on the generation model to be optimized until a preset stopping condition is met to obtain a trained generation model.

2. The model training method according to claim 1, characterized in that: The step of performing reinforcement learning on the generation model to be optimized according to the reward, the first accuracy measurement parameter, and the second accuracy measurement parameter until a preset stop condition is met to obtain a trained generation model includes: Determine a first loss function of the generative model to be optimized and a second loss function between the generative model to be optimized and the generative model according to the reward, the first accuracy measurement parameter and the second accuracy measurement parameter; Reinforcement learning is performed on the generation model based on the first loss function and the second loss function until a preset stop condition that the first loss function and the second loss function converge is met, thereby obtaining a trained generation model.

3. The model training method according to claim 2, characterized in that: The determining, according to the reward, the first accuracy measurement parameter, and the second accuracy measurement parameter, a first loss function of the to-be-optimized generation model and a second loss function between the to-be-optimized generation model and the generation model, comprises: Identifying a difference between the first accuracy measurement parameter and the second accuracy measurement parameter to obtain difference information; According to the difference information and the reward, a first loss function of the generation model to be optimized and a second loss function between the generation model to be optimized and the generation model are determined.

4. The model training method according to claim 3, characterized in that: The generation model to be optimized includes the generation model to be optimized in the current iteration and the generation model to be optimized in the previous iteration, and determining the first loss function of the generation model to be optimized according to the difference information and the reward includes: Obtain the ratio of the probability of containing the same words between the text generated by the to-be-optimized generation model of the current iteration and the text generated by the to-be-optimized generation model of the previous iteration; Obtaining a first feature vector of the text generated by the to-be-optimized generation model of the current iteration and a second feature vector of the text generated by the to-be-optimized generation model of the previous iteration; A first loss function of the generation model to be optimized is determined according to the ratio, the first eigenvector, the second eigenvector, the difference information and the reward.

5. The model training method according to claim 3, characterized in that: The determining, according to the difference information and the reward, a second loss function between the generation model to be optimized and the generation model comprises: Obtaining a third feature vector of the first text generated by the generation model, and a fourth feature vector of the second text generated by the generation model to be optimized; A second loss function between the generation model to be optimized and the generation model is determined according to the third eigenvector, the fourth eigenvector, the difference information and the reward.

6. The model training method according to claim 1, characterized in that: The obtaining of the pre-trained generative model comprises: Acquire a training sample, wherein the training sample includes a text sample, text sample prompt information corresponding to the text sample, and a true label corresponding to the text sample; Generate predicted text and predicted labels corresponding to the predicted text based on the text sample prompt information by using a generation model to be trained; Calculate the loss value according to the true label and the predicted label; The parameters of the generative model to be trained are adjusted according to the loss value to obtain a pre-trained generative model.

7. The model training method according to claim 6, characterized in that: The training sample also includes media sample prompt information corresponding to the text sample, and the generating model to be trained generates predicted text and a predicted label corresponding to the predicted text based on the text sample prompt information, including: Generate candidate texts based on the text sample prompt information through the to-be-trained generation model, and extract first feature information of the candidate texts; Generate predicted media based on the media sample prompt information through the to-be-trained generation model, and extract second feature information of the predicted media; fusing the first feature information and the second feature information to obtain fused feature information; A predicted text and a predicted label corresponding to the predicted text are generated according to the fused feature information.

8. The model training method according to claim 6, characterized in that: The real label includes a first probability value of a word corresponding to different positions in the text sample, the predicted label includes a second probability value of a word corresponding to different positions in the predicted text, and the loss value is calculated according to the real label and the predicted label, including: Obtaining the text length of the text sample; A loss value is calculated according to the text length, the first probability value, and the second probability value.

9. The model training method according to claim 1, characterized in that: The generation model to be optimized includes a pre-trained discriminant model, and the quality assessment of the second text is performed to obtain a quality score, and a reward for reinforcement learning is determined according to the quality score, including: Performing a quality assessment on the second text generated by the generation model to be optimized by using a pre-trained discriminant model to obtain a quality score; The quality score is used as a reward for reinforcement learning.

10. The model training method according to any one of claims 1 to 9, characterized in that: The method further comprises: Obtain target prompt information, and generate a target text based on the target prompt information by using the trained model; or, A text to be evaluated is obtained, and quality evaluation of the text is performed using the trained generated model.

11. A model training device, characterized in that: include: An acquisition unit, used to acquire a pre-trained generative model and copy the generative model to obtain a generative model to be optimized; A first generating unit, configured to generate text based on the prompt information by using the generating model, and obtain a first text and a first accuracy measurement parameter corresponding to the first text; A second generating unit, configured to generate text based on the prompt information by using the generation model to be optimized, to obtain a second text and a second accuracy measurement parameter corresponding to the second text; an evaluation unit, configured to perform a quality evaluation on the second text to obtain a quality score, and determine a reward for reinforcement learning according to the quality score; A learning unit is used to perform reinforcement learning on the generation model to be optimized according to the reward, the first accuracy measurement parameter and the second accuracy measurement parameter until a preset stopping condition is met to obtain a trained generation model.

12. A computer device, characterized in that: It includes a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the model training method according to any one of claims 1 to 10 is executed.

13. A storage medium, characterized in that: The storage medium is used to store a computer program, which is loaded by a processor to execute the model training method described in any one of claims 1 to 10.