Model training method, written language spoken language conversion method, device and product

Abstract

The invention provides a model training method, a written language spoken language conversion method, equipment and a product, which are applied to the field of natural language processing. The model training method comprises the steps that supervised training is conducted on a large language model based on first training data and task cues indicating to convert written languages into spoken languages, a supervised training model is obtained, and the first training data comprises supervised data pairs formed by written language texts and spoken language texts; based on second training data and a supervised training model, multiple rounds of reinforcement learning training are carried out, a written language spoken language model is obtained, and the second training data comprises a non-labeled written language data set and preference data used for generating reward signals. Therefore, the spoken language model of the written language is obtained by training after reinforcement learning of the large language model, and the quality of the spoken text transferred by the spoken language model of the written language is improved.

Description

TECHNICAL FIELD

[0001] The present application applies to the field of natural language processing, and particularly relates to a model training method, a written language oralization method, equipment and products. BACKGROUND

[0002] With the development of artificial intelligence technology, speech synthesis and speech interaction technology are widely used in intelligent assistants, voice customers, virtual anchors, education and training, etc. In the above scenarios, speech output according to written language leads to unnatural and unrealistic speech expression.

[0003] In related technologies, a data set required for supervised training is formed by annotators performing data annotation of oralization text to written language text, and a deep learning model is supervised trained based on the data set, so that the deep learning model learns the mapping relationship from written language text to oralization text.

[0004] However, the written language oralization ability of the model trained by the above method is still lacking, resulting in the quality of the oralization text converted from the written language text to be improved. SUMMARY

[0005] To solve the above problems, the present application provides a model training method, a written language oralization method, equipment and products, which can improve the quality of the oralization text obtained by the model in the written language oralization process.

[0006] The first aspect of the present application provides a model training method, comprising: based on first training data and a task prompt word indicating the conversion of written language to oralization, supervising training of a large language model to obtain a supervised training model, wherein the first training data comprises a supervised data pair formed by written language text and oralization text; based on second training data and the supervised training model, performing multi-round reinforcement learning training to obtain a written language oralization model, wherein the second training data comprises an unannotated written language data set and preference data for generating a reward signal.

[0007] In a possible implementation, the performing of the multiple rounds of reinforcement learning training based on the second training data and the supervised training model to obtain the written-to-spoken model comprises: training a reward model based on the preference data and the supervised training model; performing model initialization according to the reward model to obtain an initial first value model and an initial second value model, and performing model initialization according to the supervised training model to generate an initial first policy model and an initial second policy model; performing multiple rounds of reinforcement learning training on the initial first policy model, the initial first value model, the initial second policy model and the initial second value model based on the written data set, the supervised training model, the reward model and a preset loss function, the training of the initial first value model, the initial second policy model and the initial second value model being used to assist the training of the initial first policy model; and the written-to-spoken model being the first policy model after the multiple rounds of reinforcement learning training.

[0008] In a possible implementation, the training of the reward model based on the preference data and the supervised training model comprises: combining a model structure of the supervised training model except for an output layer with an initialized scalar output layer to construct an initial model; and training the initial model based on the preference data to obtain the reward model.

[0009] In a possible implementation, the i th round of reinforcement learning training comprises: performing multiple updates on the second policy model after the i-1 th round of reinforcement learning training and the second value model after the i-1 th round of reinforcement learning training based on the written data set, the supervised training model, the first policy model after the i-1 th round of reinforcement learning training, the first value model after the i-1 th round of reinforcement learning training and the preset loss function, to obtain a second policy model after the i th round of reinforcement learning training and a second value model after the i th round of reinforcement learning training, i being greater than or equal to 1; replacing the first policy model after the i-1 th round of reinforcement learning training with the second policy model after the i th round of reinforcement learning training to obtain a first policy model after the i th round of reinforcement learning training; and replacing the first value model after the i-1 th round of reinforcement learning training with the second value model after the i th round of reinforcement learning training to obtain a first value model after the i th round of reinforcement learning training.

[0010] In a possible implementation, the multiple times of updating the second policy model after the i-1th round of reinforcement learning training and the second value model after the i-1th round of reinforcement learning training based on the written language data set, the supervised training model, the first policy model after the i-1th round of reinforcement learning training, the first value model after the i-1th round of reinforcement learning training, and the preset loss function includes: sampling training samples participating in the i th round of reinforcement learning training in the multiple written language texts contained in the written language data set; performing written-to-spoken translation on the training samples by using the supervised training model to obtain first output data at multiple time steps and a first predicted spoken language text; performing written-to-spoken translation on the training samples by using the first policy model after the i-1th round of reinforcement learning training to obtain input data at the multiple time steps, second output data at the multiple time steps, and a second predicted spoken language text; comparing the first output data at the multiple time steps with the second output data at the multiple time steps by using a difference loss function in the preset loss function to determine a difference loss value at the multiple time steps; performing reward value evaluation on the second predicted spoken language text by using the reward model to obtain a reward value of the second predicted spoken language text; performing value evaluation on the second output data at the multiple time steps by using the first value model after the i-1th round of reinforcement learning training to obtain a single-step estimated value at the multiple time steps; determining an advantage value at the multiple time steps and a discounted return value at the multiple time steps according to the difference loss value at the multiple time steps, the reward value, and the single-step estimated value at the multiple time steps; determining multiple first sample data and multiple second sample data according to the input data at the multiple time steps, the first output data at the multiple time steps, the second output data at the multiple time steps, the difference loss value at the multiple time steps, the single-step estimated value at the multiple time steps, the advantage value at the multiple time steps, and the discounted return value at the multiple time steps; performing multiple times of updating on the second policy model after the i-1th round of reinforcement learning training according to the multiple first sample data and a policy loss function in the preset loss function to obtain the second policy model after the i th round of reinforcement learning training; and performing multiple times of updating on the second value model after the i-1th round of reinforcement learning training according to the multiple second sample data and a value loss function in the preset loss function to obtain the second value model after the i th round of reinforcement learning training.

[0011] In a possible implementation, in the plurality of first sample data, each first sample data includes the following data at a single time step: input data, second output data, difference loss value, and discounted return value, the tth update of the second policy model after the i-1th round of reinforcement learning training includes: processing the input data in the tth first sample data by the second policy model after the t-1th update to obtain third output data; determining a policy loss value by the policy loss function according to the third output data, the second output data in the tth first sample data, the difference loss value in the tth first sample data, and the discounted return value in the tth first sample data; updating the second policy model after the t-1th update according to the policy loss value to obtain the second policy model after the tth update; wherein t is greater than or equal to 1, and when t is equal to 1, the second policy model after the t-1th update is the second policy model after the i-1th round of reinforcement learning training.

[0012] In a possible implementation, in the plurality of second sample data, each second sample data includes the following data at a single time step: second output data, one-step estimated value, and discounted return value, the s th update of the second value model after the i-1th round of reinforcement learning training includes: performing value evaluation on the second output data in the s th first sample data by the second value model after the s-1th update to obtain a one-step estimated value output by the second value model after the s-1th update; determining a value loss value by the value loss function according to the one-step estimated value output by the second value model after the s-1th update, the one-step estimated value in the s th first sample data, and the discounted return value in the s th first sample data; updating the second value model after the s-1th update according to the value loss value to obtain the second value model after the s th update; wherein s is greater than or equal to 1, and when s is equal to 1, the second value model after the s-1th update is the second value model after the s-1th round of reinforcement learning training.

[0013] The second aspect of the present application provides a written language oralization method, including: obtaining a written language text to be processed; and converting the written language text into an oralization text through a written language oralization model; wherein the written language oralization model is obtained by training the model training method according to the first aspect.

[0014] The third aspect of the present application provides a model training apparatus, comprising: a first training unit configured to perform supervised training on a large language model based on first training data and a task prompt word indicating a task of transcribing written language into spoken language, to obtain a supervised training model, wherein the first training data comprises a supervised data pair formed by written language text and spoken language text; and a second training unit configured to perform multi-round reinforcement learning training based on second training data and the supervised training model, to obtain a written language spoken language model, wherein the second training data comprises an unlabeled written language data set and preference data used to generate a reward signal.

[0015] The fourth aspect of the present application provides a written language spoken language apparatus, comprising: an acquisition unit configured to acquire written language text to be processed; and a transcription unit configured to transcribe the written language text into spoken language text by using a written language spoken language model, wherein the written language spoken language model is trained according to the model training method of the first aspect.

[0016] The fifth aspect of the present application provides an electronic device, comprising a memory and a processor; the memory is connected to the processor and is configured to store a program; the processor is configured to implement the model training method according to the first aspect of the present application or any possible implementation manner of the first aspect of the present application, or implement the written language spoken language method according to the second aspect of the present application by running the program stored in the memory.

[0017] The sixth aspect of the present application provides a chip, comprising a processor and a data interface, wherein the processor reads and runs a program stored on a memory through the data interface, to execute the model training method according to the first aspect of the present application or any possible implementation manner of the first aspect of the present application, or implement the written language spoken language method according to the second aspect of the present application.

[0018] The seventh aspect of the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the model training method according to the first aspect of the present application or any possible implementation manner of the first aspect of the present application, or implement the written language spoken language method according to the second aspect of the present application.

[0019] The eighth aspect of the present application provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the model training method according to the first aspect of the present application or any possible implementation manner of the first aspect of the present application, or implement the written language spoken language method according to the second aspect of the present application.

[0020] According to the model training method, written language oralization method, device and product provided in the application, the written language oralization model is obtained through reinforcement learning post-training of a large language model. In the reinforcement learning post-training process: first, the large language model is supervised trained based on first training data and task prompt words to improve the supervised training effect and obtain a supervised training model that learns written language oralization capability; and then, based on second training data containing preference data and unlabeled written language data set and the supervised training model, multi-round reinforcement learning training is performed. The preference data is used to generate a reward signal to guide the model to perceive and judge the pros and cons of oralization text, and the written language oralization capability of the model is continuously optimized, and finally the written language oralization model is obtained. Thus, through the above reinforcement learning post-training scheme, the quality of the oralization text obtained by the model through written language oralization conversion is improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0022] Figure 1 The flowchart of supervised training of the written language oralization model.

[0023] Figure 2 The schematic diagram of the implementation environment according to the embodiments of the present application.

[0024] Figure 3 The flowchart of the model training method provided according to the embodiments of the present application Figure 1 .

[0025] Figure 4 The example diagram of sampling through the supervised training model is shown.

[0026] Figure 5 The flowchart of the model training method provided according to the embodiments of the present application Figure 2 .

[0027] Figure 6 The flowchart of the i-th round of reinforcement learning training.

[0028] Figure 7 The example diagram of reinforcement learning training is shown.

[0029] Figure 8 The structural schematic diagram of the model training device provided according to the embodiments of the present application.

[0030] Figure 9A structural schematic diagram of a written-to-spoken language conversion device according to an embodiment of the present application.

[0031] Figure 10 A structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0033] In intelligent assistant, voice customer service, virtual anchor, education and training and other human-computer interaction scenarios, the synthesized speech based on written language lacks the paralanguage components that naturally appear in real spoken language communication, resulting in unnatural and unrealistic speech expression of machines.

[0034] There are various paralanguage components, such as filler words like "just" and "this", mood words like "ah" and "ba", pauses, and inhalation sounds. In real human communication, paralanguage components frequently appear and play a key role in the naturalness and accurate expression of emotions of language. Compared with written language, spoken language expression with paralanguage components is more in line with human real communication habits. Speech synthesis based on spoken language text will significantly improve the naturalness and realism of synthesized speech, improve the degree of personification of synthesized speech, better meet the emotional and interaction needs of users, and improve the interaction experience.

[0035] Therefore, one solution to achieve real and natural speech expression is to convert formal written language text into more real and natural spoken language text through a pre-trained model.

[0036] In one way, a written-to-spoken language conversion model can be trained in a supervised manner.

[0037] Figure 1 A flowchart of supervised training of a written-to-spoken language conversion model. As shown in FIG. 1, the written-to-spoken language conversion model is trained in a supervised manner. Figure 1As shown, the supervised training process of the written-to-spoken model includes: S101, defining paralinguistic labels, i.e., manually defining paralinguistic labels appearing in the spoken expression; S102, explicitly defining paralinguistic forms, i.e., manually defining paralinguistic forms corresponding to the paralinguistic labels (for example, the paralinguistic label is filler, and the paralinguistic form is "just", "this"); S103, collecting and manually annotating audio texts, i.e., manually recording real spoken communication audios, based on the paralinguistic labels, the paralinguistic forms corresponding to the paralinguistic labels, and the spoken communication audios, annotating the data from the spoken language to the written language, and obtaining the data set required for supervised training; S104, supervised training, i.e., based on the above obtained data set, using supervised learning, such as supervised fine-tuning (SFT), to train the model, so that the model learns the mapping relationship from the written language text to the spoken language text.

[0038] However, in the above-mentioned supervised training method, the transcription ability of the written-to-spoken model depends on the coverage of the language scene and context of the manually annotated data set, but the language scene and context covered by the manually annotated data set are extremely limited, and the artificial cost is also extremely high, resulting in that the written-to-spoken model inevitably has a certain proportion of inappropriate transcription, and even unacceptable transcription errors, such as unreasonable insertion position of paralinguistic components in the spoken language text, and inconsistency between the paralinguistic components and the context, which reduces the naturalness and authenticity of the spoken language text. In the case of a small scale of the written-to-spoken model, the semantic understanding ability of the written-to-spoken model is limited, and the perception and judgment of the context semantics are insufficient, and the above-mentioned problems are more obvious.

[0039] The following Table 1 gives examples of the two types of transcription errors: unreasonable insertion position of paralinguistic components and inconsistency between paralinguistic components and context. Table 1 To solve the above problems, the embodiment of the present application provides a model training method, a written language oralization method, equipment and products, and a written language oralization model is trained through a reinforcement learning (RL) post-training scheme. In the reinforcement learning post-training scheme, a supervised training model with written language oralization capability is first obtained through supervised training, and then model optimization is performed based on reinforcement learning and the supervised training model without a large amount of labeling to obtain a written language oralization model. On the one hand, by first performing supervised training, a model with written language oralization capability is provided for reinforcement learning training, improving the efficiency and effectiveness of reinforcement learning training. On the other hand, after supervised training, the model is guided by reinforcement learning to effectively perceive and judge the pros and cons of oralization text, generating more natural, real and reasonable oralization expressions, effectively improving the written language oralization capability of the model. Thus, the accuracy of the model in oralizing written language is effectively improved, and the quality of the oralization text obtained by transcription is improved.

[0040] Example implementation environment Please refer to Figure 2 , Figure 2 The present application relates to an implementation environment. The implementation environment includes a training database 210, a first device 220 and a second device 230. The first device 220 and the second device 230 can be servers or terminals, Figure 2 For example, the first device 220 is a server and the second device 230 is a terminal.

[0041] In the embodiment of the present application, reinforcement learning post-training can be performed on the first device 220 based on the training data provided by the training database 210 to obtain a written language oralization model. The written language oralization model can be deployed on the second device 230 to convert written language text into oral language text using the written language oralization model.

[0042] The first device 220 and the second device 230 can be the same device or different devices.

[0043] Example method Please refer to Figure 3 In an exemplary embodiment, a model training method is provided, which includes the following steps: S301, based on the first training data and the task prompt word indicating the conversion of written language to oral language, the large language model is supervised trained to obtain a supervised training model, and the first training data includes a supervised data pair formed by written language text and oralization text.

[0044] The first training data can be labeled (for example, manually labeled) training data. The first training data includes a plurality of supervised data pairs. Each supervised data pair includes a written language text and a colloquial text, and the colloquial text is the labeled data of the written language text.

[0045] For supervised training, the quality and richness of the first training data are crucial. In the process of constructing the first training data, the written language text can be colloquially labeled based on a plurality of predefined sublanguage labels and sublanguage forms corresponding to the sublanguage labels, to obtain a plurality of supervised data pairs, so as to improve the richness and quality of the first training data.

[0046] In the process of colloquial labeling of the written language text, the first training data can also be balanced in the following aspects: the number distribution of colloquial texts under different sublanguage labels, and the number distribution of colloquial texts under different sublanguage forms of the same sublanguage label, to ensure the sublanguage diversity of the colloquial texts in the first training data.

[0047] In an example, the plurality of sublanguage labels can include fillers, mood words, and discourse markers. The sublanguage forms corresponding to these sublanguage labels are shown in Table 2 as follows: Table 2 The task prompt word is pre-set and can be pre-set by a professional. The task prompt word provides the large language model with an explicit instruction to transcribe written language into colloquial language, guiding the large language model to accurately perform the task of transcribing written language text into colloquial language text.

[0048] Optionally, in addition to indicating the transcription of written language into colloquial language, the task prompt word can also include examples of transcribing written language into colloquial language, requirements for the transcribed colloquial text, and the like, to provide the large language model with more context information for reference when transcribing written language text into colloquial language text, and to improve the efficiency and effectiveness of supervised training of the large language model.

[0049] Optionally, the supervised training is supervised fine-tuning (SFT), and the supervised training model obtained through the supervised training is an SFT model.

[0050] The large language model can be a pre-trained large language model, and the pre-training enables the large language model to have certain semantic understanding and text generation capabilities.

[0051] In the embodiment, the large language model is supervised trained by taking the written language text in the supervised data pair as the input sample, and taking the spoken language text in the supervised data pair as the sample label of the input sample in the supervised training phase. Thus, the large language model is adapted to the task of transcribing written language text into spoken language text through supervised training.

[0052] In an example, the written language text in the supervised data pair and the task prompt word are input into the large language model; the written language text is transcribed under the task indication provided by the task prompt word by the large language model to obtain the predicted spoken language text of the written language text; the loss value of the supervised training is determined according to the difference between the predicted spoken language text and the spoken language text in the supervised data pair; and the model parameter of the large language model is adjusted according to the loss value of the supervised training. The process can be trained for multiple rounds, and finally the supervised training model is obtained.

[0053] It should be noted that the supervised training is a multi-round training process. To prevent overfitting, the number of training rounds of the supervised training can be constrained to reserve sufficient exploration space for the subsequent reinforcement learning training phase.

[0054] S302, based on the second training data and the supervised training model, multiple rounds of reinforcement learning training are performed to obtain a written language spoken language model, and the second training data includes an unlabeled written language data set and preference data used to generate a reward signal.

[0055] The unlabeled written language data set can include a plurality of written language texts, which are input samples in the reinforcement learning training.

[0056] The preference data reflects human spoken language expression preferences and is used to generate a reward signal in reinforcement learning to guide the model to perceive and judge the pros and cons of the spoken language text, and then generate high-quality spoken language text that better meets human spoken language expression preferences.

[0057] Optionally, the preference data includes spoken language positive samples and spoken language negative samples, both of which are spoken language texts, and the quality of the spoken language positive samples is better than that of the spoken language negative samples. The spoken language positive samples and the spoken language negative samples provide more explicit reward signals for reinforcement learning, guiding the model to more obviously and accurately perceive the pros and cons of the spoken language text.

[0058] In the embodiment, the second training data can be prepared in a preliminary data preparation stage. Then, in multiple rounds of reinforcement learning training, the second training data is taken as a data basis, the unannotated written data set in the second training data is taken as an input sample of the reinforcement learning training, the preference data is used to generate a reward signal in the reinforcement learning training, a supervised training model is taken as a model basis, and the reinforcement learning algorithm is used for multiple rounds of reinforcement learning training, so as to finally train the written-to-spoken model.

[0059] In the embodiment, in the process of training the written-to-spoken model from the large language model, the large language model is first supervised trained, and then the model is further optimized through reinforcement learning training, that is, the written-to-spoken model is trained through reinforcement learning after training. Compared with completely relying on supervised training, the way of training the written-to-spoken model through reinforcement learning after training reduces the dependence of model training on annotated data, solves the problem that the limited coverage of annotated data makes it difficult to improve the model capability, improves the written-to-spoken accuracy of the written-to-spoken model, and improves the quality of the spoken text transcribed by the written-to-spoken model.

[0060] In some embodiments, the spoken positive samples and the spoken negative samples in the preference data can be generated by the supervised training model, so as to utilize the written-to-spoken capability of the supervised training sample, improve the generation efficiency of the preference data, and reduce the collection cost of the preference data.

[0061] In the embodiment, the written text prepared in advance (which can be a written text in a supervised data pair or a written text from other sources) can be transcribed multiple times by the supervised training model to obtain multiple spoken texts corresponding to the written text; and the multiple spoken texts corresponding to the written text are taken as a data basis to determine multiple preference data.

[0062] The multiple transcriptions of the written text by the supervised training model can also be understood as multiple samplings of the written text by the supervised training model; and the multiple spoken texts corresponding to the written text can be understood as multiple sampling results corresponding to the written text.

[0063] Optionally, in the process of transcribing the written text multiple times by the supervised training model, the written text and a task prompt word indicating multiple spoken transcriptions can be input into the supervised training model, so as to accurately instruct the supervised training model through the task prompt word, and the supervised training model accurately completes the multiple spoken transcription tasks under the instruction of the task prompt word.

[0064] As an example, Figure 4 An example diagram of sampling by the supervised training model is shown. As shown in FIG. 1, the written text is input into the supervised training model, and the spoken text is output from the supervised training model. Figure 4As shown, based on the task prompt word indicating multiple oralization transcriptions and the written language text "typhoon is coming", multiple samplings (such as sampling a, sampling b, and sampling c in FIG. 6) are performed by the supervised training model, and finally the oralization texts corresponding to the multiple samplings are obtained, such as the oralization text corresponding to sampling a "[utterance symbol] ah, typhoon is coming", the oralization text corresponding to sampling b "typhoon is coming [mood word]?" and the oralization text corresponding to sampling c "[utterance symbol] ah, typhoon is coming [mood word]?". Figure 4

[0065] In some embodiments, the preference data further includes sample preference annotations, which indicate that the oralization positive samples are better than the oralization negative samples, or the sample preference annotations include scores of the oralization positive samples and scores of the oralization negative samples, and the scores of the oralization positive samples are better than the scores of the oralization negative samples. Thus, the sample preference annotations accurately represent the oralization positive samples and the oralization negative samples, and even represent the score gap between the oralization positive samples and the oralization negative samples, thereby providing accurate and reliable reward signals for the reinforcement learning process.

[0066] Optionally, a sampling pair comparison annotation method is used to compare and combine the multiple oralization texts corresponding to the written language text (such as the multiple oralization texts obtained by the supervised training model performing multiple oralization transcriptions on the written language text in the foregoing embodiments) to obtain multiple preference data. For example, in the multiple oralization texts, the oralization text corresponding to sampling a is compared with the oralization text corresponding to sampling b in quality, it is determined that the oralization text corresponding to sampling a is better than the oralization text corresponding to sampling b, a sample preference annotation indicating that the oralization text corresponding to sampling a is better than the oralization text corresponding to sampling b is generated, and the oralization text corresponding to sampling a, the oralization text corresponding to sampling b, and the sample preference annotation are combined to obtain one preference data. The pair comparison annotation method has the advantages of stability, rapidity, easy understanding by annotators, and better annotation consistency, which can reduce the subjective fluctuation in the annotation process and improve the accuracy and reliability of the preference data.

[0067] Alternatively, according to a set scoring standard, the multiple oralization texts corresponding to the written language text are scored one by one to obtain scores corresponding to the multiple oralization texts respectively, the multiple oralization texts are combined two by two to obtain multiple preference data, and the scores corresponding to the two oralization texts in the preference data respectively are the sample preference annotations in the preference data. This method can improve the accuracy of scoring the oralization texts by providing an objective and professional scoring standard, so that the preference data contains more information and provides more abundant reward signals for reinforcement learning.

[0068] ​The scoring criteria can include the following multiple scoring dimensions: natural fluency of text content description (such as whether it conforms to natural spoken rhythm), frequency / quantity of paralanguage insertion (such as whether it is too little, too much, or appropriate), semantic completeness (such as whether the original meaning of written language is lost, whether there are incorrect or erroneous sentences). Thus, through multiple-dimensional scoring criteria, the accuracy of scoring oral text is improved, that is, the accuracy of preference data is improved.

[0069] It can be understood that the more scoring dimensions, the more detailed the scoring criteria are divided, whether the scoring is performed manually or by a large language model, the scoring criteria can be better understood, and the oral text can be accurately scored according to the understanding of the scoring criteria.

[0070] Optionally, for each scoring dimension, the scoring criteria further include multiple scoring levels under the scoring dimension, such as a scoring level corresponding to too little paralanguage insertion frequency, another scoring level corresponding to too much paralanguage insertion frequency, and another scoring level corresponding to appropriate paralanguage insertion frequency. Each scoring level can include a level score and a score condition corresponding to the level score. Thus, by multiple scoring dimensions and multiple scoring levels under the multiple scoring dimensions (including level scores and score conditions), detailed and reliable scoring references are provided for oral text scoring, improving the accuracy of oral text scoring, that is, improving the accuracy of preference data.

[0071] Alternatively, for one or more paralanguage types, the scoring criteria include multiple scoring levels corresponding to the paralanguage type, each scoring level can include a level score and a score condition corresponding to the level score, and the score conditions in the multiple scoring levels are determined according to the multiple scoring dimensions. Thus, multiple scoring levels are provided for paralanguage types, and the score conditions in the scoring levels are designed in combination with the multiple scoring dimensions, which refines the granularity of the scoring criteria and improves the content completeness of the scoring criteria, providing detailed and reliable scoring references for oral text scoring, improving the accuracy of oral text scoring, that is, improving the accuracy of preference data.

[0072] As an example, the scoring criteria are shown in Table 3 as follows: Table 3 If the above annotation process is manual annotation, the way of "two people annotation and one person arbitration" or "multi-person voting" can be used for manual annotation to ensure the consistency of sample preference annotation. "Two people annotation and one person arbitration" and "multi-person voting" each have advantages and disadvantages. "Two people annotation and one person arbitration" can save annotation cost, but the requirement for the arbitrator is higher, and the arbitrator needs to fully understand the scoring standard to ensure the annotation quality. "Multi-person voting" reduces the influence of subjective cognitive bias of the annotator on the annotation quality by increasing the number of annotators, but the annotation cost is higher. The appropriate annotation method can be selected according to the actual situation, such as selecting the "multi-person voting" method in the case of non-bottleneck cost.

[0073] Please refer to Figure 5 In an example embodiment, a model training method is provided, which comprises the following steps: S501, based on the first training data and the task prompt word indicating the conversion of written language to spoken language, the large language model is supervised trained to obtain a supervised training model, and the first training data comprises a supervised data pair formed by written language text and spoken language text.

[0074] The implementation principle and technical effect of S501 can be referred to the foregoing embodiments, which will not be repeated.

[0075] S502, based on the preference data and the supervised training model, a reward model is trained.

[0076] The reward model (RM) is used to provide a reward signal for multi-round reinforcement learning training. In the training process of the reward model, the preference data shows the human preference for spoken language expression, so that the reward model has the ability to perceive and judge the pros and cons of the spoken language text, and can provide accurate reward signals for multi-round reinforcement learning.

[0077] In this embodiment, after the above supervised training, the supervised training model has model capabilities related to written language spoken language, such as semantic understanding capability. Based on the supervised training model, an initial model can be constructed, so that the initial model also has model capabilities related to written language spoken language, thereby eliminating the need to train the reward model from scratch, and improving the training efficiency of the reward model. After the initial model is constructed, the preference data is used as supervised data to supervise the training of the initial model. In this supervised training process, the initial model learns the human preference for spoken language expression, and gradually has the ability to perceive and judge the pros and cons of the spoken language text, thereby obtaining the reward model.

[0078] In a possible implementation, S502 includes: combining the model structure of the supervised training model except for the output layer and the initialized scalar output layer to construct an initial model; and training the initial model based on the preference data to obtain the reward model.

[0079] In the implementation, the model structure of the supervised training model except for the output layer includes the input layer of the supervised training model and the intermediate layer (also referred to as the hidden layer) of the supervised training model, the input layer of the supervised training model, the intermediate layer of the supervised training model, and the initialized scalar output layer are combined to construct the initial model, so that the initial model has the model capability of the supervised training model related to written-to-spoken conversion, and can also output a scalar representing the reward signal through the scalar output layer. After the initial model is constructed, the initial model is supervised trained based on the preference data, in the supervised training process, the initial model scores the spokenization positive samples and the spokenization negative samples in the preference data respectively, to obtain the reward value of the spokenization positive samples and the reward value of the spokenization negative samples, and the initial model is optimized by maximizing the difference between the reward value of the spokenization positive samples and the reward value of the spokenization negative samples. The reward value of the spokenization positive samples is increased with the increase of the number of training times, and the reward value of the spokenization negative samples is decreased with the increase of the number of training times, so that the reward model can more accurately distinguish the pros and cons of the spokenization text, and give an accurate reward value for the spokenization text, to provide an accurate reward signal for the reinforcement learning training.

[0080] Optionally, the training of the initial model is sentence-level training, that is, the reward value is determined based on the hidden layer feature of the last time step of the initial model in each training.

[0081] For example, in the training process of the initial model, the oralization sample (which can be an oralization positive sample or an oralization negative sample) is converted into a token sequence. The input data of the first time step of the initial model (which can be an initial model without training or an initial model after one or more training) is determined based on the token sequence. The input data is input into the initial model, and the hidden layer of the initial model is processed to obtain the hidden layer feature of the first time step. The hidden layer feature of the first time step is combined with the input data of the first time step to obtain the input data of the second time step. The input data of the second time step is input into the initial model, and the hidden layer of the initial model is processed to obtain the hidden layer feature of the second time step. The hidden layer feature of the second time step is combined with the input data of the second time step to obtain the input data of the third time step. The above process is iterated until the hidden layer feature of the last time step is obtained. The hidden layer feature of the last time step is scored by the output layer of the initial model to obtain the reward value of the oralization sample.

[0082] Optionally, the training of the initial model is token-level training, that is, the hidden layer feature of each time step is scored to obtain the reward value of each time step, and the reward value of the input sample is determined by combining the reward value of each time step.

[0083] For example, in the training process of the initial model, the oralization sample (which can be an oralization positive sample or an oralization negative sample) is converted into a token sequence. The input data of the first time step of the initial model (which can be an initial model without training or an initial model after one or more training) is determined based on the token sequence. The input data is input into the initial model, and the hidden layer of the initial model is processed to obtain the hidden layer feature of the first time step. The hidden layer feature of the first time step is combined with the input data of the first time step to obtain the input data of the second time step. The input data of the second time step is input into the initial model, and the hidden layer of the initial model is processed to obtain the hidden layer feature of the second time step. The hidden layer feature of the second time step is combined with the input data of the second time step to obtain the input data of the third time step. The above process is iterated until the hidden layer feature of the last time step is obtained. The hidden layer feature of the last time step is scored by the output layer of the initial model to obtain the reward value of the oralization sample.

[0084] The reward model obtained by the word-level training is superior to the reward model obtained by the sentence-level training, and thus the reward model obtained by the word-level training can be used in this embodiment.

[0085] The model architecture of the reward model is an encoder-decoder architecture, such as a Transformer.

[0086] S503, model initialization is performed according to the reward model to obtain an initial first value model and an initial second value model, and model initialization is performed according to the supervised training model to obtain an initial first policy model and an initial second policy model.

[0087] In this embodiment, the model initialization and the model parameter initialization can be performed with reference to the reward model to obtain the initial first value model and the initial second value model, and the initial first value model and the initial second value model can be the same model as the reward model. The model initialization and the model parameter initialization can be performed with reference to the supervised training model to obtain the initial first policy model and the initial second policy model, and the initial first policy model and the initial second policy model can be the same model as the supervised training model.

[0088] S504, based on the written language data set, the supervised training model, the reward model, and a preset loss function, the initial first policy model, the initial first value model, the initial second policy model, and the initial second value model are trained by multiple rounds of reinforcement learning, and the training of the initial first value model, the initial second policy model, and the initial second value model is used to assist the training of the initial first policy model. The written language oralization model is the first policy model after the multiple rounds of reinforcement learning.

[0089] In the embodiment, in the multi-round reinforcement learning training, the written language dataset is used to provide unannotated written language text as a training sample, the supervised training model and the first strategy model are respectively used to transcribe the training sample into spoken language, the output data of the supervised training model is used as the contrast data of the output data of the first strategy model; the first value model is used to evaluate the output data of the first strategy model at a single time step to generate a reward signal at the single time step, and the reward model is used to evaluate the spoken language text transcribed by the first strategy model to generate a whole sentence reward of the spoken language text; the second value model and the second strategy model can be updated multiple times (i.e. multiple training) in combination with the difference between the output data of the supervised training model and the output data of the first strategy model, the reward signal at the single time step and the whole sentence reward signal of the spoken language text; then the first value model is updated based on the second value model after multiple update training, and the first strategy model is updated based on the second strategy model after multiple update training, to complete a round of reinforcement learning training of the first strategy model and the second strategy model. According to the above process, multiple rounds of reinforcement learning training can be performed. Finally, the first strategy model after multiple rounds of reinforcement learning training is determined as the written language spoken language model.

[0090] As can be seen from the above process, in each round of training process, the output difference between the supervised training model and the first strategy model, the local reward signal (the reward signal at the single time step) and the global reward signal (the whole sentence reward after multiple time steps) are used to guide the multiple updates of the second value model and the second strategy model, and then the first value model and the first strategy model are updated based on the second value model after multiple updates and the second strategy model after multiple updates. With the increase of the training rounds, the quality of the spoken language text transcribed by the first strategy model is gradually improved, the first strategy model is optimized, and finally the written language spoken language model is obtained.

[0091] In a possible implementation manner, Figure 6 The flowchart of the i-th round of reinforcement learning training is shown in FIG. 6. As shown in FIG. 6, the i-th round of reinforcement learning training includes S601-S603: Figure 6 S601: The supervised training model is used to transcribe the training sample into spoken language, and the output data of the supervised training model is used as the contrast data of the output data of the first strategy model. S601, based on the written language dataset, the supervised training model, the first policy model after the i-1th round of reinforcement learning training, the first value model after the i-1th round of reinforcement learning training and the preset loss function, the second policy model after the i-1th round of reinforcement learning training and the second value model after the i-1th round of reinforcement learning training are updated respectively for multiple times, to obtain the second policy model after the ith round of reinforcement learning training and the second value model after the ith round of reinforcement learning training, i is greater than or equal to 1; S602, the second policy model after the ith round of reinforcement learning training is used to replace the first policy model after the i-1th round of reinforcement learning training, to obtain the first policy model after the ith round of reinforcement learning training; S603, the second value model after the ith round of reinforcement learning training is used to replace the first value model after the i-1th round of reinforcement learning training, to obtain the first value model after the ith round of reinforcement learning training.

[0092] Wherein, when i is equal to 1, the first policy model after the i-1th round of reinforcement learning training is the initial first policy model, the first value model after the i-1th round of reinforcement learning training is the initial first value model, the second policy model after the i-1th round of reinforcement learning training is the initial second policy model, and the second value model after the i-1th round of reinforcement learning training is the initial second value model.

[0093] Wherein, the preset loss function can include a difference loss function, a policy loss function and a value loss function, the difference loss function is used to determine the difference between the output data of the supervised training model and the output data of the first value model, the policy loss function is used to determine the model loss of the second policy model, and the value loss function is used to determine the model loss of the second value model.

[0094] In S601, the written language data in the written language data set can be transcribed into spoken language by the supervised training model, the first strategy model after the i-1th round of reinforcement learning training, respectively, to obtain the output data of the supervised training model and the output data of the first strategy model after the i-1th round of reinforcement learning training. The difference loss value of the output data of the supervised training model and the output data of the first strategy model after the i-1th round of reinforcement learning training can be determined based on the output data of the supervised training model, the output data of the first strategy model after the i-1th round of reinforcement learning training, and the difference loss function. The output data of the first strategy model after the i-1th round of reinforcement learning training can be evaluated by the first value model after the i-1th round of reinforcement learning training and the reward model to generate local reward signals and global reward signals. The second strategy model after the i-1th round of reinforcement learning training and the second value model after the i-1th round of reinforcement learning training can be updated multiple times in combination with the difference loss value of the output data of the supervised training model and the output data of the first strategy model after the i-1th round of reinforcement learning training, the local reward signals, the global reward signals, the strategy loss function, and the value loss function, to obtain the second strategy model after the i th round of reinforcement learning training and the second value model after the i th round of reinforcement learning training.

[0095] In the above process, through multiple loss functions, rich and unlabeled written language data sets, local reward values, and global reward values, the multiple updates of the second value model and the second strategy model are guided, and the updating effect of the second value model and the second strategy model is effectively improved without relying on labeled training data, and the written language spoken language ability of the second strategy model is improved.

[0096] In S602 and S603, the first strategy model after the i th round of reinforcement learning training is obtained by replacing the first strategy model after the i-1th round of reinforcement learning training with the second strategy model after the i th round of reinforcement learning training, and the first value model after the i th round of reinforcement learning training is obtained by replacing the first value model after the i-1th round of reinforcement learning training with the second value model after the i th round of reinforcement learning training. It can be seen that one round of reinforcement learning training of the first strategy model and the first value model is realized by multiple updates of the second strategy model and the second value model. Compared with one training, multiple training can significantly improve the model capability, so this training method can quickly and effectively improve the model capability of the first strategy model and the first value model. In this way, in the next round of training, the first strategy model will realize higher quality written language spoken language, and the first value model will output more accurate local reward signals to guide further optimization and update of the second strategy model and the second value model.

[0097] Next, optional implementation schemes for S601 are provided.

[0098] In one possible implementation, such as Figure 6 As shown, S601 includes S6011 to S6018: S6011, in the written language dataset, sample training samples that participate in the i-th round of reinforcement learning training from multiple written language texts.

[0099] The sampling method can be random sampling or sequential sampling.

[0100] S6012 uses a supervised training model to perform spoken language transcription on training samples, obtaining the first output data and the first predicted spoken language text at multiple time steps.

[0101] In this optional approach, training samples can be input into a supervised training model. The supervised training model then performs spoken language transcription on the training samples. This process of transcription is a continuous series of inputs and outputs, with each input and output considered a time step. Therefore, the output data of the supervised training model at multiple time steps and the predicted spoken language text obtained by transcribing the training samples (a combination of output data from multiple time steps) can be obtained. For clarity, the output data of the supervised training model at multiple time steps is referred to as the first output data at multiple time steps, and the predicted spoken language text obtained by transcribing the training samples is referred to as the first predicted spoken language sample. The number of time steps depends on the length of the training sample text.

[0102] The first output data is a probability distribution vector, which may include the probability that each word in the vocabulary is selected as the current output.

[0103] S6013 uses the first policy model trained by reinforcement learning in the (i-1)th round to transcribe the training samples into spoken language, resulting in input data at multiple time steps, second output data at multiple time steps, and second predicted spoken language text.

[0104] In the optional manner, the training sample can be input into the first policy model after the i-1th round of reinforcement learning training, and the input data of the first policy model after the i-1th round of reinforcement learning training at multiple time steps, the output data of the first policy model after the i-1th round of reinforcement learning training at multiple time steps, and the predicted colloquial text obtained by the first policy model after the i-1th round of reinforcement learning training for the colloquial transcription of the training sample can be obtained by the first policy model after the i-1th round of reinforcement learning training for the colloquial transcription of the training sample. In order to distinguish, the output data of the first policy model after the i-1th round of reinforcement learning training at multiple time steps is called the second output data at multiple time steps, and the predicted colloquial text obtained by the first policy model after the i-1th round of reinforcement learning training for the colloquial transcription of the training sample is called the second predicted colloquial sample.

[0105] The second output data is a probability distribution vector, and the second output data can include the probability of each vocabulary token being selected as the current output.

[0106] S6014, comparing the first output data at multiple time steps with the second output data at multiple time steps by the difference loss function in the preset loss function to determine the difference loss value at multiple time steps.

[0107] In the optional manner, for each time step, the first output data at the time step and the second output data at the time step can be input into the difference loss function, and the difference loss value at the time step can be calculated by the difference loss function, thereby obtaining the difference loss value at multiple time steps.

[0108] Optionally, the difference loss function adopts KL divergence loss (KL divergence loss, KL loss for short), and the probability distribution difference between the first output data at the time step and the second output data at the time step is measured by the KL divergence loss to obtain the difference loss value at the time step. The KL divergence loss is suitable for measuring the difference between two probability distributions, and the accuracy of calculating the difference loss value at the time step can be improved by the KL divergence loss.

[0109] S6015, evaluating the reward value of the second predicted colloquial text by the reward model to obtain the reward value of the second predicted colloquial text.

[0110] In the embodiment, the second predicted colloquial text can be converted into a vocabulary token sequence, and the vocabulary token sequence can be input into the reward model, and the reward value of the second predicted colloquial text can be obtained by evaluating the reward value of the vocabulary token sequence by the reward model. The reward value is a global reward signal.

[0111] S6016, performing value evaluation on the second output data at the plurality of time steps by the first value model trained through the i-1th round of reinforcement learning, to obtain single-step estimated values at the plurality of time steps.

[0112] In this embodiment, for each time step, the second output data at the time step can be input into the first value model, and the second output data at the time step is evaluated by the reward model to obtain a single-step estimated value at the time step, which is a local reward signal.

[0113] S6017, determining advantage values at the plurality of time steps and discounted return values at the plurality of time steps according to the difference loss values at the plurality of time steps, the reward value of the second predicted colloquial text, and the single-step estimated values at the plurality of time steps.

[0114] In this embodiment, the reward value of the second predicted colloquial text is obtained through the reward model, which is a whole-sentence reward value, i.e., a comprehensive actual reward value of the plurality of time steps, but each time step has its own actual reward value. The reward value of each time step can be determined in combination with the reward value of the second predicted colloquial text and the difference loss values at the plurality of time steps. The advantage values at the plurality of time steps are determined according to the reward values at the plurality of time steps and the single-step estimated values at the plurality of time steps, wherein the reward values at the plurality of time steps and the single-step estimated values at the plurality of time steps are both absolute advantages, and the advantage values at the plurality of time steps are relative advantages. The discounted returns at the plurality of time steps are determined by statistical analysis according to the advantage values at the plurality of time steps.

[0115] Optionally, the reward value of each time step is determined in combination with the reward value of the second predicted colloquial text and the difference loss values at the plurality of time steps, including: for the last time step in the plurality of time steps, the sum of the difference loss values at the plurality of time steps and the second predicted colloquial text is determined as the reward value at the last time step; and for each time step before the last time step in the plurality of time steps, the absolute value of the difference loss value at the time step is determined as the reward value at the time step. Thus, the reward value at each time step is accurately calculated.

[0116] Optionally, the advantage values at the plurality of time steps and the discounted return values at the plurality of time steps are calculated by generalized advantage estimation (GAE) to improve the accuracy of the advantage values at the plurality of time steps and the discounted return values.

[0117] Optionally, based on the reward value at each time step and the estimated value of each step at multiple time steps, the advantage value at multiple time steps is determined. This includes: for the k-th time step, calculating the advantage value at the k-th time step based on the reward value at the k-th time step, the estimated value of each step at the k-th time step, and the estimated value of each step at the (k+1)-th time step. Then, by referring to the estimated value of each step at the (k+1)-th time step, the advantage value representing the relative advantage at the k-th time step is accurately obtained.

[0118] Furthermore, the formula for calculating the advantage value at the k-th time step is: in, This represents the reward value at the k-th time step. Let represent the single-step estimated value at the k-th time step, and let represent the single-step estimated value at the (k+1)-th time step. Indicates preset parameters. This represents the advantage value at the k-th time step.

[0119] Optionally, statistical analysis can be performed based on the advantage values ​​at multiple time steps to determine the discounted return at multiple time steps, which can be calculated using the following formula: in, This represents the discounted return at the k-th time step. Indicates the first Advantage value at each time step These are preset parameters.

[0120] S6018, based on the input data at multiple time steps, the first output data at multiple time steps, the second output data at multiple time steps, the difference loss value at multiple time steps, the single-step estimated value at multiple time steps, the advantage value at multiple time steps, and the discount reward value at multiple time steps, determine multiple first sample data and multiple second sample data.

[0121] Here, the first sample data refers to the sample data used to train the second policy model, and the second sample data refers to the sample data used to train the second value model. One first sample data point corresponds to one time step, and one second sample data point corresponds to one time step, so as to achieve single-time-step training of the second policy model and the second value model. The first sample data and the second sample data may include one or more types of data at a single time step.

[0122] In the embodiment, the sample data at the plurality of time steps can be obtained based on the input data at the plurality of time steps (i.e., the input data of the first policy model after the i-1th reinforcement learning training at the plurality of time steps), the first output data at the plurality of time steps, the second output data at the plurality of time steps, the difference loss value at the plurality of time steps, the one-step estimated value at the plurality of time steps, the advantage value at the plurality of time steps, and the discounted return value at the plurality of time steps; and the plurality of first sample data and the plurality of second sample data can be obtained from the sample data at the plurality of time steps.

[0123] In a possible implementation, the input data at the plurality of time steps, the first output data at the plurality of time steps, the second output data at the plurality of time steps, the difference loss value at the plurality of time steps, the one-step estimated value at the plurality of time steps, the advantage value at the plurality of time steps, and the discounted return value at the plurality of time steps can be stored in a storage space, and the plurality of first sample data and the plurality of second sample data can be randomly extracted from the storage space. Thus, the storage space provides rich data for the selection of the first sample data and the second sample data, and the random extraction of the first sample data and the second sample data improves the diversity of the first sample data and the second sample data.

[0124] In the storage space, the input data, the first output data, the first output data, the difference loss value, the one-step estimated value, the advantage value, and the discounted return value at one time step can be used as one sample data, and the plurality of time steps can be used as a plurality of sample data. As can be seen, one training sample obtained based on the written language database sampling can obtain a plurality of sample data, and more sample data can be obtained by sampling a plurality of training samples and processing the plurality of training samples, thereby improving the richness of the sample data.

[0125] S6019, the second policy model after the i-1th reinforcement learning training is updated a plurality of times according to the plurality of first sample data and a policy loss function in the preset loss function, to obtain the second policy model after the i th reinforcement learning training.

[0126] In the storage space, the input data, the first output data, the first output data, the difference loss value, the one-step estimated value, the advantage value, and the discounted return value at one time step can be used as one sample data, and the plurality of time steps can be used as a plurality of sample data. As can be seen, one training sample obtained based on the written language database sampling can obtain a plurality of sample data, and more sample data can be obtained by sampling a plurality of training samples and processing the plurality of training samples, thereby improving the richness of the sample data.

[0127] In the embodiment, for each first sample data, input data of the second policy model can be obtained from the first sample data, the input data is input into the second policy model, output data of the second policy model is obtained, the output data of the second policy model is another part of input data of the policy loss function; part of input data of the policy loss function can be obtained from the first sample data, the part of input data and the output data of the second policy model can be input into the policy loss function, the policy loss value is calculated by the policy loss function, and the second policy model after the i-1th reinforcement learning training is updated once based on the policy loss value. In this way, based on multiple first sample data, the second policy model after the i-1th reinforcement learning training can be updated multiple times, and the second policy model after the multiple times of updating is the second policy model after the i th reinforcement learning training.

[0128] Optionally, in the multiple first sample data, the single first sample data includes the following data at a single time step: input data, second output data, difference loss value and discount return value, the t th update of the second policy model after the i-1th reinforcement learning training includes: processing the input data in the t th first sample data by the second policy model after the t-1th update to obtain third output data; determining the policy loss value by the policy loss function according to the third output data, the second output data in the t th first sample data, the difference loss value in the t th first sample data and the discount return value in the t th first sample data; updating the second policy model after the t-1th update according to the policy loss value to obtain the second policy model after the t th update; wherein t is greater than or equal to 1, and when t is equal to 1, the second policy model after the t-1th update is the second policy model after the i-1th reinforcement learning training. In this way, the input data, the second output data, the difference loss value and the discount return value at a single time step are used as the training sample of the single update of the second policy model, accurate and rich data is provided for the single update of the second policy model, and the model update accuracy is improved.

[0129] In the optional manner, the input data in the tth first sample data is input data of a single time step, the input data in the tth first sample data is input into the second policy model updated for the t-1th time, and the input data in the tth first sample data is input into the second policy model updated for the t-1th time to perform oralization transcription to obtain output data of the second policy model updated for the t-1th time at the time step, that is, third output data. The third output data, the second output data in the tth first sample data, the difference loss value in the tth first sample data, and the discount reward value in the tth first sample data can be input into a policy loss function, and the policy loss value is calculated by the policy loss function. Based on the policy loss value, the model parameter of the second policy model updated for the t-1th time can be updated by a model optimization algorithm (such as a gradient descent method) to obtain the second policy model updated for the tth time.

[0130] Further, in the plurality of first sample data, a single first sample data further includes a supervised training loss value at a single time step. The second policy model updated for the t-1th time can be supervised trained according to the third training data to obtain the supervised training loss value at the plurality of time steps, and the third training model includes a supervised data pair formed by the written language text and the oralization text. In the tth update of the second policy model after the i-1th round of reinforcement learning training, the policy loss value can be determined by the policy loss function according to the third output data, the second output data in the tth first sample data, the difference loss value in the tth first sample data, the discount reward value in the tth first sample data, and the supervised training loss value in the tth first sample data. Thus, the supervised loss value of the second policy model is also considered in the calculation of the policy loss value, and the accuracy of the policy loss value is improved.

[0131] In an example, taking the tth first sample data corresponding to the tth time step as an example, the policy loss function can be represented as: wherein, represents an action at a time step corresponding to the tth first sample data, represents a state at the time step corresponding to the tth first sample data (that is, the input data in the tth first sample data, that is, the input data at the tth time step), represents the second policy model, represents the first policy model, represents output data of the second policy model at the time step corresponding to the tth first sample data, represents output data of the first policy model at the tth time step (that is, the second output data in the tth first sample data); This represents the discounted return value at the t-th time step (i.e., the discounted return value in the t-th first sample data).

[0132] in, The overall representation in the policy loss function is the loss value obtained by comparing the output data of the second policy model with the output data of the first policy model.

[0133] in, This represents the difference loss value in the t-th first sample data.

[0134] in, Let represent the supervised training loss value in the t-th first sample data.

[0135] in, , This indicates the preset parameters.

[0136] S6020, based on multiple second sample data and the value loss function in the loss function, the second value model after the (i-1)th round of reinforcement learning training is updated multiple times to obtain the second value model after the i-th round of reinforcement learning training.

[0137] The second sample data includes the input data of the second value model, namely the second output data at the time step, as well as some input data of the value loss function. The input data of the value loss function depends on the design of the value loss function.

[0138] In this embodiment, for each second sample data, input data for the second value model can be obtained from the second sample data. This input data is then input into the second value model to obtain its output data, which is another part of the input data for the value loss function. A portion of the input data for the value loss function can also be obtained from the second sample data. This portion of the input data and the output data of the second value model are then input into the value loss function, and the value loss value is calculated. Based on this value loss value, the second value model after the (i-1)th round of reinforcement learning training is updated once. Thus, based on multiple second sample data, the second value model after the (i-1)th round of reinforcement learning training can be updated multiple times, and the second value model after these multiple updates becomes the second value model after the i-th round of reinforcement learning training.

[0139] Optionally, in the plurality of second sample data, the single second sample data comprises the following data at a single time step: the second output data, the single-step estimated value, and the discounted return value. According to the second sample data at the plurality of time steps and the value loss function in the loss function, the s-th update of the second value model after the i-1th round of reinforcement learning training comprises: performing value evaluation on the second output data in the s-th first sample data by the second value model updated for the s-1th time to obtain the single-step estimated value output by the second value model updated for the s-1th time; determining a value loss value by the value loss function according to the single-step estimated value output by the second value model updated for the s-1th time, the single-step estimated value in the s-th first sample data, and the discounted return value in the s-th first sample data; and updating the second value model updated for the s-1th time according to the value loss value to obtain the second value model updated for the s-th time; wherein s is greater than or equal to 1, and when s is equal to 1, the second value model updated for the s-1th time is the second value model after the i-1th round of reinforcement learning training. In this way, the second output data, the single-step estimated value, and the discounted return value at a single time step are used as the training sample for the single update of the second value model, accurate and rich data are provided for the single update of the second value model, and the accuracy of the update of the second value model is improved.

[0140] In the optional manner, for the second value model, the second output data in the s-th first sample data can be input into the second value model updated for the s-1th time, value evaluation can be performed by the second value model updated for the s-1th time to obtain the single-step estimated value output by the second value model updated for the s-1th time, and the single-step estimated value output by the second value model updated for the s-1th time, the single-step estimated value in the s-th first sample data, and the discounted return value in the s-th first sample data can be input into the value loss function to obtain the value loss value by calculation of the value loss function. Based on the value loss value, the model parameters of the second value model updated for the s-1th time can be updated by a model optimization algorithm (such as the gradient descent method) to obtain the second value model updated for the s-th time.

[0141] In an example, taking the s-th second sample data corresponding to the s-th time step as an example, the value loss function can be represented as: wherein, represents the single-step estimated value output by the second value model updated for the s-1th time, represents the single-step estimated value at the s-th time step (i.e., the single-step estimated value in the s-th first sample data), represents the discounted return value at the s-th time step (i.e., the discounted return value in the s-th first sample data).

[0142] In this implementation, written language to spoken language transcription, whole-sentence reward, single-step estimated value at a single time step, and difference loss value are determined using an unlabeled written language dataset, a supervised training model, a reward model, a first value model trained in the (i-1)th round of reinforcement learning, a first policy model trained in the (i-1)th round of reinforcement learning, and a difference loss function. By combining the whole-sentence reward, single-step estimated value, and difference loss value, sample data at multiple time steps is determined, improving the accuracy and richness of the sample data across multiple time steps, thus providing abundant training samples for multiple updates of the second policy model and the second value model. Furthermore, targeted loss functions are provided for the second policy model and the second value model respectively, improving the accuracy of their updates and thereby enhancing the reinforcement learning training effect. As reinforcement learning training progresses, the written language to spoken language transcription capabilities of the first policy model and the second policy model will be effectively improved.

[0143] In one example, Figure 7 Example diagram for reinforcement learning training. (e.g.) Figure 7 As shown, the models involved in the reinforcement learning process include: the SFT model (i.e., supervised training model), the old policy model (i.e., the first policy model), the old value model (i.e., the first value model), the reward model, the new policy model (i.e., the second policy model), and the new value model (i.e., the second value model). First, training samples obtained from the reinforcement learning training data (i.e., the written language dataset) are provided to the old policy model and the SFT model. Based on the output data of the old policy model and the SFT model, the KL loss (i.e., the difference loss value) is determined. The output data of the old policy model is then provided to the reward model and the old policy model to obtain the sentence-wide reward and the value at each step (i.e., the estimated value per step). The KL loss is then... The entire sentence reward and the value at each step are provided to the GAE to obtain the advantage value and the reward discount value at each step. The previously obtained data for each step (input / output, value, advantage value, and reward discount value) are stored in a cache area. Training samples for the new policy model (i.e., the first training samples) are sampled from the cache area, and supervised training is performed on the new policy model based on supervised training data (i.e., the third training data) to obtain supervised loss. The new policy model is updated multiple times by combining the training samples, supervised loss, and policy loss function. Training samples for the new value model (i.e., the second training samples) are sampled from the cache area, and the new value model is updated multiple times by combining these training samples and the value loss function. Finally, the new policy model and the updated value model are used to replace the old policy model.

[0144] The embodiment of the present application also provides a written language oralization method, comprising: obtaining a written language text to be processed; and converting the written language text into an oralization text through a written language oralization model; wherein the written language oralization model is obtained through the model training method provided in any of the preceding embodiments. Thus, the written language oralization model obtained through the reinforcement learning improves the accuracy of converting the written language text into the oralization text, and a more natural and real oralization text is obtained, thereby improving the accuracy of the oralization text. In particular, in the voice interaction scene, voice synthesis can be performed based on the oralization text, so that the voice interaction is more in line with the human oralization expression habit, the degree of personification of the voice intelligent device is improved, and the voice interaction experience is improved.

[0145] Example apparatus Correspondingly, the embodiment of the present application also provides a model training device.

[0146] Please refer to Figure 8 In an exemplary embodiment, a model training device 800 is provided, which comprises a first training unit 801 and a second training unit 802. Wherein: The first training unit 801 is configured to perform supervised training on a large language model based on first training data and a task prompt word indicating the conversion of written language into oral language, to obtain a supervised training model, and the first training data comprises a supervised data pair formed by a written language text and an oralization text; and the second training unit 802 is configured to perform multi-round reinforcement learning training based on second training data and the supervised training model, to obtain a written language oralization model, and the second training data comprises an unannotated written language data set and preference data used for generating a reward signal.

[0147] In a possible implementation, the second training unit 802 is specifically configured to: train a reward model based on the preference data and the supervised training model; perform model initialization according to the reward model, to obtain an initial first value model and an initial second value model, and perform model initialization according to the supervised training model, to generate an initial first policy model and an initial second policy model; and perform multi-round reinforcement learning training on the initial first policy model, the initial first value model, the initial second policy model and the initial second value model based on the written language data set, the supervised training model, the reward model and a preset loss function, and the training of the initial first value model, the initial second policy model and the initial second value model is used to assist the training of the initial first policy model; and the written language oralization model is the first policy model after the multi-round reinforcement learning training.

[0148] In a possible implementation, the second training unit 802 is specifically configured to: combine the model structure of the supervised training model except the output layer with the initialized scalar output layer to construct an initial model; and train the initial model based on the preference data to obtain the reward model.

[0149] In a possible implementation, the second training unit 802 is specifically configured to: based on the written language data set, the supervised training model, the first policy model after the i-1th round of reinforcement learning training, the first value model after the i-1th round of reinforcement learning training, and a preset loss function, update the second policy model after the i-1th round of reinforcement learning training and the second value model after the i-1th round of reinforcement learning training respectively for multiple times to obtain the second policy model after the ith round of reinforcement learning training and the second value model after the ith round of reinforcement learning training, i being greater than or equal to 1; replace the first policy model after the i-1th round of reinforcement learning training with the second policy model after the ith round of reinforcement learning training to obtain the first policy model after the ith round of reinforcement learning training; and replace the first value model after the i-1th round of reinforcement learning training with the second value model after the ith round of reinforcement learning training to obtain the first value model after the ith round of reinforcement learning training.

[0150] In a possible implementation, the second training unit 802 is specifically configured to: sample, from the multiple written language texts contained in the written language dataset, a training sample participating in the i-th round of reinforcement learning training; perform written-to-spoken transcription on the training sample by the supervised training model to obtain the first output data at the multiple time steps and the first predicted spoken language text; perform written-to-spoken transcription on the training sample by the first policy model after the i-1-th round of reinforcement learning training to obtain the input data at the multiple time steps, the second output data at the multiple time steps, and the second predicted spoken language text; compare the first output data at the multiple time steps with the second output data at the multiple time steps by a difference loss function in the preset loss function, to determine the difference loss value at the multiple time steps; perform reward value evaluation on the second predicted spoken language text by the reward model to obtain the reward value of the second predicted spoken language text; perform value evaluation on the second output data at the multiple time steps by the first value model after the i-1-th round of reinforcement learning training to obtain the one-step estimated value at the multiple time steps; determine the advantage value at the multiple time steps and the discounted return value at the multiple time steps according to the difference loss value at the multiple time steps, the reward value, and the one-step estimated value at the multiple time steps; determine the multiple first sample data and the multiple second sample data according to the input data at the multiple time steps, the first output data at the multiple time steps, the second output data at the multiple time steps, the difference loss value at the multiple time steps, the one-step estimated value at the multiple time steps, the advantage value at the multiple time steps, and the discounted return value at the multiple time steps; perform multiple updates on the second policy model after the i-1-th round of reinforcement learning training according to the multiple first sample data and a policy loss function in the preset loss function, to obtain the second policy model after the i-th round of reinforcement learning training; and perform multiple updates on the second value model after the i-1-th round of reinforcement learning training according to the multiple second sample data and a value loss function in the loss function, to obtain the second value model after the i-th round of reinforcement learning training.

[0151] In a possible implementation, in the plurality of first sample data, each first sample data includes the following data at a single time step: input data, second output data, difference loss value and discounted return value, and the second training unit 802 is specifically configured to: process the input data in the tth first sample data by the second policy model updated for the t-1th time to obtain third output data; determine a policy loss value by a policy loss function according to the third output data, the second output data in the tth first sample data, the difference loss value in the tth first sample data and the discounted return value in the tth first sample data; update the second policy model updated for the t-1th time according to the policy loss value to obtain the second policy model updated for the tth time; wherein t is greater than or equal to 1, and when t is equal to 1, the second policy model updated for the t-1th time is the second policy model after the i-1th round of reinforcement learning training.

[0152] In a possible implementation, in the plurality of second sample data, each second sample data includes the following data at a single time step: second output data, one-step estimated value and discounted return value, and the second training unit 802 is specifically configured to: perform value evaluation on the second output data in the sth first sample data by the second value model updated for the s-1th time to obtain the one-step estimated value output by the second value model updated for the s-1th time; determine a value loss value by a value loss function according to the one-step estimated value output by the second value model updated for the s-1th time, the one-step estimated value in the sth first sample data and the discounted return value in the sth first sample data; update the second value model updated for the s-1th time according to the value loss value to obtain the second value model updated for the sth time; wherein s is greater than or equal to 1, and when s is equal to 1, the second value model updated for the s-1th time is the second value model after the s-1th round of reinforcement learning training.

[0153] The model training apparatus 800 provided in this embodiment belongs to the same application concept as the model training method provided in the above-mentioned embodiments of the present application, can execute the model training method provided in any of the above-mentioned embodiments of the present application, and has the corresponding function modules and beneficial effects of executing the model training method. Technical details not described in detail in this embodiment can be referred to the corresponding method embodiments of the present application, which will not be described here again.

[0154] Correspondingly, the present application also provides a written language oralization apparatus.

[0155] Please refer to Figure 9 In an example embodiment, a written language oralization apparatus 900 is provided, which includes an acquisition unit 901 and a transcription unit 902. Wherein: The acquisition unit 901 is configured to acquire a written language text to be processed. The transcription unit 902 is configured to transcribe the written language text into a spoken language text by using a written language-spoken language conversion model.

[0156] The written language-spoken language conversion apparatus 900 provided in the embodiment belongs to the same application concept as the written language-spoken language conversion method provided in the embodiments of the present application, can execute the written language-spoken language conversion method provided in any of the embodiments of the present application, and has the corresponding function modules and advantages of executing the written language-spoken language conversion method. The technical details not described in detail in the embodiment can be referred to the corresponding method embodiments of the present application, which will not be described here.

[0157] The functions implemented by each unit in the above apparatus can be implemented by the same or different processors, and the embodiments of the present application are not limited.

[0158] It should be understood that the units in the above apparatus can be implemented in the form of processor calling software. For example, the apparatus includes a processor, the processor is connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of the units of the apparatus. The processor can be a general processor, such as a CPU or a microprocessor, and the memory can be an internal memory of the apparatus or an external memory of the apparatus. Alternatively, the units in the apparatus can be implemented in the form of hardware circuit. The functions of some or all units can be implemented by designing the hardware circuit. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all units are implemented by designing the logical relationship of elements in the circuit. For another example, in another implementation, the hardware circuit can be implemented by a PLD. Taking an FPGA as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to implement the functions of some or all units. All units of the above apparatus can be implemented in the form of processor calling software, or all units can be implemented in the form of hardware circuit, or part of the units can be implemented in the form of processor calling software, and the remaining part can be implemented in the form of hardware circuit.

[0159] In the embodiments of the present application, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a CPU, a microprocessor, a GPU, or a DSP, etc. In another implementation, the processor can implement certain functions through a logic relationship of a hardware circuit, which is fixed or can be reconfigured. For example, the processor is a hardware circuit implemented by an ASIC or a PLD, such as an FPGA, etc. In the reconfigurable hardware circuit, the processor loads a configuration document to implement the hardware circuit configuration. It can be understood that the processor loads instructions to implement the functions of the above units.

[0160] It can be seen that each unit in the above apparatus can be one or more processors (or processing circuits) configured to implement the above methods, such as a CPU, a GPU, a NPU, a TPU, a DPU, a microprocessor, a DSP, an ASIC, an FPGA, or a combination of at least two of these processor forms.

[0161] In addition, each unit in the above apparatus can be integrated together or can be independently implemented. In one implementation, the units are integrated together to implement a SOC. The SOC can include at least one processor for implementing any of the above methods or the functions of the units of the apparatus. The at least one processor can be different, such as including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0162] Example electronic device Another embodiment of the present application also provides an electronic device. Referring to FIG. 10, Figure 10 As shown in the figure, the electronic device can include a memory 1000 and a processor 1010; wherein the memory 1000 is connected with the processor 1010, and is configured to store a program; the processor 1010 is configured to implement the model training method disclosed in any of the above embodiments by running the program stored in the memory 1000.

[0163] Specifically, the above electronic device can further include a bus, a communication interface 1020, an input device 1030, and an output device 1040.

[0164] The processor 1010, the memory 1000, the communication interface 1020, the input device 1030, and the output device 1040 are connected with each other through the bus. Among them: The bus can include a channel for transmitting information between various components of a computer system.

[0165] The processor 1010 can be a general processor, such as a general central processing unit (CPU), a microprocessor, or the like, or can be an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs of the solutions of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready-to-use programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0166] The processor 1010 can include a main processor, and can also include a baseband chip, a modem, etc.

[0167] The memory 1000 stores programs for executing the solutions of the present application, and can also store an operating system and other key services. Specifically, the programs can include program codes, and the program codes include computer operation instructions. More specifically, the memory 1000 can include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk memory, a flash, etc.

[0168] The input device 1030 can include a device that receives data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer or a gravity sensor, etc.

[0169] The output device 1040 can include a device that allows information to be output to a user, such as a display screen, a printer, a speaker, etc.

[0170] The communication interface 1020 can include a device using any transceiver, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc., to communicate with other devices or communication networks.

[0171] The processor 1010 executes the programs stored in the memory 1000 and calls other devices, which can be used to implement each step of any one of the model training methods provided by the above-mentioned embodiments of the present application.

[0172] The embodiments of the present application also propose a chip, which includes a processor and a data interface. The processor reads and runs programs stored on the memory through the data interface to execute any one of the model training methods provided by the above-mentioned embodiments. The specific processing process and its beneficial effects can be referred to the above-mentioned embodiments of the model training method.

[0173] Example computer program product and storage medium In addition to the method and device described above, the embodiments of the present application can also be a computer program product, which includes computer program instructions, which, when executed by a processor, causes the processor to perform the steps in the model training method according to various embodiments of the present application described in any of the embodiments of the present specification.

[0174] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of the present application, including an object-oriented programming language, such as Java, C++, and the like, and a conventional procedural programming language, such as the "C" language or the like. The program code can be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0175] In addition, the embodiments of the present application can also be a storage medium having a computer program stored thereon, which is executed by a processor to perform the steps in the model training method according to various embodiments of the present application described in any of the embodiments of the present specification.

[0176] For each of the foregoing method embodiments, in order to simply describe, it is expressed as a combination of a series of actions, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0177] It should be noted that each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts between each embodiment can be referred to each other. For device embodiments, since they are basically similar to method embodiments, they are described more simply, and the relevant parts refer to the part of the method embodiment.

[0178] The steps in the method of each embodiment of the present application can be adjusted, combined and deleted in sequence according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0179] The modules and sub-modules in the device and terminal in each embodiment of the present application can be combined, divided and deleted according to actual needs.

[0180] It should be understood that the disclosed terminal, device and method can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or sub-modules is merely a logical function division. In actual implementation, another division manner can be used. For example, a plurality of sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0181] The modules or sub-modules described as separate components can or can not be physically separate, and the components of the modules or sub-modules can or can not be physical modules or sub-modules, i.e. can be located in one place or distributed on a plurality of network modules or sub-modules. Some or all of the modules or sub-modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0182] In addition, the functional modules or sub-modules in each embodiment of the present application can be integrated into a processing module, or each module or sub-module can exist physically, or two or more modules or sub-modules can be integrated into one module. The integrated module or sub-module can be realized in the form of hardware or software functional module or sub-module.

[0183] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0184] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be directly implemented by hardware, software units executed by a processor, or a combination of both. The software units can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0185] Finally, it should be noted that, in this document, the term "only" is used simply to set off from one entity or action to another in order to avoid the use of the term "and / or" or the like for the sake of clarity. In no way should the term "only" be interpreted as implying that there is an implied exclusion of any referenced entity or action. Moreover, the terms "comprising", "including", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0186] The above description of disclosed embodiments provides enabling teaching for making or using the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model training method, characterized in that, include: Based on the first training data and the task prompts indicating that written language should be transcribed into spoken language, the large language model is trained in a supervised manner to obtain a supervised training model. The first training data includes supervised data pairs formed by written language text and spoken language text. Based on the second training data and the supervised training model, multiple rounds of reinforcement learning training are performed to obtain a written language conversational model. The second training data includes an unlabeled written language dataset and preference data for generating reward signals.

2. The model training method according to claim 1, characterized in that, The process of performing multiple rounds of reinforcement learning training based on the second training data and the supervised training model to obtain a written language-to-spoken language model includes: Based on the preference data and the supervised training model, a reward model is trained. The model is initialized according to the reward model to obtain an initial first value model and an initial second value model. The model is then initialized according to the supervised training model to generate an initial first policy model and an initial second policy model. Based on the written language dataset, the supervised training model, the reward model, and the preset loss function, multiple rounds of reinforcement learning training are performed on the initial first policy model, the initial first value model, the initial second policy model, and the initial second value model. The training of the initial first value model, the initial second policy model, and the initial second value model is used to assist the training of the initial first policy model. The written language-to-spoken language model is the first strategy model trained through multiple rounds of reinforcement learning.

3. The model training method according to claim 2, characterized in that, The process of training a reward model based on the preference data and the supervised training model includes: The model structure other than the output layer in the supervised training model is combined with the initialized scalar output layer to construct the initial model; The initial model is trained based on the preference data to obtain the reward model.

4. The model training method according to claim 2, characterized in that, The i-th round of reinforcement learning training includes: Based on the written language dataset, the supervised training model, the first policy model after the (i-1)th round of reinforcement learning training, the first value model after the (i-1)th round of reinforcement learning training, and the preset loss function, the second policy model and the second value model after the (i-1)th round of reinforcement learning training are updated multiple times to obtain the second policy model and the second value model after the (i-1)th round of reinforcement learning training, where i is greater than or equal to 1. The first policy model after the (i-1)th round of reinforcement learning training is replaced by the second policy model after the i-th round of reinforcement learning training to obtain the first policy model after the i-th round of reinforcement learning training. The first value model after the (i-1)th round of reinforcement learning training is replaced by the second value model after the i-th round of reinforcement learning training to obtain the first value model after the i-th round of reinforcement learning training.

5. The model training method according to claim 4, characterized in that, The process involves updating the second policy model and the second value model after the (i-1)th round of reinforcement learning training, based on the written language dataset, the supervised training model, the first policy model after the (i-1)th round of reinforcement learning training, the first value model after the (i-1)th round of reinforcement learning training, and the preset loss function, multiple times to obtain the second policy model and the second value model after the (i-1)th round of reinforcement learning training, including: In the written language dataset, training samples that participate in the i-th round of reinforcement learning are sampled from multiple written language texts. The supervised training model is used to transcribe the training samples into spoken language to obtain the first output data and the first predicted spoken language text at multiple time steps. The first policy model trained by the (i-1)th round of reinforcement learning is used to transcribe the training samples into spoken language, thereby obtaining the input data at the multiple time steps, the second output data at the multiple time steps, and the second predicted spoken language text. The difference loss function in the preset loss function is used to compare the first output data and the second output data at the multiple time steps to determine the difference loss value at the multiple time steps. The reward value of the second predicted spoken text is evaluated using the reward model to obtain the reward value of the second predicted spoken text. The value of the second output data at the multiple time steps is evaluated using the first value model trained in the (i-1)th round of reinforcement learning, and the single-step estimated value at the multiple time steps is obtained. Based on the difference loss value at the multiple time steps, the reward value, and the single-step estimated value at the multiple time steps, determine the advantage value and the discounted return value at the multiple time steps; Based on the input data at the multiple time steps, the first output data at the multiple time steps, the second output data at the multiple time steps, the difference loss value at the multiple time steps, the single-step estimated value at the multiple time steps, the advantage value at the multiple time steps, and the discounted return value at the multiple time steps, multiple first sample data and multiple second sample data are determined. Based on the multiple first sample data and the policy loss function in the preset loss function, the second policy model after the (i-1)th round of reinforcement learning training is updated multiple times to obtain the second policy model after the i-th round of reinforcement learning training. Based on the multiple second sample data and the value loss function in the loss function, the second value model after the (i-1)th round of reinforcement learning training is updated multiple times to obtain the second value model after the i-th round of reinforcement learning training.

6. The model training method according to claim 5, characterized in that, In the plurality of first sample data, a single first sample data includes the following data at a single time step: input data, second output data, difference loss value, and discount reward value. The t-th update of the second policy model after the (i-1)th round of reinforcement learning training includes: The second strategy model, updated at the (t-1)th time, is used to process the input data in the tth first sample data to obtain the third output data. Based on the third output data, the second output data in the t-th first sample data, the difference loss value in the t-th first sample data, and the discount reward value in the t-th first sample data, the strategy loss value is determined through the strategy loss function; Based on the policy loss value, the second policy model after the (t-1)th update is updated to obtain the second policy model after the tth update; Where t is greater than or equal to 1, and when t equals 1, the second policy model after the (t-1)th update is the second policy model after the (i-1)th round of reinforcement learning training.

7. The model training method according to claim 5, characterized in that, In the plurality of second sample data, a single second sample data includes the following data at a single time step: second output data, single-step estimated value, and discounted reward value. The s-th update of the second value model trained in the (i-1)th round of reinforcement learning, based on the second sample data at the plurality of time steps and the value loss function in the loss function, includes: The value of the second output data in the s-th first sample data is evaluated using the second value model after the (s-1)th update, and the single-step estimated value output by the second value model after the (s-1)th update is obtained. The value loss value is determined by the value loss function based on the single-step estimated value output by the second value model after the (s-1)th update, the single-step estimated value in the sth first sample data, and the discounted return value in the sth first sample data. Based on the value loss value, the second value model after the (s-1)th update is updated to obtain the second value model after the s-th update; Where s is greater than or equal to 1, when s equals 1, the second value model after the (s-1)th update is the second value model after the (s-1)th round of reinforcement learning training.

8. A method for making written language more colloquial, characterized in that, include: Obtain the written text to be processed; The written language text is transcribed into spoken language text using a written language-to-spoken language model. The written language colloquialization model is trained using the model training method according to any one of claims 1 to 7.

9. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the model training method as described in any one of claims 1 to 7, or the written language conversationalization method as described in claim 8, by running the program in the memory.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the model training method as described in any one of claims 1 to 7, or the written language conversationalization method as described in claim 8.

Images