Text regularizing method, large-scale language model training method and device and electronic equipment

Abstract

The invention provides a text regularizing method, a large-scale language model training method and device and electronic equipment, and the method comprises the steps: obtaining a fine-tuned large-scale language model and a first text, and enabling the first text to be obtained by adding a first set character at the tail of a to-be-regularized first original text, the first set character is used for indicating the large-scale language model to output a text regularizing result, inputting the first text into the large-scale language model to obtain the text regularizing result, and according to the position information of each to-be-regularized character in the text regularizing result, each to-be-regularized character adopting a second set character interval and the corresponding regularized character, performing text regularizing on the to-be-regularized character, according to the method, the first original text is adjusted to obtain the regularized text, the to-be-regularized characters do not need to be positioned and coded to mark the to-be-regularized characters in the first original text, and the fine-tuning large-scale language model is adopted to predict the positions and the regularized characters of the to-be-regularized characters in the first text, so that the complexity of text regularization is reduced, and the text regularization efficiency is improved. And the text regularizing effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a text normalization method, a large-scale language model training method, device and electronic equipment. BACKGROUND

[0002] In the text-to-speech conversion scenario, the text that needs to be converted needs to conform to the natural speech expression form, therefore, the original text needs to be normalized, that is, the non-standard text in the original text, such as abbreviations, numbers, special symbols, etc. is converted into a natural speech expression form, so as to realize more natural and smooth speech synthesis or conversion.

[0003] However, in the related art, the insertion position information in the first original text to be normalized is needed, and the model structure is complex, which increases the complexity of text normalization and reduces the effect of text normalization. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0005] To this end, the present application proposes a text normalization method, a large-scale language model training method, device and electronic equipment to reduce the complexity of text normalization and improve the effect of text normalization.

[0006] An embodiment of the present application provides a text normalization method, comprising:

[0007] obtaining a large-scale language model fine-tuned and a first text to be normalized; wherein the first text is obtained by adding a first set character at the end of a first original text to be normalized; the first set character is used to indicate the large-scale language model to output a text normalization result;

[0008] inputting the first text into the large-scale language model to obtain a text normalization result predicted by the large-scale language model under the indication of the first set character, wherein the text normalization result is obtained by predicting the position information and the normalized character of at least one character to be normalized in the first text;

[0009] adjusting each character to be normalized in the first original text according to the position information of each character to be normalized in the text normalization result, and each character original and the corresponding normalized character separated by a second set character to obtain a normalized text.

[0010] An embodiment of the present application provides a large-scale language model training method, comprising:

[0011] obtain a sample text and a label; the sample text is obtained by adding a first specified character at the end of a second original text to be regularized; the first specified character is used to indicate a text regularization result output by a large-scale language model; the label includes true value position information of each to-be-regularized character in the sample text, and each character original and a corresponding true value regularized character separated by a second specified character;

[0012] input the sample text into the large-scale language model to obtain a second text predicted by the large-scale language model, and a text regularization result obtained by predicting position information and regularized characters of at least one to-be-regularized character in the sample text under the indication of the first specified character;

[0013] determine a target loss function according to a difference between the sample text and the second text, and a difference between the text regularization result and the label;

[0014] fine-tune parameters of the large-scale language model according to the target loss function to obtain a trained large-scale language model.

[0015] Another aspect of the present application provides a text regularization device, comprising:

[0016] an obtaining module configured to obtain a fine-tuned large-scale language model and a first text to be regularized; the first text is obtained by adding a first specified character at the end of a first original text to be regularized; the first specified character is used to indicate a text regularization result output by the large-scale language model;

[0017] a prediction module configured to input the first text into the large-scale language model to obtain a text regularization result predicted by the large-scale language model for at least one to-be-regularized character in the first text under the indication of the first specified character;

[0018] an adjustment module configured to adjust each to-be-regularized character in the first original text to obtain a regularized text according to position information of each to-be-regularized character in the text regularization result, and each character original and a corresponding regularized character separated by a second specified character.

[0019] An aspect of the present application provides a training device of a large-scale language model, comprising:

[0020] An acquisition module is configured to acquire a sample text and a label, wherein the sample text is obtained by adding a first preset character at the end of a second original text to be regularized, the first preset character is used to indicate a text regularization result output by a large-scale language model, and the label includes true value position information of each character to be regularized in the sample text, and each character original and corresponding true value regularized character separated by a second preset character.

[0021] A prediction module is configured to input the sample text into the large-scale language model to obtain a second text predicted by the large-scale language model, and obtain a text regularization result in which at least one character to be regularized in the sample text is predicted in terms of position information and regularized character under the indication of the first preset character.

[0022] A determination module is configured to determine a target loss function according to a difference between the sample text and the second text, and a difference between the text regularization result and the label.

[0023] A training module is configured to fine-tune parameters of the large-scale language model according to the target loss function to obtain a trained large-scale language model.

[0024] Another aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the program, the method according to the foregoing aspect or the method according to the foregoing another aspect is implemented.

[0025] Another aspect of the present application provides a non-transitory computer readable storage medium, having a computer program stored thereon, when the computer program is executed by a processor, the method according to the foregoing aspect or the method according to the foregoing another aspect is implemented.

[0026] Another aspect of the present application provides a computer program product, having a computer program stored thereon, when the computer program is executed by a processor, the method according to the foregoing aspect or the method according to the foregoing another aspect is implemented.

[0027] The text normalization method, the large-scale language model training method, the device and the electronic equipment provided in the application obtain a large-scale language model after fine-tuning and a first text to be normalized, wherein the first text is obtained by adding a first set character at the end of a first original text to be normalized, the first set character is used to indicate that the large-scale language model outputs a text normalization result, the first text is input into the large-scale language model, and a text normalization result obtained by the large-scale language model in predicting position information and normalized characters of at least one to-be-normalized character in the first text under the indication of the first set character is obtained, each to-be-normalized character in the first original text is adjusted to obtain a normalized text according to the position information of each to-be-normalized character in the text normalization result and each character original text separated by a second set character and the corresponding normalized character, the first set character is added at the end of the first original text to obtain the first text, and the large-scale language model after fine-tuning is used to predict the position and normalized characters of each to-be-normalized character in the first text, so that the position positioning of the to-be-normalized characters in the first original text is not required, and the to-be-normalized characters are not required to be marked by coding, thereby reducing the complexity of the model structure and the complexity of text normalization and improving the effect of text normalization.

[0028] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0029] The above and / or additional aspects and advantages of the application will become apparent and be readily understood by considering the following detailed description, including the accompanying drawings, in which:

[0030] Figure 1 A flowchart of a text normalization method provided by an embodiment of the application;

[0031] Figure 2 A flowchart of another text normalization method provided by an embodiment of the application;

[0032] Figure 3 A text normalization error correction diagram provided by an embodiment of the application;

[0033] Figure 4 A diagram of to-be-normalized character replacement provided by an embodiment of the application;

[0034] Figure 5 A flowchart of a large-scale language model training method provided by an embodiment of the application;

[0035] Figure 6 A structure diagram of a large-scale language model of an embodiment of the application;

[0036] Figure 7 A schematic diagram of the structure of a text regularization device provided in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of the structure of a large-scale language model training device provided in an embodiment of the present application;

[0038] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0040] The following describes the text regularization method, large-scale language model training method, device and electronic device of the embodiments of the present application with reference to the accompanying drawings.

[0041] Figure 1 A flowchart of a text regularization method provided in an embodiment of the present application.

[0042] The executor of the text regularization method in the embodiment of the present application is a text regularization device, which can be set in an electronic device, such as a smart phone, a mobile phone, a smart wearable device, etc., which is not limited in this embodiment.

[0043] like Figure 1 As shown, the method may include the following steps:

[0044] Step 101: Obtain a fine-tuned large-scale language model and a first text to be regularized.

[0045] Among them, the large-scale language model is a pre-trained model for natural language generation, such as (Generative Pre-trained Transformer 2, GPT2) or (Large Language Model Family 2, LLAMA2), etc., among which GPT2 is a sequence-to-sequence deep learning model based on the Transformer decoder. In this application, based on the pre-trained large-scale language model, the prepared training samples are used to fine-tune the model to meet the needs of text regularization. Among them, the fine-tuning method will be explained in the subsequent embodiments.

[0046] The first text to be regularized is obtained by adding a first specified character at the end of the first original text to be regularized, and the first original text may contain characters to be regularized. The characters to be regularized can also be referred to as non-standard characters, such as characters in abbreviations, numerical values, special symbols, etc. For example, the Arabic numerals "12" appear in the Chinese text to be regularized. In Chinese, "12" is a character to be regularized, and the Arabic numerals "12" need to be regularized to Chinese "twelve" by a regularizing algorithm, so as to be accurately recognized in the text-to-speech scenario. The text-to-speech scenario includes scenarios such as intelligent voice assistants, customer service robots, audio books and article reading, navigation systems, etc. The intelligent voice assistant is, for example, a smart speaker, a car-mounted assistant, etc.

[0047] The first specified character added at the end of the first text is used to indicate the regularizing result of the text output by the large-scale language model. The first specified character is a self-defined character, for example, <tnstart>The text after the first set character is output as the regularized text, that is, the large-scale language model starts to perform text regularization operation and outputs the regularized text after learning the first set character in the process of fine-tuning. For example, the first text to be regularized is Chinese, the first text is obtained by word segmentation to include a sequence of each word segmentation result, each word segmentation in the sequence is separated by a space, and each word segmentation in the sequence is a character, which can be a word or a word or a number or a symbol, for example: one degree to 24:11 leads by 13 points <tnstart>.

[0048] It needs to be understood that in the embodiments of the present application, no additional position positioning tags or the like need to be inserted in the input first text, so that the semantic understanding of the input text will not be affected by the position positioning tags, and the complexity of the input text is reduced.

[0049] In step 102, the first text is input into the large-scale language model to obtain a text regularization result predicted by the large-scale language model under the indication of the first set of characters, the text regularization result being position information and a regularized character of at least one to-be-regularized character in the first text.

[0050] In the first text, there is at least one to-be-regularized character.

[0051] In an implementation manner of the embodiments of the present application, the large-scale language model has learned the corresponding relationship between each to-be-regularized character in the input first text to be regularized and the text regularization result corresponding to each to-be-regularized character through fine-tuning. In the text regularization result, the position information of each to-be-regularized character is included, and each to-be-regularized character and the corresponding regularized character are separated by the second set of characters. As an example, the first text is: one degree leads 24:11 by 13 points <tnstart>Wherein, the characters to be regularized are: 24:11 and 13, and the output text regularization result after prediction by a large-scale language model is: <pos3> 24:11 <norm>Twenty-four to eleven <pos6> <pos7> 13 <norm>thirteen <pos8> <eos>wherein, taking the character 24:11 to be justified as an example, the text justification result in <pos3>and <pos6>for the character to be justified 24:11 is position information, i.e. position information of the character to be justified in the first text, wherein <pos3>indicates a start position, <pos6>indicates the end position, <norm>For the second setting character, the original text of the character to be justified, i.e., 24:11, before the second setting character, and the justified character after the second setting character: twenty-four to eleven, <eos>To regularize the end mark.

[0052] In step 103, according to the position information of each to-be-regularized character in the text regularizing result, and each original character with the second set character interval and the corresponding regularized character, each to-be-regularized character in the first original text is adjusted to obtain the regularized text.

[0053] In the embodiment of the present application, the text regularizing result includes the position information of each to-be-regularized character, and the original character of each to-be-regularized character with the second set character interval and the corresponding regularized character, so that each to-be-regularized character in the first original text is adjusted according to the position information of each to-be-regularized character in the text regularizing result, and each to-be-regularized character with the second set character interval and the corresponding regularized character to obtain the regularized text.

[0054] As an example, the text regularizing result is: <pos3> 24:11 <norm>Twenty-four to eleven <pos6> <pos7> 13 <norm>thirteen <pos8> <eos>, the first original text is: once 24:11 leading 13 points, so, the first original text in the starting position for <pos3>, end position is <pos6>The irregular characters 24:11 are replaced by regular characters: twenty-four to eleven. Similarly, the start position of the irregular characters 24:11 is replaced by the start position of the regular characters: 24:11. <pos7>, end position is <pos7>The to-be-regularized character 13 is replaced by the regularized character thirteen, and the regularized text is: once leading by 13 points with 24 to 11.

[0055] In the text regularizing method of the embodiment of the present application, a fine-tuned large-scale language model and a first text to be regularized are obtained, wherein the first text is obtained by adding a first specified character at the end of a first original text to be regularized, the first specified character is used to indicate the large-scale language model to output a text regularizing result, the first text is input into the large-scale language model, and a text regularizing result obtained by the large-scale language model predicting the position information and the regularized character of at least one to-be-regularized character in the first text under the indication of the first specified character is obtained, each to-be-regularized character in the first original text is adjusted to obtain a regularized text according to the position information of each to-be-regularized character in the text regularizing result and each to-be-regularized character and the corresponding regularized character separated by a second specified character, and the first specified character is added at the end of the first original text to obtain the first text, the fine-tuned large-scale language model is used to predict the position and the regularized character of each to-be-regularized character in the first text, the position of the to-be-regularized character in the first original text does not need to be positioned, and the to-be-regularized character does not need to be marked by encoding, so that the complexity of the model structure and the complexity of the text regularizing are reduced, and the effect of the text regularizing is improved.

[0056] Based on the above embodiment, Figure 2 Another flowchart of a text regularizing method provided by the embodiment of the present application is shown in FIG. 3. Figure 2 As shown in FIG. 3, the method comprises the following steps:

[0057] In step 201, the first original text is encoded by using a byte pair encoding algorithm to obtain a first encoded text.

[0058] The byte pair encoding algorithm is a byte-level encoding algorithm, for example, a Byte Pair Encoding (BPE) encoding algorithm.

[0059] In the related art, different models need to be trained for different languages, because the same expression may be different in different languages. For example, "20th" is usually read as "second ten" in Chinese, but read as "twentieth" in English. For example, St. in English can represent Saint or Street, but St. rarely appears in Mandarin. Similarly, the period in English is usually half-width, while the period in Chinese may be half-width or full-width. These cross-language differences result in the need to train corresponding models for different languages, increasing the cost and complexity.

[0060] Therefore, in the embodiment of the present application, the first original text after segmentation is encoded by the BPE algorithm to obtain the first encoded text. Since the BPE algorithm can uniformly map the first original text of different languages at the byte level to obtain a unified encoding format, any text of any language or mixed text of multiple languages can be regularized using the large-scale language model fine-tuned in the present application, without the need to train a corresponding model for different languages, thereby reducing the cost and improving the applicable scenarios of the large-scale language model fine-tuned.

[0061] Step 202, adding a first set character at the end of the first encoded text as a first text to be regularized.

[0062] Wherein, the related explanation of the first set character can refer to the explanation in the foregoing embodiment, which has the same principle and will not be repeated here.

[0063] Step 203, obtaining the large-scale language model fine-tuned and the first text to be regularized.

[0064] Wherein, step 203 can refer to the explanation in the foregoing embodiment, which has the same principle and will not be repeated here.

[0065] Step 204, comparing the first position information and the first character original text in the sub-regularization result of the to-be-regularized character with the second position information and the second character original text in the first text.

[0066] The large-scale language model of the present application can be a GPT2 model, which is a Transformer architecture based on only a decoder. Therefore, when predicting and outputting each to-be-regularized character in the input first text, in order to avoid the problem of inaccurate output caused by abnormal model prediction, the sub-regularization result of each to-be-regularized character in the first text output in sequence is checked, i.e. the first position information of the to-be-regularized character in each sub-regularization result is compared with the second position information of the to-be-regularized character in the first text, and the first character original text of the to-be-regularized character in the sub-regularization result is compared with the second character original text of the to-be-regularized character in the first text, so as to improve the accuracy of the output.

[0067] Step 205, in response to the first position information and the second position information not matching, and / or the first character original text and the second character original text not matching, triggering the large-scale language model to re-predict the to-be-regularized character in the first text to obtain a target sub-regularization result of the to-be-regularized character.

[0068] In the embodiments of the present application, in response to the first position information and the second position information not matching, and / or the first character original text and the second character original text not matching, it is indicated that the prediction result of the large-scale language model for the to-be-normalized character in the first text is inaccurate, triggering the large-scale language model to re-predict the to-be-normalized character in the first text to obtain the target sub-normalization result of the to-be-normalized character.

[0069] Optionally, in response to the first position information and the second position information matching, and the first character original text and the second character original text matching, it is indicated that the prediction result of the large-scale language model for the to-be-normalized character in the first text is accurate, and the sub-normalization result of the to-be-normalized character is output as the target sub-normalization result.

[0070] In step 206, according to the target sub-normalization result corresponding to each to-be-normalized character, a text normalization result corresponding to at least one to-be-normalized character is output.

[0071] In one implementation manner of the embodiments of the present application, the target sub-normalization result corresponding to each to-be-normalized character can be output after the target sub-normalization result corresponding to the to-be-normalized character is determined, and the target sub-normalization result corresponding to all to-be-normalized characters is output one by one to obtain the text normalization result corresponding to the first original text.

[0072] In another implementation manner of the embodiments of the present application, the target sub-normalization result corresponding to all to-be-normalized characters can be determined, and the text normalization result is generated and output according to the order of each to-be-normalized character in the text sequence of the first original text.

[0073] As an example, as shown in Figure 3 The first text is It has offerings in the big block, Small block, circle track, LSX, LS, and E ROD categories. When the large-scale language model outputs the sub-normalization result corresponding to the to-be-normalized character LS, the attention mechanism of the large-scale language model fails due to the continuous appearance and similarity of LSX and LX, resulting in the to-be-normalized character LS being incorrectly transcribed as LSX. Through the comparison of the position information and the character content, it is confirmed that the original text prediction of the to-be-normalized character LS in the sub-normalization result is incorrect, that is, the original text "LS" is predicted as "LSX", and the model needs to be prompted to re-predict and output {l s} to correct the error. This method of verifying the position and content of each to-be-normalized character output and the to-be-normalized character in the first original text ensures the accurate prediction of the to-be-normalized character and enhances the power of the large-scale language model. At the same time, through error correction, it can also identify the missed prediction characters, such as numbers or punctuation marks, avoiding missed identification and improving the accuracy of identification.

[0074] In step 207, according to the position information of each to-be-justified character in the text justification result, and the character original text and the corresponding justified character of each to-be-justified character using the second set character interval, each to-be-justified character in the first original text is adjusted to obtain a second text.

[0075] The step 207 can refer to the foregoing embodiments for explanation, and the principle is the same, which will not be described here again.

[0076] As an example, Figure 4 A schematic diagram for performing to-be-justified character replacement provided by the embodiment of the present application, Figure 4 A Chinese example and an English example are shown in the above table, and the English example is taken as an example. In the English text, 100 corresponds to the predicted sub-justification result <pos1> 100 <norm>one hundred <pos2>Thus, according to the sub-normalization result, the following is obtained <pos1> 100 <norm>one hundred <pos2>The to-be-justified character 100 is replaced by one hundred; similarly, for the to-be-justified character "colon:", the corresponding sub-justification result is adopted <pos3> ： <norm> <sil> <pos4>To perform normalization, replace the characters to be normalized with <sil>wherein, <sil>The character is indicated as a non-pronounced character, that is, the character does not need to be pronounced when voice generation is performed. Among them, the method principle of adjusting the regular character in the Chinese example is the same, which is not listed here.

[0077] In the text regularization method of the embodiment of the application, the first original text of different languages can be uniformly mapped at the byte level by the BPE algorithm to obtain a unified encoding format, so that no matter what kind of text or mixed text of multiple languages, the large-scale language model obtained by fine-tuning can be used for regularization, and a corresponding model does not need to be specially trained for different languages, thereby reducing the cost and improving the applicable scenarios of the large-scale language model obtained by fine-tuning. At the same time, in the prediction process, the position and content of the output of each to-be-regularized character and the to-be-regularized character in the first original text are verified to ensure the accurate prediction of the to-be-regularized character, and the power of the large-scale language model is enhanced.

[0078] Based on the above embodiment, the embodiment of the application provides a large-scale language model training method, Figure 5 The flowchart of the large-scale language model training method provided by the embodiment of the application, the execution subject of the large-scale language model training method of the embodiment of the application is a large-scale language model training device, which can be arranged in a server. The specific form and type of the server are not limited in this embodiment.

[0079] As shown in the method, Figure 5 The method comprises the following steps:

[0080] Step 501, obtaining a sample text and a label, wherein the sample text is obtained by adding a first set character at the end of the second original text to be regularized.

[0081] The first set character is used to indicate the output of the text regularization result of the large-scale language model. The label includes the true value position information of each to-be-regularized character in the sample text, and each to-be-regularized character and the corresponding true value regularized character are separated by a second set character. The label includes the first prediction position information and the second prediction position information of each to-be-regularized character in the sample text, and the first prediction position information and the second prediction position information are separated by the character original text and the true value regularized character of the to-be-regularized character separated by the second set character. The first prediction position information and the second prediction position information are used to indicate the position of the to-be-regularized character in the sample text.

[0082] As one implementation method, the sample text is generated by obtaining a second source text corresponding to the sample text, encoding the second source text using a byte pair encoding algorithm to obtain a second encoded text, and appending the first set character to the end of the second encoded text to form the sample text. The method for generating the sample text can be referred to the explanation of the first text in the aforementioned embodiment; the principles are the same and will not be further elaborated here.

[0083] The explanation about the first set character in the above embodiment is also applicable to this embodiment, and the principle is the same, so it will not be repeated here.

[0084] The second original text may be obtained from a public dataset, such as the Google Text Normalization dataset (GoogleTN), the FlatTN dataset, and a self-developed Chinese dataset.

[0085] FlatTN is a public Chinese text normalization dataset sourced from Chinese Wikipedia. It uses a classification standard for Chinese characters to be normalized. We retrieved text from its GitHub repository 2 as the second source text and generated sample text and labels corresponding to the second source text according to the label format requirements. The Google Text Normalization Dataset (GoogleTN) is a public English dataset. We retrieved text from its GitHub repository 1 as the second source text and generated sample text and labels corresponding to the second source text according to the label format requirements.

[0086] As an example, the sample text is English text, and the sample text is: He usually plays games at Forest Flower Memorial Park, Glind, Cali. The corresponding tag is: [" <pos10> ",","," <sil> "," <pos11> "],[" <pos12> ",","," <sil> "," <pos13> ",],[" <pos14> ","."," <sil> "," <pos15>". The sample text is a Chinese text, and the sample text is: The army was weakened by fighting on various fronts and was pushed out of a certain location on August 30. The corresponding label is: [" <pos9> ",","," <sil> "," <pos10> "],[" <pos10>","8","eight"," <pos11> "],[" <pos12>","30","Thirty"," <pos13> "],[" <pos17> ","。"," <sil> "," <pos18>"].

[0087] wherein the position information of each to-be-justified character included in the label is determined according to the position of each to-be-justified character in the sample text, using a position marker <posx>wherein X indicates the position of the word separated by spaces, starting from 1 and ending at 128. The label includes the true value position information of each to-be-justified character in the sample text, and each to-be-justified character and the corresponding true value justified character after using the second set character interval, that is, the character before the second set character is the original character of the to-be-justified character, and the character after the second set character is the predicted justified character after the to-be-justified character, that is, the second set character plays a role in identifying the position of the predicted justified character. By taking the label as the learning target in the model training process, the model is iteratively trained to learn the representation form of the text justification result.

[0088] In step 502, the sample text is input into the large-scale language model to obtain a second text predicted by the large-scale language model, and a text justification result obtained by predicting the position information and the justified character of at least one to-be-justified character in the sample text under the indication of the first set character.

[0089] In the foregoing embodiments, the related explanations and descriptions of the large-scale language model also apply to this embodiment, and the principles are the same, which will not be repeated here.

[0090] In an implementation manner of the embodiments of the present application, the sample text and the corresponding label can generate a text sequence, and a start character is added at the beginning of the text sequence, for example: <bos>, as an example, Figure 6 A structural diagram of a large-scale language model according to an embodiment of the present application. The model is a pre-trained large-scale language model GPT2. The model is a model including only a multi-layer decoder structure. The accuracy of the regularized result of the output is increased through the multi-layer decoder structure, so that the model can generate a more natural and accurate output as Figure 6 As shown in the figure, the input information input to the large-scale language model is <bos>input_text <tnstart>TN outputs, such that the model predicts the next character based on the previous character, resulting in an output text sequence of: input_text <tnstart>TN outputs <eos>wherein, <eos>indicating the end of the output of the text normalization result; input_text is the input text to be normalized, and TN outputs is the output text normalization result, <tnstart>The first set character.

[0091] As an implementation manner, the sample text is segmented to obtain a plurality of characters, each character being a minimum segmentation unit, wherein the segmentation algorithm can use Jieba tool 3 to segment the sample text in Chinese. Further, the sequence and label generated by segmentation are input into a text sequence, the input text sequence is vectorized to obtain a semantic vector that can be processed by a large-scale language model, the semantic vector is input into a decoding module of the large-scale language model to obtain decoding features, which can effectively capture long-distance dependencies in the input data through the attention mechanism of the transform decoding module, and then, the second text corresponding to the sample text is predicted according to the decoding features, and the text normalization result is obtained by predicting the position information and the normalized character of at least one to-be-normalized character in the sample text according to the indication of the decoding feature corresponding to the first set character, thereby improving the accuracy and robustness of the normalization result.

[0092] In step 503, the target loss function is determined according to the difference between the sample text and the second text, and the difference between the text normalization result and the label.

[0093] In an implementation manner of the embodiment of the present application, the first loss function is determined according to the difference between the sample text and the second text, that is, each character in the sample text and the second text is compared to determine the difference, the second loss function is determined according to the difference between the predicted position information and the predicted normalized character of at least one to-be-normalized character in the text normalization result and the true value position information and the true value normalized character of the at least one to-be-normalized character in the sample text, and the target loss function is determined according to the first loss function and the second loss function.

[0094] In an implementation manner of the embodiment of the present application, the first weight value corresponding to the first loss function and the second weight value corresponding to the second loss function can be obtained according to the first loss function and the second loss function, wherein the first weight value and the second weight value are set weight values, the first weight value is smaller than the second weight value, for example, the first weight value is 0.01, and the second weight value is 1, the target loss function is determined by weighted addition according to the first loss function and the first weight value, and the second loss function and the second weight value, that is, in the determination of the loss function, the first loss function needs to be weighted and attenuated by the first weight value, and the second weight value does not need to be weighted and attenuated, which realizes that in the model training process, the weight of the first loss function in the loss function can be reduced, so that the model mainly adjusts the model parameters according to the target loss function determined by the second loss function, so that the model parameter adjustment focuses on learning the text normalization result after the first set character, and the output of the first loss function before the first set character is reduced, so as to improve the accuracy of the model output.

[0095] Optionally, the target loss function can be determined based on a cross-entropy loss function.

[0096] At step 504, the parameters of the large-scale language model are fine-tuned according to the target loss function to obtain a trained large-scale language model.

[0097] In the embodiments of the present application, by fine-tuning the parameters of the large-scale language model, the large-scale language model learned the corresponding relationship between the input text and the regularized result.

[0098] As an implementation manner, the large-scale language model is implemented using TensorFlow and the Transformers library of Hugging Face, and is trained using a single NVIDIA Tesla V100 GPU to improve the training effect.

[0099] Wherein, the fine-tuning of the parameters of the large-scale language model can be trained with a batch size of 16, which means that 16 sample texts are selected from the training data set each time to calculate the target loss function, and the parameters of the model are updated according to these sample texts, and the large-scale language model can be trained for 10 cycles, and the large-scale language model with the highest accuracy on the validation set is selected from all the large-scale language models trained as the final model.

[0100] In the training method of the large-scale language model of the embodiments of the present application, the model structure is simplified to a large language model structure of a pure decoder, the existing mainstream pre-trained large-scale language model can be used, the additional encoder structure is abandoned, and the model landing workload is also simplified, and in the sample text, the position of the character indicating the position of the character to be regularized is not needed to be positioned, which will not affect the semantic understanding of the input text, and at the same time, the byte-based encoding algorithm is used for encoding, which can support Chinese, English and other multi-language texts, and accurate text regularization can be realized, and the effect of text regularization is improved.

[0101] In order to realize the above-mentioned embodiments, the embodiments of the present application also propose a text regularization device.

[0102] Figure 7 A structural schematic diagram of a text regularization device provided by the embodiments of the present application.

[0103] As shown in Figure 7 , the device can include:

[0104] The acquisition module 71 is configured to acquire the fine-tuned large-scale language model and first text to be regularized; wherein the first text is obtained by adding a first specified character at the end of first original text to be regularized; and the first specified character is used to indicate the text regularization result output by the large-scale language model.

[0105] The prediction module 72 is configured to input the first text into the large-scale language model to obtain a text regularization result predicted by the large-scale language model for at least one to-be-regularized character in the first text under the indication of the first specified character, wherein the text regularization result includes position information of the to-be-regularized character and a regularized character.

[0106] The adjustment module 73 is configured to adjust each to-be-regularized character in the first original text according to the position information of each to-be-regularized character in the text regularization result and each character original text and corresponding regularized character separated by a second specified character to obtain regularized text.

[0107] Further, in an implementation manner of the embodiment of the present application, the device further comprises an encoding module configured to:

[0108] encode the first original text by using a byte pair encoding algorithm to obtain first encoded text;

[0109] add the first specified character at the end of the first encoded text as the first text.

[0110] In an implementation manner of the embodiment of the present application, the prediction module 72 is further configured to:

[0111] compare first position information and a first character original text in a sub-regularization result of each to-be-regularized character predicted by the large-scale language model with second position information and a second character original text of the to-be-regularized character in the first text;

[0112] in response to the first position information and the second position information not matching, and / or the first character original text and the second character original text not matching, trigger the large-scale language model to re-predict the to-be-regularized character in the first text to obtain a target sub-regularization result of the to-be-regularized character;

[0113] output the text regularization result corresponding to each to-be-regularized character according to the target sub-regularization result corresponding to each to-be-regularized character.

[0114] It should be noted that the foregoing explanation and description of the method embodiment are also applicable to the device of this embodiment, which will not be described here.

[0115] The text normalization device provided in the application obtains a fine-tuned large-scale language model and a first text to be normalized, wherein the first text is obtained by adding a first specified character at the end of a first original text to be normalized, the first specified character is used to indicate that the large-scale language model outputs a text normalization result, the first text is input into the large-scale language model, and a text normalization result predicted by the large-scale language model under the indication of the first specified character is obtained, the text normalization result is used to predict at least one character to be normalized in the first text, position information and a normalized character, and each character to be normalized in the first original text is adjusted to obtain a normalized text according to the position information of each character to be normalized in the text normalization result, the character original text of each character to be normalized separated by a second specified character, and the corresponding normalized character, the first specified character is added at the end of the first original text to obtain the first text, and the fine-tuned large-scale language model is used to predict the position and the normalized character of each character to be normalized in the first text, so that the position positioning of the character to be normalized in the first original text is not required, and the character to be normalized is not required to be marked by coding, thereby reducing the complexity of the model structure and the complexity of the text normalization, and improving the effect of the text normalization.

[0116] To achieve the above-mentioned embodiments, the application also provides a large-scale language model training device.

[0117] Figure 8 A structural schematic diagram of a large-scale language model training device provided by the application embodiment.

[0118] As Figure 8 indicated, the device can include:

[0119] The acquisition module 81 is configured to acquire a sample text and a label, wherein the sample text is obtained by adding a first specified character at the end of a second original text to be normalized, the first specified character is used to indicate that a large-scale language model outputs a text normalization result, and the label includes true value position information of each character to be normalized in the sample text, and each character original text separated by a second specified character and a corresponding true value normalized character.

[0120] The prediction module 82 is configured to input the sample text into the large-scale language model to obtain a second text predicted by the large-scale language model, and a text normalization result predicted by the large-scale language model under the indication of the first specified character, the text normalization result being used to predict at least one character to be normalized in the sample text, position information and a normalized character.

[0121] The determination module 83 is configured to determine a target loss function according to a difference between the sample text and the second text, and a difference between the text normalization result and the label.

[0122] The training module 84 is configured to fine-tune parameters of the large-scale language model according to the target loss function, to obtain a trained large-scale language model.

[0123] Further, in an implementation form of the apparatus according to the embodiments, the apparatus further includes an encoding module configured to:

[0124] obtain a second original text corresponding to the sample text;

[0125] encode the second original text by using a byte pair encoding algorithm to obtain a second encoded text;

[0126] add the first set character at the end of the second encoded text as the sample text.

[0127] In an implementation form of the apparatus according to the embodiments, the determining module 83 is further configured to:

[0128] determine a first loss function according to a difference between the sample text and the second text;

[0129] determine a second loss function according to a difference between predicted position information and a predicted regularized character corresponding to at least one to-be-regularized character in the text regularization result and true value position information and a true value regularized character of the at least one to-be-regularized character in the sample text;

[0130] determine a target loss function according to the first loss function and the second loss function.

[0131] In an implementation form of the apparatus according to the embodiments, the determining module 83 is further configured to:

[0132] obtain a first weight value corresponding to the first loss function and a second weight value corresponding to the second loss function;

[0133] determine the target loss function by weighted addition according to the first loss function and the first weight value, and the second loss function and the second weight value.

[0134] In an implementation form of the apparatus according to the embodiments, the label includes first predicted position information and second predicted position information of each to-be-regularized character in the sample text, and the first predicted position information and the second predicted position information are provided with a character original text of the to-be-regularized character and a true value regularized character separated by the second set character; wherein the first predicted position information and the second predicted position information are used to indicate positions of the to-be-regularized character in the sample text.

[0135] It should be noted that the foregoing explanation and description of the method embodiments are also applicable to the apparatus embodiments, which will not be repeated here.

[0136] In the training device of the large-scale language model of the embodiment, the model structure of the large-scale language model is simplified to a pure decoder, the existing mainstream pre-training large-scale language model can be used, the additional encoder and other structures are abandoned, the model landing workload is simplified, and the character in the sample text that needs to be positioned at the indicated position is not positioned, and the semantic understanding of the input text is not affected by the position character. Meanwhile, the byte-based encoding algorithm is used for encoding, so that the Chinese, English and other multi-language texts can be supported, accurate text normalization can be realized, and the effect of text normalization is improved.

[0137] To achieve the above-embodiment, the present application also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to realize the method as described in the foregoing method embodiment.

[0138] To achieve the above-embodiment, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to realize the method as described in the foregoing method embodiment.

[0139] To achieve the above-embodiment, the present application also provides a computer program product having a computer program stored thereon, wherein the computer program is executed by a processor to realize the method as described in the foregoing method embodiment.

[0140] Figure 9 A structural schematic diagram of an electronic device is provided for the embodiment of the present application. For example, the electronic device 800 can be a mobile phone, a computer, a digital broadcast terminal, a message transmission device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0141] Reference Figure 9 The electronic device 800 can include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0142] The processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete the steps of the methods described above, in whole or in part. Moreover, the processing component 802 can include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0143] The memory 804 is configured to store various types of data to support the operations of the electronic device 800. Examples of these data include instructions to operate any applications or methods on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 can be realized by any type of volatile or non-volatile storage devices, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disc, or optical disc.

[0144] The power component 806 provides power to the various components of the electronic device 800. The power component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0145] The multimedia component 808 includes a screen to provide an output interface between the electronic device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes the touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 808 includes a front camera and / or a back camera. The front camera and / or the back camera can receive external multimedia data when the electronic device 800 is in an operating mode, such as a shooting mode or a video mode. Each of the front camera and the back camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0146] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive an external audio signal when the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0147] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0148] The sensor component 814 includes one or more sensors for providing status assessments of various aspects of the electronic device 800. For example, the sensor component 814 can detect an open / closed position of the electronic device 800, relative positioning of components, such as a display and a keypad of the electronic device 800, a change of position of the electronic device 800 or a component of the electronic device 800, presence or absence of user contact with the electronic device 800, orientation or acceleration / deceleration of the electronic device 800, and temperature changes of the electronic device 800. The sensor component 814 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 814 can further include a light sensor, such as a CMOS or CCD image sensor, for use in an imaging application. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0149] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 4G, or 5G, or a combination thereof. In an example embodiment, the communication component 816 receives broadcast signals or broadcast-related information from an external broadcasting management system via a broadcasting channel. In an example embodiment, the communication component 816 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technology.

[0150] In exemplary embodiments, the electronic device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements, for performing the above-described methods.

[0151] In exemplary embodiments, a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the electronic device 800 to accomplish the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0152] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that a specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. Illustrative expressions of the above terms in the specification do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0153] In addition, the terms "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0154] Any process or method descriptions or descriptions of the flow diagrams in the specification or otherwise described herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) in the process, and the preferred embodiments of the application include additional implementations in which the order of execution of the code modules, segments, or portions of code can be changed, including substantially simultaneously, or in reverse order, or in any other order, depending on the functionality involved. Such a process or method can be understood as a self-contained software module or self-contained hardware module utilizing hardware or software components.

[0155] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions stored in a computer readable medium, which can be executed by an instruction execution system, apparatus or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination thereof. For the purposes of this specification, a "computer readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer readable medium can specifically include the following, which are non-exhaustive list: electrical connection (electrical device having one or more wires), portable computer diskette (magnetic device), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber device, and portable compact disc read-only memory (CDROM). In addition, the computer readable medium can even be paper or other suitable medium upon which the program is printed, as the program can be electronically captured, for example, by optically scanning the paper or other suitable medium, then electronically converted into a form that can be edited, compiled, or interpreted, or otherwise processed in electronic form into an executable form suitable for use in the instruction execution system, apparatus or device.

[0156] It should be understood that parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As such, if implemented in hardware, and in another embodiment, any of the following technologies known in the art or their combination can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0157] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium, and when executed, include one or a combination of steps of the method embodiments.

[0158] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0159] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.< / tnstart> < / eos> < / eos> < / tnstart> < / tnstart> < / bos> < / bos> < / posx> < / sil> < / pos17> < / pos13> < / pos11> < / pos10> < / sil> < / pos9> < / sil> < / pos14> < / pos13> < / sil> < / pos12> < / pos11> < / sil> < / pos10> < / sil> < / sil> < / sil> < / norm> < / pos3> < / norm> < / pos1> < / norm> < / pos1> < / eos> < / pos8> < / norm> < / pos7> < / pos6> < / norm> < / pos3> < / eos> < / norm> < / eos> < / pos8> < / norm> < / pos7> < / pos6> < / norm> < / pos3> < / tnstart> < / tnstart> < / tnstart>

Claims

1. A text regularization method, characterized in that: include: Obtaining a fine-tuned large-scale language model and a first text to be regularized; wherein the first text is obtained by adding a first set character to the end of the first original text to be regularized; the first set character is used to instruct the large-scale language model to output a text regularization result; Inputting the first text into the large-scale language model, obtaining a text regularization result obtained by the large-scale language model performing position information and regularized character prediction on at least one character to be regularized in the first text under the instruction of a first set character; According to the position information of each character to be regularized in the text regularization result, and the original text of each character and the corresponding regularized character using the second set character interval, each character to be regularized in the first original text is adjusted to obtain the regularized text.

2. The method according to claim 1, wherein The method further comprises: Encoding the first original text using a byte pair encoding algorithm to obtain a first encoded text; The first set character is added to the end of the first encoded text to obtain the first text.

3. The method according to any one of claims 1 to 2, characterized in that The text regularization result includes sub-regularization results corresponding to each character to be regularized, and the first text is input into the large-scale language model, and the text regularization result obtained by the large-scale language model performing position information and regularized character prediction on at least one character to be regularized in the first text under the instruction of a first set character includes: Comparing, for each character to be regularized, a sub-regularization result predicted by the large-scale language model and obtained by comparing first position information and first character original text in the sub-regularization result of the character to be regularized with second position information and second character original text of the character to be regularized in the first text; In response to the mismatch between the first position information and the second position information, and / or the mismatch between the first character original text and the second character original text, triggering the large-scale language model to re-predict the character to be regularized in the first text to obtain a target sub-regularization result for the character to be regularized; According to the target sub-regularization results corresponding to each of the characters to be regularized, the text regularization result corresponding to the at least one character to be regularized is output.

4. A large-scale language model training method, characterized in that: include: Obtaining a sample text and a label; wherein the sample text is obtained by adding a first set character to the end of a second original text to be regularized; the first set character is used to instruct the large-scale language model to output a text regularization result; wherein the label includes the true value position information of each character to be regularized in the sample text, and the original text of each character using the second set character interval and the corresponding true value regularized character; Inputting the sample text into a large-scale language model to obtain a second text predicted by the large-scale language model, and performing position information and a regularized character prediction on at least one character to be regularized in the sample text under the instruction of a first set character to obtain a text regularization result; determining a target loss function based on a difference between the sample text and the second text, and a difference between the text regularization result and the label; Fine-tune the parameters of the large-scale language model according to the target loss function to obtain a trained large-scale language model.

5. The method according to claim 4, wherein The method further comprises: Obtaining a second original text corresponding to the sample text; Encoding the second original text using a byte pair encoding algorithm to obtain a second encoded text; The first set character is added to the end of the second encoded text as the sample text.

6. The method according to claim 4, wherein Determining a target loss function based on a difference between the sample text and the second text, and a difference between the text regularization result and the label, includes: determining a first loss function according to the difference between the sample text and the second text; Determining a second loss function based on a difference between predicted position information and predicted regularized character corresponding to at least one character to be regularized in the text regularization result and true value position information and true value regularized character of the at least one character to be regularized in the sample text; A target loss function is determined based on the first loss function and the second loss function.

7. The method according to claim 6, wherein The determining of a target loss function according to the first loss function and the second loss function includes: Obtaining a first weight value corresponding to the first loss function and a second weight value corresponding to the second loss function; The target loss function is determined by performing weighted addition based on the first loss function and the first weight value, and the second loss function and the second weight value.

8. The method according to claim 6, wherein The label includes the first predicted position information and the second predicted position information of each character to be regularized in the sample text, and the original text and the true value regularized character of the character to be regularized separated by the second set character interval are arranged between the first predicted position information and the second predicted position information; wherein, the first predicted position information and the second predicted position information are used to indicate the position of the character to be regularized in the sample text.

9. A text regularization device, characterized in that: include: An acquisition module is configured to acquire a fine-tuned large-scale language model and a first text to be regularized; wherein the first text is obtained by adding a first set character to the end of the first original text to be regularized; and the first set character is used to instruct the large-scale language model to output a text regularization result. a prediction module, configured to input the first text into the large-scale language model, and obtain a text regularization result obtained by the large-scale language model performing position information and regularized character prediction on at least one character to be regularized in the first text under the instruction of a first set character; The adjustment module is used to adjust the characters to be regularized in the first original text to obtain the regularized text according to the position information of the characters to be regularized in the text regularization result, and the characters to be regularized and the corresponding regularized characters using the second set character interval.

10. A large-scale language model training device, characterized in that: include: An acquisition module is configured to acquire a sample text and a label; wherein the sample text is obtained by adding a first set character to the end of a second original text to be regularized; the first set character is used to instruct the large-scale language model to output a text regularization result; wherein the label includes the true value position information of each character to be regularized in the sample text, and each character to be regularized using the second set character interval and the corresponding true value regularized character; a prediction module, configured to input the sample text into a large-scale language model to obtain a second text predicted by the large-scale language model, and to obtain a text regularization result by predicting position information and a regularized character for at least one character to be regularized in the sample text under the instruction of a first set character; a determination module, configured to determine a target loss function based on a difference between the sample text and the second text, and a difference between the text regularization result and the label; A training module is used to fine-tune the parameters of the large-scale language model according to the target loss function to obtain a trained large-scale language model.

11. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 3 or the method according to any one of claims 4 to 8 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented, or the method according to any one of claims 4 to 8 is implemented.

Images